That's not the actual role of the Speaker. They are, and always have been, responsible for wheeling, dealing, and cajoling other members in order to pass legislation. They are the carrot to the Whip's stick.
To take one example, Nancy Pelosi was a very effective Speaker but she was not a dispassionate "judge". She was in every way a power broker, like Tip O'Neill (another very effective Speaker).
If the Speakership worked as you suggest, with no real power to push an agenda, then few people would want the job. It would be like the President of the Senate (aka Vice-President), a job which is usually a consolation prize.