Gordon Brown is facing a summer of questions about his leadership after Labour's humiliating defeat in the Norwich North by-election.
Backbench Labour MPs warned the Prime Minister he had until the party conference season in September to demonstrate he is up to the job.
The Tories overturned a Labour majority of more than 5,000 in Thursday's poll to win by 7,348 votes.
A repeat of the 16.4% swing in a general election would install David Cameron in Number 10 with a majority of more than 200 seats in the Commons.
Senior Labour backbencher Barry Sheerman said the party needs to get its "act together" and stop blaming its difficulties on the MPs' expenses scandal. "It's partly a question of leadership, it's partly a question of ideas," he said.
Mr Sheerman accused Mr Brown of failing to connect with voters, adding: "In any other human organisation I know, if the chief executive doesn't get it and doesn't deliver then he has to consider his position. Now I'm saying he's got the summer to recognise this isn't about members' expenses, it's about something much more fundamental."
The by-election was forced by the resignation of Labour's Ian Gibson as the MP for Norwich North after he was told by a party disciplinary panel he would not be allowed to stand at the next election over revelations about his expenses claims.
Former home secretary Charles Clarke, MP for the neighbouring seat of Norwich South, said the Prime Minister had not been fair in his treatment of MPs.
Writing in the Independent, Mr Clarke said: "This incompetent and unjust style has deeply damaged democratic politics. Moreover, the appalling result in Norwich illustrates the important political side-effect that Labour, as the governing party, has been injured worst of all."
Labour backbencher Kate Hoey said the Prime Minister needs to look at how he leads the party. "By-elections are always unique but there is no doubt about it that this is a bad result. The Prime Minister, I hope, will be looking at how he's looking to lead the party and to talk to the party, and a lot of party members feel that they are not listened to."
