Damian Green, the immigration minister, blamed judges after new figures revealed that nine in 10 overseas criminals who should have been sent home but are currently free in the UK.
Separate figures showed the number of foreign offenders successfully removed from the UK fell by nearly a fifth last this year.
Some of the released prisoners went on to commit new serious crimes including murder, rape, kidnapping and violence, while awaiting deportation.
The majority are able to stay and fight deportation because of human rights laws, Mr Green admitted.
During the debate Mr Green said that data protection laws had left him powerless to tell another MP whether a murderer who killed his constituent was a foreign national or not.
He was called to the House of Commons yesterday to answer an urgent question after figures from a leaked memo at the weekend said there were 4,238 foreign criminals awaiting deportation who are not in custody.
That figure had increased by almost 500 since May this year.
Foreign criminals released from custody have committed two murders, three kidnappings, 14 sexual offences and 27 other violent crimes.
Mr Green told MPs the most up to date figure for those at not in detention was actually 3,940 and that 90 per cent of those had been released by immigration judges, rather than a Home Office decision.
Most are let out by the courts because there is no immediate prospect of them being deported, he said.
That is because of human rights battles to stay in the UK, the situation in their home country or a lack of co-operation by the offender or his home government in getting essential travel documents.
Mr Green said 60 per cent were based on human rights challenges but the Home Office was powerless over court decisions and must release them.
He said: “When this happens the (UK Border) Agency works closely with both the police and the National Offender Management Service to reduce the risk of reoffending. Deportation action continues in all cases.”
Separate figures showed that in the first nine months of 2011 some 3,331 foreign criminals were deported–down 690 on the 4,021 removed in the same period last year.
Chris Bryant, the shadow immigration minister, described the drop as “astounding”.
He said: “So far on your watch we have seen numbers of staff at UKBA going down, numbers of foreign national offenders deported going down and numbers of foreign criminals in our midst going up.
“Don’t you realise that’s the wrong way round?”
Labour demanded reassurances that Home Office ministers know the “precise whereabouts” of foreign criminals awaiting deportation and to deliver “fewer words and more action”.
Mr Green insisted the Government was “doing everything in our power to increase the number and speed of removals”.
He pledged to tackle human rights laws which hinder Government action to kick out foreign criminals and that more foreign criminal-only prisons will be opened.
Labour’s David Winnick accused the Home Secretary’s department of being in “such a shambles” over matters relating to immigration control.
He said: “Can the message be made to her loudly and clearly that it’s time she got a grip on her department?”