Sukdarshan Singh, an Indian, arrived in Britain unlawfully in 1984 and raped a 59-year-old woman four years later.
He was only linked to the attack in 2006 after being arrested for drink driving and was jailed for four and a half years.
An immigration tribunal ruled he should be deported but the Court of Appeal yesterday overturned the decision because it had failed to take in to account his rights to family life as he now has a British wife and two teenage children.
In 1998, Singh, 55, raped a woman in Bridgend, Wales, where he had a market stall.
He later married in 1991 and his wife did not know about the crime until 2006 when he was arrested for driving with excess alcohol and DNA evidence linked him to the rape.
He was sentenced to four years six months jail at Cardiff Crown Court and his application for indefinite leave to remain in the UK was refused in 2008 because of his rape conviction.
Yesterday, he was appealing against the Asylum and Immigration Tribunal (AIT) finding which upheld the decision of the Home Secretary to return him to India.
But Lord Justice Aikens said the “overall question” in the case was whether deportation would be a disproportionate interference with his private and family life protected by the European Convention on Human Rights.
He said the AIT had concluded that if family life was to continue, Mr Singh’s wife and children, who are all British citizens, would also have to go to India.
But the tribunal had not considered the effect of the move on the two children, who had lived all their lives in the UK and this was an error of law, the judge said.
He said the AIT should have balanced the threat of family breakdown, the effect on the children and feasibility of a move to India with the aim of deportation in protecting the public.
“I am not convinced that when the balance is struck it would inevitably be on the side of deportation,” he said.
He allowed the appeal and ordered that another tribunal should reconsider the case.