A Kent builder claims he was called racist by Jobcentre staff, after his advert for a carpenter said the person must be able to speak and read English.
Dean Osmon, of Teynham, was told it might breach the Race Relations Act.
He said spoken English was vital for workers to deal with clients and explain technical details.
The Department for Work and Pensions said each vacancy was judged on its own merits and Mr Osmon’s language restrictions did not breach the act.
The advert read: “Must be able to understand English, written and verbal, as will be dealing with clients in residential and business premises.”
But a letter from the Jobcentre Plus diversity and race equality team to Mr Osmon said: “You informed the contact centre you were not willing to employ anyone who did not speak English.
“Where employers discriminate against job applicants on the basis of race, they may be in breach of the Race Relations Act.”
Mr Osmon said he thought it was “ridiculous”.
He said: “I think it’s disgusting when you go into a Jobcentre, and they say change that word ‘English-speaking’, that you can’t advertise for what you want.”
The Commission for Racial Equality told BBC South East Today it was reasonable for an employer to require employees to speak English in the work setting, particularly in customer-facing roles.
The issue is being taken up by Shadow Work and Pensions Secretary Philip Hammond.
He said: “I think it’s nonsense.
“The ability to speak English, like the ability to speak Spanish or French, is a qualification.
“I will be seeking an assurance that Jobcentre Plus will be instructed to treat a requirement by an employer for a linguistic skill, as precisely that, a requirement for a qualification and not a sign of latent racism.”