In NYC, Companies Will Have to Prove Their AI Hiring Software Isn’t Sexist or Racist

New York City businesses that use artificial intelligence to help find hires now have to show the process was free from sexism and racism.

A new law, which takes effect Wednesday, is believed to be the first of its kind in the world. Under New York’s new rule, hiring software that relies on machine learning or artificial intelligence to help employers choose preferred candidates or weed out bad ones — called an automatic employment decision tool, or AEDT — must pass an audit by a third-party company to show it’s free of racist or sexist bias.

Companies that run AI hiring software must also publish those results. Businesses that use third-party AEDT software can no longer legally use such programs if they haven’t been audited.

{snip}

AI-infused hiring programs have drawn scrutiny, most notably over whether they end up exhibiting biases based on the data they’re trained on. Studies have long found that programs that use machine learning or artificial intelligence often exhibit racism, sexism and other biases.

As flashy generative AI applications like ChatGPT and Midjourney have surged in popularity, federal lawmakers and even many tech company executives have repeatedly called for regulation. But so far, there’s little sense from Congress what that might look like.

{snip}

Julia Stoyanovich, a computer science professor at New York University and a founding member of the city’s Automatic Decisions Systems Task Force, said it’s an important start but still very limited.

{snip}

It’s also not clear how the law will be enforced or to what extent.

New York’s Department of Consumer and Worker Protection, charged with enforcing the law, “will collect and investigate complaints” of companies accused of violating it, an agency spokesperson said.

Jake Metcalf, a researcher specializing in AI for Data & Society, a nonprofit group that studies the effects of technology on society, said the wording of the law — it defines AEDT as technology that will “substantially assist or replace discretionary decision making” — has led lawyers that advise large companies not to take it seriously.

{snip}

“It’s very hard to figure out what ‘substantially’ means,” he said.

{snip}