AI in Medicine Needs to Be Carefully Deployed to Counter Bias – And Not Entrench It

Doctors, data scientists and hospital executives believe artificial intelligence may help solve what until now have been intractable problems. AI is already showing promise to help clinicians diagnose breast cancer, read X-rays and predict which patients need more care. But as excitement grows, there’s also a risk: These powerful new tools can perpetuate long-standing racial inequities in how care is delivered.

“If you mess this up, you can really, really harm people by entrenching systemic racism further into the health system,” said Dr. Mark Sendak, a lead data scientist at the Duke Institute for Health Innovation.

These new health care tools are often built using machine learning, a subset of AI where algorithms are trained to find patterns in large data sets like billing information and test results. Those patterns can predict future outcomes, like the chance a patient develops sepsis. These algorithms can constantly monitor every patient in a hospital at once, alerting clinicians to potential risks that overworked staff might otherwise miss.

The data these algorithms are built on, however, often reflect inequities and bias that have long plagued U.S. health care. Research shows clinicians often provide different care to white patients and patients of color. Those differences in how patients are treated get immortalized in data, which are then used to train algorithms. People of color are also often underrepresented in those training data sets.

“When you learn from the past, you replicate the past. You further entrench the past,” Sendak said. “Because you take existing inequities and you treat them as the aspiration for how health care should be delivered.”

A landmark 2019 study published in the journal Science found that an algorithm used to predict health care needs for more than 100 million people was biased against Black patients. The algorithm relied on health care spending to predict future health needs. But with less access to care historically, Black patients often spent less. As a result, Black patients had to be much sicker to be recommended for extra care under the algorithm.

“You’re essentially walking where there’s land mines,” Sendak said of trying to build clinical AI tools using data that may contain bias, “and [if you’re not careful] your stuff’s going to blow up and it’s going to hurt people.”

{snip}

Over the last several years, hospitals and researchers have formed national coalitions to share best practices and develop “playbooks” to combat bias. But signs suggest few hospitals are reckoning with the equity threat this new technology poses.

Researcher Paige Nong interviewed officials at 13 academic medical centers last year, and only four said they considered racial bias when developing or vetting machine learning algorithms.

{snip}

The Biden administration over the last 10 months has released a flurry of proposals to design guardrails for this emerging technology. The FDA says it now asks developers to outline any steps taken to mitigate bias and the source of data underpinning new algorithms.

The Office of the National Coordinator for Health Information Technology proposed new regulations in April that would require developers to share with clinicians a fuller picture of what data were used to build algorithms. {snip}

The Office for Civil Rights at the U.S. Department of Health and Human Services last summer proposed updated regulations that explicitly forbid clinicians, hospitals and insurers from discriminating “through the use of clinical algorithms in [their] decision-making.” {snip}

{snip}

Many experts in AI and bias welcome this new attention, but there are concerns. Several academics and industry leaders said they want to see the FDA spell out in public guidelines exactly what developers must do to prove their AI tools are unbiased. Others want ONC to require developers to share their algorithm “ingredient list” publicly, allowing independent researchers to evaluate code for problems.

Some hospitals and academics worry these proposals — especially HHS’s explicit prohibition on using discriminatory AI — could backfire. “What we don’t want is for the rule to be so scary that physicians say, ‘OK, I just won’t use any AI in my practice. I just don’t want to run the risk,'” said Carmel Shachar, executive director of the Petrie-Flom Center for Health Law Policy at Harvard Law School. Shachar and several industry leaders said that without clear guidance, hospitals with fewer resources may struggle to stay on the right side of the law.

{snip}