National

HUD recently proposed a rule that would protect financial institutions from liability for using algorithms to make lending decisions, as long as the technology used was produced or distributed by a recognized company.

Why it matters: AI can inadvertently rely on characteristics that include or are correlated with race, gender and socio-economic class, so under the proposed rule, financial institutions could make illegal determinations and hide behind an AI product.

The big picture: Financial institutions are increasingly using AI to detect suspicious activity, optimize portfolios, recommend strategic investments and assess creditworthiness. 

The impact: There are factors the financial sector may use that — while not explicitly equivalent to race or gender — correlate with those characteristics and can result in discrimination. 

How it works: Institutions may decide, for example, that unbanked individuals are less creditworthy, and using this factor in loan decisions could disadvantage people of color and women. 

Nearly 17% of African Americans and 14% of Hispanic Americans are unbanked, compared to just 3% of white Americans.

15% percent of unmarried female-headed family households are also unbanked.

What's happening: HUD released a proposed rule that would eliminate the disparate impact standard, which prohibits policies or procedures that result in a disproportionate adverse impact on protected groups. It would also shield financial institutions from liability that arises when they use AI-based tools from third parties, like tech companies for instance, whether or not there was knowledge of the problematic algorithm. 

What to watch: If the HUD rule is enacted, algorithms could obscure the reason for a credit denial. 

But if denial notices are required to be made clear to borrowers, in compliance with the Fair Credit Reporting Act, potential homeowners may have some means of identifying illegal or inappropriate grounds for a determination, even if the financial institution is shielded from liability. 

The bottom line: There are already challenges in applying anti-discrimination laws to AI-based determinations. The newly proposed HUD rule would make this considerably more difficult.

Miriam Vogel is the executive director of Equal AI, an adjunct professor at Georgetown Law and a former associate deputy attorney general at the Department of Justice.

A proposed HUD rule on AI could allow for housing discrimination

Financial institutions are increasingly using AI to assess creditworthiness and more.

A black and white photo of a house broken up by pixelated static.

Cities

Artificial intelligence

Expert Voices

Technology

Automation and AI

HUD recently <a class="gtmContentClick" data-vars-link-text="proposed" data-vars-click-url="https://s3.amazonaws.com/public-inspection.federalregister.gov/2019-17542.pdf" data-vars-content-id="3d778ea7-8230-4f21-935b-b6b7f384a310" data-vars-headline="A proposed HUD rule on AI could allow for housing discrimination" data-vars-event-category="story" data-vars-sub-category="story" data-vars-item="in_content_link" href="https://s3.amazonaws.com/public-inspection.federalregister.gov/2019-17542.pdf" target="_blank">proposed</a> a rule that would protect financial institutions from liability for using algorithms to make lending decisions, as long as the technology used was produced or distributed by a recognized company.Why it matters: AI can inadvertently rely on characteristics that include or are correlated with race, gender and socio-economic class, so under the proposed rule, financial institutions could make illegal determinations and hide behind an AI product.

From the Valley to D.C., Big Tech players like Facebook, Google and Amazon are under more scrutiny than ever as new technology develops and privacy and antitrust concerns grow in lockstep with companies’ ambitions.

Many health-related AI technologies today are biased because they're built on datasets largely comprised of men and individuals of European descent.

Why it matters: An AI system trained to identify diseases, conditions and symptoms in people in these datasets could fail when presented with data from people with different characteristics. 

Background: AI-powered disease detection technology is part of the health care AI market expected to exceed $34 billion by 2025.

Researchers recently demonstrated that AI used in breast cancer screenings correctly identified more cancers, reduced false positives and improved reading times. 

What's happening: Most medical research tends to focus on men, and most genetic data publicly available is from individuals of European descent. As AI is increasingly used in medicine, it could result in misdiagnoses of patients based on their gender, race and/or ethnicity.

While heart attacks generally strike men and women equally, they are more likely to be fatal in women, which can be caused by a delay in care due to gender-based differences in symptoms. 

Similarly, if a person is not of European descent, AI medical technologies may incorrectly diagnose that person, as their symptoms and disease manifestations could differ. 

Recent studies and mishaps have shown that our current data and programs that rely on AI, like search engines and image recognition software, are biased in ways that can cause harm.

What we're watching: Some steps are being taken to ensure that AI is evaluated for bias, including proposed legislation. 

The National Institutes of Health launched a new program last year to expand diversity in medical research and data by soliciting volunteers from populations that are currently underrepresented.

Go deeper: Scientists call for rules on evaluating predictive AI in medicine 

Miriam Vogel is the executive director of Equal AI, a professor at Georgetown Law and a former associate deputy attorney general at the Department of Justice.

How bias creeps into health care AI

AI training skewed toward some populations can be unreliable for others.

A man's face geometrically segmented with some portions highlighted

Health Care

Future

Axios Pro: Deals

Many health-related AI technologies today are biased because they&#x27;re built on datasets largely comprised of <a class="gtmContentClick" data-vars-link-text="men" data-vars-click-url="https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1910554" data-vars-content-id="a382814b-6bb5-4294-b5e1-fd80bd5cccda" data-vars-headline="How bias creeps into health care AI" data-vars-event-category="story" data-vars-sub-category="story" data-vars-item="in_content_link" href="https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1910554" target="_blank">men</a> and individuals of <a class="gtmContentClick" data-vars-link-text="European descent" data-vars-click-url="https://www.nature.com/news/genomics-is-failing-on-diversity-1.20759" data-vars-content-id="a382814b-6bb5-4294-b5e1-fd80bd5cccda" data-vars-headline="How bias creeps into health care AI" data-vars-event-category="story" data-vars-sub-category="story" data-vars-item="in_content_link" href="https://www.nature.com/news/genomics-is-failing-on-diversity-1.20759" target="_blank">European descent</a>.Why it matters: An AI system trained to identify diseases, conditions and symptoms in people in these datasets could fail when presented with data from people with different characteristics.

Miriam Vogel

Stories

A proposed HUD rule on AI could allow for housing discrimination

How bias creeps into health care AI