Canopy goes to FAT*

Last week, a group of us from Canopy traveled to Atlanta to attend FAT*, a multi-disciplinary conference on fairness, accountability and transparency in socio-technical systems.

Do you know what that means? We weren’t sure either, but last year’s line-up was a stellar collection of computer scientists trying to prove bias, philosophers trying to define bias and lawyers trying to get judges to make decisions based on p-values. So we decided to face the hordes of football fans who were descending on Atlanta for the Super Bowl and check out this year’s program.

We had a delightful time, learned a lot, and have decided to share with you some of our takeaways.

(And just to get it out of the way early, yes, the conference acronym is terrible. They’re working on it.)

Let’s move from Fairness and Transparency to Justice and Ethics. Or maybe simply Human Rights?

During the town hall discussion on the second day of the conference, someone suggested that the FAT* community change its focus from Fairness and Transparency to Justice and Ethics.

That comment reminded me of a discussion I attended a few months ago where Philip Alston from the NYU School of Law said (jokingly!) that he wanted to “strangle ethics”.

What I always see in the AI literature these days is ‘ethics’. I want to strangle ethics... ethics are completely open ended. You can create your own ethic. But human rights, you can’t. They're in the Constitution, they’re in the Bill of Rights, they’ve been interpreted by courts, there are certain limits. And until we start bringing those in to the AI discussion, there’s no hard anchor.

So what should we focus on?

Most people at FAT* would probably agree that the AI community needs to address the negative, real world consequences of discriminatory algorithmic decision making. But from the wide range of talks, there appears to be substantial disagreement about what addressing these problems looks like.

I agree with Alston that fairness and ethics are fuzzy concepts, and that this community should challenge itself to be accountable for how our work impacts real people. This does not mean that we need to forgo mathematical and algorithmic approaches, but that we should pull on threads until we get down to the human level.

I am not familiar with a lot work that does this successfully, so I am going to reach outside of this year’s FAT* program to highlight an example. Last year, Virginia Eubanks, published a phenomenal book titled Automating Inequality that investigates the impacts of software automation and algorithmic decision making on poor and working-class people in America. What makes Eubanks’ work so compelling is that she combines historical context with data analysis and a tremendous amount of on-the-ground reporting. As computer scientists, we might not have the training to do that, but we should be educating ourselves about the context that our work exists in and reaching out to historians, journalists, case workers and lawyers who engage with the human rights impact of the systems we work on.

Eubanks shows that it is possible to do “full-stack” work on algorithmic human rights. Now we just need more people to give it a try.

In practice, the biggest challenges and biggest wins come down to data.

One thing that stood out to me as the talks progressed was how different the theoretical work was from the applied work. The CS literature on algorithmic fairness largely focuses on metrics, algorithms and what can be proven about the two. We saw talks proposing new metrics, new algorithms, and a very good empirical study of the effectiveness and robustness of combinations of the two -- which sadly concluded that “fairness interventions might be more brittle than previously thought”.

But the real-world applications had a very different flavor. They tended to revolve around data. What type of data would we need to empirically prove bias? What domain-specific metrics should we look at? Can we get around practical hurdles like missing information?

There were two papers addressing these issues that really stood out.

The first was an analysis of the Bayesian Improved Surname Geocoding (BISG) method, titled “Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved”. BISG is a technique that is widely used to impute someone’s racial group given their last name and location. Why would anyone do this? Because some industries are legally required to have fair decision practices, but are at the same time forbidden from collecting information about protected attributes from their customers or applicants. Both regulators and the companies themselves rely on these “proxy models” to guess an unknown protected class based on observed variables in order to audit for fair practices. The paper analyzes the bias introduced by these proxy models and proposes an improvement to the most commonly used models. This area of research is both mathematically interesting and has the potential for broad impact.

In the following session, Sendhil Mullainathan presented the incredible work he did with Ziad Obermeyer in which they identified “significant racial bias” in an automated selection process for a beneficial medical care program that impacts 70 million people. The bias stemmed from the decision to train the selection algorithm to keep costs consistent across racial groups, but not to keep healthcare needs consistent across racial groups. Black patients tend to have lower healthcare costs, and thus were being enrolled in this beneficial program at half the rate they would have been without the bias. The authors write that while insurers may focus on costs, “from the social perspective, actual health – not just costs – also matters.”

In addition to identifying this problem, Sendhil and Ziad reached out to the company that designed the algorithm and have worked with them to reduce the bias against black patients by 84%!

These two papers speak to the heart of the questions the FAT* community is trying to address. How do we identify when algorithms are discriminating against historically marginalized groups? What can we do about it in practice? And when are the stakes too high, the potential damage too much, should we toss the algorithm out the window?

These “rubber-hits-the-road” papers were a reminder that it is often access to data and a deep understanding of a domain that leads to real impact.

How we’re contributing to this work at Canopy

Here at Canopy, we are creating a personalized content discovery architecture that keeps your data on device. We know that we are complicating our ability to audit for fairness by not storing any user data on the server, but we are not scared of complicated problems. We believe that people deserve to be in control of their digital identities. And for too long, companies have presented us with false choice when it comes to our privacy. At Canopy, we are committed to building delightful products that are both private and fair.

If you want to follow us along on that journey, we’re on Twitter. We’ll be sharing more about our approach to fairness in an age of privacy there. And please come to FAT* next year in Barcelona, we’ll be sure to be there.

published by:
Erica Greene
February 5, 2019