Machines Behaving Badly

The Morality of AI

Toby Walsh

• Publisher: Flint
• Year: 2022
• Print ISBN:
  9780750999366
• eBook ISBN:
  9781803990842
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Can we build moral machines?

Artificial intelligence is an essential part of our lives – for better or worse. It can be used to influence what we buy, who gets shortlisted for a job and even how we vote. Without AI, medical technology wouldn't have come so far, we'd still be getting lost in our GPS-free cars, and smartphones wouldn't be so, well, smart. But as we continue to build more intelligent and autonomous machines, what impact will this have on humanity and the planet?

Professor Toby Walsh, a world-leading researcher in the field of artificial intelligence, explores the ethical considerations and unexpected consequences AI poses. Can AI be racist? Can robots have rights? What happens if a self-driving car kills someone? What limitations should we put on the use of facial recognition? Machines Behaving Badly is a thought-provoking look at the increasing human reliance on robotics and the decisions that need to be made now to ensure the future of AI is a force for good, not evil.

FAIRNESS

There are plentiful examples of problems involving the fairness of AI-enabled decision-making. To compound matters, we don’t yet have a precise playbook for fixing them. In part, this is because it is still early days in dealing with some of these issues. But it is also because there probably aren’t going to be simple fixes for many of these problems.

Fairness goes to the heart of what it means to live in a just and equitable society. This is something we’ve struggled with ever since we started living alongside each other, and the answers we have continue to evolve. AI puts some of these fairness issues on steroids. However, despite there not being good answers to many of these questions, there are a dozen valuable lessons that I will pull out.

Before we look into these challenges, I want to mention the many possible upsides to automating decision-making that could help make the world a fairer place. First, handing over decisions to computers could lead to greater consistency. Humans can be capricious and random in coming to decisions. Computer programs, on the other hand, can be frustratingly consistent. We are often most aware of this when they’re consistently doing the wrong thing.

Second, automated decision-making has the potential to be more transparent than human decision-making. Humans are far from transparent in how they make decisions, and it’s not certain we’ll ever truly understand and be able to record how we do so. Even though many automated systems are not easily understood today, there are no fundamental reasons why we can’t make them more transparent in the future.

Third, human decision-making is full of unconscious biases. We can work hard at eliminating these, but even the best of us struggles to do so. All of us subconsciously make decisions based on gender, race and other attributes, even though we know we shouldn’t and try hard not to do so. When we automate decisions, we can simply not include those attributes in the data given to the machine. Eliminating bias isn’t as simple as this, but this may at least be a first step to fairer decisions.

Fourth, and perhaps most crucially, automated decision-making can be much more data-driven and evidence-based. There are many situations where humans make decisions based on intuition. But in many of these settings, we are now collecting and analysing data. We can therefore determine for the first time whether our decisions are fair. And when they are not, we can consider adjusting the decision-making to improve the fairness.

MUTANT ALGORITHMS

In August 2020, we saw what were perhaps the first (but I suspect not the last) protest marches over an algorithm. Due to the COVID-19 pandemic, students at UK schools weren’t able to sit A-level or GCSE exams. Instead, an algorithm was used to allocate grades based on the predictions of teachers.

According to the official regulator Ofqual, the algorithm adjusted around four in ten marks down by one grade or more from the grades predicted by teachers. Students took to the streets of London to protest. The government quickly capitulated and reverted to the grades predicted by teachers. The prime minister, Boris Johnson, blamed the fiasco on ‘a mutant algorithm’. But there was nothing mutant about the algorithm. As far as we can tell, it did exactly what it was meant to do. The problem was that Ofqual hadn’t thought carefully enough about what the public would find fair.

We shall never know how accurate the algorithm was, since the students didn’t sit their final exams. But it is worth pointing out that teachers aren’t that good at accurately estimating the performance of their students. In Scotland, teacher estimates are collected every year, and only 45 per cent of students in a normal year achieve their estimated grade. We shouldn’t expect teachers to have done any better in estimating grades in an exceptional year like 2020. It is also worth pointing out that human markers often don’t agree with each other. Outside of maths and the sciences, 30 per cent of A-level markers disagree on what grade to give a paper.

So let’s be generous and not too critical of the accuracy of Ofqual’s algorithm. After all, the algorithm was designed to give an overall distribution of grades that looked similar to previous years, with similar proportions of grades for each subject. In fact, mirroring recent years, it even allowed a slight increase in A and A* grades. Ofqual even went so far as to check that the proportion of grades handed out to different subpopulations (by gender, ethnicity and income, for example) matched that in recent years.

However, what soon became apparent was that even if Ofqual’s algorithm was as accurate as human markers, it was nevertheless biased in favour of certain groups and against others. In particular, it was biased in favour of pupils at schools that had done well in previous years, and in subjects where class sizes were small. Or, to put it another way, it was biased against students in poor public schools and in favour of students at rich private schools.

How did this happen? The algorithm tried to ensure that the range and distribution of grades achieved by a group of students in a class was similar to students in the same class over the previous three years. The exception to this was small classes – those with less than 15 students in most cases – where the teacher-predicted grades were used because of the lack of a data algorithm. The net result was that the algorithm was fair on average, providing a similar distribution of grades as in past years. But the unfairness was highly biased against students in public schools, especially in more deprived areas, and highly biased in favour of students in private schools. Small teaching groups, and less popular A-levels such as Law, Ancient Greek and Music, are more common at private schools, which insulated those students from being marked down. And, historically, A-level results in selective and private schools have been higher, bequeathing a higher range of grades to the 2020 students.

I suspect that even if Ofqual had avoided these problems of bias, they would have run into trouble. Some students were going to win, some were going to lose. The winners won’t shout out, but you can be sure the losers will complain loudly. Ofqual therefore had an impossible task. The government should have been generous, accepted that Ofqual was going to have to give out some higher grades, and funded more university places to compensate.

If we apply the sort of ethical principles used to inform decision-making in other areas like medicine, we can see that the marking algorithm failed the principle of justice. The burdens and benefits were not distributed equally across all groups in society. Students from poor state schools were more likely to have their grades marked down than students from rich private schools. There is no justice in this.

Actually, the marking algorithm exposed two more fundamental problems with the examination system. First, it highlighted how important it is to retain human agency, especially in high-stakes decisions. In the case of exams, people felt they had agency as they could hope to ace the exam. But having an algorithm give students a predicted grade without an exam, however accurate the prediction might have been, denied them this agency.

Second, the marking algorithm exposed and magnified a fundamental problem with the public examination system that had existed even when humans were doing the marking. Ranking students nationwide on a simple scale, when this ranking would decide life-changing events like university places, is inherently questionable. Should your life options be decided by how well you perform in an exam on one particular afternoon? That seems no better than the whims of an algorithm. The truth is that algorithms cannot fix broken systems. They inherit the flaws of the systems into which they’re placed.

PREDICTIVE POLICING

Another problematic area where AI algorithms have been gaining traction is in predictive policing. This sounds like the film Minority Report but is actually far simpler. We cannot predict when someone is going to commit a crime. Humans are not that predictable. But we can predict where, on average, crime will take place.

We have lots of historical data about crime: police incident reports, sentencing records, insurance claims and so on. And in most districts there are insufficient police resources to patrol all the neighbourhoods we would like. So why not use machine learning to predict where and when crimes are most likely to take place, and focus police resources on those places and times?

In late 2011, Time magazine identified predictive policing (along with the Mars Rover and Siri) as one of the year’s 50 best inventions.¹ It is now used by police departments in a number of US states, including California, Washington, South Carolina, Alabama, Arizona, Tennessee, New York and Illinois. In Australia, the NSW police force has an even more sinister and secret algorithm, which predicts not when and where crime might take place, but who is likely to commit a crime. These individuals are then given extra scrutiny and monitoring by local police. A report analysing police data found that people under the age of 25, along with Indigenous citizens, were disproportionately targeted, and that the algorithm makes decisions based on ‘discriminatory assumptions’.²

There are several fundamental problems here. First, we don’t have ground truth. We want to predict where and when crime will take place, and who is going to be responsible, but we simply can’t know this. We only have historical data on where crime was reported. And there is a lot of crime that took place that we don’t know about.

The data we have reflects the biases of the system in which it was collected. Perhaps the police disproportionately patrolled poorer neighbourhoods. The greater prevalence of crime reported in such neighbourhoods might therefore be simply a consequence of this greater number of patrols. Or it might be the result of racism within the police force, which meant that more Black people in such neighbourhoods were stopped and searched.

Predicting future crime on such historical data will then only perpetuate past biases. Let me adapt a famous quotation from writer and philosopher George Santayana: ‘Those who use AI to learn from history are doomed to repeat it.’ In fact, it’s worse than repetition. We may construct feedback loops in which we magnify the biases of the past. We may send more patrols to poorer and predominately Black neighbourhoods. These patrols identify more crime. And an unfortunate feedback loop is set up as the system learns to send even more patrols to these neighbourhoods.

SENTENCING

AI algorithms have been gaining traction in another part of the judicial system, helping judges decide who might or might not ...

Table of contents

• Cover
• Title
• Copyright
• Contents
• AI
• The People
• The Companies
• Autonomy
• Humans v. Machines
• Ethical Rules
• Fairness
• Privacy
• The Planet
• The Way Ahead
• Epilogue: The Child of Our Brains
• About the Author
• Acknowledgements
• Notes