Cathy O’Neil: Weapons of Math Destruction
Cathy O’Neil is a data scientist who studied maths at Harvard, taught at MIT & worked at D. E. Shaw.
In her book, Weapons of Math Destruction, she expresses her concern about algorithmic influence in a number of fields. I’ll summarise the book here for anyone who hasn’t read it.
Going to college, borrowing money, getting sentenced to prison, finding a job & holding that job. All of these are increasingly controlled, or influenced, by mathematical algorithms that wield seemingly arbitrary punishments. O’Neil calls this the dark side of Big Data.
Not all algorithms have a “dark side”. But a few have significant, and often unintended, consequences on peoples lives. These “Weapons of Math Destruction” (WMDs) have three defining characteristics: opacity, scale and damage
Let’s review each in turn.
Opacity: With WMDs, a lack of transparency and accountability is the norm. We’re analysed and ranked without knowing the variables upon which we are judged. For example, teachers have been fired after receiving low “value added” scores but couldn’t be told exactly where they went wrong.
Scale: Scale turns WMDs from local nuisances into forces that define and destroy lives: For example, banks have defined high-risk borrowers based on their social circles and addresses. This, in turn, affects their ability to get an apartment, a job, or a car to get from one to the other.
Damage: WMDs are based on historic, often biased, data. Worst still, the models create destructive feedback loops. For example. as most tech executives are male, hiring algorithms deprioritise female applicants which, in turn, further skews the dataset from which the algorithm bases its decisions.
These WMDs all feed off each other. A model that sends more police officers to poor neighbourhoods causes more arrests to be made there. In turn recidivism algorithms give longer sentences to people who live in this location. In turn hiring models reject them as candidates and creditors jack up their interest rates.
Overall, Weapons of Math Destruction is a short but important book. I’d highly recommend reading it in full. Pick up your copy here.