Computer says No: How the EU's AI laws cause new injustice

"You have two little children, and you want to give them everything," Derikx said, "but every time they ask something, you have to say 'no' - that's hard."

Derikx was one of 26,000 parents accused of making fraudulent benefit claims by the Dutch tax authority, which used an algorithm for fraud detection. The system was "unlawful and discriminatory," an official investigation found, eventually leading Dutch prime minister Mark Rutte to resign earlier this year.

For Derikx, the investigation came too late. The financial hardship took such a toll on his marriage that he ended up getting a divorce. "That's a big price," Derikx said.

EU proposes AI regulation

The child benefits scandal in the Netherlands is not unique. Reports of abuse have surfaced across Europe.

Danish authorities also used artificial intelligence to identify children in "vulnerable families" that disproportionately targeted parents with foreign origins.

And the Netherlands, one of Europe's frontrunners in this field, used algorithms to create risk profiles of children under the age of 12.

Critics have long advocated against such uses of AI. In January, 61 advocacy organisations sent an open letter to the EU demanding "red lines" on AI applications that threatened fundamental rights.

Meanwhile, industry representatives argued that regulating AI would stifle innovation.

Upon taking office, European Commission president Ursula von der Leyen had made regulating artificial intelligence a political priority, and the EU claimed to seek compromise between addressing the risks and protecting innovation.

By April 2021, the EU Commission announced a long-awaited proposal to regulate artificial intelligence.

Corporate ascendency

Data compiled for this investigation shows that commission officials logged 152 official lobby meetings on AI over the time period from December 2019 until August 2021.

Corporations and business associations accounted for over 100, or two-thirds, of those meetings; less than one third was with civil society, academics, and trade unions.

Most organisations only got to meet the EU once. Google topped the list with nine meetings, the first of which occurred just three days after the Commission took office.

Alongside Google were Microsoft and Big Tech-funded business associations such as DigitalEurope. In total, 10 of the most active organisations had business interests.

European consumer organisation Beuc and European Digital Rights, an advocacy organisation, were the only non-profits near the top of the list.

"This is about democracy," said Margarida Silva from Corporate Observatory Europe, a Brussels-based lobby watchdog. "And we are giving more of a voice to those that have the most money."

An EU spokesperson declined to comment on why industry accounted for the majority of meetings.

And these numbers tell just a part of the story. Only high-ranking officials are obliged to disclose their meetings, but much of the lobbying occurs at lower levels.

DG Connect, the EU commission department responsible for drafting the proposal, stated by email that they had held almost 1,000 meetings on AI policy with "external stakeholders."

'Ethics-washing'

Lobby papers submitted to the EU, and seen by EUobserver's correspondents, detailed how tech firms pushed for more AI in the public sector.

Google encouraged "governments to promote public-sector adoption of AI to help create a thriving ecosystem."

Microsoft wrote that "the public sector could benefit enormously from the adoption of AI solutions." And DigitalEurope, the tech lobby that enjoys close relations with the European Commission, told the EU to "focus on [the] public sector for trust."

Tech firms also played up their capacity to self-regulate, minutes of meetings between tech firms and the commission, obtained through a freedom of information request, showed.

Facebook underlined it spends $3.7bn on "integrity." Google and Microsoft highlighted their corporate AI principles. And tech lobby DigitalEurope demanded that the EU adopted a "voluntary process-based certification scheme" - a practice whereby companies would themselves assess that their AI products lived up to the EU requirements before putting them on the market.

Critics have called this practice "ethics-washing": relying on claims of ethical corporate behaviour to stave off political oversight.

In fact, the European Consumer Organisation told the EU in late 2019 exactly that: "Avoid a purely 'ethics' narrative, which implies self-regulation."

"Public authorities often do not have the capacity to develop their own AI systems," said Ursula Pachl of Beuc, "which means that we are again dependent on these companies."

Public-sector AI

Regardless, the EU proposal conceded to key industry demands on self-regulation and the promotion of AI in the public sector.

When the Commission released its draft AI regulation there was much attention for the proposed ban on social scoring - ranking people for their trustworthiness - and some uses of facial recognition techniques.

But many public-sector uses of AI would still be permitted, even though they were classified as high-risk.

This included the use of algorithms in policing, the judicial system, and approval of social benefits.

Producers would only need to do a self-assessment to make sure their "high-risk" system complied with rules on transparency, risk management, and human oversight.

Presenting the proposal, Margrethe Vestager, EU commissioner for the digital portfolio, said the "legal framework shapes the trust we must build if we want people and businesses to embrace AI solutions."

The commissioner said she was worried about the uptake of artificial intelligence. She feared, for instance, that municipalities would not want to take the risk of using AI to support social workers in finding solutions for vulnerable citizens. "I think that would be a pity," Vestager said.

The draft AI regulation was accompanied by a commitment to invest €1bn per year in AI and making "the public sector a trailblazer for AI."

What had happened to Derikx in the Dutch child-benefit case was a clear illustration of the risks the EU intended to prevent with the regulation, an EU official with close insight into the policy process admitted.

Nonetheless, "the point is not that we shouldn't use AI," the official added. "We should use trustworthy AI that is properly developed. It's in the interest of everybody, including industry."

But concerns remain over whether the proposed certification system and self-assessment through tech providers themselves offers sufficient protection for EU citizens.

A recently published response by members of the EU expert group on AI drew a clear conclusion: "The proposal does not provide adequate fundamental rights protection, nor does it provide sufficient protection to maintain the rule of law and democracy, and thus fails to secure Legally Trustworthy AI."

Invisible injustice

The harm of algorithms is notoriously difficult to demonstrate. Derikx spent years in court to prove his innocence. Others lost custody of their children or committed suicide.

"Most of the cases will go undetected," said Sarah Chander, policy lead at European Digital Rights. "People will not even know that the decisions about their life are being made by an automated system."

"The burden is still on the people who were affected," adds Daniel Leufer of Access Now, one of the rights-advocacy groups.

In the Netherlands, victims were promised €30,000 in compensation, but the Dutch government has struggled to keep its commitment.

"Parents still need to fight for every penny," said Derikx, who has teamed up with a group of victims in their fight for compensation - and recognition. "It's an ugly, nasty process. But I won't let it go."