Wednesday

28th Feb 2024

Investigation

Computer says No: How the EU's AI laws cause new injustice

  • Big Tech companies pushing for self-regulation, while their algorithms create new injustice in European public services (Photo: Sacha Fernandez)

When Roger Derikx applied for childcare benefits, he thought he would be receiving money — not losing it.

Derikx, 46 years old, was asked to repay the Dutch government €68,000. The problem: he was never told why he had to return the benefits, to which he was entitled. Authorities repossessed his car, and took a 40 percent cut of his salary for years.

Read and decide

Join EUobserver today

Get the EU news that really matters

Instant access to all articles — and 20 years of archives. 14-day free trial.

... or subscribe as a group

"You have two little children, and you want to give them everything," Derikx said, "but every time they ask something, you have to say 'no' — that's hard."

Derikx was one of 26,000 parents accused of making fraudulent benefit claims by the Dutch tax authority, which used an algorithm for fraud detection. The system was "unlawful and discriminatory," an official investigation found, eventually leading Dutch prime minister Mark Rutte to resign earlier this year.

For Derikx, the investigation came too late. The financial hardship took such a toll on his marriage that he ended up getting a divorce. "That's a big price," Derikx said.

EU proposes AI regulation

The child benefits scandal in the Netherlands is not unique. Reports of abuse have surfaced across Europe.

Danish authorities also used artificial intelligence to identify children in "vulnerable families" that disproportionately targeted parents with foreign origins.

And the Netherlands, one of Europe's frontrunners in this field, used algorithms to create risk profiles of children under the age of 12.

Critics have long advocated against such uses of AI. In January, 61 advocacy organisations sent an open letter to the EU demanding "red lines" on AI applications that threatened fundamental rights.

Meanwhile, industry representatives argued that regulating AI would stifle innovation.

Upon taking office, European Commission president Ursula von der Leyen had made regulating artificial intelligence a political priority, and the EU claimed to seek compromise between addressing the risks and protecting innovation.

By April 2021, the EU Commission announced a long-awaited proposal to regulate artificial intelligence.

Corporate ascendency

Data compiled for this investigation shows that commission officials logged 152 official lobby meetings on AI over the time period from December 2019 until August 2021.

Corporations and business associations accounted for over 100, or two-thirds, of those meetings; less than one third was with civil society, academics, and trade unions.

Most organisations only got to meet the EU once. Google topped the list with nine meetings, the first of which occurred just three days after the Commission took office.

Alongside Google were Microsoft and Big Tech-funded business associations such as DigitalEurope. In total, 10 of the most active organisations had business interests.

European consumer organisation Beuc and European Digital Rights, an advocacy organisation, were the only non-profits near the top of the list.

"This is about democracy," said Margarida Silva from Corporate Observatory Europe, a Brussels-based lobby watchdog. "And we are giving more of a voice to those that have the most money."

An EU spokesperson declined to comment on why industry accounted for the majority of meetings.

And these numbers tell just a part of the story. Only high-ranking officials are obliged to disclose their meetings, but much of the lobbying occurs at lower levels.

DG Connect, the EU commission department responsible for drafting the proposal, stated by email that they had held almost 1,000 meetings on AI policy with "external stakeholders."

'Ethics-washing'

Lobby papers submitted to the EU, and seen by EUobserver's correspondents, detailed how tech firms pushed for more AI in the public sector.

Google encouraged "governments to promote public-sector adoption of AI to help create a thriving ecosystem."

Microsoft wrote that "the public sector could benefit enormously from the adoption of AI solutions." And DigitalEurope, the tech lobby that enjoys close relations with the European Commission, told the EU to "focus on [the] public sector for trust."

Tech firms also played up their capacity to self-regulate, minutes of meetings between tech firms and the commission, obtained through a freedom of information request, showed.

Facebook underlined it spends $3.7bn on "integrity." Google and Microsoft highlighted their corporate AI principles. And tech lobby DigitalEurope demanded that the EU adopted a "voluntary process-based certification scheme" — a practice whereby companies would themselves assess that their AI products lived up to the EU requirements before putting them on the market.

Critics have called this practice "ethics-washing": relying on claims of ethical corporate behaviour to stave off political oversight.

In fact, the European Consumer Organisation told the EU in late 2019 exactly that: "Avoid a purely 'ethics' narrative, which implies self-regulation."

"Public authorities often do not have the capacity to develop their own AI systems," said Ursula Pachl of Beuc, "which means that we are again dependent on these companies."

Public-sector AI

Regardless, the EU proposal conceded to key industry demands on self-regulation and the promotion of AI in the public sector.

When the Commission released its draft AI regulation there was much attention for the proposed ban on social scoring — ranking people for their trustworthiness — and some uses of facial recognition techniques.

But many public-sector uses of AI would still be permitted, even though they were classified as high-risk.

This included the use of algorithms in policing, the judicial system, and approval of social benefits.

Producers would only need to do a self-assessment to make sure their "high-risk" system complied with rules on transparency, risk management, and human oversight.

Presenting the proposal, Margrethe Vestager, EU commissioner for the digital portfolio, said the "legal framework shapes the trust we must build if we want people and businesses to embrace AI solutions."

The commissioner said she was worried about the uptake of artificial intelligence. She feared, for instance, that municipalities would not want to take the risk of using AI to support social workers in finding solutions for vulnerable citizens. "I think that would be a pity," Vestager said.

The draft AI regulation was accompanied by a commitment to invest €1bn per year in AI and making "the public sector a trailblazer for AI."

What had happened to Derikx in the Dutch child-benefit case was a clear illustration of the risks the EU intended to prevent with the regulation, an EU official with close insight into the policy process admitted.

Nonetheless, "the point is not that we shouldn't use AI," the official added. "We should use trustworthy AI that is properly developed. It's in the interest of everybody, including industry."

But concerns remain over whether the proposed certification system and self-assessment through tech providers themselves offers sufficient protection for EU citizens.

A recently published response by members of the EU expert group on AI drew a clear conclusion: "The proposal does not provide adequate fundamental rights protection, nor does it provide sufficient protection to maintain the rule of law and democracy, and thus fails to secure Legally Trustworthy AI."

Invisible injustice

The harm of algorithms is notoriously difficult to demonstrate. Derikx spent years in court to prove his innocence. Others lost custody of their children or committed suicide.

"Most of the cases will go undetected," said Sarah Chander, policy lead at European Digital Rights. "People will not even know that the decisions about their life are being made by an automated system."

"The burden is still on the people who were affected," adds Daniel Leufer of Access Now, one of the rights-advocacy groups.

In the Netherlands, victims were promised €30,000 in compensation, but the Dutch government has struggled to keep its commitment.

"Parents still need to fight for every penny," said Derikx, who has teamed up with a group of victims in their fight for compensation — and recognition. "It's an ugly, nasty process. But I won't let it go."

Author bio

Alina Yanchur, Camille Schyns, Greta Rosén Fondahn and Sarah Pilz are freelance investigative journalists.

This is the first of a two-part investigation into the lobbying on the landmark EU proposal to regulate artificial intelligence, which leaves the door open to widespread use of the technology in the public sector.

Stakeholder

Podcast: 'Big Tech' and the threat to democracy

Do tech giants like Google, Apple or Facebook have too much power? Is their growing power a threat to democracy? These topics are highlighted in the latest Nordic Talks podcast, featuring three specialists - among them EU commissioner Margrethe Vestager.

Investigation

'Redacted' - what Google and Microsoft told Mogherini on AI

Is Mogherini's 'Global Tech Panel' just a talking shop, or does it contribute to EU policy? New documents suggest the latter, but give little clue how technology leaders from Microsoft and Google influence new AI strategy.

Investigation

How Big Tech dominates EU's AI ethics group

Despite the responsibility for drafting the EU's artificial intelligence ethics guidelines, few of the expert group members were ethicists. In fact, 26 experts — nearly half of the group's 56 members — represented business interests.

Investigation

Who is Kris Roman, the Kremlin's man in Belgium?

In Flanders, Russia's access to the Belgian far-right is facilitated by Kris Roman. What is far less known are his more than decade-long connections with Russian intelligence.

Latest News

  1. EU supply chain law fails, with 14 states failing to back it
  2. Joined-up EU defence procurement on the horizon?
  3. Macron on Western boots in Ukraine: What he really meant
  4. Amazon lobbyists banned from EU Parliament
  5. MEPs adopt new transparency rules for political ads
  6. EU nature restoration law approved after massive backlash
  7. Memo from Munich — EU needs to reinvent democracy support
  8. For Ukraine's sake, pass the EU due diligence directive

Stakeholders' Highlights

  1. Nordic Council of MinistersJoin the Nordic Food Systems Takeover at COP28
  2. Nordic Council of MinistersHow women and men are affected differently by climate policy
  3. Nordic Council of MinistersArtist Jessie Kleemann at Nordic pavilion during UN climate summit COP28
  4. Nordic Council of MinistersCOP28: Gathering Nordic and global experts to put food and health on the agenda
  5. Friedrich Naumann FoundationPoems of Liberty – Call for Submission “Human Rights in Inhume War”: 250€ honorary fee for selected poems
  6. World BankWorld Bank report: How to create a future where the rewards of technology benefit all levels of society?

Stakeholders' Highlights

  1. Georgia Ministry of Foreign AffairsThis autumn Europalia arts festival is all about GEORGIA!
  2. UNOPSFostering health system resilience in fragile and conflict-affected countries
  3. European Citizen's InitiativeThe European Commission launches the ‘ImagineEU’ competition for secondary school students in the EU.
  4. Nordic Council of MinistersThe Nordic Region is stepping up its efforts to reduce food waste
  5. UNOPSUNOPS begins works under EU-funded project to repair schools in Ukraine
  6. Georgia Ministry of Foreign AffairsGeorgia effectively prevents sanctions evasion against Russia – confirm EU, UK, USA

Join EUobserver

EU news that matters

Join us