Tuesday

21st Sep 2021

Investigation

Computer says No: How the EU's AI laws cause new injustice

  • Big Tech companies pushing for self-regulation, while their algorithms create new injustice in European public services (Photo: Sacha Fernandez)

When Roger Derikx applied for childcare benefits, he thought he would be receiving money - not losing it.

Derikx, 46 years old, was asked to repay the Dutch government €68,000. The problem: he was never told why he had to return the benefits, to which he was entitled. Authorities repossessed his car, and took a 40 percent cut of his salary for years.

Read and decide

Join EUobserver today

Become an expert on Europe

Get instant access to all articles — and 20 years of archives. 14-day free trial.

... or subscribe as a group

"You have two little children, and you want to give them everything," Derikx said, "but every time they ask something, you have to say 'no' - that's hard."

Derikx was one of 26,000 parents accused of making fraudulent benefit claims by the Dutch tax authority, which used an algorithm for fraud detection. The system was "unlawful and discriminatory," an official investigation found, eventually leading Dutch prime minister Mark Rutte to resign earlier this year.

For Derikx, the investigation came too late. The financial hardship took such a toll on his marriage that he ended up getting a divorce. "That's a big price," Derikx said.

EU proposes AI regulation

The child benefits scandal in the Netherlands is not unique. Reports of abuse have surfaced across Europe.

Danish authorities also used artificial intelligence to identify children in "vulnerable families" that disproportionately targeted parents with foreign origins.

And the Netherlands, one of Europe's frontrunners in this field, used algorithms to create risk profiles of children under the age of 12.

Critics have long advocated against such uses of AI. In January, 61 advocacy organisations sent an open letter to the EU demanding "red lines" on AI applications that threatened fundamental rights.

Meanwhile, industry representatives argued that regulating AI would stifle innovation.

Upon taking office, European Commission president Ursula von der Leyen had made regulating artificial intelligence a political priority, and the EU claimed to seek compromise between addressing the risks and protecting innovation.

By April 2021, the EU Commission announced a long-awaited proposal to regulate artificial intelligence.

Corporate ascendency

Data compiled for this investigation shows that commission officials logged 152 official lobby meetings on AI over the time period from December 2019 until August 2021.

Corporations and business associations accounted for over 100, or two-thirds, of those meetings; less than one third was with civil society, academics, and trade unions.

Most organisations only got to meet the EU once. Google topped the list with nine meetings, the first of which occurred just three days after the Commission took office.

Alongside Google were Microsoft and Big Tech-funded business associations such as DigitalEurope. In total, 10 of the most active organisations had business interests.

European consumer organisation Beuc and European Digital Rights, an advocacy organisation, were the only non-profits near the top of the list.

"This is about democracy," said Margarida Silva from Corporate Observatory Europe, a Brussels-based lobby watchdog. "And we are giving more of a voice to those that have the most money."

An EU spokesperson declined to comment on why industry accounted for the majority of meetings.

And these numbers tell just a part of the story. Only high-ranking officials are obliged to disclose their meetings, but much of the lobbying occurs at lower levels.

DG Connect, the EU commission department responsible for drafting the proposal, stated by email that they had held almost 1,000 meetings on AI policy with "external stakeholders."

'Ethics-washing'

Lobby papers submitted to the EU, and seen by EUobserver's correspondents, detailed how tech firms pushed for more AI in the public sector.

Google encouraged "governments to promote public-sector adoption of AI to help create a thriving ecosystem."

Microsoft wrote that "the public sector could benefit enormously from the adoption of AI solutions." And DigitalEurope, the tech lobby that enjoys close relations with the European Commission, told the EU to "focus on [the] public sector for trust."

Tech firms also played up their capacity to self-regulate, minutes of meetings between tech firms and the commission, obtained through a freedom of information request, showed.

Facebook underlined it spends $3.7bn on "integrity." Google and Microsoft highlighted their corporate AI principles. And tech lobby DigitalEurope demanded that the EU adopted a "voluntary process-based certification scheme" - a practice whereby companies would themselves assess that their AI products lived up to the EU requirements before putting them on the market.

Critics have called this practice "ethics-washing": relying on claims of ethical corporate behaviour to stave off political oversight.

In fact, the European Consumer Organisation told the EU in late 2019 exactly that: "Avoid a purely 'ethics' narrative, which implies self-regulation."

"Public authorities often do not have the capacity to develop their own AI systems," said Ursula Pachl of Beuc, "which means that we are again dependent on these companies."

Public-sector AI

Regardless, the EU proposal conceded to key industry demands on self-regulation and the promotion of AI in the public sector.

When the Commission released its draft AI regulation there was much attention for the proposed ban on social scoring - ranking people for their trustworthiness - and some uses of facial recognition techniques.

But many public-sector uses of AI would still be permitted, even though they were classified as high-risk.

This included the use of algorithms in policing, the judicial system, and approval of social benefits.

Producers would only need to do a self-assessment to make sure their "high-risk" system complied with rules on transparency, risk management, and human oversight.

Presenting the proposal, Margrethe Vestager, EU commissioner for the digital portfolio, said the "legal framework shapes the trust we must build if we want people and businesses to embrace AI solutions."

The commissioner said she was worried about the uptake of artificial intelligence. She feared, for instance, that municipalities would not want to take the risk of using AI to support social workers in finding solutions for vulnerable citizens. "I think that would be a pity," Vestager said.

The draft AI regulation was accompanied by a commitment to invest €1bn per year in AI and making "the public sector a trailblazer for AI."

What had happened to Derikx in the Dutch child-benefit case was a clear illustration of the risks the EU intended to prevent with the regulation, an EU official with close insight into the policy process admitted.

Nonetheless, "the point is not that we shouldn't use AI," the official added. "We should use trustworthy AI that is properly developed. It's in the interest of everybody, including industry."

But concerns remain over whether the proposed certification system and self-assessment through tech providers themselves offers sufficient protection for EU citizens.

A recently published response by members of the EU expert group on AI drew a clear conclusion: "The proposal does not provide adequate fundamental rights protection, nor does it provide sufficient protection to maintain the rule of law and democracy, and thus fails to secure Legally Trustworthy AI."

Invisible injustice

The harm of algorithms is notoriously difficult to demonstrate. Derikx spent years in court to prove his innocence. Others lost custody of their children or committed suicide.

"Most of the cases will go undetected," said Sarah Chander, policy lead at European Digital Rights. "People will not even know that the decisions about their life are being made by an automated system."

"The burden is still on the people who were affected," adds Daniel Leufer of Access Now, one of the rights-advocacy groups.

In the Netherlands, victims were promised €30,000 in compensation, but the Dutch government has struggled to keep its commitment.

"Parents still need to fight for every penny," said Derikx, who has teamed up with a group of victims in their fight for compensation - and recognition. "It's an ugly, nasty process. But I won't let it go."

This is the first of a two-part investigation into the lobbying on the landmark EU proposal to regulate artificial intelligence, which leaves the door open to widespread use of the technology in the public sector.

Author bio

Alina Yanchur, Camille Schyns, Greta Rosén Fondahn and Sarah Pilz are freelance investigative journalists.

Stakeholder

Podcast: 'Big Tech' and the threat to democracy

Do tech giants like Google, Apple or Facebook have too much power? Is their growing power a threat to democracy? These topics are highlighted in the latest Nordic Talks podcast, featuring three specialists - among them EU commissioner Margrethe Vestager.

Investigation

'Redacted' - what Google and Microsoft told Mogherini on AI

Is Mogherini's 'Global Tech Panel' just a talking shop, or does it contribute to EU policy? New documents suggest the latter, but give little clue how technology leaders from Microsoft and Google influence new AI strategy.

Investigation

Revealed: loopholes letting EU firms 'break' arms embargoes

Most arms deals include "post-sale services", such as training, maintenance and know-how. Often up to 50 percent of the value of a multi-year arms contract is related to post-sale services, which leave companies with an unseen stake in controversial conflicts.

Investigation

France opposed guidelines on EU presidency sponsorships

Internal document shows diplomats questioned whether the secretariat of the Council of the EU was legally allowed to write guidelines on the financing of the six-month rotating EU presidency activities.

Stakeholders' Highlights

  1. Nordic Council of MinistersNATO Secretary General guest at the Session of the Nordic Council
  2. Nordic Council of MinistersCan you love whoever you want in care homes?
  3. Nordic Council of MinistersNineteen demands by Nordic young people to save biodiversity
  4. Nordic Council of MinistersSustainable public procurement is an effective way to achieve global goals
  5. Nordic Council of MinistersNordic Council enters into formal relations with European Parliament
  6. Nordic Council of MinistersWomen more active in violent extremist circles than first assumed

Latest News

  1. First refugee deaths confirmed on Belarus-EU border
  2. EU kept in dark on ex-commissioner's new lobby job
  3. Fraud against EU dropped 20% last year
  4. French outrage over US security deal exposes EU frustrations
  5. Auditors slam EU Commission on green investments
  6. Youth migration 'costing West Balkans up to €5.5bn a year'
  7. Central & Eastern Europe: What Merkel did for us
  8. Netherlands against more rights for rejected asylum-seekers

Join EUobserver

Support quality EU news

Join us