Saturday

4th Feb 2023

Magazine

Forecasting and profiling, or bias and discrimination?

  • As Artificial Intelligence becomes an essential, albeit invisible, feature of our daily interactions, brace yourselves for even more EU-wide political wrangles (Photo: Markus Winkler)
Listen to article

Artificial Intelligence has already brought down one government in Europe.

In 2019, the Dutch tax authority used self-learning algorithms to create risk profiles in an attempt to spot fraud involving child-care benefits. As it became clear that the families, mainly from ethnic-minority communities, had been identified on suspicion of fraud and then penalised because of algorithm-generated profiles, a massive political scandal brought down the government of Dutch prime minister Mark Rutte.

Read and decide

Join EUobserver today

Become an expert on Europe

Get instant access to all articles — and 20 years of archives. 14-day free trial.

... or subscribe as a group

  • United Nations' former human rights chief, Michelle Bachelet, has warned AI systems could carry 'negative, even catastrophic' risks (Photo: Wikimedia)

Rutte survived. But thousands of ordinary lives were ruined.

As Artificial Intelligence becomes an essential, albeit invisible, feature of our daily interactions, brace yourselves for even more EU-wide political wrangles, centring on European Commission proposals to regulate the sector.

The proposals, made in 2019, are now making their way through the European Parliament — where political groups have tabled over 3,000 amendments — and are expected to receive the Council greenlight in December. The EU's AI Act will then be agreed next year, after intra-institutional negotiations.

Expect a showdown between the parliament and the council, however. MEPs are expected to push for stricter regulation and better protection of rights, while governments will likely argue for fewer rules, citing the need for competitiveness and security concerns.

"The success of the text will lie in the balance we find between the need to protect the interests and rights of our citizens, and the interest to stimulate innovation and encourage the uptake and development of AI," Romanian liberal MEP Dragoș Tudorache, one of the European lawmakers in charge of the file, said recently. "The real political discussions are yet to come."

Bulgarian socialist MEP Petar Vitanov, one of the negotiators on the file, says the focus must be on ensuring that "fundamental rights and freedoms are safeguarded, there can be no innovation without fundamental rights."

Key issues include the governance of the act, and the legislation's definition of risk.

Lawmakers want to give the commission the power to extend the list of so called "high-risk areas", and to raise fines for non-compliance to €40m or seven percent of annual turnover.

Some EU governments are seeking exemptions for the use of AI by migration authorities and law enforcement, which could lead to more control over communities, including ethnic communities, which are already more policed than others.

While some critics, such as the United Nations' former human rights chief, Michelle Bachelet, say governments should put a moratorium on the sale and use of AI systems until the "negative, even catastrophic" risks they pose can be addressed.

A report by the UN on AI's use as a forecasting and profiling tool warns that the technology could have an impact on "rights to privacy, to a fair trial, to freedom from arbitrary arrest and detention and the right to life."

Bachelet acknowledged that AI "can be a force for good, helping societies overcome some of the great challenges of our times," but suggested that the harms it could bring outweigh the positives. "The higher the risk for human rights, the stricter the legal requirements for the use of AI technology should be," she said.

The commission proposal calls for a ban on AI applications that manipulate human behaviour (such as toys which use voice assistance encouraging dangerous activities in children) or systems that allow Chinese-style 'social-scoring'. The use of biometric identification systems, such as facial-recognition, for law enforcement in public spaces, is also prohibited.

Exceptions are allowed in the case of a search for victims of kidnapping, the identification of a perpetrator or suspect of a criminal offence, or for the prevention of imminent threats, such as a terrorist attack.

Digital-rights activists warn, however, that there are "loopholes" that allow mass surveillance.

Any exemption given to governments and companies for use of an AI system which is "purely accessory" and used in a minor matter could, in fact, "undermine the entire act", warns Sarah Chander, who leads the policy work on Artificial Intelligence for European Digital Rights (EDRi), a network of non-profit organisations working to defend digital rights in the EU.

High risk

The Commission proposal focuses on so-called "high-risk" AI systems which may jeopardise people's safety or fundamental rights, such as education (for example, scoring of exams), employment (like CV-filtering software for recruitment), or public services (for instance, credit-scoring to deny people the opportunity to obtain a loan).

Companies who want to compete using AI systems in this "high-risk" category, would have to meet EU requirements, such as explainability, risk-assessment, and human oversight.

Some worry, however, that these requirements could discourage start-ups and businesses from investing in Europe in such AI systems, giving the competitive advantage to the US or China.

Companies that fail to comply with the legislation could face fines of up to €30m or six percent of their global turnover.

Chander highlighted that some of the biggest harm from AI systems could come in the delivery of public services, such as social services, policing (where predictive-policing based on mass surveillance is a major concern), and migration. AI-based social decisions are dangerous because AI systems make assumptions seem like facts.

Chander says the commission's proposal do not go far enough on restricting the use of facial recognition. Her organisation also wants to ban the use of AI for predictive policing and migration as well as for predicting emotions.

Rights-defenders argue that companies should be obliged to make fundamental rights impact-assessments, and provide information on where and how an AI system would be used and its impact on individuals.

They also want clear information provided to the public, and say citizens should be able to seek explanations from public authorities or companies. Citizens should also be able to claim remedies if a company or an authority has violated the AI Act, or an individual has been impacted by a banned system.

Chander said there is a common misunderstanding that AI system can be "perfected", and policy-makers often ask her how to make these systems better, less prone to bias. But that is the wrong question, she argues because the problem is AI systems are replicating an already discriminatory system.

"Some systems cannot be made better," she says, adding: "We don't want to make a perfect predictive-policing system, or perfect lie detector".

This article first appeared in EUobserver's magazine, Digital EU: the Good, the Bad — and the Ugly, which you can now read in full online.

Magazine

The challenge of artificial intelligence

The fast-growing impact of artificial intelligence will be the biggest challenge for business and consumers in Europe's single market of tomorrow.

Eight EU states miss artificial intelligence deadline

Pan-European strategy "encouraged" member states to publish national artificial intelligence strategies by mid-2019. Germany, France and the UK have already done so - others are lagging behind.

To lead in cyberspace, the EU needs to avoid digital tribalism

To avoid digital tribalism the EU needs a strategy to better engage with the Global South, including the emerging digital powers such as Brazil, Egypt, Ghana, India, Indonesia, Jamaica, Kenya, Mexico, Singapore, South Africa, and Senegal.

Digital EU: the Good, the Bad — and the Ugly

The European Union has impressive digital ambitions and an equally-impressive array of initiatives, proposals, directives and regulations, all designed to make the bloc 'fit for the digital age'. But what do they all mean — and will they all work?

Vestager: 'Technology must not steal our time'

Given the rapid pace of digitalisation, the EU has rushed to set and regulate digital standards. Many new initiatives are led by Margrethe Vestager, EU commissioner for competition, who says the focus must be on making technology work for people.

Latest News

  1. Greece faces possible court over 'prison-like' EU-funded migration centres
  2. How the centre-right can take on hard-right and win big in 2024
  3. Top EU officials show Ukraine solidarity on risky trip
  4. MEPs launch anonymous drop-box for shady lobbying secrets
  5. Hawkish ECB rate-rise 'puts energy transition at risk'
  6. MEPs push for greater powers for workers' councils
  7. How Pavel won big as new Czech president — and why it matters
  8. French official to take on Islamophobia in EU

Stakeholders' Highlights

  1. Party of the European LeftJOB ALERT - Seeking a Communications Manager (FT) for our Brussels office!
  2. European Parliamentary Forum for Sexual & Reproductive Rights (EPF)Launch of the EPF Contraception Policy Atlas Europe 2023. 8th February. Register now.
  3. Europan Patent OfficeHydrogen patents for a clean energy future: A global trend analysis of innovation along hydrogen value chains
  4. Forum EuropeConnecting the World from the Skies calls for global cooperation in NTN rollout
  5. EFBWWCouncil issues disappointing position ignoring the threats posed by asbestos
  6. Nordic Council of MinistersLarge Nordic youth delegation at COP15 biodiversity summit in Montreal

Stakeholders' Highlights

  1. Nordic Council of MinistersCOP27: Food systems transformation for climate action
  2. Nordic Council of MinistersThe Nordic Region and the African Union urge the COP27 to talk about gender equality
  3. Friedrich Naumann Foundation European DialogueGender x Geopolitics: Shaping an Inclusive Foreign Security Policy for Europe
  4. Obama FoundationThe Obama Foundation Opens Applications for its Leaders Program in Europe
  5. EFBWW – EFBH – FETBBA lot more needs to be done to better protect construction workers from asbestos
  6. European Committee of the RegionsRe-Watch EURegions Week 2022

Join EUobserver

Support quality EU news

Join us