Subcommittee on Human Rights Joint meeting Committee on Security and Defence Workshop addressing the topic of human control and AI-supported weapon systems: human rights risks, accountability and poli
Legend
Transcript markers
- text Sentence being played
- text Search result
- text Intervention cited by the link you opened
Click a sentence to play the video from that moment. Select text to quote it with the speaker, timestamp and link.
More options
Offset
Transcript offset
Export
Live capture
Versions of this video
Transcripts may differ slightly from one version to another.
DROI European Parliament
Follow its sittingsMarie-Agnes Strack-Zimmermann
AI has changed the speed of war. It has changed who decides. It has changed the very meaning of accountability in the use of force. Europe cannot stand by while Washington, Beijing, and Moscow write, or rather fail to write, the rules of AI-driven warfare. This is a question of values, and at the same time, a question of humanity's survival.
I will not go any longer because our time today is limited, and what we all want is to hear the voices of the experts joining us. So thank you very much for being here. Let me now give the floor to our speakers, first to the author of the study, Neta Goussac, for a presentation of about eight minutes, and after then to Ms. Mense. So the floor is yours.
She's online, sorry. There she is. Good morning and the floor is yours.
Netta Goussac
Good morning, Chair Saturi, Chair Strack-Zimmermann. Thank you for your remarks, honorable members and participants. Thank you for the invitation to join you. I regret that I can't be with you in Brussels and I'm grateful for the opportunity to join remotely. My task is to set out the main findings of the subcommittee that the sub- the main findings of the study that the subcommittee commissioned.
Ms. Manson and Dr. Dorsey will then bring perspectives from their respective reporting and scholarship. Two developments in my view explain why the subcommittee commissioned this study, echoing the remarks by the two chairs. The first is the technology. Artificial intelligence is not a discrete category of weapon awaiting development. It's an enabling technology already integrated across military activity, from logistics to intelligence to targeting, and its use is shaping the conduct of current conflicts.
For more than a decade, international discussion of these systems has been anchored in international humanitarian law. The human rights dimension remains underexamined, and the study addresses that gap and what it means for EU policy. The human rights lens matters for two reasons. First, AI-enabled military systems may be deployed outside of armed conflict. They may be used in operations to which international humanitarian law does not apply, like counter-terrorism, border and migration control, maritime law enforcement, and surveillance.
The second is a gap in the debate.
There, the governing framework is international human rights law. Second, the human rights standard for using force can be stricter than the standard in armed conflict. An operation that might be defensible under the laws of armed conflict could violate human rights law. Predictive and target recommendation systems can readily fall short of the human rights law requirements of necessity, imminence, and proportionality, especially where their outputs influence decisions to use force.
And disparate accuracy in AI systems can translate into disparate and even discriminatory exposure to the use of force, deprivation of liberty, and intrusive surveillance. Meaningful human control has become the organizing concept of the international policy debate on autonomous weapons and more recently AI in the military domain. It runs through more than a decade of discussions at the UN and through the EU Parliament's own resolutions. Across the various formulations over the years, three terms recur: control, oversight, and judgment. They're frequently used interchangeably, but the study treats them as distinct.
Control is exercised within a system, oversight sits outside it in like reviews and authorizations, and judgment is the reasoning that the law presupposes. The study gives the concept of human control operational content, identifying five elements. Systems must be predictable and reliable. Operators must understand both the system and the situation. Operations must be constrained in time, in space, and in task.
Humans must retain a real capacity to intervene.
And outcomes must be traceable so that someone can be held to account. Three features of AI-enabled military systems constrain each of these five elements. The first is speed. AI is valued precisely because it turns sensor data into outputs like target nominations, recommended courses of action, at scale and at speed. But the obligations I've described presuppose time for judgment.
Speed that outruns that time can make these obligations impossible to discharge. The second reason is automation bias. People tend to defer to machine outputs even when those outputs are wrong. So human-in-the-loop type safeguards can be weak precisely under the kinds of time pressure where they matter the most. And the third reason is algorithmic opacity.
Modern machine learning systems produce outputs whose internal logic is not readily intelligible, and that makes control harder to exercise in the moment and afterwards harder to investigate what went wrong, attribute responsibility, and provide victims with a remedy. The study examines the principal mechanisms for reacting when something goes wrong. Individual criminal responsibility, for example, presupposes an identifiable person acting with intent and with knowledge. But where an officer authorizes a strike against targets nominated by a decision support system, it's an open question whether that mental element can be established, and causation might be diffused across training data that was assembled years earlier, design choices, procurement decisions, and operational choices. State responsibility is in principle better suited to harms of this diffuse character.
It requires no proof of mental state, and a state that deploys a system with insufficient testing, weak safeguards, or no provision for investigation might be held responsible regardless of whether any specific responsible individual can be identified.
But here the constraint lies in enforcement, which is slow, weak, and largely inaccessible to victims. The study's conclusion is that the responsibility gap, which is often mentioned, is not that no one is responsible, but that responsibility is either misallocated or unallocated. The study finds that the EU is well placed to act, and the EU Parliament in particular. The study makes 12 recommendations, and I'm going to highlight just four. First, a fresh Parliament resolution, updating the 2018 and 2021 positions in light of technical developments both and policy discussions, developments in the policy discussions, spelling out the real operational content of meaningful human control along the line of the five elements I've described.
Second is scrutiny.
So for example the EU AI Act excludes military defense and national security systems from its scope and Parliament can monitor how that boundary operates in practice to ensure it operates as intended and it can use its budgetary powers over the European Defense Fund and European Defense Industry Program to ensure that funded systems meet the standards that the EU is seeking to meet. Third, a legislative initiative requesting the Commission to propose a framework for human rights due diligence by providers of military AI. And fourth, beyond Parliament, the Council could adopt an EU common position on autonomous weapons and military AI or failing consensus it could adopt Council conclusions and the Commission could issue guidance on Article 10 paragraph 6 of the EDF regulation, the provision that makes EU funding ineligible for autonomous weapons that lack meaningful human control. To conclude, chairs and honorable members, existing law of course fully applies to AI-enabled military systems. The question is whether the institutions that must apply this law retain the practical capacity to do so in light of the special characteristics of AI and the study's recommendations aim to ensure that they do.
I'm very grateful for the opportunity to discuss this important matter with you today and I look forward to the discussion.
Marie-Agnes Strack-Zimmermann
Thank you very much.
And now I give the floor to Katrina Manson for seven minutes, also online. So welcome and the floor is yours.
Katrina Manson
Thank you very much to everyone listening. I am author of the new book Project Maven: A Marine Colonel, His Team, and the Dawn of AI Warfare, which investigates the US pursuit of AI warfare over the past decade and gives concrete examples of how AI has performed in real combat situations. I am glad to describe some of my findings today. I am speaking to you in an independent capacity and emphasize that as a reporter I take no specific policy position myself. The US has been pursuing the development of AI warfare in earnest since 2017 when it started an effort named Project Maven.
It was ostensibly an effort to use AI to analyze drone footage too copious for human eyes to process. But the Deputy Defense Secretary at the time has since in retirement explained to me that this was always part of a broader pursuit of autonomy.
He described this autonomy as autonomy at rest and autonomy on the move, and I'll explain what he meant by both. Autonomy at rest came to mean a mission control platform to integrate intelligence, operations, and visual display, much like Google Earth for war. Today, the platform that has emerged from this effort is called Maven Smart System. It sucks in more than 150 separate intelligence and other data feeds, and the US is using it in every combatant command or military theater around the world, including for US operations against Iran. It can pair a target to a weapon system, and US military commanders have told me that ultimately they expect this process to become automated.
Autonomy on the move has come to mean drones that could navigate autonomously in groups or swarms and ultimately select a target and fire at it without human intervention.
The backers of these efforts argue they would save the lives of US military personnel and could also safeguard allies and civilians too and dissuade adversaries from fighting the US in the first place. Others have argued, as you've just heard, that AI could speed and scale war and sacrifice accuracy by giving too little time for human review of targets, make war a more appealing policy tool, and introduce inaccuracies related to AI that can span bias, hallucination, algorithmic drift, and over-reliance on machine outputs without checking underlying data. I reveal examples in my book where computer vision algorithms go wrong. Some algorithms confused objects, misidentifying people and buildings. In the opening days of Russia's 2022 invasion of Ukraine, I report in the book that algorithmic accuracy used by the US to identify potential targets dropped precipitously from 70% accuracy to 30% and sometimes as low as 10%.
That's because the algorithms weren't familiar with this territory. Maven's algorithms failed to identify Russian tanks in the snow, in part because they were used to identifying military objects against a backdrop desert or jungle. Gathering new data helped retrain models and model performance did start to improve, but there were other problems.
AI couldn't recognize battle damage such as a Russian tank with the turret blown off. In one early case that predates the Russian invasion of Ukraine, Microsoft, which provides algorithms for Maven, decided to produce algorithms for Maven that could only recognize person or building. I learned military operators were unhappy with this, worried that the AI could not distinguish between men, women and children and only identify person. In August 2021, the US struck a compound in Kabul as part of its withdrawal from Afghanistan. The US initially said it was a righteous strike but later reframed this as a tragic mistake.
Seven children and three adult civilians were killed. That strike I learned did not use Maven, but afterwards the Maven team ran its AI over the drone video footage and the AI also could not identify any of the children on the footage. Turning up the sensitivity levels for the algorithms did not help. The light in the early evening was too low and the shadows too long. A former US defense official who visited the frontline of Ukraine came back shaken.
Atrocities are possible in the era of AI, the former official told me. I have reported that AI targeting has already sped up the kill chain. With the integration of computer vision into Maven smart system, I learned the US can now hit 1,000 targets in a day, up from under 100 targets before. With the addition of large language models to speed up other processes, including intelligence and admin, the US can now potentially hit 5,000 targets a day, I was told. Some military ethicists within the US system worry about the potential gamification of war.
The pursuit of autonomy, autonomy on the move, has come with technological challenges. I reported in my book that the US has developed autonomous jet skis as part of a program named Whiplash and that the CIA smuggled these into Ukraine.
When a jet ski packed with explosives landed up on the Turkish coast in 2024 on the southern side of the Black Sea, I discovered it sparked consternation inside the Pentagon that the program would be revealed. In other tests, an autonomous drone boat went rogue and upturned a captain into the water. I also found examples where a drop of ocean spray on the lens of a drone boat camera could interrupt the computer vision tracking. These of course are examples as part of research development trying to improve the technology. More recently, US Central Command has said it used three autonomous drone boats to explode targets in Iran, and today the US is seeking to expand both AI targeting and autonomous weapons systems, including for voice-controlled drone swarming technology.
I recently reported on a non-public Pentagon document known as the Joint Targeting Doctrine 360 that I wanted to draw your attention to in light of the discussion of meaningful human control, which is not the standard that the US military uses. The document includes, it was most recently updated in April 2026, and the document I discovered includes an entirely new chapter that foresees a time where AI will initiate actions in war, not humans.
Humans under this system would only monitor, rather than current human-in-the-loop systems in which humans initiate actions according to this doctrine document. This would mark a significant departure for US doctrine and for the future of war. The US doctrine argues that the speed of future warfare, along with US adversaries' own advances in AI, may require the adoption of "completely autonomous systems." The same US doctrinal document came with warnings. It says there is a need to exercise caution to avoid over-reliance on AI output without human oversight.
And the new US targeting doctrine also says that the risk of relying on AI for autonomy poses serious moral and legal dilemmas and requires the establishment of unambiguous ethical guidelines to mitigate concerns with AI-enhanced decisions. My understanding is that those guidelines are currently being revised and not yet public. Thank you, Katrina.
Marie-Agnes Strack-Zimmermann
Thank you very much. And finally, now the floor is for Jessica Dorsey for seven minutes.
Jessica Dorsey
Thank you very much to the chairs Sartori and Strack-Zimmermann for hosting us here today and for all of you for being here.
It's a pleasure to join Katrina Manson and Neta Goussac in this important and timely discussion. My thanks as well to Neta for a comprehensive study on this topic and to Katrina for Project Maven, which is an incredibly detailed and eye-opening account of how AI is already reshaping contemporary military operations. I echo much of what these brilliant women have already highlighted and I hope to build out from their insights. As we've heard, AI is no longer the future of warfare alone. It is increasingly being integrated into intelligence analysis, target identification, prioritization, logistics, and battlefield decision support today.
AI is not only changing the weapons we use, it is changing how military decisions themselves are made. Against this backdrop, I would like to complement the perspectives we've already heard by offering three observations that I hope are useful by traversing both human rights law and international humanitarian law perspectives.
As Neda mentioned, much of the discussion around autonomous weapons and AI in warfare has naturally focused on the application of IHL during armed conflict, but human rights law continues to provide an essential framework for transparency, accountability, democratic oversight, and the protection of human dignity alongside it. AI raises questions not only about how force is used, but also about how decisions are made, how they're scrutinized, and ultimately how people are held to account. I hope these observations will also help frame our discussion with you afterwards. The first observation is that the conversation is no longer nor should it be limited to autonomous weapons. Much of the public debate has understandably focused on fully autonomous weapons, the so-called killer robots, or weapons once activated that can select and engage a target without human intervention.
But many of the systems currently being fielded are not fully autonomous at all. They are the AI-enabled decision support systems that are built to assist humans in gathering data, identifying objects, analyzing intelligence, generating and prioritizing targets, or coming up with planned courses of action. These systems often technically leave the final decision to a human operator, yet that does not mean the legal or ethical questions disappear. International humanitarian law and more broadly the protection of fundamental human rights is built upon the exercise of qualitative human judgment, not merely human presence or blindingly accepting a computer's recommendation.
Understanding this is crucial to understanding where the risks and opportunities lie. If AI systems shape what information decision makers see, how they see it, and how they interpret uncertainty, or which options appear most persuasive, then they are also shaping the judgments upon which the legal obligations ultimately depend. Therefore, the main area of focus ought to be whether the human rights retain, sorry, whether humans retain the time, authority, information and institutional conditions necessary to exercise appropriate levels of human contextual legal judgment. The second observation is that AI can create what I have elsewhere called the illusion of precision. Katrina talked about the possibility of gamification of warfare, which we need to absolutely wrestle with.
Additionally, AI systems frequently produce confidence scores, probabilities, rankings and highly sophisticated visualizations. These outputs can create a perception of certainty, neutrality and objectivity. But technological precision is not the same as legal precision.
We heard from Katrina about how the accuracies of these algorithms can vary greatly. But even if, and this is a big if, we get much improved technology, greater computational accuracy does not necessarily translate into greater protection for civilians or better respect for the right to life. Determining whether an operation complies with IHL requires contextual assessments that cannot simply be reduced to numerical or algorithmically calculated outputs. Distinguishing between combatants and civilians, for example, taking precautions to avoid civilian harm, determining military necessity versus anticipated civilian harm in proportionality assessments, and constantly changing battlefield conditions are part of that qualitative judgment. Indeed, there's a risk that quantification itself can obscure rather than illuminate the uncertainty inherent in these decisions.
The elements of speed and scale complicate these issues greatly. One consequence is that we're beginning to confuse computable information with legally relevant information. This is first and foremost a governance challenge rather than being just a technological one. My third observation and perhaps, at least in my view, the most important one is that accountability begins long before deployment. When we discuss accountability, we often think about investigations after harm has occurred, and those mechanisms remain essential.
But many of the decisions that determine whether military AI will be used responsibly have already been made months or even years earlier. Investment in research and development, planning decisions, design choices, procurement choices, training data, system architecture, testing, evaluation, verification and validation, the assumptions that we embed into optimization objectives. All of these shape how systems will perform long before they're ever deployed. For that reason I increasingly think we should understand accountability more broadly, not only assigning legal responsibility after harm occurs but as ensuring that those making decisions throughout the life cycle of military AI systems can explain, justify and where necessary revise those decisions to avoid or minimize harm in subsequent operations. Accountability is therefore not an endpoint, it's a continuous process.
That continuous process is what allows institutions to identify risks early, prevent foreseeable harm where possible and provide meaningful scrutiny when decisions affect people's lives. So what might this mean for you as members of the European Parliament? I would suggest four practical questions for you to think about and to ask.
The first, when we're discussing military AI, we need to think beyond autonomous weapons. Ask how AI is influencing military decision-making across the entire targeting process and broader operational lifecycle. Second, ask for evidence, not assumptions, particularly where claims are made that AI will improve compliance with IHL or human rights or reduce risks to civilians. When governments or companies claim that AI improves precision or reduces civilian harm, Parliament should ask compared to what? Under what conditions?
What evidence supports that claim? Third, please scrutinize investment, procurement, acquisition, and development, and not only deployment. Governance is often discussed as something that happens once a capability exists. In reality, many of those consequential governance choices are made long before during procurement and finalized once systems enter operational service. Finally, I ask that you remember when we talk about responsible AI, it is not simply about keeping a human nominally involved.
It's about responsibility by design. It is about preserving the quality of human judgment, protecting human dignity, ensuring meaningful oversight, and maintaining democratic accountability from the very beginning of the planning phases throughout the entire lifecycle of these systems, rather than just thinking of it when things go wrong.
Europe has long sought to demonstrate that security and the protection of fundamental rights are not competing objectives but mutually reinforcing ones. We're now at an important geopolitical inflection point. We are seeing military AI become more deeply embedded in defense, but as we observe this, we also should realize Europe has an opportunity now to show leadership, not simply by developing these technologies, but by shaping how they are governed. This means anchoring our values, human rights and fundamental freedoms, our legal commitments and democratic accountability from the earliest stages of the military AI life cycle, rather than trying to retrofit them after systems have been designed or deployed. The most consequential choices in AI-enabled warfare will not be made only on the battlefield.
They will be made in investor boardrooms, across research agendas, procurement decisions, oversight mechanisms and the governance frameworks we establish today. Those choices will shape not only how military AI systems perform, but how effectively they protect civilians and fundamental rights, what kind of military decision-making they encourage, and ultimately, what kind of Europe we want to defend. Thank you.
Mounir Satouri
Bien, merci mesdames les expertes pour tous ces apports.
On va passer au débat avec les membres. On va commencer par les coordinateurs DROIT et SEDE quand ils sont là, et je crois que c'est Monsieur Gomart qui commence au nom du PPE.
Christophe Gomart
Thank you for your insights. We have heard that military AI is already a reality, while the AI Act does not apply to military systems. In this context, is the European Union falling behind on an issue that has major implications for human rights? As these technologies are increasingly used in military operations, what concrete steps should the EU take to guarantee accountability and preserve meaningful human control over the use of force? This meeting confirms the urgent need for a SEDE report on defense AI, which I have been calling for over the past few months.
Our committee cannot remain a spectator when the next MFF does not, at this stage, identify this as a strategic priority. We must start preparing today for what will become one of the key pillars of the next war. Thank you.
Mounir Satouri
Merci Monsieur Gomart. Monsieur Galère. Non. Ah non. So quickly.
Okay, okay. J'ai compris que vous étiez pas loin de là où était assise Isabelle, je pensais que vous parliez dans ce temps-là. Donc on va passer au groupe socialiste, Sven Mikser et puis Nacho. Une minute trente.
Sven Mikser
Yes, thank you, thank you very much for the presentations. Obviously there are very important legal and moral implications that have to be taken into account when we consider the future of AI-powered weapon systems and data analysis and so on. When it comes to assessing the accuracy and the quality of AI-powered systems, we cannot only take into consideration the absolute safety or accuracy, but we have to assess it against the alternatives, so non-AI-powered or non-digital or analog solutions and also ask the question of whether they are more or less accurate and more or less safe than the alternatives.
I think, but obviously it's more difficult, or it's easier in banking, for example, than it is when human lives are at stake. But my question about the what was referred to as gamification of war, it also has implications on a strategical level because obviously the AI-powered war lowers, because it reduces the not it not only improves the lethality, but it also reduces the risk to one's own servicemen and women. And that actually lowers the threshold of taking the political decision to use force. And also the same actually happens with the extreme speed of data processing. It also lowers the threshold of using force.
Do you see this being reflected in the doctrine in the EU countries, but also elsewhere?
The fact that it's actually easier to wage war because of the lower threat, a lower risk to your to your own people.
Nacho Sánchez Amor
Merci, Nacho.
Thank you. This is a very difficult very difficult balance and with a moral implication to maximize the human control on any kind of system used in war and to maximize effectiveness from the military point of view and at the extreme of this line is what our guest called tragic mistakes and I'm recalling the case of the school girls in in Iran. The problem is the narrative behind this illusion of accuracy because apparently in the military system and the military defense ecosystem they are selling to the public opinion this is a way to be more accurate and then to avoid tragic mistakes. And we don't know as we are not experts till what extent this is real or is just a narrative to cover the maximize the implication or the use of autonomous weapons. I would like to know from any of you what is your assessment as my colleague from the PPE said about the exclusion of the AI Act of the military aspect that apparently is going to be in some way reversed in the in the future.
Your opinion on the idea of the United Nations of create a legally binding agreement on globally regulate and ban lethal autonomous weapon or at least to impose a moratorium. And regarding your proposals, Mrs. Gosak, about the eligibility rationale on the European Defense Fund regulation. What are we going to pay with European money regarding autonomous weapons? These are the things I would like to know from you some more explanation or information. Thank you.
Catarina Vieira
Thank you very much and thanks to all the speakers. It's very clear that by all of your interventions that AI is already shaping our reality in military operations and that we have a responsibility to act here.
Something I'd like to highlight from the report that I don't think was mentioned until now was the use of Israel's military AI-assisted targeting systems such as Lavender and Where's Daddy, which led to the mass generation of 37,000 human targets with virtually no human oversight. And especially at the beginning of the genocide, these decisions, these AI-led decisions seemed to point out that targeting these 37,000 people in their own homes while they were asleep and their families were with them in the bedroom or in the house could be okay as long as the casualties other than the targets were 15 to 20 civilians. And what I want to say with this is that this is a catastrophic automated violation of humanitarian law and something that we need to pay a lot of attention with concrete cases and concrete examples. It also shows, as your interventions did, that our international legal frameworks that were developed before these technologies are struggling to address with the new challenges in decision-making and accountability and they need to be updated and revised. And that's a tough pathway, but in which we all have a role to play.
And here in the European Parliament, I do believe that we have a key role in ensuring robust democratic scrutiny, overseeing how the EU defense funding is used, demanding transparency and helping shape this new international framework that is up to date to these technological advances.
I thought the recommendations of the study were very interesting and I'd like to specifically ask about recommendation number four, when it comes to a framework requiring human rights due diligence by providers of AI. What more concretely in more detail would you expect the European Parliament and the European Commission to bring forward when it comes to human rights due diligence for AI? Thank you.
Mārtiņš Staķis
Thank you chair and thank you for your presentations. I have to say I fully agree with everything that was said that autonomous weapon systems should be operated only under meaningful control of the humans. However, my question is how to persuade about that also the war criminal Putin, Xi Jinping, Kim Jong-un and others because I think the arguments you presented here will carry not, you know, much weight to them. Thank you.
Lynn Boylan
Thank you chair and thanks to all of the witnesses and for their presentations. Really, really fascinating stuff and I think that this question of AI-enabled weapons is one of the most profound moral questions of our time.
And I think the answer to the question I would find is with UN Secretary-General António Guterres when he said there's no place for lethal autonomous weapons systems in our world, that machines that have the power and discretion to take human lives without human control should be prohibited by international law. I also note that Amnesty International, Access Now, and many other organizations and international experts initiated a joint statement last month calling for tech companies and states to halt the provision of AI systems for use in the military kill chain and to take all steps to ensure that other AI systems they provide do not cause or contribute to violations of international humanitarian law and international human rights law. And I think that this is important for the framing of the debate. When you have the UN Secretary-General and the world's leading human rights organizations clearly calling for strong international action and political leadership on this issue, I think we should really be listening to them. I would also, as my colleague in the Greens has raised, I would like to ask the witnesses specifically about Israel's use of AI-enabled weapons because we know and multiple reports have demonstrated that Israel utilizes AI at the targeting and repression of Palestinians in Gaza, and Palestine has been referred to as the laboratory for the field testing of these weapons.
So do the witnesses agree that Europe and the United States are in fact profiting from the reprehensible use of AI against Palestinians?
Mounir Satouri
Alors, pour conclure le tour des coordinateurs, je donne la parole à Hans Nohoff, et ensuite les experts vont répondre, et on pourra faire un deuxième tour de prise de parole pour les collègues qui le souhaitent. Allez-y.
Hans Neuhoff
Vielen Dank, ich werde Deutsch sprechen. Ich werde Deutsch sprechen, auch aus dem Grund, dass die Themen, die Sie hier aufgeworfen haben, außerordentlich komplex sind und alte Fragen betreffen, nämlich die Fragen von Krieg, Macht und Moral. Ich habe weniger eine konkrete Frage an Sie als ein paar Beobachtungen, die Sie aber natürlich auch zu einer Stellungnahme veranlassen können. Zum einen geht es ja auch militärisch betrachtet um das Thema der Abschreckung, also eines neuen Instruments von Abschreckung, von Deterrence, wobei mir ein wesentlicher Unterschied darin zu bestehen scheint, in Abgrenzung zu den Atomwaffen, dass bei den Atomwaffen niemand sie wirklich einsetzen will, während bei den KI- oder AI-gestützten Systemen offenbar eine hohe Bereitschaft besteht, sie einzusetzen.
Und in der Ukraine findet das ja gegenwärtig schon statt. Und der Punkt, auf den ich zu sprechen kommen möchte, betrifft die Rolle von Privatunternehmen, insbesondere US-amerikanischen Hightech-Konzernen in diesem Zusammenhang, denn wir wissen, dass sie es sind, die die Entwicklung hier entscheidend vorantreiben. In der Ukraine ist es vor allen Dingen Palantir, das die Möglichkeit eben hat, hier seine Datenanalysesysteme unter Realbedingungen einzusetzen, zu erproben und auf diese Art und Weise eben einen entscheidenden Entwicklungsvorsprung zu erlangen. Die Frage, die sich mit Blick auf die von Ihnen aufgeworfenen Kategorien aus meiner Sicht ergibt, ist, welche Möglichkeiten haben wir denn überhaupt, das hier zugrunde liegende Verhältnis, nämlich das, was gerne als Public-Private-Partnership beschrieben wird, also die Tatsache, dass immer mehr Macht, immer mehr Zuständigkeit auch bei Privatunternehmen liegt, wie wir das überhaupt in den Griff bekommen können. Das internationale Recht, das Völkerrecht zeichnet sich ja bekanntlich dadurch aus, dass es zwar moralische Kategorien aufstellen kann, faktisch aber machtlos ist.
Wie wollen wir unter den gegebenen Bedingungen, insbesondere in den USA, nämlich dass die treibenden Kräfte Privatunternehmen sind, das überhaupt in den Griff bekommen? Vielen Dank.
Mounir Satouri
Bien, je vais donc donner la parole aux experts pour répondre. On va commencer par Madame Goussac dans l'ordre de tout à l'heure et vous avez cinq minutes chacune pour répondre.
Netta Goussac
Thank you very much for those questions and for those very astute and helpful observations. In the five minutes available, I will try to address the key questions I was able to note as much as possible.
First, in relation to the exclusion in the AI Act of systems that, of AI capabilities that are specifically for defense or security applications, I don't want to speculate on where the legislation might go and whether that exclusion will remain. But certainly it would be helpful, as I noted in the study, to clarify the boundaries between military systems that are for military defense and national security and systems that are not. I think this would be helpful both from an industrial perspective, but also to ensure that the Act's substantive transparency and risk management standards are both being applied to the systems to which they are intended to apply and are being used as a benchmark even for excluded systems. I think there is this opportunity to really scrutinize how the Act is being applied in practice and developing ideas for how the intent and the spirit of the legislation can be ensured in practice.
When it comes to regulation of autonomous weapons systems at the international level, I wanted to mention that in general the recommendations in the study could be put into three categories. The first is about standard setting, the second is about scrutiny, and the third is about accountability. In this is my personal view, just my personal reflection is that I would rarely wish to put all of my trust in one of those three categories of action. And so accountability alone is for me not enough of an action item for any actor, including for the EU, because it really only is something that comes up when something goes wrong. But standard setting and scrutiny over the implementation of those standards can have a marked effect on preventing things from going wrong.
So at the international level, of course, the EU has been a significant voice in calls for the regulation of autonomous weapons systems, the specific regulation beyond existing international humanitarian law. And I think then at the EU and at the national level among EU countries, the positions are more mixed.
And so for me as a researcher, I'm certainly interested in in that tension that exists within the EU and would be interested to to know from the members how they they see it too. But certainly, I think at the very least clarifying standards at the international level, and by this I mean not just under international humanitarian law, but also under other areas of applicable international law, would certainly be a good start and definitely a good basis on which to decide whether and how normative development should take place. On the EDF, the question regarding eligibility for funding, again, this is an example of a tension, right? Because of course there is a desire to, it's a very evident desire to accelerate defense industrial activity within the EU. And AI-enabled military systems and systems with autonomous functions are part of that push.
The EU has a clear policy position around meaningful human control, but what's less clear, at least from as an outsider, is how that is being implemented, how you can ensure that that requirement is both clarified and strengthened through a clearer understanding of what it requires, clearer screening obligations, both at the procurement stage and later in the operational and decommissioning phases of systems, and transparency and reporting duties as well. The final point I can address, and hopefully I've not run out of time, is just in relation to the number of members who mentioned Israel's use of AI in its conflict with Hamas recently in Gaza. I don't want to make a comment on the legality or otherwise of the behavior, but of this use of AI, but certainly from an EU perspective, I think it demonstrates the findings of the report around speed and opacity and also automation bias.
So certainly very starkly illustrates the risks associated with the use of AI from an international humanitarian law perspective, but also from a human rights perspective. I think in terms of how to react to that from a regulatory perspective in the EU, it's that's where the report's recommendations are sort of trying to are trying to land because I think the risks are clear and very clearly demonstrated by recent conflicts, but what's less clear is exactly how to prevent that kind of system from being developed, how to prevent it from being adopted, how to ensure that systems that are adopted are used appropriately and how to ensure accountability when things like this go wrong. The report does address the accountability, there was a specific question regarding the role of the private sector, the report does address the accountability and responsibility of private sector actors, but this is much less, I don't know, it's much less clear and it's much more challenging to react to from an international law perspective.
I don't know if I could resolve it neatly in the report, but at least there is some content there to try to at least identify the fact that you cannot rely on existing international law measures alone in order to ensure responsibility and accountability, and specific research and study into the other available mechanisms might be useful here. Thank you.
Katrina Manson
Thank you very much. I thought I could pick up on a few elements that some of your contributors mentioned. One contributor mentioned Minab, the school in Iran that was struck on the opening day of US operations against Iran. I wanted to share my most recent reporting on that, which of course is incomplete.
The US is undertaking an investigation into what happened in Minab and my latest reporting is that it was submitted, the report was submitted in April and it is currently under review at US Central Command. The element that I recently reported is that an intelligence analyst in 2019 noticed changes at that site and flagged those changes into a military database, but that military database was not connected to the authoritative foundational military database that the US uses for much of its targeting work. I mention that because at the moment I am not aware of what role, if any, AI played in that strike. I am aware that the Pentagon is now seeking to use more AI to try to knit up very disparate intelligence databases to try to make sure that they are as up to date as they can be and that these disparate systems and pieces of information that can prove very critical are noticed.
And so from the Pentagon perspective, there is an argument that AI may be able to play a role in revealing information that might not otherwise be caught in disconnected digital systems. For the people who mentioned Israel, my primary reporting is on the US, so I don't have anything specific to add to that. But I would say that from the US perspective, I thought it was very interesting that former Chairman of the Joint Chiefs of Staff, retired General Mark Milley, referenced that reporting in 972 Magazine, that it was reported that military operators were reviewing targets sometimes for as little as 20 seconds according to that reporting from 972 about Israel. And that General Milley, retired General Milley himself, referred to AI as a Pandora's box and a double-edged sword. And so the top US military experts and officials are considering AI both as something they wish to pursue, but also something they see comes attached with great risk as well.
Those are the two contributions I thought I could usefully offer.
Jessica Dorsey
I've so many things to say about this, so I'll try to keep it organized and address some of the comments in turn. First, starting off with conversations around the IDF use, I think it's very important to think about using these case studies as far as what we know in the public domain about how these systems are again shaping the way decisions are made. Design choices that are made, excuse me, to quantify certain variables, to think that law can be coded in and to anchor civilian harm levels at what has been reported sometimes in proportionality assessments that for one low-level target that there would be 15 to 20 civilians accepted. That is a very poor reading of what the law requires. In fact, it eschews, it avoids, it skips over obligations to take precautions in attack and the duty of constant care to avoid or to any extent minimize civilian harm prior to moving into a proportionality assessment.
Thanks very much.
And what I think is part and parcel of how AI fits into this and how this sort of interacts with the legal framework, and I wrote about it and I couldn't leave the stage without doing the ultimate academic flex, I just wrote a book about this kind of stuff.
I did a deep dive into the IDF's reported use on these systems and what they are doing to legal assessments. And I think that what comes out of this, what's really actually quite enlightening and why I co-direct something called the Realities of Algorithmic Warfare research projects rather than sort of the promises or the, you know, one day AI may do this. Well, what is it doing right now? And I think wrestling with these kinds of things, not only the IDF use, the reported use at enormous time compression, the speed and scale is what we see. And you know, this is sort of against the backdrop of again that precision narrative, the idea that these are the most technologically advanced militaries in the world using some of the most technologically advanced munitions, systems, etc.
But it's very difficult to see the visual evidence of what Gaza looks like right now because if we're talking about precision in warfare, we can't accept that at face value unless we mean that we're destroying precisely everything.
And I think that that we need to wrestle with what speed and scale looks like on the ground. Fast forward to more recent discussions about Minab, the conflict in Iran, the use by the United States, the, you know, the Israel conflict. Starting in 2023, you know, we were looking at unprecedented numbers of generated targets and then now we're even to a scale that's just very difficult to comprehend. The US itself CENTCOM has acknowledged in the first 96 hours to have hit 5,000 targets. I mean, I'm a lawyer, so I don't love math, but doing the math on that really shows that there's just not enough time.
I don't know how it's, you know, that you can build in enough judgment to verify and validate each of those targets. And then that leads to continuous incidents of civilian harm. Minab is one of many, of hundreds, right? I think we rightly need to talk about Minab, but it is again, you know, it's often written off as sort of a tragic mistake, that's the term that they use, and it is tragic. But mistake implicates that it was something that couldn't have been prevented.
But the idea, of course, leaning on extraordinarily good reporting by Katrina and others regarding the fact that these outdated satellite images had been available to be updated and were updated, but then didn't propagate through a system that might have been used. That is problematic.
And the idea is if you don't focus on what your legal obligations require you to do as far as precautions are concerned, then these are the outcomes that we're going to see. We're going to see, unfortunately, many more MINOBs. And I think that this is not in line with what we in Europe understand as fundamental freedoms and human rights. So I want to kind of set that as a background. I think this echoes maybe and and links into some other comments I'd like to make.
It's not all new what we're talking about today. The last time I had the honor of being in this building and maybe even in this same room, I presented in 2017 a report I wrote on the use of armed drones. And at this during this particular discussion as well, at that point we were talking about the effects of the physical distancing, the remoteness in warfare. And this goes to the questions about lowering the threshold of violence or or the lowering the threshold to go to war. And that was, you know, nine years ago, also one of the main concerns.
And we're seeing the exact same thing now, not only just with the physical distancing, but a new layer of distance that I also write about, the psychological distance. We're farming out our decision-making, our cognitive processes to machines and we are prioritizing speed and scale.
And oftentimes when I engage with military actors or operators who are interested in acquiring this stuff, I think there are many reasons why acquisition of these these systems is important militarily. Often at the center of the reasoning is the speed, you know, what used to take years now take weeks. What used to take weeks now take seconds, right? And I think then we need to interrogate, well, what are we doing with that extra time? What you've won back in time, are you using that to make other choices, to generate more targets, to to maximize lethality, or are you using it to carry out your legal obligations, which are to take precautions in attack, to verify and validate, but that's part of a military compound and not an elementary school with more than 120 children inside of it.
And I think that just kind of highlights these problems that we've had over the last several decades haven't gone away, they've just become exacerbated. The ICRC put out a 2024 challenges report, and I cite it so often I already know the page number. On pages seven and eight, they highlight this sort of as a bifurcated issue. It is a erosion of the protective mechanisms that are built into the law by political choices that actors in these conflict zones are making to minimize who counts as a civilian and maximize who might count as a combatant, or to interpret law to the very minimal floor rather than using a more robust good faith reading of it. These are then exacerbated by these new technologies that we're seeing integrated into warfare.
I think that links into the role of private companies question. I think we need to be wrestling with that. I think that came from the German colleague. I think we're seeing a shift afoot, right? The role of US companies has been quite prominent and will continue to be prominent if we allow it to be prominent.
This is contract law. We are entering into at the national level contracts with particular companies like Palantir, Anduril, others, and we are creating lock-in, we are creating these kinds of agreements. I just want to empower people to realize this is a choice we are making. We're seeing though, and this is where I want to identify that this is that geopolitical shift that's afoot, and I think it offers opportunities in the European context. I see a sea change happening on this side of the ocean, and as an American and Dutch citizen, I have a very interesting perspective, I think, of kind of being insider-outsider on both sides.
But I think what we're seeing that this shift is trying to move us away from dependencies on US tech and defense companies, more for a European-first approach. And I think this is actually quite promising. And why I think that's promising, not just because if you create an alternative for Palantir's Maven Smart and you stamp a European flag on it, that's not good enough. I wouldn't say that those kind of technologies are what we need to build the exact replication of them. But it offers the opportunity when we are building, and we are building various alternatives to these capabilities on the European continent, that we can build them with that responsible-by-design approach, foregrounding and upstreaming our legal obligations, our considerations of ethics, this responsible-by-design from the very earliest planning stages.
And this again leads to those opportunities for leadership, I think, from the European side of things. And I want to finally get to the last point about the rule of law. I think it's, I think someone also mentioned that international law is powerless. And I think it's sort of my job up here, pretty on-brand for an international law professor, to say no, that's wrong. International law is not powerless.
It's quite a robust tool. And I think we ought not to forget about that. We don't need to start from scratch.
We have robust legal frameworks of human rights and international humanitarian law. They require by law also how we interpret the treaties is a robust good faith reading of them based on the object and purpose of why those treaties were agreed to to begin with. Going back to basics I think we also ought not to forget on the European continent that and this is the law professor in me quoting Louis Henkin most states follow most international law most of the time. And I think that's really great. So again we're not having to start from scratch.
I think we need to ensure that we lead by example by giving the robust reading that IHL requires to preface precautions over proportionality especially in design and development of these systems. And I think we shouldn't forget that international law was actually one of the quietest but most powerful ways to get the Trump administration to walk itself back from the cliff on Greenland right? Europe stood up with an economic bazooka don't love the militarization of the term but it was a way without firing a shot to stand up and assert ourselves under the scope of international law.
So I think we need not forget that in our own positioning in Europe about that.
Marie-Agnes Strack-Zimmermann
Thank you very much, Jessica. We could see how important it is that we talk about this huge problem. So we have two questions more and we have 15 minutes left. So first Michael Gahler and then Pekka Toveri and then you have the chance to answer. Please.
Michael Gahler
Thank you. Thank you very much. Being aware of our moral and and legal dilemmas that were expressed here and also being aware of the discussions that we had years ago prior to the full-scale war and prior to the current technical development. I think the positioning of the Parliament was very clear. There cannot be, for instance, money provided on defense research for developing autonomous weapons.
I think that was the clear position at the time, that we, and of course everybody said, there cannot be, so to say, in a war, there cannot be killings by an automated weapon without human interference. Now, I think in the current situation, we also, I think we have to differentiate between, for instance, a situation like in Gaza, where you have a densely populated civilian area, where those who fight each other, on the Hamas side, they don't even wear uniforms, so they can hardly be identified, and one can arguably not leave it, any shots done by an autonomous weapon and create a collateral damage that is far bigger than any, so to say, target that you actually hit. I mean, that is what we have seen, even without autonomous weapons, I think the practice there was one. On the other side, when it is on Ukraine, I think I have no moral dilemma when it is about setting up an Iron Dome, so an air defense system that should, that decide autonomously to shoot down Iskander or Kinzhal weapons, and that is done in this way. There are no humans involved, there is nobody inside, it's all weaponry only, so I think there is no dilemma to allow an autonomous decision to identify these flying missiles.
I don't see a problem in this regard.
And currently when I'm watching... so I wonder whether we should not really differentiate the theaters and allow autonomous things to happen if the frontline is very clear: who is the aggressor, who is the victim, and who is defending and who is attacking. Thank you.
Pekka Toveri
Thank you. Well, we are living in a world where we have to be ready to defend democracy, our people, with arms. And that means we have to be ready to kill people and to be killed. I'm father of a free Finnish army reservist and I really want them to have the best possible tools if they have to go to the harm's way to protect us all. And AI is just a tool.
I'm not a fan of AI in any way, but AI is a tool that military is using. And we've seen new tools coming to military. Collateral damage has happened every time we have new, more effective weapon systems, we have more collateral damage. What is crucial here is training of the military to do the right thing. And I've been in Afghanistan, I've seen that military is not actually as bloodthirsty as many people think.
They do a lot of restraint to save people and avoid unnecessary losses to civilian population.
One thing, this paper has good suggestions, but I think one thing is missing: the dialogue with military and industry. You know, we are talking here in high halls and pointing fingers and telling what they should do and shouldn't do, but we have to find a compromise because we need better tools to defend ourselves against nations like Russia where human life doesn't mean anything. And in order to find that balance, the right technical means, the right doctrinal means, the right training of the military, we need to have the dialogue earlier than later. And so, you know, what's your view? Should we have more dialogue with military and more dialogue with industry so that we understand better each other's needs so that the end result will be better?
Thank you.
Speaker 23
Thank you very much. Netta Goussac, you want, you have two minutes to answer if you want.
Netta Goussac
Thank you for the members' comments and questions. They were poetic comparisons of different concepts. So I wanted to just reflect on those comparisons and then offer one response to the direct question on the report's recommendations.
In relation to comparisons, I think it's important to compare things of a comparable nature. Um, when it comes to the use of military AI in, um, Gaza by Israel against Hamas, we're primarily talking about what is broadly considered a decision support system, that is an AI tool that militaries use in order for, um, to support human decisions about when, where and how to apply force to targets. Uh, in Ukraine, the air defense system that the member referred to is, is, the example was of an autonomous air defense system. Of course, there are a range of AI tools being used in the context of the Russia-Ukraine conflict, um, but a, an autonomous air defense system, a close-in weapon support system, um, is an autonomous weapon system. It may be AI-enabled.
There, there is no person who is necessarily deciding exactly where, when to apply force to a target. These are similar but different technologies, which raise some same issues but also some distinct issues. And when it comes to the question of human control, which of course is relevant to both kinds of uses of AI in the military domain, the member very helpfully pointed out the exact kinds of constraints that differentiate between what you might say morally acceptable or legally acceptable uses and uses that fall below that standard. That is a use of an AI-enabled military system that is bounded in time, that is under supervision, that has a very strict task, that is being used in a very limited geographic space like the air domain. These kinds of constraints are in fact the operationalization of a concept of human control which can and often do differentiate between uses that meet legal standards and moral standards and uses that don't.
Similarly, I think when it comes to the comparison between using an AI system and not using an AI system, which the second member mentioned in his question, I don't think that there is a binary between these two options, but rather it's about how AI tools are being used by militaries, how they are adopted, how they are developed and how they are used. And in relation to the specific question about a dialogue with the military and the need for training, I could not agree more. Some of this is slightly outside of the scope of the paper, which was very much focused on the competency of the European Parliament, but I wanted to point out one aspect which is relevant, which is the role of the EU Military Committee, specifically referred to in recommendations five and ten. I understand that the EU Military Committee is the highest military body within the Council and it comprises chiefs of defense and is supported by EU military staff. And in the recommendations of the report, I have suggested that the EUMC be included in efforts to strengthen institutional oversight and also in efforts to develop guidance that operationalize high-level principles around human control, to ensure that those operationalization guidelines are reflective of the reality of military experiences and that military planners can apply them in doctrine, in training and in concepts of operation.
Thank you.
Marie-Agnes Strack-Zimmermann
Thank you very much, and Jessica for a short answer please.
Jessica Dorsey
Sure, just to add a bit to the last comment and question. I think it's right to talk about, I don't think that anyone thinks that any of these military actors are bloodthirsty, as you mentioned. They're not. They're trying to do the right thing. Sometimes, though, when things go wrong in warfare, they will, they always have, they will continue to, there must be accountability in a broad sense, right, to not only account for what went wrong, perhaps in a legal avenue, not always, though, but also to improve operational activity for subsequent attacks.
So this is the point of understanding that when things go wrong, we must have accountability mechanisms in place. And I think that's why it's absolutely crucial, that's also the work that we do in Responsible by Design thinking, is having a multi-stakeholder approach. Everyone needs to be at the table to talk about the way forward.
I engage with military operational personnel at the Dutch MOD, in the US, in the UK, in Germany, etc. in order to understand their perspectives better, to ask sharper questions, to understand what they're running up against. I also am in conversation with industry leaders and representatives, right? So Palantir, Fokker, Microsoft, Helsing, etc. in order to also understand where they're coming at this from, right? Often times these disparate communities don't know what they don't know, and so the idea is if we're communicating and having a multi-stakeholder dialogue, that that feeds into seeing the issues from the 360 view that I think that this kind of discussion needs. So only underscoring your point about training, understanding that this is the way forward, but it, you know, diplomacy and deliberation take time, and that is often at tension right now with the speed at which this technology is accelerating.
To slow things down isn't to disadvantage ourselves. I think we need to realize, you know, also from a military operational perspective, tactical patience, slowing it down in order to gather more information to make sure your targeting procedures are indeed targeting legitimate military objectives. That is a win for slowing down. So deliberation, diplomacy takes time, so building in friction points is I think something we need to understand how to attain this very delicate balance. Thank you.
Bien, merci.
Mounir Satouri
On a parmi nous le conseiller pour les aspects internationaux de l'intelligence artificielle à la Direction générale des réseaux de communication à la Commission, Monsieur Ikila, vous avez la parole.
Ikila
Merci beaucoup. Thank you very much. I heard many comments and references to the AI Act, so maybe I will just clarify the military exclusion, the rationale for it. So this is based on the fact that the AI Act is effectively internal market legislation which is based on Article 114 of the Treaty of the Functioning of the European Union. And it was felt that the military exclusion was justified both by Article 4 of the Treaty of the European Union and by the specificities of the Member States and the common Union defense policy which are covered in a particular chapter of the Treaty of the European Union.
And these are subject to public international law, which is therefore it was considered to be the more appropriate legal framework for the regulation of AI systems in the context of the use of lethal force and other AI systems in the context of military and defense activities.
So this is the the rationale for the military exclusion in the AI Act. I would also like to take the opportunity to to thank the writers, the authors of the report for the recommendations to the Commission. And there was particularly one on a code of conduct. We'll take note of that. We have noticed, however, that the codes of practice we have published as implementation tools for the AI Act, they have actually attracted a lot of following, which shows that we do have actually influence in the international discussions.
Thank you.
Mounir Satouri
Merci. Monsieur Apostolov, responsable pour les questions liées à l'intelligence artificielle à la division des droits de l'homme du SEAE, vous avez la parole.
Apostolov
Thank you, Chair. Honorable members, distinguished speakers. Thank you for the invitation and the very rich discussions today. The EEAS welcomes the report and today's discussion and particularly their attention to the human rights dimension of AI-enabled military systems. EU's own approach already reflects this dimension.
The EU and its member states promote a human-centric, human rights-based, multi-stakeholder and trustworthy AI in line with international law, which includes international humanitarian law and international human rights law, and with the EU values in all relevant international initiatives on global AI governance and its implications for security and defense. EU member states and the EEAS coordinate EU's input to multilateral processes on AI in security and defense, notably the Group of Governmental Experts on lethal autonomous weapons systems within the Convention on Certain Conventional Weapons, on the basis of forbidding lethal autonomous weapons systems that cannot respect international humanitarian law and allowing the use of others. The 11 guiding principles which were agreed by state parties in 2019 remain relevant and as you know have also guided NATO's AI strategy. The rolling text which is currently under negotiation in the Group of Governmental Experts has the potential to become another important building block in the global governance of lethal autonomous weapons systems. Beyond the Convention on Certain Conventional Weapons, all member states are signatories of the Responsible AI in the Military Domain Blueprint for Action and are also participants in the process implementing the Political Declaration on Responsible Military Use of AI and Autonomy.
These complementary processes are highly important for developing global thinking, governance, and practical solutions. And last but not least, the EU, as you know, also promotes the global implementation of the UN principles of business and human rights, which highlight the responsibility of all companies, including private technology companies, to respect human rights. We thank the author and the two speakers for the reflection today, and we look forward to a continued dialogue. Thank you.
Marie-Agnes Strack-Zimmermann
Okay, thank you very much, colleagues, and thank you to our speakers for this really rich discussion, which leaves us with a lot to think about, and we will keep working on this. So thank you all for your time today, and please, colleagues from the SEDE, we have to run to 6Q2 for the vote on hybrid warfare INI. So thank you very much and have a good day.
Mounir Satouri
Je vous propose de vous installer, on va on va commencer, que celles et ceux qui souhaitent quitter la salle le fassent. Et alors pour poursuivre après cette discussion importante et passionnante qu'on a eue, on a trouvé que le débouché naturel était de donner la parole à Nacho pour l'examen du projet de rapport sur justement les droits de l'homme et les défis posés par l'intelligence artificielle.
Nacho, tu as la parole.
Nacho Sánchez Amor
Merci, Pluton Naturel. Puisqu'on continue à parler de l'intelligence artificielle. Bon, I'm using English in this case. This initiative resolution is called Shaping the EU External Action to Address Challenges to Human Rights Posed by Artificial Intelligence. This is indeed the first AFET and DROI non-legislative report on artificial intelligence, and it's in my opinion a very timely report because anything related to the artificial intelligence is increasingly present and relevant for the daily life of our citizens.
Every week we can read the news concerning, for example, armed conflict.
We have been dealing with the issue extensively prior to this point, but also the artificial intelligence being used as a tool for strategic influence, economic competition, or foreign interference. It was due time for our subcommittee to address this topic, especially because the issue has been raised recently with very prominent cases as the Grok nudity case, that means every person in the world could be affected by a misuse of artificial intelligence, and because we have been a little bit lazy on artificial intelligence and human rights compared to the United Nations and the Council of Europe. These institutions, they started to working human rights and artificial intelligence more than a decade ago. We at the European Union level, we have for sure, and we have been talking about the AI Act, now the AI Omnibus, but as we heard a minute ago from the Commission representative in the prior debate, the approach to the artificial intelligence has been mainly related to internal market. Then we have to extend the scope and introduce our rational, our political thinking about artificial intelligence and the external dimension of the European Union.
And in the current moment of the world with this geopolitical climate, it could be an opportunity to really develop with some friends, some partners in the world, some AI systems that are compliant with human rights as a complementary way to tackle the growing presence of the authoritarian narratives.
The core of the paper is to reflect on challenges but also on the opportunities of the artificial intelligence. I try to avoid this Luddite approach thinking that any progress is inherently dangerous. In my view, artificial intelligence is not inherently evil, but has, on the contrary, could help to advance human rights if there is a careful design, test, and deployment of the system. The report is structured as follows: first, address general considerations, then list the main risks, and then some provisional proposals. Among the challenges to human rights posed by artificial intelligence is well-known data privacy and discrimination, surveillance, women's rights and gender equality, children's rights, migration, labor and environmental risks, transnational repression, business and human rights, and we talked recently, automated weapon systems.
The report also addresses artificial intelligence and democracy and the role of artificial intelligence in electoral processes. And finally, some provisional proposals include an holistic strategy for the EU external action to reinforcing the EU toolbox, the need to develop a strong and resilient European Union ecosystem that reduces strategic dependencies and reinforces strategic autonomy, maybe developing open source AI system, supporting the United Nations and the Council of Europe, and calling for the United Nations effort to create a legally binding agreement to globally regulate the use of lethal autonomous weapons. I would like to thank my shadow fellow rapporteurs because the work I distribute to them what we call the long draft version, the working draft version, and we expect to start negotiation after the summer and vote the paper, if it is the case, to the end of the year. Thank you for your amendments or your improvements to this provisionally draft paper. Thank you.
Mounir Satouri
Merci Nacho, je vais donc donner la parole au shadow rapporteur ou à leur représentant et j'imagine que c'est toi Isabelle qui parle au nom du shadow PPE.
Isabel Wiseler-Lima
Tout à fait. Je me fais tout simplement sa voix donc Adrián Vázquez m'a demandé si je pouvais lire ces quelques lignes. Good morning.
Thank you to the rapporteur, my fellow countryman Nacho Sanchez Amor, for this draft report. My team and I have read it. There are elements we welcome and others we want to refine. Our approach to this file is simple: balance rights with technological development and safeguards with the needs of the 21st century digital economy where the EU must move forward. We cannot ignore the new world of AI, but we cannot ignore its clear risks either.
We will be tabling amendments accordingly. Thank you and have a good summer for all of you. From my side, I want only to add that I'm really looking forward to this report on a subject I consider of the utmost importance. So thank you very much Nacho for the work you have done and will be doing and to all the shadow rapporteurs. Thank you.
Mounir Satouri
Merci Isabelle. Je n'ai pas vu le groupe ECR représenté, ni même le groupe Renew. Je crois qu'il n'y a personne. Et bien je vais donner la parole à Katarina Viera pour le groupe Vert.
Catarina Vieira
Thank you very much and thank you to the rapporteur for presenting the draft report and for the work that has gone into it.
We are very welcoming the report as the Greens group and we look forward to discussions with the political groups. Obviously it's a timely contribution that wants to ensure that the technological innovation remains firmly grounded in the protection of fundamental rights as it should be. And for us AI must remain human-centered, transparent and accountable with human rights serving as the guiding principles to how the technologies are developed and then later used. From a perspective I also like to point out that the environmental impact of AI deserves greater attention. The rapid expansion of AI infrastructure can bring increasingly energy and water consumption as well as growing demand for critical raw materials.
This is something we cannot ignore if we want to fuel the digital transition in a sustainable manner consistent to Europe's climate and environmental objectives. Looking at the report I really would like to highlight the importance of AI's impact on mental health particularly for children and young people. It can bring real benefits but also it can amplify misinformation, encourage harmful online behavior and contribute to social isolation.
And as the rapporteur said, there's also an angle to the impact that it can have on our democracies. Also another important point that I think will be well covered in the report is the fact that it is creating new forms of gender-based violence with deepfakes, image-based abuse, and online harassment, and it is increasingly being used to target women and girls. So we need stronger safeguards. Finally, I'd like to add that I really appreciate the trade angle also present in the report, the call for an export ban for AI-assisted mass surveillance to countries where they are likely to use this technology to violate human rights, but also the call to include AI in the dual-use regulation in the future. So, I'm very excited to work on this after the summer and to work with the rapporteur and all the shadow rapporteurs to further strengthen the report.
Thank you.
Isabel Serra Sánchez
Gracias, presidente. Bueno, con respecto al informe, en primer lugar agradecemos pues el enfoque, pero pensamos también que es importante en un contexto como en el que estamos, en una situación internacional como en la que estamos, de avance hacia régimen de guerra, de avance hacia, bueno, pues una incertidumbre cada vez mayor, entre otras cosas porque Estados Unidos e Israel pues están siendo una amenaza al conjunto de la humanidad, poner nombre.
Cuando hablamos de inteligencia artificial yo creo que es evidente, ¿no?, que la inteligencia artificial podría servir para garantizar los derechos humanos, para hacernos vivir mejor, para reducir, bueno, pues el tiempo de trabajo, para, bueno, en todo lo que tiene que ver con el bienestar de las personas, pero en este momento concreto estamos viendo situaciones tremendamente horrorosas. Quiero hablar concretamente de un sistema de inteligencia artificial que está poniendo en marcha, que lleva poniendo en marcha Israel en los últimos años, que es el Lavender. Es un sistema que según un estudio que ha salido en el que han entrevistado propiamente a los oficiales del ejército permite seleccionar de forma instantánea hasta 37.000 objetivos, 37.000 objetivos en apenas 20 segundos de revisión por parte de las personas, de los oficiales del ejército de Israel, y que ese sistema tiene, bueno, pues una identificación de hasta 20 víctimas civiles como daños colaterales. Este es el sistema que está utilizando Israel en este momento para generar destrucción y masacre en Gaza, supongo que también en este contexto en Líbano. Y en ese, en tanto, bueno, pues que esto es lo que está sucediendo y cómo se está utilizando la inteligencia artificial pensamos que hay que poner nombre también a la gravedad y a quién está utilizando esto, ¿no?.
Por tanto, bueno pues haremos nuestras enmiendas pero pensamos que es fundamental en un contexto como este abordarlo poniendo nombre a quién y qué países y qué estados en concreto Israel está utilizando de esta forma. Y quiero aprovechar los últimos segundos simplemente para denunciar que Abu Safiya, que es un médico palestino, está secuestrado, está encarcelado por salvar vidas por parte del Estado de Israel. Y quiero aprovechar mi intervención en esta comisión para decir que la Unión Europea no puede callar ante este ante este secuestro por parte de Israel, ante este encarcelamiento injusto y exigir su libertad y decir que no en nuestro nombre y no con nuestro silencio. Muchas gracias.
Mounir Satouri
Merci chers collègues, je crois que le groupe ESN n'est pas représenté. Très bien. Et ben je vais donner maintenant la parole à Monsieur Ikila pour la Commission européenne. Vous avez la parole.
Ikila
Merci beaucoup, thank you chair.
So I would like to thank you for this draft and it's a very broad draft covering many bases and many important aspects and it is comprehensive work. I would like to make some general comments first which explain the role of the European Commission and the AI Office in particular and the activities we're doing and then I will make some references to specific sections in the report. Obviously I cannot go through all of it because of lack of time. First of all, just a minor point on the definitions or use of terms. It talks about AI systems, just a terminological point.
In many cases it would be better to talk about AI systems and models or just AI because we make this distinction in the AI Act between AI systems and general-purpose AI models and often what is referred concerns models just as well or even more. That's just a minor terminological point. Now as regards then our international engagement, so we have very active international involvement which is the external dimension of our internal policy where we have Union legislation and that Union legislation is an important element in our international engagement also based on the fact that it is based on our values and based on our principles that are dear to us.
We are part of all the international AI governance discussions that matter. We have been doing this for years now, so the Commission and the AI Office has a role and a seat in all these key discussions that are going on worldwide. In some cases in the UN system of course our coordination is done by the EU delegations run by the External Action Service and there the coordination often goes also beyond the EU Member States extending to like-minded countries. We want to advocate our values, our human-centric trustworthy AI approach. This is a key aspect of our international engagement and we want to ensure that any international outcome statements, documents, whatever are aligned with our values and our AI approach.
And we position ourselves as a proactive, cooperative and reliable partner that wants to lead by example and collaborate internationally protecting our interests, security and values. And we advocate the responsible stewardship of AI based on values and rights. We have both bilateral and multilateral cooperation in all those forums where it matters.
And we have noticed that we are a strong voice in the international discussions. We do get listened to. Other countries actively seek our views, want to hear our positions, want to know what we do, what position the EU is taking in this regard. I would just maybe highlight a couple of things. We do coordinate of course our positions also between the institutions.
And as I said, the coordination also extends to countries, like-minded countries like Canada and Japan and so on. We are actively supporting the UN processes, the Global Digital Compact, the Global Dialogue on AI Governance which took place last year, and the scientific panel. And we fought very hard to have our values represented in those texts, and that was not very easy in the UN context because of course most of the UN countries do not necessarily share our values, unfortunately. There was reference to open source.
I would just like to highlight that the Commission just published an open source strategy at the beginning of June and that also includes a reference to using the EU tech business offer as a way of propagating that promoting open source because that was quite strongly highlighted in the in the text. I would also like to mention that in all these interactions the rights aspect is important. Just last week at the Global Dialogue we the EU statement made on behalf of the EU by the Director General of DG Connect in the Commission highlighted also the human rights aspect which is of course the point of the report. Now another thing that was highlighted in the report was the potential competitive advantage that trustworthy AI based on guardrails can can bring. And we fully agree with that.
In fact there is this misconception misguided notion that regulation hampers innovation. We we reject that firmly. We think it is actually an advantage and I think there is a growing consensus or growing voices number of voices which show that some guardrails are necessary. And our third way which is between the totalitarian approach of China and the no no regulation approach of the US has attracted a lot of attention and has been actively followed and adopted by South Korea for example whose legislation is very similar to ours. Brazil is basing its bill very closely to to our regulation.
So we have certainly a lot of goodwill and following and influence worldwide. So these are maybe the most important points that I wanted to make. I could go on much longer, but I think I've already exceeded my time. So just want to emphasize that we have very active international engagement. It is being followed.
It is getting a lot of attention. Other countries actively seek our positions, want to know what we do and want to coordinate with us. So we have a very significant voice in the world in the area of AI. Thank you.
Mounir Satouri
Bien, nous avons parmi nous la responsable pour les questions liées à l'intelligence artificielle à la division des droits de l'homme au SEAE. Madame Kimerko, je vous donne la parole.
Kimerko
Merci. Mr. Chair, honorable members of the European Parliament and dear colleagues, thank you. It's a privilege to be here on behalf of the European External Action Service. First of all, allow me to thank the rapporteur, Mr. Nacho Sánchez Amor, for this report. I welcome the European Parliament's continued attention to the opportunities and risks for human rights that arise from artificial intelligence.
This is an issue that has become central to the European Union's internal policies, external action and multilateral engagement. And last week precisely, the representatives of all United Nations member states gathered in Geneva for the first global dialogue on artificial intelligence governance. We welcome the establishment of the global dialogue on AI governance at the UN level, as envisaged by the UN Global Digital Compact. This mechanism can help develop an inclusive approach to AI governance globally, amplifying the voices of the Global South in a discussion dominated by a few global powers. In this vein, the landmark meeting last week demonstrated that the governance of AI requires international cooperation grounded in international law and universal human rights, which we heard in the previous session indeed quite a lot.
In Geneva, we promoted a multi-stakeholder approach to AI governance and we reaffirmed EU's commitment to a trustworthy AI that is, I quote, "both human-centric and grounded in international human rights law, placing individuals, their dignity, rights, freedoms and well-being at the center of AI development and deployment."
Against this backdrop, the European Parliament's discussion could not be more timely. The report rightly stresses that human rights must remain at the center of artificial intelligence and that existing international human rights obligations continue to apply fully in the context of AI. It also rightly recognizes the broad spectrum of risks that AI systems may pose to fundamental rights, including risks to, indeed, like Mr. Rapporteur, you precisely reminded, some of them are privacy, data protection, the rights of the child, democracy, the rule of law, as well as algorithmic discrimination, mass surveillance, the suppression of dissent, gender-based violence, and transnational repression. These, in fact, are all concerns that we continue to raise during our diplomatic engagements. Through our international digital strategy, digital partnerships, digital dialogues, human rights dialogues at bilateral level, but also in multilateral fora like Geneva last week, we continue to reaffirm that human rights apply both offline and online.
Artificial intelligence must be human-centric, grounded in international human rights law, and designed to serve people, not the other way around. Thank you.
Nacho Sánchez Amor
Very briefly to thank my shadow rapporteurs fellows for the improvement they are announcing in their amendments. I'm fully open to examine every of them, including what Isabel said about mentioning particular cases, even if the normal practice in DROI is in the horizontal report is not particularly pointed or mentioned cases, but I'm open to examine the things in the process of negotiation. Thank you to the Commission and the External Service, thank you for your very precise improvement, including the terminology precision, and the closeness of the approach between the Parliament and your institutions in the risks and the challenges, but also in the advantages of having AI on our side in a world in which our principles and values approach sometimes could be of help to try to create global governance that has been started to be discussed under the umbrella of United Nations. Thank you everyone for the contributions.
Mounir Satouri
Très bien. Et bah à suivre, je suis persuadé que vous aurez un travail de synthèse et de négociation passionnant sur un sujet important. Et bah on est pile-poil à l'heure pour le point d'après, juste quelques minutes pour accueillir nos invités au podium et Monsieur Assis Francisco est déjà prêt.
Bien, chers collègues, je vous propose qu'on reprenne nos travaux avec ce septième point de l'ordre du jour qui est à l'agenda et consacré à la situation des droits humains et le rétrécissement de l'espace civique et démocratique en Guinée-Bissau depuis le coup d'État de novembre 2025.
Il est organisé en association avec la délégation du Parlement européen à l'Assemblée parlementaire paritaire ACP-UE et je salue notre cher collègue que nous connaissons bien à cette commission, Francisco Assis, qui préside cette délégation et qui est à mes côtés sur le podium. Je vous rappelle que le 22 novembre 2025, la Guinée-Bissau a été frappée par un coup d'État militaire à la veille de l'annonce des résultats d'élections présidentielles. Le Parlement européen a adopté le 18 décembre dernier une résolution qui condamne notamment la prise de pouvoir par les militaires, l'interruption du processus électoral et le changement anticonstitutionnel du gouvernement. La résolution exige également la libération immédiate et inconditionnelle des dirigeants de l'opposition politique et de toutes les autres personnes détenues arbitrairement depuis le coup d'État. Comme l'exige le mandat de notre sous-commission, nous avons assuré le suivi des demandes exprimées dans cette résolution lors d'un échange de vues en janvier dernier.
Mais depuis, la situation dans le pays ne s'est pas améliorée, bien au contraire. Dernier épisode en date, vendredi dernier, 10 juillet, l'ancien Premier ministre Domingos Pereira a été de nouveau placé en détention. C'est à la lumière de cette dégradation de la situation que nous tenions à faire un état des lieux en se focalisant en particulier sur le rétrécissement de l'espace civique et démocratique que connaît la Guinée-Bissau depuis le coup d'État.
Merci beaucoup par avance à nos invités pour leur éclairage sur la situation. Bienvenue à Madame Monteiro, directrice du bureau du président de l'Assemblée nationale populaire et ancien ministre de la Justice et des Droits de l'homme qui est avec nous sur le podium. Nous sommes par ailleurs connectés avec Monsieur Victorino Inde, coordinateur national du réseau des défenseurs des droits de l'homme de Guinée-Bissau, association membre du réseau d'ONG CIVICUS. Et nous entendrons enfin Madame Jolita Pons, chef de la division de l'Afrique occidentale pour le Service européen pour l'action extérieure. Mais avant tout cela, je donne la parole à Francisco pour des propos préliminaires.
Francisco Assis
Merci beaucoup Monsieur le Président, j'irai comme d'habitude parler en portugais et en ce cas c'est particulièrement important parce que nous parlons de la situation d'un pays où la langue officielle c'est justement le portugais, la Guinée-Bissau. Bom, eu queria naturalmente saudar todas e todos os presentes e salientar, como já aqui foi referido, que esta troca de pontos de vista ocorre num momento muito crítico da vida da Guiné-Bissau. Como aqui já foi referido, há poucos dias ocorreu de novo a prisão de uma importantíssima figura da vida cívica e política da Guiné-Bissau, o antigo primeiro-ministro Domingos Simões Pereira.
Olhamos, aliás, com profunda preocupação para a situação política e a segurança na Guiné-Bissau. O que se está a passar, e muito concretamente o caso Domingos Simões Pereira, interpela profundamente a nossa consciência humanista internacional e creio que nós não podemos permanecer em silêncio perante uma situação desta natureza. Temos, como é óbvio, uma obrigação moral e política de denunciar o que está a acontecer e não apenas denunciar como reagir perante esta situação. A comunidade internacional não deve, ou o que resta da comunidade internacional, não deve assistir passivamente aos ataques à democracia e aos direitos humanos, que é o que se está hoje a verificar de forma muito clara na Guiné-Bissau. Nesse sentido, procurando passar enfim das palavras a alguns atos, embora sejam atos também constituídos por palavras, dirigi na semana passada um apelo ao Secretário-Geral das Nações Unidas, António Guterres, que conhece muito bem a situação na Guiné-Bissau por variadíssimas razões, apelando a uma intervenção firme e consequente das Nações Unidas que ponha fim a este silêncio institucional.
Que aliás é uma das questões que nos tem vindo a ser colocada a partir da Guiné-Bissau, que é o silêncio das Nações Unidas em relação a este assunto.
Dirigi também já uma pergunta prioritária à Comissão Europeia com o objetivo de aferir com precisão o grau de empenhamento da Comissão na defesa do pluralismo e da democracia na Guiné-Bissau e acima de tudo quais as consequências que estes atropelos graves poderão ter nas relações diplomáticas e sobretudo na cooperação entre a União Europeia e a Guiné-Bissau. Apelamos ainda à CPLP e à CEDEAO para que de acordo com os seus princípios e valores atuem com urgência no restabelecimento da ordem democrática e constitucional na Guiné-Bissau. Não poderemos também deixar obviamente de apelar ao Parlamento Europeu para que continue a ser a voz, como tem sido, dos que são silenciados na Guiné-Bissau. A nossa passividade, que estou certo não vai acontecer, seria obviamente inaceitável. Quero também declarar em meu nome e em nome da delegação à Assembleia Parlamentar Paritária OEACP-UE a nossa solidariedade a todo o povo da Guiné-Bissau.
Estamos ao lado dos cidadãos guineenses na sua luta pela paz, pelo desenvolvimento e pelas regras democráticas. É nossa responsabilidade garantir que as suas aspirações a um futuro livre e digno nunca sejam esquecidos da nossa parte.
Por fim, dizer, e faço sempre questão de dizer isso até enquanto português, antiga potência colonial da Guiné, que evidentemente não há aqui nenhuma intenção sobranceira de intervir num estado soberano que alcançou a sua independência na sequência de uma luta armada trágica para grande parte desse povo guineense. Nunca esqueça que a Guiné é a pátria de Amílcar Cabral, que é uma das maiores figuras dos movimentos de libertação nacional africanos. Não há aqui nenhuma interferência nesse sentido. A nossa solidariedade, pelo contrário, é uma solidariedade direta com o povo guineense, com as mulheres e os homens da Guiné-Bissau, que ao longo das suas vidas passaram por circunstâncias extremamente difíceis e que neste momento estão novamente confrontados com uma situação intolerável. E seria isto que eu teria a dizer, senhor presidente, caro presidente, nesta fase.
Mounir Satouri
Merci, monsieur le président. Je donne maintenant la parole à Madame Montero pour une intervention de sept minutes. Vous avez la parole et bienvenue.
Monteiro
Bom dia a todos. Eu começo por perguntar como morre uma democracia.
Morre num único dia? Morre quando se interrompe a ordem constitucional ou começa a morrer muito antes, quando aqueles que exercem o poder deixam de aceitar um princípio simples, mas absoluto, que a soberania pertence ao povo e que a vontade popular livremente expressa nas urnas constitui o único fundamento legítimo do exercício do poder? Creio que é esta a pergunta que deve orientar a nossa reflexão sobre a situação na Guiné-Bissau. O golpe de Estado de novembro de 2025 não marcou o início da crise dos direitos humanos no meu país. Foi o momento em que um processo de degradação institucional iniciado anos antes atingiu o seu ponto culminante.
Importa, porém, esclarecer um aspecto essencial. A erosão da democracia não começa de forma silenciosa. A violência política esteve presente desde o início. O que se agravou ao longo do tempo foi a sua banalização, a sua generalização e a captura progressiva das instituições que tinham precisamente a missão de a impedir.
Desde 2020, assistimos ao enfraquecimento dos mecanismos de fiscalização do poder, à instrumentalização da justiça, à perseguição de opositores políticos, jornalistas, magistrados e membros da sociedade civil, à limitação da liberdade de imprensa e à utilização da intimidação e da violência como instrumento de governação. Nenhum destes acontecimentos isoladamente destrói uma democracia. Mas todos eles, quando persistem e se acumulam, acabam por transformá-la profundamente. Tudo isto foi sucessivamente documentado. Organizações da sociedade civil, partidos políticos, jornalistas, ativistas e cidadãos denunciaram repetidamente graves violações dos direitos humanos.
Foram elaborados relatórios. Foram enviados apelos à CEDEAO, à União Africana, às Nações Unidas, à União Interparlamentar e a diversas outras instituições internacionais. Os sinais estavam lá.
Ask anything about this video — I analyse its transcript to answer.