Autonomous Weapon Systems and International Humanitarian Law: A Reply to the Critics

By Michael N. Schmitt*

Click here to read the full text as a PDF

“A sword is never a killer, it is a tool in the killer’s hands.”
Seneca[1]

Introduction

In November 2012, Human Rights Watch, in collaboration with the International Human Rights Clinic at Harvard Law School, released Losing Humanity: The Case against Killer Robots.[2] Human Rights Watch is among the most sophisticated of human rights organizations working in the field of international humanitarian law. Its reports are deservedly influential and have often helped shape application of the law during armed conflict. Although this author and the organization have occasionally crossed swords,[3] we generally find common ground on key issues. This time, we have not.

“Robots” is a colloquial rendering for autonomous weapon systems. Human Rights Watch’s position on them is forceful and unambiguous: “[F]ully autonomous weapons would not only be unable to meet legal standards but would also undermine essential non-safeguards for civilians.”[4] Therefore, they “should be banned and . . . governments should urgently pursue that end.”[5] In fact, if the systems cannot meet the legal standards cited by Human Rights Watch, then they are already unlawful as such under customary international law irrespective of any policy or treaty law ban on them.[6]

Unfortunately, Losing Humanity obfuscates the on-going legal debate over autonomous weapon systems. A principal flaw in the analysis is a blurring of the distinction between international humanitarian law’s prohibitions on weapons per se and those on the unlawful use of otherwise lawful weapons.[7] Only the former render a weapon illegal as such. To illustrate, a rifle is lawful, but may be used unlawfully, as in shooting a civilian. By contrast, under customary international law, biological weapons are unlawful per se; this is so even if they are used against lawful targets, such as the enemy’s armed forces. The practice of inappropriately conflating these two different strands of international humanitarian law has plagued debates over other weapon systems, most notably unmanned combat aerial systems such as the armed Predator. In addition, some of the report’s legal analysis fails to take account of likely developments in autonomous weapon systems technology or is based on unfounded assumptions as to the nature of the systems. Simply put, much of Losing Humanity is either counter-factual or counter-normative.

This Article is designed to infuse granularity and precision into the legal debates surrounding such weapon systems and their use in the future “battlespace.” It suggests that whereas some conceivable autonomous weapon systems might be prohibited as a matter of law, the use of others will be unlawful only when employed in a manner that runs contrary to international humanitarian law’s prescriptive norms. This Article concludes that Losing Humanity’s recommendation to ban the systems is insupportable as a matter of law, policy, and operational good sense. Human Rights Watch’s analysis sells international humanitarian law short by failing to appreciate how the law tackles the very issues about which the organization expresses concern. Perhaps the most glaring weakness in the recommendation is the extent to which it is premature. No such weapons have even left the drawing board. To ban autonomous weapon systems altogether based on speculation as to their future form is to forfeit any potential uses of them that might minimize harm to civilians and civilian objects when compared to other systems in military arsenals.

I. Autonomous Weapon Systems

Before turning to the law, it is necessary to frame the issue definitionally. A weapon system consists of a weapon and the items associated with its employment.[8] An example is the Predator unmanned aerial combat system armed with a weapon such as a Hellfire missile. The Department of Defense has recently defined an autonomous weapon system as:

a weapon system that, once activated, can select and engage targets without further intervention by a human operator. This includes human-supervised autonomous weapon systems that are designed to allow human operators to override operation of the weapon system, but can select and engage targets without further human input after activation.[9]

The crux of full autonomy is a capability to identify, target, and attack a person or object without human interface. Although a human operator may retain the ability to take control of the system, it can operate without any control being exercised. Of course, a fully autonomous system is never completely human-free. Either the system designer or an operator would at least have to program the system to function pursuant to specified parameters.

U.S. forces have operated two human-supervised autonomous systems for many years–the Aegis at sea and the Patriot on land–both designed to defend against short notice missile attacks.[10] Another human-supervised autonomous weapon system, Israel’s Iron Dome, is presently receiving a great deal of attention as it very effectively destroys incoming rockets.[11] However, the United States is currently not fielding any fully autonomous weapon systems.[12] Nor are there any “plans to develop lethal autonomous weapon systems other than human-supervised systems for the purposes of local defense of manned vehicles or installations.”[13] That said, Human Rights Watch is correct in noting that this fact does “not preclude a change in that policy as the capacity for autonomy evolves.”[14] At some point in the future, such systems will find their way into the battlespace.

Fully autonomous weapon systems must be distinguished from those that are semi-autonomous, which are commonplace in contemporary warfare.[15] The latter engage specific targets or categories of targets that a human operator selects. For instance, a “fire and forget” missile on an aircraft locks onto a target identified by the pilot and then attacks it without human involvement. Losing Humanity does not address semi-autonomous systems–appropriately so since the systems differ qualitatively from those that are fully autonomous. Nor does the report examine what it calls “automatic weapons defense systems.”[16] These systems respond automatically (or near automatically) when they detect incoming threats. An example is the “close-in weapon system” (CIWS or “Sea Whiz”).[17] Used for point-defense of warships against incoming missiles, the Sea Whiz can be programmed to detect inbound missiles based on parameters that include speed and altitude, and can automatically engage them.

Before turning to the legal issues surrounding autonomous weapon systems, it is necessary to debunk a number of myths about the systems that are clouding public debate. First, the idea of “robot wars” is pure science fiction. As noted by a Department of Defense Task Force, “the true value of these systems is not to provide a direct human replacement, but rather to extend and complement human capability by providing potentially unlimited persistent capabilities, reducing human exposure to life-threatening tasks, and, with proper design, reducing the high cognitive load currently placed on operators/supervisors.”[18] Autonomous weapon systems will be integrated into human warfare, but are highly unlikely to replace it.

Second, neither the United States nor any other country is contemplating the development of any systems that would simply hunt down and kill or destroy enemy personnel and objects without restrictive engagement parameters, such as area of operation or nature of the target. Moreover, the Defense Science Board points out that:

[A]ll autonomous systems are supervised by human operators at some level, and autonomous systems’ software embodies the designed limits on the actions and decisions delegated to the computer. Instead of viewing autonomy as an intrinsic property of an unmanned vehicle in isolation, the design and operation of autonomous systems needs to be considered in terms of human–system collaboration.[19]

As presently envisaged, autonomous weapon systems will only attack targets meeting predetermined criteria and will function within an area of operations set by human operators.

The U.S. Department of Defense is exceptionally sensitive to the human interface issue. It has recently promulgated policy guidance that requires the Secretaries of the military departments, the Commander of U.S. Special Operations Command, and certain other high level officials to:

[d]esign human-machine interfaces for autonomous and semi-autonomous weapon systems to be readily understandable to trained operators, provide traceable feedback on system status, and provide clear procedures for trained operators to activate and deactivate system functions . . . ; [c]ertify that operators of autonomous and semi-autonomous weapon systems have been trained in system capabilities, doctrine, and [tactics, techniques, and procedures] in order to exercise appropriate levels of human judgment in the use of force and employ systems with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable ROE . . . ; [and e]stablish and periodically review training, and [tactics, techniques, and procedures], and doctrine for autonomous and semi-autonomous weapon systems to ensure operators and commanders understand the functioning, capabilities, and limitations of a system’s autonomy in realistic operational conditions, including as a result of possible adversary actions.[20]

Finally, robots will not “go rogue.” While autonomous and semi-autonomous weapon systems will be susceptible to malfunction, that is also the case with weapon systems ranging from catapults to computer attack systems. Like a missile that “goes ballistic” (loses guidance), future autonomous systems could fall out of parameters. But the prospect of them “taking on a life of their own” is an invention of Hollywood.

The one real risk is tampering by the enemy or non-State actors such as hackers. As an example, the enemy might be able to use cyber means to take control of an autonomous weapon system and direct it against friendly forces or a civilian population. Those developing the systems are acutely aware of such risk. U.S. policy on the matter is that, “[c]onsistent with the potential consequences of an unintended engagement or loss of control of the system to unauthorized parties, physical hardware and software will be designed with appropriate: (a) [s]afeties, anti-tamper mechanisms, and information assurance [and] (b) [h]uman machine interface.”[21]

II. Unlawful Weapon Systems

Losing Humanity concludes that “[a]n initial evaluation of fully autonomous weapons shows . . . such robots would appear to be incapable of abiding by key principles of international humanitarian law. They would be unable to follow the rules of distinction, proportionality, and military necessity and might contravene the Martens Clause.”[22] While it is true that some autonomous weapon systems might violate international humanitarian law norms, it is categorically not the case that all such systems will do so. Instead, and as with most other weapon systems, their lawfulness as such, as well as the lawfulness of their use, must be judged on a case-by-case basis. To assert, as Human Rights Watch does, that “[f]ull autonomy would strip civilians of protections from the effects of war that are guaranteed under the law” is to melodramatically oversimplify international humanitarian law, while failing to accord it the respect it is due.

The international humanitarian law governing weaponry proceeds along two tracks. One focuses on the legality of the weapon system itself. The legal reviews discussed below respond to this issue. A separate and distinct family of prohibitions (labelled “conduct of hostilities” rules) applies to a weapon system’s use irrespective of whether or not the weapon system is lawful per se.

Among the earliest prohibitions with respect to the legality of weapons per se is the ban on means or methods of warfare that are of a nature to cause superfluous injury or unnecessary suffering.[23] It first appeared in the regulations annexed to the 1899 Hague Convention II and its 1907 counterpart.[24] Article 35(2) of Additional Protocol I to the 1949 Geneva Conventions affirms the prohibition and irrefutably reflects customary international law.[25] Therefore, the norm binds even States that are not Party to the Protocol, such as the United States. Substantively, it outlaws those means and methods of warfare that unnecessarily aggravate the suffering of combatants, that is, which cause suffering serving no military purpose. In that it is otherwise unlawful to attack civilians, this norm applies only to suffering or injury experienced by combatants.[26]

Losing Humanity does not mention the prohibition. Perhaps this is because autonomy is unlikely to present unnecessary suffering and superfluous injury issues since the rule addresses a weapon system’s effect on the targeted individual, not the manner of engagement (autonomous). Nevertheless, an autonomous system could be used as a platform for a weapon that would violate the prohibition, such as a bomb containing fragments that are designed to be difficult to locate during the treatment of wounded combatants.[27] The combination of the platform and the weapon would render the autonomous weapon system unlawful per se. But this possibility is not a valid basis for imposing an across-the-board preemptive ban on the systems.

International humanitarian law also prohibits weapon systems that cannot be directed at a specific military objective.[28] These weapons are unlawful per se in that they are of a nature to strike combatants, military objectives, civilians, and civilian objects without distinction. The principle of distinction is a norm of customary international law,[29] and the companion treaty prohibition appears in Article 51(4)(b) of Additional Protocol I.[30]

The prohibition on weapon systems that are indiscriminate because they cannot be aimed at a lawful target is often confused with the ban on use of discriminate weapons in an indiscriminate fashion. The classic case is that of the SCUD missiles launched by Iraq during the 1990-91 Gulf War. While it is true that the missiles were inaccurate, they were not unlawful per se because situations existed in which they could be employed discriminately. In particular, the missiles were capable of use against troops in open areas such as the desert, and they actually struck very large military installations without seriously endangering the civilian population.[31] However, when launched in the direction of cities, as repeatedly occurred during the conflict, their use was undeniably unlawful. Even though the cities contained military objectives, the missiles were insufficiently accurate to reliably strike any of them.

The SCUD example has particular resonance with respect to autonomous weapon systems. A misperception exists that without a “man in the loop” there is a high risk of misidentifying civilians or civilian objects as lawful military objectives that either may be directly attacked or that do not have to be considered when assessing whether an attack on a military objective will comply with the rule of proportionality (discussed below). In fact, Human Rights Watch categorically claims that “[f]ully autonomous weapons would not have the ability to sense or interpret the difference between soldiers and civilians, especially in contemporary combat environments.”[32]

What Human Rights Watch appears to have missed is that even an autonomous weapon system that is completely incapable of distinguishing a civilian from a combatant or a military objective from a civilian object can be lawful per se. Not every battlespace contains civilians or civilian objects. When they do not, a system devoid of any capacity to distinguish protected persons and objects from lawful military targets can be used without endangering the former. Typical examples would include the employment of such systems for an attack on a tank formation in a remote area of the desert or from warships in areas of the high seas far from maritime navigation routes. The inability of the weapon systems to distinguish bears on the legality of their use in particular circumstances (such as along a roadway on which military and civilian traffic travels), but not their lawfulness per se.

Human Rights Watch’s apprehension is also counterfactual. Military technology has advanced well beyond simply being able to spot an individual or object. Modern sensors can, inter alia, assess the shape and size of objects, determine their speed, identify the type of propulsion being used, determine the material of which they are made, listen to the object and its environs, and intercept associated communications or other electronic emissions. They can also gather additional data on other objects or individuals in the area and, depending on the platform with which they are affiliated, monitor a potential target for extended periods in order to gather information that will enhance the reliability of identification and permit target engagement when the target is relatively isolated. Even software for autonomous weapon systems that enables visual identification of individuals, thereby enabling precision during autonomous “personality strikes” against specified persons, is likely to be developed. These and related technological capabilities auger against characterization of autonomous weapon systems as unlawful per se solely based on their autonomous nature.[33]

The organization actually seems to ask more of autonomous weapon systems than of human-operated systems. For example, Human Rights Watch points to the possibility that an autonomous weapon system might be deceived through the concealment of weapons or exploitation of the system’s sensor limitations.[34] Yet, asymmetrically disadvantaged enemies have been feigning civilian or other protected status to avoid being engaged by human-operated weapon systems for decades (even centuries).[35] The fact that the techniques sometimes prove successful has never merited classifying those systems as indiscriminate per se. In fact, it would be counter-productive to take such an approach because it would incentivize the enemy’s use of the tactic in order to keep weapon systems off the battlefield.

Human Rights Watch also observes that “fully autonomous weapons would not possess human qualities necessary to assess an individual’s intentions.”[36] While it is true that human perception of human activity can enhance identification in some circumstances, human-operated systems already frequently engage targets without the benefit of the emotional sensitivity cited by the organization. For example, human-operated “beyond visual range” attacks are commonplace in modern warfare; no serious charge has been levelled that the weapon systems that conduct them are unlawful per se.[37]

In fact, human judgment can prove less reliable then technical indicators in the heat of battle. For instance, during the 1994 friendly fire shootdown of two U.S. Army Blackhawks in the no-fly zone over northern Iraq, the U.S. Air Force F-15’s involved made a close visual pass of the targets before engaging them.[38] Pilot error (and human error aboard the AWACs monitoring the situation) contributed to their misidentification as Iraqi military helicopters. Similarly, in 1988 the USS Vincennes engaged an Iranian airliner that it mistakenly believed was conducting an attack on the ship. The warship’s computers accurately indicated that the aircraft was ascending. Nevertheless, human error led the crew to believe it was descending in an attack profile and, in order to defend the ship, they shot down the aircraft.[39] Such tragedies demonstrate that a man in the loop is not a panacea during situations in which it may be difficult to distinguish civilians and civilian objects from combatants and military objectives. Those who believe otherwise have not experienced the fog of war.

Losing Humanity also highlights the unique emotional character of human beings. It suggests “robots would not be restrained by human emotions and the capacity for compassion, which can provide an important check on the killing of civilians.” The report concludes that “[e]motionless robots could, therefore, serve as tools of repressive dictators seeking to crack down on their own people without fear their troops would turn on them.”[40] While it is certainly correct that emotions can restrain humans, it is equally true that emotions can unleash the basest of instincts. From Rwanda and the Balkans to Darfur and Afghanistan, history is replete with tragic examples of unchecked emotions leading to horrendous suffering. Reliance on this factor by Human Rights Watch is empirically suspect.

An autonomous weapon system only violates the prohibition against weapons incapable of being directed at a lawful target if there are no circumstances, given its intended use, in which it can be used discriminately. Consider an autonomous anti-personnel weapon system designed for employment in urban areas. Because it is contemplated for use where civilians and combatants are regularly co-located, the system must have sufficient sensor and artificial intelligence capability to distinguish them; otherwise, it qualifies as indiscriminate per se. An autonomous weapon system unable to reliably distinguish between civilians and combatants but planned for use where civilians are not present would still have to be capable of geographical restriction (either based on system constraints such as maximum range or on human operator pre-programming). This would be needed to prevent it from passing into areas where civilians are located. Systems can be limited temporally to achieve the same end.[41]

A second form of prohibition on indiscriminate weapons is codified in Article 51(4)(c) of Additional Protocol I, and reflects customary international law.[42] It disallows weapon systems that, despite being able to strike their targets accurately, have uncontrollable effects. The paradigmatic example is a biological contagion used to infect combatants, the subsequent spread of which is uncontrollable. A weapon with such effects could be mounted on an autonomous platform. Another example is an autonomous weapon system that searches for and conducts cyber attacks against dual-use infrastructure (cyber infrastructure used by both the military and civilians). The malware used to conduct the attacks could be indiscriminate if designed in a way that makes it likely to spread into the civilian network.[43]

III. Unlawful Use of Lawful Weapon Systems

As should be apparent, the likelihood of an autonomous weapon system being unlawful per se is very low. This being so, the question becomes whether international humanitarian law provides sufficient safeguards with respect to the use of these weapon systems. Sadly, the unlawful use of lawful weapons is far from rare during armed conflicts.

The seminal principle of international humanitarian law is distinction. It is one of two principles in this body of law recognized as “cardinal” by the International Court of Justice, which has also characterized it as “intransgressible.”[44] The principle of distinction serves as the fount for the international humanitarian law rules, including those regarding the use of weapon systems, that seek to safeguard civilians, civilian objects, and other protected persons and places during the conduct of hostilities. Article 48 of Additional Protocol I codifies this customary law principle[45]: “In order to ensure respect for and protection of the civilian population and civilian objects, the Parties to the conflict shall at all times distinguish between the civilian population and combatants and between civilian objects and military objectives and accordingly shall direct their operations only against military objectives.”

Distinction is operationalized in a number of rules, the two most fundamental being the customary law prohibitions on making civilians and civilian objects the object of attack.[46] They are codified in Articles 51(2) and 52(1) of Additional Protocol I respectively.[47] Self-evidently, it would be unlawful to use an autonomous weapon system to directly attack civilians or civilian objects. In this regard, note that the same issues that present themselves with regard to other weapons systems also appear in the case of autonomous weapon systems. For instance, the exception to the prohibition on attacking civilians that exists for those who directly participate in hostilities also applies to the use of autonomous weapon systems against them.[48] Similarly, the universally accepted definition of military objectives found in Article 52(2) of Additional Protocol I pertains equally to attacks by autonomous weapon systems on objects, as does the controversy over whether war-sustaining objects qualify as military objectives.[49]

Article 51(4)(a), which reflects customary international law, sets forth a further prohibition on indiscriminate attacks by banning those that are not directed at an unlawful target, and, as a result, are of a nature to strike lawful targets and civilians or civilian objects without distinction.[50] This prohibition differs from that on indiscriminate weapons in that in this case the weapon is capable of being aimed at a lawful target, but the attacker does not do so.[51]

Doubt–the lack of certainty that a person is a lawful target–is of particular importance in respect to this prohibition. During an attack, doubt as to status must be resolved in favor of treating the individual in question as immune from attack. Article 50(1) of Additional Protocol I codifies this presumption, which is generally characterized as customary in nature.[52]

The mere existence of some doubt does not bring the presumption into operation.[53] Rather, the degree of doubt that bars attack is that which would cause a reasonable attacker in the same or similar circumstances to hesitate before attacking. Restated, attackers must act responsibly as a matter of law when conducting military operations. They must consider “the information from all sources which is reasonably available to them at the relevant time,”[54] as well as factors like force protection, the military value of the target, and the likelihood that subsequent opportunities to conduct an attack will present themselves.

The fact that the doubt threshold is framed in terms of human reasonableness complicates translation into the autonomy context. Obviously, development of an algorithm that can both precisely meter doubt and reliably factor in the unique situation in which the autonomous weapon system is being operated will prove highly challenging. After all, artificial intelligence is artificial.

Detailed discussion of the technical mechanisms for determining doubt is beyond the scope of this article. Nevertheless, algorithms attributing values to sensor data, thereby enabling the autonomous weapon system to compute doubt (or, since it is a machine, the likelihood of being a lawful target), are theoretically achievable. For instance, autonomous weapon systems could be equipped with sensors that enable them to determine when a potential target is a child. Such a determination would substantially decrease the probability that the target is a combatant. On the other hand, if the sensors ascertain that a potential target is carrying a weapon or engaging in hostilities (for instance, by launching a missile or firing a weapon), the likelihood of the target being a combatant increases. These are overly simplistic examples offered for the sake of illustration; the actual sensor capabilities of autonomous weapon systems will be much more advanced.

Since such determinations are highly contextual, it will prove more problematic to decide upon the doubt threshold at which an autonomous weapon system will be programmed to refrain from attack. For instance, more doubt can be countenanced on a “hot” battlefield than in a relatively benign environment. The key is human interaction with the system. In theory, human operators could program these and other factors into an autonomous weapon system. Should they set unreasonably high thresholds of doubt (that is, the point where the systems will not attack), the system would violate the prohibition on indiscriminate attacks.

No equivalent presumption exists in the lex scripta for objects. Still, clearly an attack based on an unreasonable conclusion that an object is a military objective violates international humanitarian law. The mode of analysis described above for persons would apply equally to objects. In addition, Article 52(3) of Additional Protocol I sets forth a separate rule with regard to objects normally dedicated to civilian purposes: “In case of doubt whether [such] an object . . . is being used to make an effective contribution to military action, it shall be presumed not to be so used.”[55] Some difference of opinion exists over whether this presumption reflects customary law.[56] In light of that disagreement, the international group of experts that drafted the Tallinn Manual took the admittedly tautological position that in case of doubt such an object may only be attacked “following a careful assessment.”[57]

Ultimately, there is little difference between application of the rules governing attacks on individuals and objects. An autonomous weapon system that lacks any capability to distinguish lawful from unlawful targets may not be used where the two are co-located. In addition, human operators must be able to program acceptable levels of doubt, based on the circumstances in which they will be used, into the systems. Failure to comply with these requirements would constitute an indiscriminate attack.

An attack that is directed against a lawful target must nevertheless comply with the rule of proportionality. The customary international humanitarian law rule of proportionality, codified in Articles 51(5)(b) and 57(2)(iii) of Additional Protocol I, prohibits “an attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated.”[58] This rule is among the most complex and misunderstood in international humanitarian law with respect to both interpretation and application.[59]

Selection of targets by autonomous weapon systems would not absolve humans of responsibility for attacks conducted in violation of the rule of proportionality. Most importantly, a human operator who launches an autonomous weapon system into a situation in which the ensuing attacks are likely to cause excessive collateral damage has violated the rule. When making judgments as to possible violations, both the system’s capabilities and the environment in which it will operate have to be considered. For instance, a violation would occur if the autonomous system served as a platform for weapons that are insufficiently precise to be used in the particular setting, such as a city, and, as a result, harm to civilians and civilian objects was probably going to be excessive to the military gains.

A matter that has drawn a great deal of attention in discussions about autonomous weapon systems is their capability to perform proportionality calculations. Recall that autonomous weapon systems employed in an area where civilians and civilian objects are present must possess some means of distinguishing them from lawful targets. If an autonomous weapon system locates a person or object, but cannot satisfactorily identify it as a lawful target, the system must treat that person or object as civilian. Any harm expected to befall the person or object during an attack on a valid military target would have to be factored into the proportionality calculation as collateral damage.

Proportionality calculations require consideration of both expected collateral damage and anticipated military advantage. A system already exists for determining the likelihood of collateral damage to objects or persons near a target. The “collateral damage estimate methodology” (CDEM) is a procedure whereby an attacking force considers such factors as the precision of a weapon, its blast effect, attack tactics, the probability of civilian presence in structures near the target, and the composition of structures to estimate the number of civilian casualties likely to be caused during an attack.[60] CDEM does not resolve whether a particular attack complies with the rule of proportionality. Instead, it is a policy-related instrument used to determine the level of command at which an attack causing collateral damage must be authorized. Oversimplified, the policy of those armed forces utilizing the methodology is that the higher the likely collateral damage, the higher the required approval authority.

The commander with authority to authorize the attack makes the proportionality determination as part of the attack’s approval process. It is this individual who factors in the other essential element of a proportionality calculation, that is, the anticipated military advantage of the attack.[61] Such determinations are contextual. For instance, an attack on a command-and-control facility expected to cause five civilian deaths at an early stage of the conflict yields greater military advantage than an attack on the same facility that occurs after enemy forces are in disarray and nearing defeat. Similarly, the destruction of a tank that is distant from the frontlines does not yield as much military advantage as destruction of one effectively firing on friendly forces. Because it is contextual, the military advantage element of the proportionality rule generally necessitates case-by-case determinations.

There is no question that autonomous weapon systems could be programmed to perform CDEM-like analyses to determine the likelihood of harm to civilians in the target area. Moreover, these weapon systems would usually be no less likely to generate a reliable result than CDEM since the latter is heavily reliant on scientific algorithms. The more difficult task for the autonomous weapon system would be assessing military advantage. Given the complexity and fluidity of the modern battlespace, it is unlikely in the near future that, despite impressive advances in artificial intelligence, “machines” will be programmable to perform robust assessments of a strike’s likely military advantage. In part, this leads Human Rights Watch to conclude that the proportionality test “requires more than a balancing of quantitative data, and a robot could not be programmed to duplicate the psychological processes in human judgment that are necessary to assess proportionality.”[62]

Yet, military advantage algorithms could in theory be programmed into autonomous weapon systems. For example, the systems could be pre-programmed with unacceptable collateral damage thresholds for particular target sets or situations. As an example, an autonomous weapon system could be pre-programmed with a base maximum collateral damage level of X for a tank; a human would have already made the determination that X generally comports with the proportionality rule. Such thresholds would have to be adjustable by human operators based on the military situation at a particular phase in the conflict, in a particular area of operations, and so forth. Of course, determining the appropriate threshold would be a very subjective endeavour. However, as noted by the ICRC commentary to Additional Protocol I, and as acknowledged in Losing Humanity, proportionality determinations necessarily involve a “fairly broad margin of judgment” and “must above all be a question of common sense and good faith for military commanders.”[63]

Because military advantage is such a context specific value, compliance with the rule of proportionality would require that the base maximum collateral damage threshold be very conservative. Algorithms could be then be developed that would permit the autonomous weapon system to refine the base level threshold to account for specified variables it encountered on a mission. As an example, it would be reasonable to allow the system to increase the level of acceptable collateral damage if it identifies a concentration of enemy tanks, as distinct from a single tank. The concentration poses a greater threat and therefore the military advantage of destroying individual tanks making up the concentration is greater than that of destroying the same tanks when they are operating alone. Similarly, it would be reasonable for the system to adjust the level of acceptable collateral damage based on whether a targeted tank is headed towards or away from the battlefront.

These highly simplistic examples are offered solely for the sake of illustrating the point that the requirement to assess military advantage need not be an insurmountable obstacle. Obviously, the actual algorithms used would need to be much more sophisticated. While they would likely not be able to account for all imaginable scenarios and variables that might occur during hostilities, the same is true of a human confronted with unexpected or confusing events when making a time sensitive decision in combat. Neither the human nor the machine is held to a standard of perfection; on the contrary, in international humanitarian law the standard is always one of reasonableness.

Human Rights Watch also cites the principle of “military necessity” as a basis for finding autonomous weapon systems unlawful.[64] In the author’s view, the organization mischaracterizes military necessity as a distinct rule of international humanitarian law, rather than a foundational principle that undergirds the entire body of law.[65] Despite the different approaches to military necessity adopted by the author and Human Rights Watch, it is clear that even military necessity as understood by that organization would not render autonomous weapon systems unlawful. They would not be unlawful per se because it is clear that autonomous weapon systems may be used in situations in which they are valuable militarily—that is, militarily necessary. As to prohibitions based on use, the condition that military objectives yield some military advantage would make any separate requirement for military necessity superfluous.[66] With regard to situations raising proportionality issues, any strike lacking military advantage but causing harm to civilians or civilian objects would violate the rule.[67] Taking these observations together, the result is that military necessity has no independent valence when assessing the legality of autonomous weapon systems or their use.

Losing Humanity somewhat inexplicably fails to examine a central element in every assessment of a weapon system’s use: the international humanitarian law requirement that the attacker take precautions in attack.[68] Set forth in Article 57 of Additional Protocol I, the rule, which reflects customary international law, requires an attacker to exercise “constant care . . . to spare the civilian population, civilians and civilian objects.”[69] The article goes on to articulate the means by which this obligation is to be carried out. In particular, an attacker is required to “do everything feasible to verify that the objectives to be attacked are neither civilians nor civilian objects and are not subject to special protection but are military objectives”; cancel an attack if it becomes apparent that the rule of proportionality will be breached; provide “effective advance warning” of an attack if it may affect the civilian population, “unless circumstances do not permit”; “[w]hen a choice is possible between several military objectives for obtaining a similar military advantage, [select] that the attack on which may be expected to cause the least danger to civilian lives and to civilian objects”; and “take all feasible precautions in the choice of means and methods of attack with a view to avoiding, and in any event to minimizing, incidental loss of civilian life, injury to civilians and damage to civilian objects.”[70]

Each of these obligations applies fully to the use of autonomous weapon systems. The requirement to do everything feasible to verify that the target is a military objective would, for example, require full use of on-board or external sensors that could boost the reliability of target identification. In fact, an autonomous weapon system could not be used in isolation if additional external means of identifying the target would measurably improve identification and their use was militarily feasible in the circumstances. As an illustration, such a situation might present itself if an unmanned aerial system could be used to narrow down the location of enemy forces before the autonomous weapon system is launched into that area. This would reduce the likelihood of the system’s misidentification of civilians as combatants.

The fulcrum of the verification requirement is the term “feasible.” Feasible has been interpreted as that which is “practicable or practically possible, taking into account all circumstances ruling at the time, including humanitarian and military considerations.”^[71] Military considerations include both technical and operational factors. Thus, in the previous example, sending an unmanned aerial system into an area of operations that could place it at a degree of risk not justified by the extent of enhanced identification capability would not be feasible. This might be because the aerial system is needed for operations elsewhere of greater military value or because its use in other operations may have a greater prospect for the avoidance of civilian casualties.

The requirement to select among military objectives to minimize civilian casualties and damage to civilian objects would likewise apply to autonomous weapon systems. For instance, an autonomous weapon system could not be employed to attack electrical substations if attacking transmission lines was militarily feasible, would achieve the same military objective (such as temporarily disrupting enemy command and control during friendly operations), and placed civilians and civilian objects at less risk.

However, it is the requirement to select the means of warfare likely to cause the least harm to civilians and civilian objects without sacrificing military advantage that is the key to the controversy over autonomous weapon systems. Indeed, it is the oft-ignored linchpin to various other weapon controversies, such as that surrounding the use of unmanned aerial combat systems. Consider the practical implications of this prescriptive norm: if the use of an autonomous weapon system can be expected to cause greater collateral damage than the use of a weapon system under human control, and the use of the latter is neither likely to diminish the probability that the desired military objective will be achieved nor poses a significant risk to the human operator, use of the autonomous weapon system would be forbidden as a matter of law. Restated, the only situation in which an autonomous weapon system can lawfully be employed is when its use will realize military objectives that cannot be attained by other available systems that would cause less collateral damage. Of course, there is a fair degree of elasticity in application of the norm given that it is based on the feasibility of the competing systems’ use. Nevertheless, the requirement to select among means of warfare should significantly temper the concerns of those who would prophylactically prohibit use of autonomous weapon systems.

Indeed, contemplate the consequences of prohibiting autonomous weapon systems completely. What critics miss is that an autonomous weapon system may be able to achieve a military objective with less threat of collateral damage than a human controlled system. For example, an autonomous weapon system could be armed with non-lethal weapons unavailable on manned systems, its sensor suite could be more precise or discriminatory than one available on manned systems, or its decision-making capability could be better than that of a human in a particular environment (such as a very dangerous one). If the use of the human controlled system in question comports with the rule of proportionality, it would be lawful for an attacker to use it in the absence of the autonomous weapon system. Therefore, the prohibition of autonomous weapon systems would actually place civilians and civilian property at greater risk of incidental harm than if the autonomous weapon system had been available to the attacker.

Some States are beginning to set forth autonomous weapon systems guidelines that are meant to foster compliance with international humanitarian law in addition to achieving other objectives, such as avoiding mistaken engagements. For instance, the current U.S. policy, which distinguishes between semi-autonomous weapon systems, human-supervised autonomous weapon systems, and autonomous weapon systems, provides:

(1) Semi-autonomous weapon systems (including manned or unmanned platforms, munitions, or sub-munitions that function as semi-autonomous weapon systems or as subcomponents of semi-autonomous weapon systems) may be used to apply lethal or non-lethal, kinetic or non-kinetic force. Semi-autonomous weapon systems that are onboard or integrated with unmanned platforms must be designed such that, in the event of degraded or lost communications, the system does not autonomously select and engage individual targets or specific target groups that have not been previously selected by an authorized human operator.

(2) Human-supervised autonomous weapon systems may be used to select and engage targets, with the exception of selecting humans as targets, for local defense to intercept attempted time-critical or saturation attacks for:

(a) Static defense of manned installations.

(b) Onboard defense of manned platforms.

(3) Autonomous weapon systems may be used to apply non-lethal, non-kinetic force, such as some forms of electronic attack, against materiel targets in accordance with [applicable directives].

The policy acknowledges that autonomous or semi-autonomous weapon systems might be intended for use in a manner falling outside these policies. In such cases, it mandates high-level approval before formal development and then again before fielding the system.[72] This requirement is in addition to the legal review requirements set forth below.

IV. Legal Review of Weapon Systems

Since the prospect of autonomous weapon systems is so new, the requirement to conduct a review of their legality looms large.[73] Codified in Article 36 of Additional Protocol I, the rule provides that “in the study, development, acquisition or adoption of a new weapon, means or method of warfare, a High Contracting Party is under an obligation to determine whether its employment would, in some or all circumstances, be prohibited by this Protocol or by any other rule of international law applicable to the High Contracting Party.”[74] Means of warfare are weapons and weapon systems, whereas method of warfare refers to the tactics, techniques and procedures (TTP) by which hostilities are conducted. An autonomous weapon system is a means of warfare. Employing multiple autonomous weapon systems to conduct, for example, a siege by targeting all vehicular traffic into or out of a populated area illustrates their use as a method of warfare.

Although Human Rights Watch suggests that disagreement exists as to whether Article 36 restates customary international law,[75] the obligation to conduct legal reviews of new means of warfare before their use is generally considered—and correctly so—reflective of customary international law.[76] Consensus is lacking as to whether an analogous requirement exists to perform legal reviews of methods of warfare.[77]

States that are Party to Additional Protocol I are clearly required to conduct a legal review of both the autonomous weapon system and any TTP that its user develops. Non-Party States are arguably only bound by the obligation to review the system itself in light of its envisaged usage. This author is of the opinion that both reviews—whether or not legally mandated—are well-advised whenever feasible.

Human Rights Watch also advises that reviews of autonomous weapon systems “should take place at the earliest stage possible and continue through any development that proceeds.”[78] For States Party to Additional Protocol I, this is, as is clear from the plain text of Article 36, a legal requirement. Since there is no corresponding customary international humanitarian law requirement, non-Party States, such as the United States, are only required to ensure weapons are lawful before use. Nevertheless, the organization is right to stress that early legal reviews can shape the development stage of a weapon system and resultantly avoid unnecessary effort and cost associated with components and capabilities that may not pass legal muster. It is U.S. policy to conduct two legal reviews, once prior to taking the decision to enter into formal development, and again before an autonomous weapon system is fielded.[79]

Losing Humanity appears to infer that the United States has taken the position that only a weapon—and not the weapon system—need be examined. Putting individual cases where the United States may or may not have complied with the requirement for legal reviews aside, it is presently U.S. policy to review both weapons and weapon systems. This obligation is unambiguously confirmed in a Department of Defense directive which provides that “[t]he acquisition and procurement of DoD weapons and weapon systems shall be consistent with all applicable domestic law and treaties and international agreements . . . , customary international law, and the law of armed conflict . . . . An attorney authorized to conduct such legal reviews in the Department shall conduct the legal review of the intended acquisition of weapons or weapons systems.”[80]

Human Rights Watch expressed particular concern regarding the possibility of autonomous weapon systems modification. It noted “robots . . . are complex systems that often combine a multitude of components that work differently in different combinations.”[81] The organization also highlighted “the fact that some robotic technology, while not inherently harmful, has the potential one day to be weaponized.”[82] A fair reading of the international humanitarian law norm is that any significant modification to a weapon system requires legal review. The United States agrees. For example, the U.S. Air Force policy on weapon reviews specifically mandates “a timely legal review of all weapons and cyber capabilities, whether a new weapon or cyber capability at an early stage of the acquisition process, or a contemplated modification of an existing weapon or cyber capability, to ensure legality under LOAC, domestic law and international law prior to their acquisition for use in a conflict or other military operation.”[83] To summarize, U.S. policy is to review all weapons, their associated delivery systems, and any significant modification of them. This policy unquestionably applies to autonomous weapon systems.

Losing Humanity cites an array of legal prohibitions for consideration during the legal review. However, the report is overinclusive in that a number of the prohibitions cited bear primarily on the unlawful use of lawful weapons. Legal reviews do not generally consider use issues since they are contextual by nature, whereas the sole context in a determination of whether a weapon is lawful per se is its intended use in the abstract.[84] For instance, the rule of proportionality does not factor into a weapons review because compliance depends on the situational risk to civilians and civilian objects and the anticipated military advantage in the attendant circumstances. Because the assessment is contextual, it is generally inappropriate to make ex ante judgments as to a weapon’s compliance with the rule. Only if the weapon system were necessarily employed in situations where injury to civilians or harm to civilian objects is inevitable and predictable in scope—as in a cyber malware weapon developed for a particular attack—would such an assessment have to be made prior to fielding of the weapon. The requirement that an attacker take feasible precautions in attack to minimize harm to civilians and civilian objects is likewise context specific and, therefore, any assessment of compliance with the norm can only occur with respect to its use in particular circumstances, not as part of the legal review.

The Air Force guidance delineates the legal issues that must be examined when determining the legality of a weapon system being considered for acquisition:

3.1.1. Whether there is a specific rule of law, whether by treaty obligation of the United States or accepted by the United States as customary international law, prohibiting or restricting the use of the weapon or cyber capability in question.
3.1.2. If there is no express prohibition, the following questions are considered:

3.1.2.1. Whether the weapon or cyber capability is calculated to cause superfluous injury, in violation of Article 23(e) of the Annex to Hague Convention IV; and
3.1.2.2. Whether the weapon or cyber capability is capable of being directed against a specific military objective and, if not, is of a nature to cause an effect on military objectives and civilians or civilian objects without distinction.[85]

The requirements set forth in paragraph 3.1.2 mirror analogous provisions appearing in Additional Protocol I, all of which are customary in nature.[86]

The Air Force guidance’s extension of its substantive requirements to cyber capabilities, a relatively recent revision to the basic document, is noteworthy. It illustrates the principle that the rules of international humanitarian law regarding the legality of weapon systems apply fully to weapons that did not exist at the time a particular treaty norm was crafted or customary law crystallized.[87] It is incontrovertible that all of the norms discussed apply equally to autonomous weapon systems.

Finally, Human Rights Watch asserts that legal reviews “should assess a weapon under the Martens Clause,” a proposition echoed by the International Committee of the Red Cross.[88] The clause originally appeared in the 1899 Hague Convention II and was subsequently included in both the 1907 version of that treaty and Additional Protocol I.[89] The International Court of Justice recognizes it as customary in nature and has observed that the Martens Clause “proved to be an effective means of addressing rapid evolution of military technology.”[90]

By its own terms, though, the clause applies only in the absence of treaty law.[91] In other words, it is a failsafe mechanism meant to address lacunae in the law; it does not act as an overarching principle that must be considered in every case. Today, a rich fabric of treaty law governs the legality of weapon systems. Certain of these treaties bear directly on the development of autonomous weapon systems. The restrictions on incendiary weapons, air delivered antipersonnel mines, and cluster munitions, for example, limit their employment on autonomous weapon systems by States Party to the respective treaties.[92] As discussed above, general principles and rules of international humanitarian treaty law, particularly those contained in Additional Protocol I, further restrict weaponry. Emergence of many customary international humanitarian law norms since 1899 also measurably diminishes the significance of the clause. By the turn of the 21st century, the likelihood that future weapon systems, including those that might be autonomous, would not violate applicable treaty and customary law, but be unlawful based on the Martens Clause, had become exceptionally low.

Losing Humanity is correct to accentuate the importance of weapons reviews in the process of developing and fielding new weaponry. However, it must be cautioned that such reviews examine only the legality of a weapon system as such, not its use in any particular circumstance. Therefore, it is doubtful whether the requirement for the reviews will serve as an impediment to the development of autonomous weapon systems as a class of weapons.

V. Accountability

Human Rights Watch expresses anxiety regarding accountability for the activities of fully autonomous weapons. It asks the very reasonable question: “If the killing were done by a fully autonomous weapon, . . . the question would become: whom to hold responsible?” The organization concludes that, “[s]ince there is no fair and effective way to assign legal responsibility for unlawful acts committed by fully autonomous weapons, granting them complete control over targeting decisions would undermine yet another tool for promoting civilian protection.”[93]

The problem with this conclusion is that it is based on a false premise.[94] The mere fact that a human might not be in control of a particular engagement does not mean that no human is responsible for the actions of the autonomous weapon system.[95] A human must decide how to program the system. Self-evidently, that individual would be accountable for programming it to engage in actions that amounted to war crimes. Moreover, the commander or civilian supervisor of that individual would be accountable for those war crimes if he or she knew or should have known that the autonomous weapon system had been so programmed and did nothing to stop its use, or later became aware that the system had been employed in a manner constituting a war crime and did nothing to hold the individuals concerned accountable.[96]

It is hopefully improbable that an autonomous weapon system would be programmed to commit war crimes. Much more likely would be a case in which a system that has not been so programmed is nevertheless used in a manner that constitutes such crimes. For example, the operator of an autonomous weapon system that cannot distinguish civilians from combatants who employs the system in an area where the two are intermixed has committed the war crime of indiscriminate attack. Any commander or supervisor who ordered the attack would likewise be criminally responsible for committing a war crime. So too would a commander or supervisor who knew the operation was about to be mounted and failed to suppress it or who later learned of the operation and failed to take action to hold those responsible accountable.

The United States accepts the premise that those involved in autonomous weapon system operations may be held accountable for their decisions. In its most recent guidance on the use of the systems, the Department of Defense has emphasized that “[p]ersons who authorize the use of, direct the use of, or operate autonomous and semi-autonomous weapon systems must do so with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable rules of engagement (ROE).”[97] The policy imposes identical requirements on Commanders of the U.S. Combatant Commands.[98]

Conclusion

This Article has demonstrated that autonomous weapon systems are not unlawful per se. Their autonomy has no direct bearing on the probability they would cause unnecessary suffering or superfluous injury, does not preclude them from being directed at combatants and military objectives, and need not result in their having effects that an attacker cannot control. Individual systems could be developed that would violate these norms, but autonomous weapon systems are not prohibited on this basis as a category.

International humanitarian law’s restrictions on the use of weapons would nevertheless limit their employment in certain circumstances. This is true of every weapon, from a rock to a rocket. Of course, the fact that autonomous weapon systems locate and attack persons and objects without human interaction raises unique issues. These challenges are not grounds for banning the systems entirely. On the contrary, international humanitarian law’s restrictions on the use of weapons (particularly the requirements that they be directed only against combatants and military objectives, they not be employed indiscriminately, their use not result in excessive harm to civilians or civilian objects, and they not be used when other available weapons could achieve a similar military advantage while placing civilians and civilian objects at less risk) are sufficiently robust to safeguard humanitarian values during the use of autonomous weapon systems.

But might States ban the weapons in the foreseeable future? As noted in the 1868 St. Petersburg Declaration, international humanitarian law fixes “the technical limits at which the necessities of war ought to yield to the requirements of humanity.”[99] Virtually every rule therein reflects a balance States have made between these two seminal factors–humanitarian concerns and military necessity. The humanitarian concerns that are factored into the equation reflect the interest States have in maximizing international humanitarian law’s protection of their combatants and civilian population during armed conflict. In that States are self-interested entities, these concerns are tempered by their desire to retain the ability to fight effectively in order to achieve national interests. The result of this dialectic interplay is international humanitarian law, either in the form of treaty law that has been negotiated by States on the basis of their assessment of the balance or customary law resulting from State practice and opinio juris that reflects the balancing.

Given this process of norm formulation, the Human Rights Watch position is unlikely to find traction. Clearly, and as illustrated in the new Department of Defense directive on autonomous weapon systems, States are sensitive to the humanitarian implications of these systems. Yet, autonomy in combat is in its infancy. Until both their potential for unintended human consequences and their combat potential are better understood, it is unlikely that any State would seriously consider banning autonomous weapon systems. Indeed, there is little historical precedent for banning weaponry before it has been fielded.[100]

Counterintuitive though it may seem, it would arguably be irresponsible to prohibit autonomous weapons at this stage in their development. As noted, such weapons may offer the possibility of attacking the enemy with little risk to the attacker. Although this “value” has sometimes been criticized with respect to unmanned combat aerial systems like the Predator, there is no basis in international humanitarian law for suggesting that attacking forces must assume risk. On the contrary, what is often forgotten is that international humanitarian law affirmatively protects combatants. The paradigmatic example is the “cardinal” prohibition on weapons that cause unnecessary suffering.[101] In fact, international humanitarian law was almost exclusively concerned with the protection of combatants until adoption of the fourth Geneva Convention in 1949.[102] It runs counter to the object and purpose of this body of law to suggest that a weapon system that reduces harm to combatants in situations in which its use does not aggravate civilian risk should be unlawful.

An even more compelling argument is that banning autonomous weapon systems before their potential is understood may have the effect of denying commanders a tool for minimizing the risk to civilians and civilian objects in certain attack scenarios. Admittedly, autonomous weapon system development is not at the point where one can authoritatively conclude the systems will offer less harmful options than human-operated systems. However, it is equally not at the point where such a possibility can be ruled out.

Human Rights Watch is to be commended for drawing attention to the issue of fully autonomous weapon systems. However, in the absence of even a single such system being fielded, it is premature to draw conclusions either as to their legality or to the broader issue of whether they should be banned as a matter of policy. Understanding of the systems’ potential for both positive and negative ends is simply too primitive at this time to comfortably draw conclusions as to their legal, moral, and operational costs and benefits.

---