The IAEA Solution: Knowledge Sharing to Prevent Dangerous Technology Races

Each player chooses whether to research (Ri,t ∈ {0, 1}).

Players learn whether each has made a discovery (Xi,t ∈ {0, 1}) and whether each has caused a failure (Fi,t ∈ {0, 1}).

Players learn whether either player has reached TAI (A = max{Ai,t + Xi,t}) or not (A > max{Ai,t + Xi,t}).

If one player has more knowledge than the other (Ai,t + Xi,t > Aj,t + Xj,t), the more advanced player chooses an amount of knowledge to share (Tt ∈ {0, ..., (Ai,t + Xi,t) − (Aj,t + Xj,t)}). Payoffs Each player gets a per-period payoff based on their AI knowledge at the beginning of the period. We follow models in the economics literature in assuming that short-term benefits increase when a player has more AI knowledge than others do (Budd, Harris and Vickers 1993). Players split the benefit of the knowledge they have in common, while a leading player gets all the benefit of their extra knowledge that the lagging player lacks. Each unit of knowledge produces a total benefit α > 0. Suppose that Ai,t ≥ Aj,t. Both players get α/2 from each of the first Aj,t units of knowledge. The leader gets an additional α for each of the Ai,t − Aj,t units of knowledge that only they have discovered. The player who reaches TAI (Ai,t = A) first gets a benefit τ > 0 in each subsequent period, while the losing player pays a cost κ > 0 each period. (If both players reach the TAI threshold in the same period, we assume that each has a 50% chance of winning.) Players therefore have two motives for maintaining a technological lead for two reasons: to get more benefit from their basic AI knowledge and to get to TAI first. These conflicting interests represent the idea that having access to innovations in AI allows a state to compete more effectively with its rivals. Each player suffers a loss λ > 0 for each failure caused by either player. A player who invests in research (Ri,t = 1) pays a cost normalized to 1. This cost can be thought of as representing expenditures on R&D (by the state or companies under its influence) as well as the expected costs of failures that do not impose externalities on other states. Players have a common discount factor δ ∈ (0, 1). 23Players continue to get these payoffs from their cumulative knowledge in each period forever, even after the race ends. Equilibria of the Baseline Model: Racing and TechnologySharing If players do not condition on each other’s past actions, they simply race to develop more advanced AI. We distinguish two different kinds of races that can occur in equilibrium. In a Perpetual Race equilibrium, both players invest in research until one of them discovers TAI (R1,t = R2,t = 1 ∀ t such that A1,t, A2,t < A). In a Limited Race equilibrium, a player will drop out (permanently) if they fall too far behind. That is, there is some gap g⋆ ≥ 1 such that, if Ai,t − Aj,t ≥ g⋆, then Rj,t = 0.24 However, if players condition on what has happened in previous periods, there can be a cooperative equilibrium in which they avoid a race. In such an equilibrium, the leader shares basic AI technology with the laggard, who in turn refrains from pursuing TAI. Each period, if the leader discovers a new unit of knowledge (X1,t = 1), they share one unit of less advanced knowledge with the laggard (Tt = 1), maintaining a constant gap G between the two players’ knowledge levels. The only exception is the final period, when the leader reaches TAI (X1,t = 1 and A1,t + 1 = A), at which point they do not share their last basic AI discovery (Tt = 0). This deal is enforced by a grim trigger. If the leader doesn’t share any technology after making a discovery (X1,t = 1 and Tt = 0), or if the laggard does any research (R2,t = 1), then the game enters a race: players play either the Limited or Perpetual Race subgame equilibrium in every subsequent period.26 24There can also be a “Never Race” equilibrium, in which players don’t research regardless of how far behind or ahead they are. 25That is, at the beginning of each period, the leader and laggard have respective knowledge levels A1,t and A2,t, with A1,t = A2,t + G. If the leader makes a discovery, their knowledge level in the next period will be A1,t+1 = A1,t +1. Since they share a unit of knowledge with the laggard, the latter will have A2,t+1 = A2,t +1. 26For all parameter values, at least one of the three non-cooperative equilibria exists – Perpetual Race, Limited Race, or the equilibrium in which players never invest. Therefore, in presenting our results from the baseline The leader benefits from this cooperative arrangement because they are guaranteed to reach TAI first, while the laggard benefits by gaining access to a growing stock of knowledge without having to invest in research themselves. In addition, both players benefit from a reduction in the expected number of failures, since only the leader is doing research, while the laggard is only implementing designs that the leader has verified work as intended. Mind the Gap: Results of the Baseline Model Intuition might suggest that a dangerous race can be avoided if one of the competitors is far enough ahead. Let’s call this idea the Hopeless Laggard effect: If the lagging player is far enough behind, and so their chance of winning the race is small enough, they would accept a tech-sharing bargain. If they are even further behind, it isn’t worth it for them to race even in the absence of a tech-sharing bargain. This Hopeless Laggard effect sometimes happens, but not always. In our model, we find that, in some cases, a race can only be avoided if the gap between the players is neither too small nor too big. We call this the Goldilocks effect. In these cases, the benefits of basic-AI advances are so large that the laggard is always willing to invest in research, no matter how far behind they are. The leader is only willing make a technology-sharing deal if the laggard is close enough to pose a credible threat of winning a race. The laggard would make a deal, rather than try to overtake the leader, as long as they are not too close. model, we assume that players always follow one of these non-cooperative equilibria if they do not follow the tech-sharing equilibrium. 27The essential features of this kind of technology-sharing are that the lagging state gives up its pursuit of some powerful technology in exchange for assistance from the leader with some other technology, and that the transfer of technical knowledge from one state to another reduces the shared risks associated with the technology. In practice, such an agreement might allow a lagging state to do research in related areas, so long as that research did not bring them too close to the prohibited transformative technology. Indeed, in the case of the IAEA itself, non-nuclear weapons states can research civilian nuclear technologies, so long as they do not try to develop nuclear weapons. In this section, we present conditions for when a large or a small knowledge gap is needed to prevent a race. When the benefits from basic-AI advances are small, a race can be avoided if the gap between the players is large enough (the Hopeless Laggard effect). When the benefits from basic AI are in a middle range, a race can only be avoided — if at all — when the gap is in a middle range (the Goldilocks effect). We present sufficient conditions for the Goldilocks effect. Among these conditions is that the stakes of acquiring TAI must be sufficiently high. Finally, if the benefits of basic AI are too high relative to the other parameters, such as the players’ subjective evaluations of the risks of the race, it becomes impossible to avoid a race, no matter how small or large the gap is. These results follow from how the benefits of basic AI determine what the players do in the absence of an agreement. A Perpetual Race only occurs if the benefits of basic AI are so large that it is worth investing in research even as a player’s probability of winning the race to TAI approaches zero. For any values of the other parameters, there is always some such threshold α such that, if the basic-AI benefit is above α, Perpetual Race is an equilibrium, while below it, Limited Race is an equilibrium. If the basic-AI payoff is low enough that Limited Race is an equilibrium, the Hopeless Laggard effect must hold. If the laggard starts off far enough behind, the race will be over before it’s begun: even though the leader will not share technology, the laggard will not invest in research. The Hopeless Laggard effect can hold even if the basic-AI payoff exceeds the threshold α, so that a Perpetual Race would occur in the absence of a cooperative agreement. The larger the knowledge gap, the lower the chance that the laggard will ever catch up, which reduces the leader’s incentive to make a deal. However, the leader has another reason to make a deal besides guaranteeing their victory in the TAI race: avoiding the spillover costs of failures the laggard would cause if they researched. If these costs are significant, the leader can credibly commit to sharing technology despite an arbitrarily large gap. The laggard may not be willing to make a deal if the gap is small enough to give them a significant shot at catching up, but for a large enough gap, they will be willing to cooperate. By contrast, if the basic-AI payoff is too high relative to the expected cost of failures, then for some large enough gap the leader would defect to not sharing discoveries. Let us call the threshold basic-AI payoff above which the leader prefers to defect for a large enough gap α. If the expected cost of failures is high enough, then α < α: there is a range of basic-AI payoffs in which a Perpetual Race would occur in the absence of a technology-sharing deal, but for a large enough gap, both players will agree to a deal. If and only if the basic-AI payoff is below either α or α, then the Hopeless Laggard effect holds. If the basic-AI payoff is higher than both these thresholds, then the Hopeless Laggard effect does not hold: for all sufficiently large gaps, the equilibrium outcome is a Perpetual Race. Figure 1 illustrates the different equilibrium outcomes that occur when the knowledge gap becomes very large. The Hopeless Laggard effect holds only in the regions where no race occurs if the gap is sufficiently large. In the “No Race, No Sharing” region, the basic-AI payoff is low enough that the laggard doesn’t race if they are far enough behind, even though the leader does not share technology. For basic-AI payoffs above the “No Race, No Sharing” region, no matter how large the gap in players’ knowledge levels, the laggard prefers to race if the leader does not share technology. In the “No Race, Sharing” region, the basic-AI payoff is nonetheless low enough relative to the high expected cost of failures that the leader shares technology to keep the laggard from conducting research. In the “Race” region, the basic-AI payoff is so high that the leader can’t commit to sharing technology, so both players carry 28The pace of discoveries (ν) must also be high enough for this range to exist. 29For low expected failure costs, the first threshold is binding: a large gap can only prevent a race if the benefits of basic AI alone do not justify investment for a laggard who is too far behind. For higher expected failure costs, the second threshold is binding: a large gap can prevent a race if either a distant laggard wouldn’t race or a distant laggard would race but the leader would share technology. 30This figure holds fixed δ = 0.95 and ν = 0.25. Figure 1: Equilibrium outcome for large knowledge gaps out research, resulting in a Perpetual Race. Proposition 1 provides the necessary and sufficient condition for the Hopeless Laggard effect. The proposition highlights the importance of the returns from basic AI. When these returns are low, the players can avoid dangerous races when one is sufficiently ahead. The method of avoiding the race could be either through a sharing agreement or through the 31The threshold α is the upper bound of the “No Race, No Sharing” region, and it is upward sloping because the higher the expected cost of failures, the more valuable advances must be for a distant laggard to invest in research perpetually. The threshold α is the upper bound of the “No Race, Sharing” region. It slopes upward because the higher the expected cost of failures, the greater the cost to the leader of triggering a race by defecting. laggard unilaterally dropping out. Proposition 1 (Hopeless Laggard Effect): There is a threshold α⋆ > 0 such that, if and only if the benefit of each basic-AI advance (α) is at most α⋆, then there is a threshold G ≥ 1 such that, if the initial gap (G) is at least G, there exists an equilibrium in which the lagging player never researches (R2,t = 0 for all periods t). If, on the other hand, the basic-AI payoff is so large that the Hopeless Laggard effect doesn’t hold, it may nonetheless be possible to avoid a race if the gap is not too large. In a close race, the leader may be willing to make a technology-sharing bargain because the chance of the laggard overtaking them is very high. Depending on the other parameters, there may be a middle range in which the gap is small enough that the leader would cooperate and large enough that the laggard will cooperate too. This is the Goldilocks effect mentioned above. Figure 2 illustrates a case where the Goldilocks effect is in operation. Each line represents the difference between a player’s expected utility from cooperating and their expected utility if they defect, triggering a race. If the difference is positive, the player would cooperate; if it’s negative, they prefer to defect. The leader’s relative payoff from cooperating declines as the gap between the players increases: as the leader becomes more likely to get to TAI first even if a race occurs, cooperation becomes less appealing. The reverse is true for the laggard, whose relative payoff from cooperating increases in the gap size. When the 32We present and prove a full statement of Proposition 1 in Online Appendix A.1. 33There are cases where cooperation is possible even if the laggard is only one unit of knowledge behind. In particular, such cases occur when the stakes of TAI are in a middle range. However, the implication that any small enough gap is sufficient for cooperation in such cases may be an artifact of using discrete knowledge levels. If we were to model knowledge levels as continuous, we might find that there is always some range of gaps close to zero at which the laggard would be unwilling to cooperate. 34This figure holds fixed δ = 0.95, ν = 0.4, θ = 0.1, λϕ = 0.25, α = 0.9, τ = 1, and κ = 4. Figure 2: Incentive to cooperate by player (high-stakes TAI) knowledge gap is 2, both players would cooperate. If the gap is only 1, the laggard would defect. If the gap is 3 or more, the leader would defect. A second proposition summarizes a set of sufficient conditions for the Goldilocks effect:36 Proposition 2 (Goldilocks Effect): If the following conditions are true:

the probability of making a basic-AI discovery each period (ν) is high enough 35For some other parameter values, the middle range in which cooperation is possible includes more than one gap size. 36In Online Appendix A.2, we present a full statement of Proposition 2 and explain how we derived it on the basis of the results established in Proposition 1 and numeric computations. We describe our numerical methods in Online Appendix B.

the basic-AI payoff (α) is in a middle range, the lower bound of which is at least α⋆

the probability that any given discovery leads to TAI (θ) is low enough

the stakes of the race for TAI (τ + κ) are high enough then there is some range of gap sizes defined by 1 ≤ G′ ≤ G′′ such that:

if the initial gap G ∈ {G′, ..., G′′}, the tech-sharing equilibrium exists

and otherwise, the tech-sharing equilibrium does not exist and a Perpetual Race occurs (R1,t = R2,t = 1 in all periods t until one player discovers TAI). There is an upper bound on the basic-AI benefit because both players’ incentives to defect are increasing in that parameter. The higher the basic-AI benefit, the smaller the range of knowledge gaps in which both players would cooperate. If it is too high, there is no gap at which both players will cooperate. The last condition, about the benefit of winning and cost of losing the TAI race, is perhaps the most surprising. Why would raising the stakes of TAI improve the feasibility of cooperation? A key point for understanding this result is that there is always some gap at which the laggard would cooperate. As the gap gets bigger and their chance of catching up approaches zero, they would be strictly better off in expectation receiving technology from the leader, rather than bearing the costs (from investment and additional failures) of making advances for themselves. Given the lower bound on the basic-AI payoff, the leader will only cooperate if the gap is small enough. The bigger the difference between the payoffs of winning and losing the race to TAI, the higher the maximum gap at which the leader would cooperate. If the TAI stakes are large enough, the range of gap sizes in which the leader would cooperate overlaps the range in which the laggard would cooperate. Figure 3 is analogous to Figure 2, but with a much smaller difference in payoffs between Figure 3: The players cannot cooperate through sharing because TAI is low stakes winning and losing the race for TAI.37 With the stakes of TAI lowered, the laggard has less to gain from racing and would cede TAI to the leader no matter how small the gap is. However, a race has become less costly for the leader, and so no matter how small the gap, they prefer to defect and not share any discoveries they make. In short, when the stakes of TAI are high enough and the benefits of basic AI are in a middle range, the players can only avoid a race if the knowledge gap is small enough. 37This figure holds the parameters fixed at the same values as Figure 2 except that the cost of losing the race to TAI is lower: κ = 1. Extensions Incomplete Information A simplifying assumption of our model is that actors have complete information. In the real world, it may be difficult for rival companies, states, or other competitors to monitor each other’s research activities or discoveries. The laggard might cheat on the agreement by secretly carrying out research or the leader might cheat by concealing their technological advances rather than sharing them with the laggard. If either actor had significant doubts about their ability to monitor the other, they might be unable to agree to an otherwise mutually beneficial technology-sharing agreement. Such difficulties with monitoring compliance would certainly make a technology-sharing bargain more difficult in practice. However, we analyzed two extensions of the model with incomplete information and found that technology sharing can be feasible even in the absence of perfect monitoring. In the Hidden Research extension, the laggard has the option of carrying out research in secret, so the leader cannot monitor the laggard’s compliance with the technology-sharing deal. If their research goes well, they can potentially beat the leader to TAI, and if it does not, they continues to receive technology from the leader, ensuring that they never fall further behind than they were when the agreement began. We assume that making extensive use of any knowledge they have acquired that the leader has not yet shared with them would be observable, so the laggard can choose temporarily to not deploy those discoveries. Thus, while their research is hidden, they do not get any benefit from their knowledge beyond the level the leader has shared. Once they choose to carry out their research openly, they receive the full benefits of their knowledge. There is still a risk, however, that the laggard will cause a failure when they test new discoveries, and if a failure happens, the leader infers that the laggard was carrying out research, triggering the end of the technology-sharing agreement. Through numerical computation, we find that there are indeed some parameter values for which the laggard can profitably cheat in this way. Anticipating that the laggard will cheat, the leader will not agree to share technology in the first place. So, there are some cases where technology sharing is possible in the baseline model but not in this extension. However, we find that there are other parameter values for which the laggard cannot gain from secretly researching, so a technology-sharing agreement can be possible even when the leader is unable to verify that the laggard is not conducting research. For instance, if in expectation it will take many periods to reach TAI, or if failures are very likely or costly, the possibility of winning the TAI race is outweighed by the cost of investment and increased failure risk. We also analyze a Hidden Discoveries extension, in which the laggard cannot observe whether the leader is complying with the deal. Specifically, the laggard does not observe whether the leader makes a discovery each period. The exact technology-sharing arrangement we analyze in the baseline model is clearly not feasible in this version of the model, since the laggard cannot condition on whether the leader shares an old technology every time they make a new discovery. Instead, we check the existence of an equilibrium in which the leader shares one unit of technological knowledge with the laggard every K periods. If the leader 38We assume that the probability and cost of a failure is the same when the laggard is secretly testing new designs as when players openly implement them for the first time. It might be more realistic to assume that the risks are smaller since the laggard would presumably have to conduct testing on a smaller scale if they are to keep it secret. However, relaxing our assumptions in this way shouldn’t qualitatively effect the result, because the laggard would still have to tradeoff the chance of overtaking the leader with the cost of investing in research and at least some increased failure risk. Indeed, even if there was no chance of causing a failure while conducting secret research and therefore no chance of getting caught, for some parameter values the laggard would still be better off giving up their shot at TAI in order to avoid paying for their own research, since they will receive the leader’s basic AI discoveries either way. 39See Online Appendix C.1 for a formal description of the model and our numerical methods for analyzing it. ever fails to transfer a unit of technology in a scheduled period or if the laggard ever carries out research, the bargain ends and the players follow whichever non-cooperative subgame equilibrium exists. Since discoveries are stochastic and the leader may make more than one discovery in any given K periods or make none, the gap may grow or widen over time. As a result, there is always a chance that the deal will collapse. The gap can close completely as a result of the leader failing to make discoveries in multiple periods. If there is no gap in a period when the erstwhile leader is scheduled to transfer technology to the other player, they will not have any technology to share that the laggard doesn’t already have. Because the laggard can’t observe whether the leader has been making discoveries, if the leader does not share technology, the laggard can’t tell whether the leader is reneging or has simply been unlucky. Nonetheless, we find that this kind of technology-sharing agreement is an equilibrium for some parameter values. Numerical analysis suggests that a tech-sharing equilibrium exists as long as the probability and benefits of each basic-AI discovery are in a middle range and the probability of any given discovery leading to TAI is sufficiently low. Thus, both of these extensions provide a strong test for the feasibility of tech-sharing agreements: in each one, monitoring is not just imperfect, but non-existent for one of the players. Wealth Transfers So far, we have only considered one way that competitors could avoid a race: by sharing technology. In practice, there are other agreements they might strike. One possibility is that the leader might simply pay the laggard not to research, without transferring technology. Such a solution would be more robust against unanticipated events that might motivate the laggard to rejoin the race. As the technological gap between leader and laggard grows, the 40See Online Appendix C.2 for a formal description of the model and our numerical methods for analyzing it. 41Another way the leader might compensate the laggard would be to allow them to use the technology without sharing the knowledge of how the technology works, e.g. through an API. risk of a race breaking out would decline. We created an extended version of the model in which the leader can transfer wealth, rather than technology, to the laggard in exchange for the laggard refraining from research. We find that a wealth-sharing agreement is indeed self-enforcing for some parameter values. There are cases in which both a technology-sharing agreement and a wealth-sharing agreement are feasible, but there are also cases in which only a wealth-sharing agreement is feasible. However, we also find cases in which a technology-sharing agreement is possible, but a wealth-sharing agreement is not. Because the technology gap grows under a wealth-sharing agreement, the prospect of a race becomes less costly to the leader and so the maximum payment they would make steadily declines. If a Limited Race is a subgame equilibrium, the leader could eventually stop making payments altogether without triggering a race, and so the laggard would have an incentive to defect and begin racing before the gap reached that point. Anticipating the laggard’s defection, the leader would defect when the gap was even smaller, and so on, so that players would not agree to the deal in the first place in equilibrium. These cases, in which tech-sharing agreements are feasible but wealth-sharing agreements are not, occur when advances in basic AI occur rapidly but the discovery of TAI is a long way off in expectation. Wealth-sharing deals unravel if the probability of a discovery each period is too high, because the laggard anticipates with a high degree of confidence when the gap is about to widen to the point where the leader would stop making payouts. Because a rapid 42However, even when the non-cooperative equilibrium is a Limited Race, a wealth-sharing agreement can still be feasible if discoveries occur with a low probability, so that even when the leader is just one unit away from stopping transfers, the payments continues for such a long time in expectation that the laggard wouldn’t want to defect. 43There are also cases in which a wealth-sharing equilibrium exists but the leader eventually stops paying even though the resulting race is perpetual. In these cases, while a technology-sharing agreement could guarantee that the laggard would never race, a wealth-sharing agreement could not. pace of basic AI discoveries does not affect the gap between the players under a technologysharing agreement, the laggard would cooperate so long as the probability that any one discovery leads to TAI is low, because the agreement lasts a long time in expectation. The NPT: A Case Study The fact that there are only nine nuclear weapons states today — rather than the dozens Kennedy warned of — is likely in part due to the global nuclear nonproliferation regime that is centered around the Treaty on the Non-Proliferation of Nuclear Weapons. Also known as the Nuclear Non-Proliferation Treaty (NPT), this international agreement was a technologysharing bargain in which nuclear-armed states provided civilian nuclear technology to other states, and the latter agreed in turn not to acquire nuclear weapons themselves. The case confirms that dynamics similar to those described in the model can be important elements in the governance of dangerous new technologies. It also demonstrates that such trades can be necessary to convince states to give up the pursuit of a powerful technology. Technology sharing was acceptable to the US and Soviets because some states were close to acquiring nuclear technology. It was acceptable to those other states because they got things in return, including civilian nuclear technology and security guarantees. The deal thus reconciled the goals of (1) enabling less technologically-advanced powers to reap the benefits of civilian nuclear energy technology, (2) reducing the global risks of nuclear energy programs, and (3) discouraging beneficiary states from pursuing nuclear weapons. Some scholars reject the idea that the NPT was meant to be a bargain to trade civilian nuclear tech for giving up nuclear weapons. Swango (2014) argues that when the United States ratified the treaty in 1968, it was not committed to providing assistance to states 44See Online Appendix D for a formal description of the model and our numerical methods for analyzing it. 45M¨uller 2005; Cirincione 2007; Findlay 2012 The NPT requires NWS themselves to give up nuclear weapons eventually, albeit without specifying a timeframe. However, possible limitations on states’ use of an advanced tech (represented by TAI in our model) is beyond the scope of our model. that renounced nuclear weapons. At the same time, the US was willing to share nuclear technology with some states that remained outside the NPT. We show, however, that the promise of civilian nuclear assistance and cooperation was essential to the selling of the agreement to the powers who would be denied the possibility of pursuing nuclear weapons. Civilian and military applications of nuclear science play a similar role to basic and transformative AI, respectively, in our model. This is so for several reasons. First, nuclear technology is dual-use: the technical knowledge required in nuclear energy and other civilian applications can also make it easier for a state to develop nuclear weapons. Second, even the civilian applications of nuclear technology pose risks that spill across borders, from accidents to the theft of nuclear material by terrorist groups,47 risks that are heightened if states do not have access to lessons learned from others’ experience. Third, since much of even civilian nuclear technology is not publicly available, we can meaningful speak of gaps between different states in terms of their access to advanced nuclear technology. There are also some differences between the nuclear case and the strategic context we describe in the model. A significant one is that, unlike TAI in our model, nuclear weapons technology already existed at the time that the NPT was created. This is essentially a simplification of the modeling context in which the leader achieves the major advance (TAI) immediately; the fundamental strategic dynamics are similar. Another difference between the case and model is in the number of actors. While our model is a two-player game, the NPT is a global multilateral agreement. However, it codifies a bargain between two groups: the original five nuclear weapons states (NWS) and the world’s many more non-nuclear weapons states (NNWS). Larger numbers of states add complexity, but we believe that some fundamental strategic problems are evident in both contexts. 46M¨uller 2005; Pilat and Busch 2015 47Pilat and Busch 2015; Findlay 2011 48Findlay 2011; Taebi and Mayer 2017 49Kroenig 2009 The NPT emerged from negotiations between the United States and the Soviet Union aimed at limiting the spread of nuclear weapons to other countries. The two superpowers reached agreement by the end of 1966 that the treaty should prohibit the transfer of nuclear weapons technology to any states that did not already have it. However, many of the NNWS objected to the two-tiered system this proposed treaty enshrined: a permanent division between the nuclear weapon “haves” (the US, the USSR, the United Kingdom, France, and China) and “have-nots” (the rest of the world).52 Swango (2014, 211) — who, as noted earlier, rejects the claim that the NPT can be seen as a technology-sharing bargain — remarks: “Non-aligned states and the superpowers’ own allies argued they could not join a discriminatory system that permitted the superpowers to possess nuclear weapons unless they received something in exchange for adhering.” Among the countries objecting to the US-Soviet draft treaty were several nations that a 1958 US National Intelligence Estimate judged to be capable of developing nuclear weapons within a decade. Getting such countries to join the NPT was a “particular concern for the superpowers,” but these close laggards “realized that signing the treaty could have economic and security drawbacks” and insisted that the treaty guarantee their access to “nuclear technology for peaceful purposes” (Hilborne 2012, 254). In response to the demands of the NNWS, Washington and Moscow revised the NPT, starting in the second half of 1967. The UN General Assembly approved the final version of the treaty in June 1968 only after two further articles had been added guaranteeing, as Bourantonis (1997, 356) summarizes, “the sharing of benefits from the peaceful application of nuclear energy” with states that agreed not to pursue nuclear weapons. Articles III 50Cirincione 2007, 30 51Goldschmidt 1980, 75 52M¨uller 2005, 46 Glenn Seaborg, a former chair of the US Atomic Energy Commission later observed: “The non-nuclear countries were not about to accept without resistance a pact that they believed to be highly discriminatory against them” (quoted in Cirincione 2007, 31). 53Goldschmidt 1980, 74; Cirincione 2007, 27 and IV not only permitted NNWS to engage in civilian nuclear activities but explicitly ensured the transfer of knowledge related to civilian nuclear technology. While the NNWS had to submit to verification by the IAEA that they were not developing nuclear bombs, Article III required that the verification measures must “avoid hampering the economic or technological development of the Parties or international co-operation in the field of peaceful nuclear activities.” Article IV creates a positive obligation of signatories to share peaceful nuclear technology: “All the Parties to the Treaty undertake to facilitate, and have the right to participate in, the fullest possible exchange of equipment, materials and scientific and technological information for the peaceful uses of nuclear energy.”54 Whereas the original US-Soviet negotiations focused exclusively on banning the proliferation of nuclear weapons, “It was not until the rest of the world got involved with the negotiations that its most contentious provisions — those calling for sharing nuclear technology for peaceful purposes and for the gradual elimination of the nuclear weapon states’ arsenals — were debated and finally accepted.”55 Assessing the overall impact of the NPT on nuclear proliferation is complex. Some evidence suggests that states which receive technical assistance with civilian nuclear activities are more likely to pursue and ultimately acquire nuclear weapons. However, one of these studies also shows that signing up for the NPT reduces the chances states will start a nuclear weapons program. And Gibbons (2020, 282) finds that it is precisely the prospect of receiving civilian nuclear technology that has persuaded some states to join the NPT, because: “After the risks of nuclear assistance became well-known following India’s nuclear explosion in 1974, most major suppliers conditioned their assistance on recipients joining nonproliferation agreements.” Whether or not the US intended, at the time of the NPT’s 54The text of the NPT is available at: www.un.org/disarmament/wmd/nuclear/npt/text 55Cirincione 2007, 30 56Kroenig 2009; Fuhrmann 2009; Brown and Kaplow 2014 57Fuhrmann 2009 ratification, to use civilian nuclear assistance to induce individual states to join the treaty, the offer of such assistance seems to have been crucial to both the passage of the the treaty and the large number of states that subsequently joined it. Thus, like the players in the technology-sharing equilibrium of our model, the signatories of the NPT know that NNWS could use their knowledge of civilian nuclear technology to pursue a weapons program. But they calculate that most NNWS will have a strong incentive not to jeopardize their continued access to peaceful nuclear technology — and that many of them would pursue weapons in the absence of such assistance and cooperation. Information sharing with the NNWS serves a second purpose beyond inducing states to renounce nuclear weapons: it improves the safety and security of nuclear facilities, reducing risks that spill across borders. In the last three decades the IAEA has played a growing role in providing that assistance, advising states on preventing accidents at nuclear reactors and securing nuclear fuel against theft. In particular, the IAEA’s Global Nuclear Safety and Security Network supports “the transfer of knowledge from countries with mature nuclear energy programmes to countries that have only just started to embark on such programmes.”60 As in our model, knowledge-sharing under the nuclear non-proliferation regime reduces the global risks of nuclear energy production in less developed countries. The NPT case illustrates how the results of our model can occur in practice. A bargain to share one kind of technology can prevent competition for another, related technology. The negotiations that led to the NPT also illustrate the Goldilocks Effect. The nuclear-armed powers were willing to concede civilian nuclear assistance because several states posed a 58A 2004 UN report notes: “Almost 60 states currently operate or are constructing nuclear power or research reactors, and at least 40 possess the industrial and scientific infrastructure which would enable them, if they chose, to build nuclear weapons at relatively short notice if the legal and normative constraints of the Treaty regime no longer apply” (emphasis added; quoted in Cirincione 2007, 106). 59Findlay 2011; Taebi and Mayer 2017 60International Atomic Energy Agency, “Global Nuclear Safety and Security Network (GNSSN)”, accessed May 20, 2022, https://www.iaea.org/services/networks/global-nuclear-safety-and-security-network credible threat of developing their own atomic weapons in the near future. Most of the non-nuclear-armed countries were willing to join the NPT because they were not too close and assistance with nuclear energy was a valuable inducement. Discussion For most of the Cold War, the United States and the Soviet Union continued to deploy increasing numbers of nuclear warheads atop intercontinental ballistic missiles. In the late 1980s, this nearly continuous increase was replaced by a nearly continuous decline on both sides. This change coincided with an agreement between the sides, the Strategic Arms Reduction Treaty (START I), which was negotiated in the 1980s and signed by George H. W. Bush and Mikhail Gorbachev in 1991. How far did the sides reduce their arsenals of deployed weapons? Down to about 6,000 on each side, the number allowed by the treaty. Agreements have an important role to play in arms control, but the strategic context determines which can be self-enforcing in the absence of a common governing authority. The results of the models presented here confirm that even in the highly adversarial context of technology competition, and even without the mitigating factor of the decreasing returns from weapons technology investment in the nuclear race, agreements between adversaries that reduce the level of arms or technology competition are possible. These results contrast with those that derive from static race models. The role of information about each other’s behavior, for instance, is different. Dynamic models allow agents the possibility of punishing adversaries who fail to adhere to agreements. Implementing such strategies requires a certain amount of information, and this implies that information can play a welfare-increasing role that is absent from static models. While perfect information is not required for cooperative equilibria, the absence of any expectations about an adversary’s behavior precludes agreements that require conditioning on it. Another conceptual issue, however, is whether the actors know exactly what is required to achieve TAI. This sort of information is subtly different from knowledge of relative position in the search for TAI. The welfare decreasing effect of information in the Armstrong, Bostrom and Shulman 2016 model, for instance, comes not from information about relative position directly, but from the heightened competition that erupts when actors know exactly what they need to do to outdo each other, and from the assumption that doing slightly more leads to all the benefits with certainty. At least in the near term, however, since it remains unknown what is required to produce TAI, this seems not to be a faithful representation of the strategic context. This is another reason to expect information about actors’ actions and knowledge to have a more beneficial effect, at least in the short term when actors do not know the precise set of steps required for TAI. Such static models, and dynamic models that are analyzed through Markov perfect equilibria, place the emphasis on the material strategic context. The dynamic game that we analyze puts the emphasis instead on the social world that actors could create. It allows us to ask what agreements would be self-enforcing in the absence of a common authority to compel compliance. We stated at the outset that this paper would provide a baseline model, and indeed, there are many extensions that would be useful for understanding these dynamics. We might analyze larger numbers of players or the dynamics of a model with the possibility of conflict or sabotage. One could also study the degree of resource investment in order to understand, for instance, how investment changes as players are closer and farther from each other in the AI knowledge race (Bimpikis, Ehsani and Mostagir 2019), and how this impacts safety dynamics. One could study learning dynamics as players observe aspects of each other’s progress, such as how players might be given an incentive to invest or not based on other players’ success. Innovations may provide investment and racing incentive because they represent proof of the possible. This modeling direction would enable the generation of hypotheses about when to expect secrecy versus product development that employs recent innovations. A promising direction for future work is to model a tradeoff between advancing quickly and advancing safely. This research direction includes models of NGO, governmental, and intergovernmental actors who seek to increase race safety. Such actors may have the paradoxical problem of not being able to credibly threaten to take actions that risk safety. They may not wish, for instance, to threaten to race unsafely themselves unless other actors race safely. On the other hand, in a situation with multiple potential racers, an altruistic actor could offer to help the safest actor with the goal of influencing the behavior of all. In the best-case equilibrium, the threat from the altruistic actor would deter other actors from racing unsafely without the altruistic actor even having to commit any resources at all. The threat of committing resources if another actor deviated could be enough. This would mirror some models of lobbying in which lobbyists achieve their objectives without committing any funds — because of the threat to fund political adversaries. How these dynamics would interact with different relative positions in the research race is something for modeling to investigate. Although considerations about the social and global risks of advanced technologies to societies at large motivate this study, our model only allows direct analysis of the welfare of the potential participants in a technology race. The balance of the benefits and risks to societies is likely to vary considerably across different applications of AI, not to mention other advanced technologies, and has crucial implications for whether a technology-sharing agreement that is mutually beneficial to its participants would be in the interests of affected third parties. For the most dangerous technologies, such agreements may not go far enough: a total ban on research in some areas may be preferable for humanity as a whole. For other technologies, the benefits to third parties may be so large that an agreement which reduces competition and delays advances in expectation may be detrimental to global welfare.