The Artefacts of Intelligence: Governing Scientists' Contribution to AI Proliferation

Governing scientific research outputs 7

Knowledge-control regimes in AI research 11

Risks 17

The case study 21

What is special about AI? 31

Introducing the central topics of analysis 35

Contribution to literature 49 CHAPTER 1: STAGED RELEASE 54

Introduction 54

The lead-up to the GPT-2 release decision 61

The identification problem 70

Coordination within the AI community 74

Distributing the responsibility to identify and address risks 100

The relationship between science and society 111 Vision 1: Scientists as knowledge-producers, society as the beneficiary 112 Vision 2: Science as powerless; society as powerful in determining the technology’s impact 116 Vision 3: Society as a threat to science, requiring management of the public perception of AI 125 Vision 4: Society as a decision-maker within the KCR 128 Conclusion 131 CHAPTER 2: INTERFACE 134

Introduction 135

The origins of the GPT-3 API 139

Scale as a governance strategy 146

The API as a platform for governance 170

The relationship between science and society: open source vs API 192

The API and the relationship between industry and academia 208

Introduction 237

Background 240

How does the regime work? 246

Teaching an old dog new tricks 255

Level of analysis 257

Incentive to avoid controversial topics in BIS 264

Accept/reject as a lever 266

Gap between research and applications 272 Upshot 278 Conclusion 280 CONCLUSION 283

KCRs and risk 285

The challenges of governing AI 298 2.1 Governability: opportunities and limits 298

Governing scientific research outputs This dissertation follows recent attempts to impose order on the dissemination of AI research artefacts. A number of institutional entrepreneurs within the AI research community have attempted to integrate their concerns about AI risks into the way AI research outputs are shared. This has involved departing from the standard paradigm in which AI researchers are expected to promptly open source the software created during a published research project. There have been some minor successes in reconfiguring the way AI research is shared, although on the whole, progress has been slow. Those concerned with the risks of widespread AI proliferation have been met with resistance, both from the AI research community and, on a deeper level, from the institutions they seek to reform. On a general level, I am trying to understand whether science — as a place where new, potentially dangerous technologies sometimes get developed — can be governed in the 1 European Commission, Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts (COM(2021) 206 final). 2021. interests of society. This question, of the governability of science, has already been studied in various forms. Some scholars have studied how public opinion can be afforded a greater role in the setting of scientific priorities. Others have focussed on the epistemic problem of anticipating the impacts of new technologies before they become irreversible. Other scholars have studied how scientists use rhetoric to insulate themselves against government intervention. Scholars have also looked at attempts to integrate into major scientific projects the expertise of social scientists, ethicists, and lawyers — can scientific development be guided, from the outset, by these voices?5 My dissertation contributes to this general body of literature on the governability of science. More specifically, I am investigating a particular type of governance: structuring who gets what access to the knowledge and artefacts created during the research process. Here, I rely on Stephen Hilgartner’s concept of a “knowledge-control regime”.6 I explain this concept below, but for now, a knowledge-control regime is a governance regime that allocates rights, obligations, and restrictions over knowledge — for example, military classification or patent protection. The central knowledge-control regime operative within AI research is the conference publication regime, through which AI researchers publish papers. This goes hand-in-hand with the second-most central knowledge-control regime in AI research, which is that researchers are expected to open source the AI software that they write about in 6 Stephen Hilgartner, Reordering Life : Knowledge and Control in the Genomics Revolution, Inside Technology (Cambridge, Massachusetts: The MIT Press, 2017). 5 Stephen Hilgartner, Barbara Prainsack, and J. Benjamin Hurlbut, ‘Ethics as Governance in Genomics and Beyond’, in The Handbook of Science and Technology Studies, ed. Ulrike Felt et al., Fourth, Ebook Central (Cambridge, Massachusetts: The MIT Press, 2017), 823–51. 4 Thomas F. Gieryn, ‘Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists’, American Sociological Review 48, no. 6 (1983): 781–95, https://doi.org/10.2307/2095325; J. Benjamin Hurlbut, ‘Remembering the Future: Science, Law, and the Legacy of Asilomar’, in Dreamscapes of Modernity, ed. Sheila Jasanoff and Sang-Hyun Kim (University of Chicago Press, 2015), 126–51, https://doi.org/10.7208/chicago/9780226276663.003.0006. 3 David H Guston, ‘Understanding “Anticipatory Governance”’, Social Studies of Science 44, no. 2 (15 November 2013): 218–42, https://doi.org/10.1177/0306312713508669; David Collingridge, The Social Control of Technology (London: Frances Pinter, 1980). 2 Alan Irwin, ‘Constructing the Scientific Citizen: Science and Democracy in the Biosciences’, Public Understanding of Science 10, no. 1 (1 January 2001): 1–18, https://doi.org/10.3109/a036852. papers. These regimes are designed such that scientists can effectively evaluate and build upon each other’s work. My dissertation addresses the question: how have these scientific knowledge-control regimes been adapted to address AI risks? In other words, how does risk become a feature of the knowledge-control regimes that govern AI research? Stephen Hilgartner’s 2017 book, Reordering Life, studies knowledge-control regimes in the Human Genome Project. The study is about the relationship between knowledge-control regimes and transformative scientific change. Hilgartner studies how knowledge-control regimes, such as libraries of genetic data, coevolve alongside changes in the underlying science, such as new genome mapping technologies. My research is a simple extension to this work, in that I have added an extra ingredient: risk. Many people, including many AI scientists, are concerned about various risks of AI, ranging from concrete harms taking place today, to risks of future, more powerful systems (see below). I study the interaction between AI risks and knowledge-control regimes. As with Hilgartner’s study of the Human Genome Project, I am simultaneously attentive to how changes in the underlying science can also have an impact upon knowledge-control regimes. The question is how AI risks are integrated into the knowledge-control regimes (“KCRs”) that govern AI research. A range of lower-level questions fall underneath. What do risk-focussed KCRs look like? How do these KCRs distribute responsibilities for tackling risks across different actors? How do they rewrite the social contract between the scientific community and broader society? Are some scientific knowledge objects easier to control than others — and so how might changes in the underlying science produce opportunities for governance? Is it possible to build risk-focussed KCRs on top of the field’s existing (academic) regime for sharing research outputs, or is it impossible to teach an old dog new tricks? How does the anarchic structure of the scientific field limit the kinds of changes that can be effectively made to KCRs? To further clarify the focus of the thesis, the table lays out different possible research questions and whether or not they are the focus: Question Focus of this thesis? Why govern? I.e. What is the best rationale for governing the outputs of AI research? No. In the case study, the motivation for governance is the societal risks posed by AI (described below). I do not focus on substantiating these risks. Who should govern? I.e. which actors should be empowered to govern the outputs of AI research? No. The case study focuses on governance coming from within the AI research community rather than government regulation (see below). But I do not study the normative question of which actors would be most appropriate to govern AI research. How does governance take place? I.e. What form do the various governance attempts take, and how do they evolve? Yes. This is the main focus of the thesis, with an emphasis on the interplay between risk and knowledge-control regimes. Can AI research be governed? I.e. is the field of AI research, in its current form, capable of being governed? Secondary focus. A recurring theme is the difficulty of governing how AI researchers share their work. The thesis is mainly descriptive, in that I describe new, risk-focussed KCRs and how they function. The thesis is also partly explanatory, in that I offer some explanation for the emergence of these new KCRs – beyond the obvious explanation of “researchers and labs were simply responding appropriately to the real risks of AI”. For example, I argue that the existing institutional landscape – including the existing KCRs – constrains the development of new, risk-focussed KCRs (see below, section 5). The thesis is not seeking to be normative, although in the concluding chapter I do briefly offer some reflections on how AI should be governed given the persistent difficulties with addressing AI risks via the channels attempted in the case study.

Knowledge-control regimes in AI research Hilgartner defines a knowledge-control regime as “a sociotechnical arrangement that constitutes categories of agents, spaces, objects, and relationships among them in a manner that allocates entitlements and burdens pertaining to knowledge.”7 The Creative Commons licence is an example, as are therapist-patient confidentiality rules. Knowledge-control regimes can be created by legal rules, but other substrates are available, such as informal norms and practices. Some knowledge-control regimes restrict the flow of knowledge, while others aim to spread knowledge more widely. A knowledge-control regime positions different agents, such as the author of a conference paper or the owner of a patent. The regime structures the relationship between agents analogous to a contractual agreement. Hilgartner borrows from the legal philosopher Wesley Hohfeld in unpacking these different possible relationships. For example, one scientist might have a right to access the data behind a published experiment, and this 8 Wesley Newcomb Hohfeld, Fundamental Legal Conceptions, ed. Walter Wheeler Cook (New Haven: Yale University Press, 1919). 7 Hilgartner, 9. entails that the authors have a duty to share the data upon request. That is one category of relationship: rights and corresponding duties. Another category is where one agent is free to take an action (which Hohfeld calls a “privilege”), e.g. he is free to pass on information to trusted confidantes; and this entails that other agents have no right to prevent the behaviour. These relationships parcel out control over knowledge, delineating who can do what with certain knowledge objects, such as accessing, editing, using, and sharing the knowledge. Sometimes, this has the effect of creating some kind of ownership over property — for example, patent ownership, or the ownership of digital art via non-fungible tokens. In AI research, the central knowledge-control regime is the system for publishing papers at conferences, which I will call the “conference publication regime”. There are certain high-prestige AI conferences that are international in nature and cover AI or machine learning very broadly: NeurIPS, ICML, and ICLR. There are also conferences known for particular areas within AI, such as CVPR for computer vision, and NAACL for natural language processing. Authors submit short papers, typically around 8 pages. Papers usually have several authors; often, the first author will be a junior researcher who did most of the painstaking labour, and the last author will be a senior researcher who supervised the project. The authors can come from anywhere in the world, and they do not have to be based at a university. The papers are peer reviewed on the basis of scientific merit. The accepted papers are published on the conference’s web page. The authors can be invited to present their work at a poster session or a talk at the conference. Authors also often upload their papers to the preprint site arXiv, in tandem with going through the conference publication regime, and this is generally tolerated by the conferences. 9 Hohfeld’s framework also contains second-order relationships: an agent may (or may not) have the power to make changes to the existing setup of rights and privileges. Under the conference publication regime, nobody owns the knowledge contained in the papers — it “belongs to the literature”. The authors have no right to step in and prevent other researchers from using the ideas in later work; they can only demand that their contribution be acknowledged through a citation. If the authors want control over how their work is applied in the world, they have no recourse through the conference publication regime. They would have to turn to other regimes, where possible, such as patent protection. Patent protection is compatible with the conference regime, in that it is possible to publish and then apply for a patent on the published technique, although this is not a widespread practice (the industry labs sometimes do it).10 The conference publication regime plays a foundational role in structuring AI research. First, it provides strong incentives for researchers to be productive. Publishing at a major conference gives a researcher status, a sense of achievement, and it looks good on the CV. Job advertisements for AI researcher roles will often prefer that the applicant has, for example, ‘First-author publications at peer-reviewed AI conferences (e.g. NeurIPS, CVPR, ICML, ICLR, ICCV, and ACL)’.11 Second, and relatedly, somebody’s existing publications provides a way of assessing their expertise or progress in an area. Internally, corporate research labs will sometimes look at publications as a way of assessing career progression for promotion, and the same thing happens at universities. Third, the conference publication regime helps the researcher to organise their workflow. A researcher’s calendar will have the submission deadlines for their favourite conferences; they can spend a few months working on a paper, submit it to a conference, and then move onto the next. Finally, conference papers allow for collaboration between researchers across time and space. 11 This example is from a job advert for a Research Scientist at Facebook’s AI research team (accessed Jan 2022). 10 Nathan Calvin and Jade Leung, ‘Who Owns Artificial Intelligence? A Preliminary Analysis of Corporate Intellectual Property Strategies and Why They Matter’ (Working paper, February 2020), https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-working-paper-Who-owns-AI-Apr2020.pdf. Papers are narrowly focussed and build upon one another. Polanyi compared the scientific process to different people collectively building a jigsaw puzzle. The conference publication regime (and the broader culture of publishing papers) hosts this large-scale collaboration. Overall, this regime is the most important institutional structure in AI research. It allows for a kind of structured anarchy, comparable to (as Polanyi pointed out) the market as a form of social organisation. In just a single institutional framework, AI researches are given incentives to produce research, a way of assessing each other’s expertise and status, a way to structure the working year, a means of collaborating across time and space, and even, in many cases, a social life — after all, the conferences are great opportunity to hang out with people you haven’t seen in a while. It is little wonder, then, that AI researchers appear to be obsessed with papers. Alongside conference publication, there is an informal regime where researchers upload their code and models to online repositories like GitHub. This dovetails with the conference publication regime: authors will often include links to such repositories at the bottom of their papers, and this is viewed favourably by reviewers. The NeurIPS conference has a code submission policy, where they ask that, where relevant, authors submit the code that they used to train and evaluate their models. Sometimes, it will be easy to take the code and use it to train a model that is very similar to that featured in the paper. However, sometimes – even if the code is very comprehensive – running this computation will take many weeks and thousands of dollars or more. This is becoming increasingly expensive 14 The NeurIPS 2021 code submission policy reads: “If any of the main contributions of your accepted paper depends on an experimental result, it’s best practice for responsible research to include code that produces this result.” See: ‘NeurIPS 2021 Code and Data Submission Guidelines’ (NeurIPS, 2021), https://neurips.cc/Conferences/2021/PaperInformation/CodeSubmissionPolicy. 13 On Twitter, common topics of conversations are, for example: (a) “I am excited to share my latest paper…”; (b) complaints about shoddy peer reviewing; and (c) solidarity with other researchers in the face of an upcoming submission deadline. Also, if you ask an AI researcher, “how do you keep up to date with what’s going on in the community?” they will often assume you are asking: “how do you keep up to date on reading the latest literature?” 12 Michael Polanyi, ‘The Republic of Science’, Minerva 1, no. 1 (1 September 1962): 54–73, https://doi.org/10.1007/BF01101453. with the trend towards training large models. In such cases, researchers are generally expected to open source the model. Practically speaking, this means uploading a file containing code that defines the structure of the model alongside a very long list of numbers. Those numbers are called the “parameters” or “weights” of the model, and they specify the strength of the connections between different neurons inside the neural network. These weights are discovered during the training process. Even though researchers use GitHub to share their models, they do not build these models as open source projects, with online developers chipping in along the way. Rather, the researchers upload their finished project to GitHub. If researchers directly collaborate with individuals from different labs, this is an academic collaboration, as coauthors on a paper. The project’s GitHub page will refer to the paper – for example, linking to a preprint listed on arXiv, listing the authors, and showing how to cite the paper. In other words, scientific publication is still the dominant KCR. Finally, models are usually trained using publicly available datasets. However, sometimes researchers will create their own datasets; if so, they are generally expected to share this too. As with code and models, this expectation is not always strictly adhered to; for example, Google and Facebook have certain in-house datasets that their researchers use without sharing. I have briefly sketched out the existing knowledge-control regimes within AI research. Risks to society are not given any specific role in the setup. 16 For example, Google has the JFT-300M internal dataset of labelled images. 15 This reflects the ‘cathedral’ rather than the ‘bazaar’ model of development. See Eric S Raymond, The Cathedral and the Bazaar : Musings on Linux and Open Source by an Accidental Revolutionary, First edition (Beijing; Cambridge, Mass.: O’Reilly, 1999).

Risks In this thesis, “risk” refers broadly to a possibility of future harm. The sociological literature on risk sometimes takes a narrower concept: using ‘risk’ to refer to a particular cultural approach towards assessing potential harm, emphasising harms that are quantifiable or lend themselves to technical assessment. I use the broader concept in order to capture the diversity in how people in my case study think about risks from AI, which are often difficult to quantify or technically assess, especially when they relate to risks of future AI systems. Nonetheless, the thesis still retains a sociological approach to studying risk in one key respect: risks are treated as partly social phenomena. In other words, risks are subject to debate and contestation between actors, they motivate changes to knowledge-control regimes, and they can be reflected in the resultant new knowledge-control regimes. It is this social process that is of primary interest; and so I do not seek to give my own position on what the risks of AI look like. The conversation about AI and risk is very diverse, with many different kinds of risks discussed. One way of carving up the space of risks is to look at potential applications of AI that could lead to harmful outcomes:

Weapons. AI has many potential military applications, including missile targeting, drones, and battlefield analysis and decision support. The concern is that AI could increase the destructive potential of military force, or remove elements of human 18 Kai-Fu Lee, ‘The Third Revolution in Warfare’, The Atlantic, 11 September 2021, https://www.theatlantic.com/technology/archive/2021/09/i-weapons-are-third-revolution-warfare/620013/. 17 Langdon Winner, The Whale and the Reactor: A Search for Limits in an Age of High Technology (Chicago, UNITED STATES: University of Chicago Press, 1986), http://ebookcentral.proquest.com/lib/oxford/detail.action?docID=557593; Hurlbut, ‘Remembering the Future’. judgement (and restraint) from the use of force. There are also indications that AI could be used for the creation of new chemical weapons.19

Surveillance and political oppression. AI is applicable to various forms of surveillance: identifying individuals (e.g. facial recognition) and making sense of what is happening in a stream of data (e.g. a video).20 AI for processing text and image data is also applicable to censorship. Therefore, there are concerns that AI systems could strengthen authoritarian regimes in their political oppression; and that biases in AI systems will lead to discriminatory policing.

Misinformation. AI could be used to create “bots” that spread misinformation online. This is because today’s AI systems are becoming capable of generating human-like text. There are also concerns about AI-generated media such as videos and images, which could also be used to spread misinformation.

Automated decision-making. AI can be used to automate decisions on behalf of companies and government authorities. AI can introduce defects into these decisions, including gender or racial biases.

Labour displacement. A wide range of potential AI applications involve the substitution of human labour, including (for example) truck driving. One risk is if such automation leads to unemployment and economic inequality.24 24 Morgan R. Frank et al., ‘Toward Understanding the Impact of Artificial Intelligence on Labor’, Proceedings of the National Academy of Sciences 116, no. 14 (2 April 2019): 6531–39, https://doi.org/10.1073/pnas.1900949116. 23 Karen Yeung, ‘Algorithmic Regulation: A Critical Interrogation.’, Regulation & Governance 12, no. 4 (December 2018): 505–23. 22 Alec Radford et al., ‘Better Language Models and Their Implications’, OpenAI Blog (blog), 24 February 2019, https://openai.com/blog/better-language-models/. 21 Samuel C. Woolley and Philip N. Howard, eds., Computational Propaganda: Political Parties, Politicians, and Political Manipulation on Social Media, Oxford Studies in Digital Politics (New York: Oxford University Press, 2018), https://doi.org/10.1093/oso/9780190931407.001.0001. 20 Toby Shevlane and Allan Dafoe, ‘AI as a Social Control Technology’ (Unpublished manuscript, 2021). 19 Fabio Urbina et al., ‘Dual Use of Artificial-Intelligence-Powered Drug Discovery’, Nature Machine Intelligence 4, no. 3 (March 2022): 189–91, https://doi.org/10.1038/s42256-022-00465-9. Another approach to carving up different AI risks is to look at the mechanism leading up to the harm. One helpful framework separates between accidents, deliberate misuse, and “structural risks”:25

Accidents. These are harms not intended by the people using the AI system. This would include cases where the AI system is biassed (e.g. on racial or gender grounds) and so produces unfair outcomes when deployed. It also includes the concern that highly capable, future AI systems will pursue goals not intended by the developers (see below).

Misuse. These are harms intended by the users, e.g. cases where AI systems are used for criminal activity, terrorism, or disinformation. The category of “misuse” will, for many people, include states using AI for unethical forms of surveillance and for warfare; although there is no clear consensus on how to draw the line between use and misuse in these cases (which involves political or moral judgements).

Structural. The structural perspective is distinctive from the other two in that it does not focus on the immediate harms from an AI system misfiring or being misused, but instead looks at the (often more long-run) impact of AI on social dynamics. This would include possible impacts of AI on: employment, nuclear stability, and the resilience of authoritarian regimes. This category also includes 27 Ben Garfinkel, ‘Is Democracy a Fad?’, The Best That Can Happen (blog), 26 February 2021, https://benmgarfinkel.wordpress.com/2021/02/26/is-democracy-a-fad/; Keir A. Lieber and Daryl G. Press, ‘The New Era of Counterforce: Technological Change and the Future of Nuclear Deterrence’, International Security 41, no. 4 (1 April 2017): 9–49, https://doi.org/10.1162/ISEC_a_00273. 26 Zwetsloot and Dafoe; Paul Christiano, ‘What Failure Looks Like’, AI Alignment Forum, 17 March 2019, https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like. 25 Remco Zwetsloot and Allan Dafoe, ‘Thinking About Risks From AI: Accidents, Misuse and Structure’, Lawfare, 11 February 2019, https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure. the concern that humans gradually lose control over the future by delegating more and more decision-making to AI systems. Finally, another way of mapping the space of risks is to look at harms already occurring today and harms which would require stronger AI capabilities than we have today. A much-discussed cluster of risks in the latter category would be catastrophic or existential risks from AGI (artificial general intelligence: an AI system that has the wide-ranging cognitive capabilities of a human, or greater). The Oxford philosopher Nick Bostrom argued in his 2014 book, Superintelligence, that AI systems that exceed the cognitive capabilities of humans would pose an existential risk to humanity, either through seizing control of the world or through wielding weapons of mass destruction. The idea that AI could in future pose catastrophic or existential risks to humanity has some support among AI researchers: in a 2022 survey, 36% of the AI researchers surveyed agreed that it was plausible that AI could cause a catastrophe on the level of ‘all-out nuclear war’.30

The case study My case study goes from 2019 to 2021. Fortuitously, 2019 was both when I began collecting empirical data but also the natural starting-point to the story: I am not aware of any previous attempts to reconfigure AI research KCRs around risk. My empirical material can be broken down into three episodes, i.e. three attempts to change how AI research outputs are shared. My thesis has a chapter devoted to each. 30 Julian Michael et al., ‘What Do NLP Researchers Believe? Results of the NLP Community Metasurvey’ (arXiv, 26 August 2022), https://doi.org/10.48550/arXiv.2208.12852. 29 Nick Bostrom, Superintelligence : Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014); See also Russell, Human Compatible. 28 See Stuart Russell, Human Compatible: AI and the Problem of Control, 1st edition (London: Allen Lane, 2019). First, there was the “staged release” of GPT-2, announced in February 2019. GPT-2 is the name of a large, pretrained language model. This means the model is fed a very large quantity of text data (scraped from the internet), but with certain words hidden, and the training task is to predict the missing words. In order to succeed in this task, the model must learn generally applicable skills and knowledge: initially, this involves basic things like knowledge of syntax, but to squeeze out further performance, the model must learn something about how the world works, and about the underlying concepts expressed in language. After that pretraining process, the model can then be adapted to perform a range of “downstream” tasks, such as answering reading comprehension questions or classifying text into categories. GPT-2 was also well-suited to generating text, because, given an initial chunk of text, the user could repeatedly sample from the model’s predictions about what the next word would be. GPT-2 was surprisingly capable of generating coherent stories and articles. Members of the research lab that made the model, OpenAI, were concerned that it could be misused to flood the internet with fake text. They decided to only open source a small version of the model, which was less capable. The plan was to gradually release larger versions of the model, while watching to see if people were misusing the models that had been released. This was partially successful, and a few other research groups followed OpenAI’s lead, withholding similar models. However, at the same time, the effort was undermined by a succession of GPT-2-like models that other research groups released. The second episode concerns GPT-3, the successor to GPT-2, which was much larger and more powerful. The GPT-3 paper came out in May 2020. By this time, OpenAI had transitioned from being a non-profit to a for-profit company. GPT-3 became the company’s 31 Alec Radford et al., ‘Language Models Are Unsupervised Multitask Learners’, OpenAI Blog, 2019, https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. first product: users could interact with the model through a web interface or an application programming interface (API). This was an attempt to make money from the model, but simultaneously an opportunity to experiment with a new governance regime. OpenAI set rules on how the model could be used. For example, GPT-3 cannot be used for targeted political messaging. OpenAI set up a system for monitoring how people are using the model, allowing the company to step in and prevent misuse. This was not common for commercial APIs in AI, and not possible under the standard scientific practice for sharing models, i.e. uploading open source models. Through this new regime, the lab took on responsibility for shaping how GPT-3 would be used in the world. AI researchers are ordinarily quite ignorant of how their work gets used in the real world, often viewing this as falling outside of their purview. For technical reasons, GPT-3 was harder to replicate than GPT-2, and at the time of writing there exists no similar open source model. A few other companies do offer API access to similar models, though likely without the same level of resources devoted to monitoring for misuse. Finally, I look at changes to the way conference papers are published. In 2020, NeurIPS — the most prestigious deep learning conference — began reviewing papers on ethical grounds. They also required that authors submit a “broader impact” statement at the end of their papers, discussing the possible societal consequences of the work. For a conference of nearly 2000 accepted papers, 13 were flagged for ethical review, and four of those were rejected on ethical grounds. As with the staged release of GPT-2, the initiative was met with some protestation from within the research community, especially the broader impact statement requirement. Researchers argued that the vast majority of papers had no ethical or societal significance – or at least, nothing that was at all unique to specific papers. The broader impact statements were generally low quality, and the 2021 conference organisers watered down the requirement. Throughout the project, my analysis is focussed on the level of the field. Although two of my substantive chapters begin with the actions of a single AI lab (OpenAI), my project is not an organisational analysis, and I do not devote much attention to the internal decision-making or politics of that lab. Not only did I lack unfettered access to the within-organisational dynamics, but more importantly, a single lab cannot make important changes to the field’s KCRs by acting alone. In both those chapters, the most interesting thing was how the experimental KCRs attempted to reorganise, and depended upon, the wider research field. It is difficult to define, in a clean way, the boundaries of the field that I am studying. The conference publication regime helps with this task: one starting point would be ‘the English speaking researchers who regularly publish papers at the NeurIPS conference’. However, in many cases this definition will be artificially narrow, leaving out, for example: (a) researchers who do similar work but prefer to publish at other conferences or do not speak English; and (b) people working in non-research roles within important AI labs, and cloud computing providers, who help to shape the events in my case study. Also, the boundaries of the relevant field are partly endogenous to the events I am studying – see, for example, my discussion in chapter 2 about the range of actors that have the ability to replicate GPT-2. I prefer to expand my view of the field to accommodate these various actors, even if 32 See Paul J. DiMaggio and Walter W. Powell, ‘The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields’, American Sociological Review 48, no. 2 (1983): 147–60, https://doi.org/10.2307/2095101; Paul DiMaggio and Walter W Powell, The New Institutionalism in Organizational Analysis (Chicago: University of Chicago Press, 1991); Huseyin Leblebici et al., ‘Institutional Change and the Transformation of Interorganizational Fields: An Organizational History of the US Radio Broadcasting Industry’, Administrative Science Quarterly, 1991, 333–63; W. Richard Scott, Institutions and Organizations, vol. 2 (Sage Thousand Oaks, CA, 1995); Neil Fligstein and Doug McAdam, A Theory of Fields (New York: Oxford University Press, 2012). Fligstein and McAdam define ‘strategic action fields’ as ‘mesolevel social orders, as the basic structural building block of modern political/organizational life in the economy, civil society, and the state’. practically speaking it is difficult for me to fully capture all the different actors within the field. The research field has a mix of university-based academic researchers and industry labs. While top conferences still have a higher number of papers from academia, some of the biggest contributors come from industry. Google AI Research, for example, publishes more papers at top AI conferences than any university group, with 178 papers accepted to NeurIPS 2020 (Microsoft was also at 95 – slightly less than Stanford and MIT).33 Industry labs also train an especially high proportion of the largest models (i.e. models with a high number of parameters), which requires greater computational resources and engineering effort than is normally within reach for academic researchers. These industry labs have been surprisingly academic in their culture and behaviour, with researchers focussing heavily on contributing to the scientific literature, often free from the pressure to work on commercial products (although this varies from place to place). I do not have a strong answer for what motivates the large technology companies to house and fund these labs, but some possibilities are: (1) the labs offer prestige and marketing for the companies as leaders in AI; (2) it is helpful to have leading in-house AI experts who can be called upon for more product-led AI projects; (3) sometimes the models trained by researchers will be used in production; (4) the companies are waiting for a later moment when AI becomes more commercially valuable, at which point they will move to a less academic approach; and/or

the leaders of the technology companies are genuinely interested in advancing AI. It is therefore difficult to draw neat distinctions between different categories of actor, although I 34 Deep Ganguli et al., ‘Predictability and Surprise in Large Generative Models’, in 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, 1747–64, https://doi.org/10.1145/3531146.3533229. 33 See Sergei Ivanov, ‘NeurIPS 2020. Comprehensive Analysis of Authors, Organizations, and Countries.’, Criteo R&D Blog (blog), 15 October 2020, https://medium.com/criteo-engineering/neurips-2020-comprehensive-analysis-of-authors-organizations-and-c ountries-a1b55a08132e. do highlight where there are important differences between industry and academic labs, especially where these differences are relevant to governance. In building the case study, my primary data source was interviews. I interviewed both people designing and running the new KCRs (e.g. members of OpenAI), and with the AI researchers who reacted to, and interacted with, these new KCRs. I conducted 42 interviews with 37 different people between 2019 and 2021, most of which lasted around an hour. My interviewees have been a mix of university-based researchers and researchers based at industry AI labs. My interviewees are not a random sample of the population of AI researchers. Some of them I met organically at conferences: in 2019 I attended ICML, ICLR and NeurIPS, and in 2020, I attended AAAI. Often I would go through conference proceedings (again, NeurIPS, ICLR and ICML) and take authors’ email addresses from their papers, sending out bulk emails. The vast majority did not get back to me; those that did were probably disproportionately interested in AI risk or issues around publication. Indeed, a few researchers have declined my interview requests by saying that they do not think about my topics at all and just prefer to get on with their research. I did not take everything my interviewees told me at face value. I felt a number of them probably saw me as an outsider, almost like a journalist, and therefore naturally adopted a defensive position (which normally meant defending the existing publication regime). Nonetheless, this still made for useful data. I have given most of my interviewees pseudonyms, even though many of them consented to be named. I have tried to find gender-neutral pseudonyms to help protect the identities of those who wanted to remain anonymous. 35 In chapter 1, I highlight the bureaucratic environment of industry labs as conductive to more top-down rules on what artefacts should not be made openly available. In chapter 2 I highlight how the large compute resources of certain industry labs affords them a kind of governance power. I also collected thousands of tweets during 2019-2021. I tried to follow as much of the Twitter-active, English-speaking AI research community as possible; in 2019, I made a social network graph of this community, which allowed me to find members who I was not following (i.e. AI researchers who were followed by many other AI researchers). With both my interview and Twitter data, I have targeted AI researchers, i.e. members of the technical, scientific community developing and studying novel forms of AI, especially deep learning. This community is split between universities and industry labs (above), which is reflected in my sample. I have not been studying: software developers who use AI inside their companies; humanities and social science academics who work on AI; or policy-makers interested in AI. I have interviewed people with social science or humanities backgrounds who are now closely integrated into the AI research community — for example, members of the OpenAI Policy team, or staff at the Partnership on AI (a non-profit that features in my case study). I have focussed on the scientific community because this is where the KCRs governing scientific research outputs are made and applied. Also, most of the world’s novel, advanced AI techniques and systems come from this community, and so it is an important source of AI risk. An alternative focus for my thesis could have been policymakers grappling with AI: for example, those contributing to the EU AI Act. There are three reasons why I did not cover this. (1) Considerations of focus and time: focussing specifically on the AI research community has allowed me to go into greater depth on that world. (2) During the time period I studied (2019-2021), policymakers were not openly contemplating controls on the proliferation of AI. For example, the EU AI Act, in its current form, focuses primarily on the provision of AI services, rather than the underlying proliferation of AI capabilities. My research focuses on this more upstream issue of proliferation, which is still an important site for governance, not least because the nature and magnitude of AI proliferation sets the stage for the downstream use of AI (and how easily those uses can be governed). (3) Relatedly, it comes down to an interesting question of power and influence. The AI research community has the de facto power to decide how much AI proliferation occurs, because it is they who produce the insights and artefacts, and there do not exist go-to policy levers for governments to modulate this. During the “crypto wars”, the US government struggled to effectively regulate (using export controls) the proliferation of cryptographic research from the 1970s onwards, and this became a freedom of speech issue. Government efforts to regulate the dissemination of AI research could be met with a similar backlash, or framed as an attack on scientific progress. Similarly, in one of my interviews with an employee of the Partnership of AI interested in the risks of AI proliferation, a major theme was their lack of power over what AI researchers choose to do, and their unwillingness to adopt a strong position before AI researchers had been “brought along”. (As a different interviewee put it: “Leverage in this space is being attached to a big computer.”) As additional sources of data, I rely upon publicly available online sources, such as company blogs, personal blogs, reports, job adverts, research papers, API documentation, application forms for accessing models, discussions on forums like Reddit and Hacker News, podcast interviews, talks, and conference web pages. With all my data sources, I have qualitatively coded the data into thematic categories using Roam Research. These categories are relevant across all three of the experimental KCRs that I examine in the case study. I developed the categories continuously throughout the research, going between my data and my research question, and adjusting the categories as I collected more data. The high-level categories are now reflected in the six central themes 36 See Barney G. Glaser and Anselm L. Strauss, The Discovery of Grounded Theory : Strategies for Qualitative Research, 1967; Matthew B Miles and A. M Huberman, Qualitative Data Analysis : A Sourcebook of New Methods (Beverly Hills: Sage, 1984). that I introduce in section 4, below. For example, one category was ‘large models and the relationship between industry and academia’. I also had more specific categories that were useful for organising my data, such as ‘GPT-2 decision-making’ and ‘GPT-2 replications’. I have tried to resist my project becoming a study in how AI researchers talk about knowledge-control regimes and AI risks, despite collecting a wealth of such data. My study is more institutionalist in nature: I use the qualitative data to gain a deeper understanding of the new KCRs and how they rubbed against the existing field. This follows Hilgartner, whose study was likewise quietly informed by institutional theory. This is a natural position given that knowledge-control regimes are institutional in nature: they are persisting social structures that can be maintained or disrupted by social actors. In fact, the case study could be alternatively framed as a study in institutional change, in that the relevant KCRs have undergone a process of being problematised and adapted in light of potential risks from AI. (Nonetheless, the aim of the thesis is not to contribute to the literature on institutional change, at least when pitched at such a high level of abstraction; see section 7 for the intended contributions.) Despite studying three different episodes, I view the thesis as a single case study project. The different episodes overlap in terms of the people involved and the time period, and (at least with GPT-2 and GPT-3) lessons from one KCR endogenously inform the next. Nonetheless, I still draw comparisons between these three different KCR experiments. My aim with such comparison, which I will use frequently throughout the dissertation, is to add depth to my description of the new KCRs, bringing out aspects that I might otherwise take 38 See Charlene Zietsma and Thomas B. Lawrence, ‘Institutional Work in the Transformation of an Organizational Field: The Interplay of Boundary Work and Practice Work’, Administrative Science Quarterly 55, no. 2 (1 June 2010): 189–221, https://doi.org/10.2189/asqu.2010.55.2.189. 37 Similar to Hilgartner, I adopt new institutionalist theory as a background theoretical paradigm. This literature brings a focus on the ‘field’ as a level of analysis, and institutions and institutional change as key objects of interest. See DiMaggio and Powell, The New Institutionalism in Organizational Analysis. for granted. Also, there are commonalities between the different KCRs (at least in the challenges they must address), and this gives me some added confidence in the robustness of my arguments, even though I do not claim that my findings will necessarily generalise to other fields of science and technology.

What is special about AI? Focussing on AI as a single case study allows for deeper exploration, but comes at the cost of missing potentially interesting connections with historical cases. In this section, I briefly touch upon some relevant parallels to other fields of science and technology. This is not intended to substitute for the benefits of a multiple case study design, but to help situate AI as a case study. AI is not the first time that KCRs within a scientific field have had to adapt to concerns about societal risks from the technology being produced. I would highlight a few other examples: nuclear science in the 1930s and 1940s; computer security (in particular, the disclosure of software vulnerabilities) in the late 1980s through to the 2000s; and gain-of-function virology research in the early 2010s. All these examples, alongside AI, involve risks from sharing research knowledge and artefacts, and “upstream” (i.e. addressed to the researchers) norms or interventions in an attempt to govern these risks. In comparing them to AI, we can look at both (a) the nature of the field and the actors involved, and (b) the nature of the technology and its risks. The field and actors. All these examples have involved academic research communities, and often with private sector actors also doing research in the area. What stands out about AI is the absence of state involvement in publication decisions, which we can see in each of the other examples. For nuclear research, there was a fast transition, at least in the US, from an academic field to total state involvement and control over the research via the Manhattan Project. The US Department of Defense thereby imposed a very strict regime of secrecy on nuclear research. The geopolitical context for such state intervention in nuclear research was World War II. In virology, the geopolitical context (in the 2000s and 2010s) was the post-9/11 “war on terror”. In the US, a 2004 report from the National Academy of Sciences led to the creation of the National Science Advisory Board for Biosecurity (NSABB), a federal-level panel of experts reporting into the US government. (The 2004 report was titled: “Biotechnology Research in an Age of Terrorism”.) The NSABB then played a central role in a 2011 controversy over a couple of papers describing how to increase the transmissibility (between mammals) of H5N1 influenza. The NSABB recommended against publication of these papers, and the US government supported this decision. The issue was even discussed between states at the international level, at the WHO and the Australia Group. In the computer security case, too, the US government has played some role in governing how software vulnerabilities are published. In 1988, there was the spread of the first computer worm (the “Morris worm”), created by a graduate student, which caused much disruption on the early internet. When researchers at UC Berkeley recovered the code for the worm, the US Department of Defense reportedly encouraged them not to publish this code. Then DARPA, a federal defence funding agency, funded the creation of a new centre for coordinating the defence against computer viruses. The geopolitical context for AI is currently the tensions between the US and a rising China. This has not yet led the US or other Western governments to intervene in the AI research 43 E. H. Spafford, ‘Crisis and Aftermath’, Communications of the ACM 32, no. 6 (1 June 1989): 678–87, https://doi.org/10.1145/63526.63527. 42 Katie Hafner and John Markoff, CYBERPUNK: Outlaws and Hackers on the Computer Frontier, Revised, Updated edition (New York: Simon & Schuster, 1995). 41 Jade Leung, ‘Who Will Govern Artificial Intelligence? Learning from the History of Strategic Politics in Emerging Technologies’ (PhD Thesis, University of Oxford, 2019). 40 Ian Sample, ‘Bird Flu: How Two Mutant Strains Led to an International Controversy’, The Guardian, 28 March 2012, sec. World news, https://www.theguardian.com/world/2012/mar/28/bird-flu-mutant-strains. 39 Alex Wellerstein, Restricted Data (University of Chicago Press, 2021). field to regulate publication, although it is possible that this may occur in future (the AI case study is far from closed). With AI, economic and strategic competition between states has so far played out in different fora. We have seen national AI strategies; state sponsorship of AI research projects;44 and there has been some discussion of export controls on AI technologies, which has not yet led to controls on AI research publications. The EU’s proposed AI Act does not focus on AI research (see chapter 2). All this means that the case study in this thesis is notably isolated from state involvement, relying much more heavily on private actors. With time, we could learn that this was a function of the case study covering the early stages of AI’s rise as a risky (and strategically important) technology. The technology and its risks. In all the examples discussed, there has been some disagreement or uncertainty over the nature and magnitude of the risks, although perhaps none more so than AI. With the nuclear example, some people had impressive foresight about the nature of the risk. In 1914, nearly three decades before the Manhattan project, H.G. Wells wrote The World Set Free, a science fiction novel in which atomic bombs are invented, leading to a devastating nuclear war. The physicist Leo Szilard read the book in 1932, a year before he had the idea for a neutron chain reaction. Szilard foresaw the destructive potential of his discovery, especially given the rise of Nazism in Germany, and encouraged other nuclear scientists against publishing on the chain reaction. The main disagreement between scientists during this period was about the feasibility of creating nuclear weapons, with some thinking it was impossible or many years away. 46 Richard Rhodes, The Making of the Atomic Bomb (New York: Simon & Schuster, 1986), 24. 45 Jade Leung, Sophie-Charlotte Fischer, and Allan Dafoe, ‘Export Controls in the Age of AI’, War on the Rocks, 28 August 2019, https://warontherocks.com/2019/08/export-controls-in-the-age-of-ai/. 44 For example, the French government has provided the computational resources for BigScience’s effort to replicate GPT-3: see chapter 2. In the H5N1 controversy in 2011, the main disagreement was over the risks and benefits of sharing the research, rather than (for example) whether the virus was dangerous or possible to produce. Some scientists argued that it was important to publish the research so that researchers could better understand the virus and what mutations to watch for in the wild; and that this benefit outweighed the risk of bioterrorism. Likewise in computer security, the main focus of the debate was on the risks and benefits of publication. Many computer security experts argued that open publication of software vulnerabilities incentivised software-makers to patch the software more quickly. AI is characterised by deep ambiguity about the nature of the technology and its risks. For example, there is variation across different expert communities in how we should think about AI. Some experts view AI as a (sometimes defective) tool that human decision-makers put too much faith in, leading to unfair and non-transparent decisions – and question whether AI is a form of “intelligence”. Other experts view AI in terms of a set of capabilities that are increasing and becoming more general over time, trending towards human-level and then super-human level intelligence. Even among experts with the latter view, where many worry about existential risks to humanity’s future,49 there is a diversity of opinion over how these risks come about (what technical failures might occur; the role of human misuse; whether the risk would be sudden or gradual; and how close in time we are from dangerous AI being developed). These risks are often pinned to the future technology of “artificial general intelligence” (AGI), but it is difficult to define or precisely envision what AGI will look like. As we shall see in the case study, it is also not settled which kinds of AI insights or artefacts are most risky, whether from the perspective of AGI risks or present-day risks. Finally, AI is a very general-purpose technology, with a very wide range 49 See Toby Ord, The Precipice : Existential Risk and the Future of Humanity (London: Bloomsbury, 2020). 48 Bruce Schneier, ‘Crypto-Gram: November 15, 2001’, Schneier on Security (blog), 15 November 2001, https://www.schneier.com/crypto-gram/archives/2001/1115.html. 47 Sample, ‘Bird Flu’. of potential applications. This adds to the difficulty of thinking about what the impact of AI will be in society. Overall, then, AI is an interesting case study of an emerging technology, where the roles of different actors have not yet been settled – but where the state is not yet heavily involved – and where the risks of the technology are not settled either. This is the context for the development of new, risk-based KCRs, where private actors are forced to rely on their own powers of institutional entrepreneurship, in a context of deep ambiguity about what the relevant risks are and how they should best be tackled.

Introducing the central topics of analysis Here, I introduce six central topics for the dissertation. The first five are all important dimensions along which the different KCRs will be analysed. The sixth topic, in contrast, is about understanding the forces that shape the KCR development process. Everything here will be further substantiated in the rest of the dissertation. First, what source of power does the new KCR rely upon? For GPT-2 and GPT-3, an important factor was that, for a limited time period, OpenAI was the only actor that had those models. The company could therefore pick and choose what capabilities to make available and when. Replication by other research groups threatened to undermine the temporary control over proliferation that OpenAI had established, although this threat was moderated by the size of the models. Especially in the case of GPT-3, there was a limited number of research groups who had the ability to train such a large model, including paying the cost of compute (estimated to be around $5 million).50 The scientific trend towards “bigger is better” creates new opportunities for controlling AI research models. 50 Chuan Li, ‘OpenAI’s GPT-3 Language Model: A Technical Overview’, Lambda Blog, 3 June 2020, https://lambdalabs.com/blog/demystifying-gpt-3/. In addition to establishing a bottleneck, the KCR governing GPT-3 also relies upon strong surveillance capabilities. In March 2021, GPT-3 was generating 4.5 billion words per day. Effective monitoring of such activity can only be possible using natural language processing AI systems – it is likely that language models are being used to monitor the use of language models. The KCR therefore relies upon a technical (as well as social) machinery of governance, at a level that has only recently become technically possible. Finally, ethical review of conference publications relies upon an existing bottleneck in the conference publication regime, namely that papers must be approved by reviewers before being published by the conference. Second, how does the KCR construct a pipeline for expert knowledge about risk to feed into decision-making? KCRs normally include decision-making processes – for example, peer reviewers assess a paper for scientific knowledge, or a patent office assesses the novelty of an invention. In my case study, decision-making processes are constructed within the new KCRs, and these serve as a means for controlling risk. What do these decision-making processes look like — who are the experts, and how are they built into the KCR? In the case of ethical review of conference papers, if a paper is flagged by reviewers as raising ethical issues, dedicated “ethics reviewers” are called in to make a judgement. These reviewers are drawn from the broader fields of AI governance and AI ethics. In the case of the GPT-3 API, OpenAI’s in-house policy team crafts the rules on what constitutes safe and unsafe uses of the model. Partly, this was informed by early, in-house experiments with GPT-3: for example, can humans distinguish GPT-3-generated news articles from real ones? Then, on an ongoing basis, OpenAI’s policy team sought to understand misuse of the model, both by (a) encouraging external researchers to do further studies on the capabilities 51 OpenAI and Ashley Pilipiszyn, ‘GPT-3 Powers the Next Generation of Apps’, OpenAI Blog, 25 March 2021, https://openai.com/blog/gpt-3-apps/. and limitations of the model via the API, but also (b) monitoring how users are interacting with the model. They can ask, for example: are many users generating political content, and in what context? The API helps to produce knowledge about misuse, and this knowledge then feeds directly back into the rules governing use of the API, which the company updates over time. A lawyer might say that the API evolves as a “living tree”; in the same way, a cybernetician might say that the API sets up feedback loops between the governance of the model and the outside world. As the case study developed, I realised that it was more about production of knowledge (about models and their risks) than it was about the suppression of knowledge. At the outset, I expected this to be a project about secrecy, and I read literature on, for example, the organisation of secrecy at Bletchley Park during World War II. To some extent, this was right: most obviously, the KCRs used for GPT-2 and GPT-3 both required that the weights of the models be withheld, and this could be considered a form of secrecy. And more generally, the industry labs regularly use non-disclosure agreements to protect the secrecy of ongoing research – even if the same research will often be published openly in papers soon after. Nonetheless, the bigger story was the endless drive to produce more knowledge. Actors were asking themselves questions like: “Can this model actually be misused?” “Instead of withholding the model, shouldn’t we produce another model that counteracts the misuse?” “Is the model being misused in the real world?” And “How can we inform policymakers and users about the risks of today’s AI systems?” The KCRs I have studied all seem more devoted to producing knowledge, in answer to such questions, than concealing knowledge. My explanation for this is one half cynical: secrecy is often not in the interests of researchers, and so it is systematically ignored as a tool for preventing the proliferation of dangerous technology, with researchers preferring to do what they do best (knowledge production). Nevertheless, in equal measure, it is also a function of some inherent connection between knowledge and governance. Governing new technologies does require that decision-makers are informed about risks. Third, how does the KCR position the relationship between science and society? As I have noted, the prevailing KCR landscape is built around the principle that scientists should be able to communicate with each other effectively. Forums for scientific publication, such as conference proceedings, were not originally built as channels for disseminating knowledge and technology to “the public” or governments and companies. Indeed, many early scientists were reluctant to have their work read by the masses. Roger Bacon, in his Opus Majus, written in the 13th Century, wrote: “the secrets of the sciences are not written on the skins of goats and sheep so that they may be discovered by the multitude.”53 Even today, when I ask AI researchers why scientific openness is important, they usually foreground the within-science benefits, i.e. researchers can validate and build upon each other’s work. Especially since the internet, it has become very easy for actors outside the scientific community to eavesdrop on this process, but AI researchers often seem to view this technological diffusion as a welcome bonus rather than the reason they share their work. “Society” is somewhere “out there” – an object that might get transformed by AI, or a jungle where the uses of AI will be thrashed out. It is not a force that actively shapes the way AI research is shared. In the case study, this basic premise is challenged, and actors have sought ways of putting society in the driving seat. There are two main ways in which this has occurred. We have already seen the first: feedback loops. With GPT-2, OpenAI was watching how actors would use the smaller versions of the model, and using that to inform the release schedule for the larger, more capable versions. For example, they monitored “anonymous forums 53 Roger Bacon and Robert Burke, The Opus Majus of Roger Bacon (New York: Russell & Russell, Inc., 1962), 11–12. Bacon attributes this view to Aristotle and Socrates. 52 William Eamon, ‘From the Secrets of Nature to Public Knowledge: The Origins of the Concept of Openness in Science’, Minerva 23, no. 3 (1 September 1985): 321–47, https://doi.org/10.1007/BF01096442. with a history of spreading disinformation and organising hate movements” to see if these people were planning on using GPT-2.54 With GPT-3, as I have mentioned, the API setup allowed the company to monitor misuse of the model directly. The other mechanism for bringing society into the KCR is “representational”: representatives of society are built into the decision-making process. This is analogous how, in a representative democracy, the legislature is supposed to represent the population. Of course, the link between the representation and the represented is not always very strong. One example of the representational method is how the ethical reviewers for conference papers are selected; as the organisers for one conference put it, “we wanted to ensure that potential societal impacts of work published at NAACL were considered from multiple different cultural perspectives.”55 Also, a research team that replicated and open sourced a GPT-2-like model claimed that their decision to do so had drawn upon “emerging norms and governance processes that incorporate a broad set of values from across society”56 (a tenuous claim). In other words, people, organisations, and institutional processes are constructed as representing some notion of society, and this is then built into the functioning of the KCR. This has occurred in parallel with a related drive (outside of my scope) to engineer AI models with one eye on society – for example, removing societal prejudice from datasets used to train the models, or the general effort to align AI systems with human values (a.k.a. alignment research). The difference is that, in my case study, it is KCRs that are putatively aligned with human values, rather than the models themselves. As well as making the scientific process more responsive to society, there has also been some effort to make society more responsive to the science. In the case of the NeurIPS 56 Nitish Shirish Keskar et al., ‘CTRL: A Conditional Transformer Language Model for Controllable Generation’, ArXiv:1909.05858 [Cs.CL], 2019, 12. 55 Emily M. Bender and Karën Fort, ‘NAACL Ethics Review Process Report-Back’, NAACL 2021, 20 May 2021, https://iuliaturc-google.github.io/naacl-org/naacl-2021-website/blog/ethics-review-process-report-back/. 54 Irene Solaiman et al., ‘Release Strategies and the Social Impacts of Language Models’, ArXiv:1908.09203 [Cs.CL], 2019, 5. changes, the broader impact statements partly functioned to position scientists as public educators, the idea being that policy makers and civil society actors might read the statements and learn something about the technical realities of AI risk. That said, the requirement was also imagined as a way of forcing self-reflection upon AI researchers, the idea being that if they contemplated the social impact of their work, this might lead them to produce more socially responsible research. Fourth, how does the KCR position different actors inside the scientific community? One of the things that upset many AI researchers about GPT-2 was that the research community had no special access to GPT-2 (there was an academic access programme but it was limited in scope and did not give full model access). Researchers got access to the models on the same schedule as everyone else. The KCR did not really honour the boundary between scientists and non-scientists. GPT-3 was similar: there is not one API for researchers and another for software developers, for example; although OpenAI appeared quite keen to grant scientists API access when they applied. GPT-3 and related developments have contributed to an anxiety among some university-based researchers that they are being left out of certain areas of research, partly due to the high computational costs of training large models, but partly because they do not perceive that these models are being shared in a fully “open” way. Fifth, what types of research outputs count as risky? One risk philosophy that is common in my case study is a concern that dangerous AI systems will proliferate and be used in a careless or malicious way. This is the key focus of the KCRs used for GPT-2 and GPT-3, 57 In the Stanford HAI Virtual Workshop on Foundation Models (23-24 August, 2021), one of the leaders of the initiative said: “Sadly, it’s becoming harder and harder for the community to engage. A big but often underappreciated factor driving the deep learning AI revolution is the incredibly open culture and open ecosystem. It’s the norm that researchers release their datasets and code and tutorials so others in academia and industry can learn and build on top of them, and this has led to astonishing progress in the field. But with foundation models, this openness is being eroded, with datasets and code increasingly not being released. For example, while some models like GPT-2 and T5 are public, the code for training them is not. And with GPT-3, even though a few people could get API access, the models are not available.” See: Workshop on Foundation Models: Day 1 (Stanford HAI, 2021), https://www.youtube.com/watch?v=dG628PEN1fY. which both aim to keep the models out of the hands of “bad actors”. The idea was that the models themselves were the hardest thing for bad actors to recreate, given the high compute costs required to produce the models. This was assuming that the papers, in contrast, did not contain truly novel insights. However, this focus on models has been questioned, with some risk-conscious AI researchers arguing that the GPT-3 paper was actually the risky thing. The paper, alongside other papers that OpenAI published in 2020, demonstrated to many onlookers the benefits of scale: if you throw a large amount of compute and data at a model with a very high number of parameters, you can get very impressive capabilities. Some people viewed this as dangerous in that it accelerates the field’s progress towards advanced AI, thus giving the world less time to prepare. This concern was actually discussed before the publication of these papers within OpenAI, and they even delayed publication for a number of months. However, we will see that, in the AI research field, insights can be harder to govern than artefacts; and moreover, there are stronger incentives for researchers and labs to publish papers. In fact, a general finding of the case study might be the robustness of the research paper as an institution: the paper is still going strong, having successfully deflected or absorbed concerns around risk. In the case of ethical review, the concern is not about proliferation of risky technology; often, the concern is that the paper contributes negatively to the discourse around AI – for example, authors claiming they have trained an AI system that predicts criminality from a photo of someone’s face. The mechanism is to deny such papers the prestige of conference publication. Overall, I will argue that the three new KCRs each govern different objects: the staged release of GPT-2 governed the model as information (i.e. weights that could be uploaded to online repositories); the GPT-3 API governed the model as technology (i.e. a piece of machinery with which users could have arm’s length interactions); and the NeurIPS ethical review governed papers as contributions to discourse. Moreover, I have found that actors generally start with the question: what kinds of research outputs can we actually, practically govern? They work backwards from there, identifying the most governable objects as the relevant source of risk. This contrasts with a naive theory where the process of deeming risky certain kinds of research outputs (papers, models, code) would begin with an assessment of the risks of sharing that kind of output. For example, one might claim that models are the most risky thing, because they are the easiest thing for bad actors to weaponise. Or alternatively, one could claim that sharing the code carries greater risks, because it allows bad actors greater flexibility to customise models. However, if we ask, why – in chapters 1 and 2 – do we find such an emphasis on models as a source of risk? the answer, I will argue, has a lot to do with the governability of models. Equally, in chapter 3, if we ask why papers became subject to ethical review, I would argue that the primary reason is that the conference publication system (including peer review) conveniently provides an existing bottleneck through which papers must pass. In other words, the social construction of risk is driven not only by deeper risk philosophies, but largely by the opportunities and barriers to governance (which themselves have technical and institutional roots). The new KCRs I study are all opportunistic in their approach to governance. As a mirror image, when I interview researchers who clearly do not want new, risk-focussed KCRs to enter the field, they usually focus very little on arguing that AI is not risky per se. Instead, they focus heavily on the futility of trying to govern AI research outputs. They say AI research is too easy to replicate, including large models. Or they say that it is impossible to extricate the negative applications of AI from the positive ones. Such discussions about risk must be put in context: they might look like they are arguing against the riskiness of AI, but often they are actually arguing about the structural inability of their field to take action. Finally, stepping back, I argue that the AI research field is an environment that is hostile to governance, due to its anarchic nature. My study could be usefully read alongside Thomas Gieryn’s work on how scientists use rhetoric to defend science against political interference. My empirical material contains many examples of this, especially if you expand “political interference” to include any actor, including AI researchers themselves, who wish to govern AI research. However, the governance-hostile environment is based on much more than rhetoric. As an analogy, I would draw upon James Scott’s 2009 book, The Art of Not Being Governed. Scott studies the people living in Zomia, a very large, mountainous region that crosses several Asian countries. He argues that their way of life can be explained in the context that, for thousands of years, they have been trying to avoid being integrated into states. Scott writes: ‘Virtually everything about these people's livelihoods, social organization, ideologies, and (more controversially) even their largely oral cultures, can be read as strategic positioning designed to keep the state at arm's length.’60 For example, Scott argues that rice is the perfect crop for state control: it grows above ground, so its value can be assessed by tax collectors; it matures at a specific time of year, and so the state knows when to show up and appropriate the harvest; it travels well, so the farmers can be integrated into national trade; and it requires that a large group of farmers live and work in a fixed location. Scott argues that state-fleeing populations avoid rice in favour of crops like sweet potatoes that are difficult to appropriate (for example, they grow below ground, and are heavy to transport) and do not require farmers to settle in fixed, high-density settlements. Scott argues that the people of Zomia are ‘barbarians by design’.61 We can analyse the AI research field in a similar way. This involves looking at the institutional setup of the AI research field – including its existing KCRs – and asking whether the institutions are conductive or resistant to governance. Such governance might 61 Scott, 8. 60 Scott, x. 59 James C Scott, The Art of Not Being Governed : An Anarchist History of Upland Southeast Asia, Yale Agrarian Studies (New Haven, Conn.: Yale University Press, 2009). 58 Gieryn, ‘Boundary-Work’; Thomas F Gieryn, Cultural Boundaries of Science: Credibility on the Line (Chicago: University of Chicago Press, 1999). come from states but also might be driven by other actors, or might arise organically within the field. At first glance, the conference publication regime might look like the equivalent of rice: something that makes governance easy. It is transparent, with authors listing their names on papers, and with the papers openly describing the relevant innovation; the conferences occur at a regular schedule; and there is an existing bottleneck, i.e. peer review, through which the paper must travel. These features help to explain the ease with which NeurIPS established their broader impact statement requirement and ethical review. However, upon closer inspection, the conference publication system – and the broader framework surrounding it – make governance very difficult. The transparency of conference publications and open source code might look like they help accountability, but in a way, this openness helps scientists avoid responsibility for the consequences of their work. As I noted above, the researchers deny ownership of their work, claiming that it belongs to the literature. The techniques and artefacts wind their way into many different applications, but if they are open source, the researchers can maintain complete ignorance of this. If instead the researchers sought greater ownership over their work – and, say, built it into products themselves, or sold access to the work to specific companies and governments – they would quickly run into moral and policy questions. In other words, the openness of scientific research supports researchers in the challenge of – as Hilgartner, Prainsack, and Hurlbut put it – “positioning the creation of new knowledge (the domain of science) as institutionally separate from making societal choices about how to use knowledge (the domain of markets, law, policy, and ethics).”62 This positioning is especially necessary (and tenuous) in AI research, where — in the case of models and code — the research outputs are usable, technical artefacts. 62 Hilgartner, Prainsack, and Hurlbut, ‘Ethics as Governance in Genomics and Beyond’, 837. In addition, the research process is very piecemeal: it unfolds as a very long chain of 8-page papers. As Michael Polanyi pointed out, this can be a fruitful way of coordinating scientific development across time and space63 (although one of my interviewees would disagree, arguing that the field needs more large-scale, long projects, which are not incentivised under the current system).64 But scientific efficiency aside, the piecemeal nature of the current system makes it very hard to pinpoint a particular paper as responsible for a particular AI capability. AI capabilities accumulate over time, thanks to thousands of tightly focussed contributions. (This also combines with the fact that AI capabilities are often multi-purpose, useful in many different domains and by actors with various intentions.) As an illustration, many researchers criticised the NeurIPS broader impact statement requirement by arguing either that (a) much AI research is too theoretical or abstract in nature to be linked to specific, real world effects; or (b) papers only have “broader impact” when aggregated into large sub-fields, or even the whole AI field, and so the different statements will be very repetitive. I believe these criticisms point to genuine weaknesses of the initiative, rather than being purely a rhetorical strategy to avoid governance. The crucial point is that these weaknesses come from the underlying foundations of the KCR: it is built as a layer on top of the existing publication regime. The researchers, therefore, do not need to rhetorically invent their own lack of accountability (pace Gieryn) — the existing institutional framework has already done the work for them. Another source of anarchy is that the ability to develop a given AI system is distributed across a wide range of research groups who generally hold no obligations to one another. This becomes very important when one research group wants to prevent others from not releasing a particular type of AI system – a recurring issue in my case study. A few 65 See Nick Bostrom, Thomas Douglas, and Anders Sandberg, ‘The Unilateralist’s Curse and the Case for a Principle of Conformity’, Social Epistemology 30, no. 4 (3 July 2016): 350–71, https://doi.org/10.1080/02691728.2015.1108373. 64 See Chapter 1. 63 Polanyi, ‘The Republic of Science’. structural factors contribute to the fragmented and disorganised nature of the research field.

Nearly all the knowledge needed to build the relevant AI system will already be publicly available. This is thanks to the piecemeal nature of the current publication regime (and its high degree of openness). (2) Researchers are quite thinly spread across a wide range of universities and companies. Again, the existing publication regime contributes to this situation, allowing researchers to passively collaborate and learn from one another even when they are geographically and organisationally disparate. (3) Researchers do not sit within overbearing, hierarchical organisations that could coerce them into not pursuing or publishing some line of research. This is most obviously the case in university settings, and less true in corporate labs; although these labs are generally quite academic environments. The field does not have many strong, local power structures that can be knitted together into a field-wide governance regime. In my case study, all these different factors combine to make the anti-proliferation attempts (as with GPT-2 and GPT-3) difficult to maintain.

Contribution to literature As well as contributing to longstanding discussions within the field of science and technology studies (below), my dissertation contributes to the growing literature on the governance of AI.66 The AI governance literature could benefit from more in-depth, qualitative research on contemporary attempts to govern AI. Such research can help us to better understand the AI field and the social dynamics that operate there. The present time is ripe for this kind of research, now that new AI capabilities are coming online and various actors are considering how the risks of these capabilities should be addressed. My research was fortuitously timed, given that the staged release of GPT-2 (see chapter 1) occurred during the first year of my DPhil studies. Looking back, this episode symbolised the start of 66 Allan Dafoe, ‘AI Governance: A Research Agenda’, July 2017, https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf. a new era in AI research: the rise of large language models, trained by industry labs with large computing resources, and in parallel the concerns over the risks of proliferating these models. This era in AI research is still ongoing, and when trying to understand the latest AI developments and how they might be governed, it will be useful to draw upon the reservoir of experience and insight that comes from a detailed, three-year case study. In addition, my thesis raises a normative question for AI governance: is academic science the right institutional home for the development of advanced AI? I will return to this question at the conclusion of the dissertation. I argue that the current academic system, where stronger AI capabilities are developed and shared paper by paper, provides a very insecure footing for AI governance. The current system precludes meaningful control over the AI development and deployment, leaving governance efforts with no foothold. This could become a serious problem if the risks from AI – and even AGI – are as grave as some fear. I do not offer concrete solutions to this structural problem, although I highlight some of the possible strengths of corporations as focal points for governance. There are many who are naturally and healthily sceptical of corporations playing this role, including many working on open source AI. One contribution of my thesis could be to highlight that the status quo regime – which revolves around the academic paper at its centre – has its own problems too, and greater adherence to the ideal of academic openness is far from a cure-all for AI governance. The dissertation also contributes to the STS literature, by building links between Hilgartner’s work on KCRs and the broader literature on the governance of science and technology. I especially have in mind three related strands of literature. First, there is the work of Sheila Jasanoff and others on the role of scientific knowledge in making regulation67 and (in reverse) the role of regulation in shaping how scientific knowledge is produced. In other words, at a high level, Jasanoff studies the two-way relationship between knowledge and governance. Hilgartner’s work on KCRs in genomics speaks to this literature, because he studies the relationship between knowledge (namely, scientific research knowledge) and governance (namely, KCRs). By adding a focus on risk, as I do, we can give the study of KCRs an even stronger link back to Jasanoff’s work. The KCRs I study shape, and are shaped by, the ongoing production of knowledge about AI risks. This gives a new dimension to the operation of KCRs, which can play a central role in commissioning and channelling risk knowledge. The thesis therefore contributes to the Jasanoffian project of understanding the relationships between science, risk, and regulation and governance. Second, there is the literature on the social construction of risks from science and technology;70 and third, as I have mentioned, there is Thomas Gieryn’s work on rhetorical attempts by scientists to draw a boundary between scientific and non-scientific arenas. I 70 Mary Douglas and Aaron B Wildavsky, Risk and Culture: An Essay on the Selection of Technological and Environmental Dangers, Ebook Central (Berkeley: University of California Press, 1982); Stephen Hilgartner, ‘The Social Construction of Risk Objects: Or, How to Pry Open Networks of Risk’, in Organizations, Uncertainties, and Risk, ed. James F. Short and Lee Clarke (Boulder, CO: Westview Press, 1992), 39–53; Andrew Lakoff, ‘Preparing for the Next Emergency’, Public Culture 19, no. 2 (1 May 2007): 247–71, https://doi.org/10.1215/08992363-2006-035; Peter Andrée, Genetically Modified Diplomacy : The Global Politics of Agricultural Biotechnology and the Environment (Vancouver, Canada: UBC Press, 2007), http://ebookcentral.proquest.com/lib/oxford/detail.action?docID=3412552; Steve Maguire and Cynthia Hardy, ‘Riskwork: Three Scenarios from a Study of Industrial Chemicals in Canada’, in Riskwork (Oxford: Oxford University Press, 2016), https://doi.org/10.1093/acprof:oso/9780198753223.003.0007. 69 On this high-level theme, see also Foucault’s later works: Foucault. Michel Foucault, Michel Senellart, and Graham Burchell, Security, Territory, Population : Lectures at the Collège de France, 1977-78 (Basingstoke: Palgrave Macmillan, 2007); Michel Foucault, The History of Sexuality, Volume I : The Will to Knowledge (1976; repr., UK: Penguin, 2020). 68 Herbert Gottweis, Governing Molecules: The Discursive Politics of Genetic Engineering in Europe and the United States, Inside Technology (Cambridge, Mass.: MIT Press, 1998); Sheila Jasanoff, States of Knowledge : The Co-Production of Science and the Social Order (New York: Routledge, 2004); Natalie Hannah Porter, ‘Ferreting Things out: Biosecurity, Pandemic Flu and the Transformation of Experimental Systems’, BioSocieties 11, no. 1 (March 2016): 22–45. 67 Sheila S. Jasanoff, ‘Contested Boundaries in Policy-Relevant Science’, Social Studies of Science 17, no. 2 (1987): 195–230; Sheila Jasanoff, Science at the Bar : Law, Science, and Technology in America (Cambridge, Mass: Harvard University Press, 1995). would argue that both these two literatures can be enriched by an institutionalist lens (even if I am not the first to attempt this). Much of the literature on risk construction focuses on how different ways of measuring risk, and different ways of thinking and talking about risk, contribute to actors’ understanding of risk. While valuable, this must be complemented by research into the role played by the institutional context. My research pushes against the idea that expert communities, relying on their toolbox of epistemic devices (such as the precautionary principle, cost-benefit analysis, or scenario-planning exercises), construct ideas about risk that are then simply plugged into governance regimes. Rather, the governance regimes I study – and the structure of the AI research field more broadly – continuously help to identify, study, and package risks, in tandem with governing those risks. As Herbert Gottweis put the point, governance ‘is never simply a reaction to a problem and is always co-productive of the problems to which it seems to react’.72 The social construction of risks from AI cannot be separated from the (shifting) institutional and technical context of AI research; and in particular I would argue that the tools we have for sharing research (conference publications; open source repositories; and now APIs) have significant and unnoticed flow-through effects on the construction of risks from AI. This is interesting because the AI research field is currently going through a period of institutional change – with research becoming much more resource-intensive, and with industry labs taking an increasingly prominent role – which provides new opportunities for governing (and therefore socially constructing) AI risks. Similarly, Gieryn’s focus is on scientists’ rhetorical boundary drawing. I would argue that the boundary between scientists and non-scientists, as it relates to AI risks, is made of 72 Gottweis, Governing Molecules, Location 1942. The quotation refers to ‘policymaking’ rather than ‘governance’ more generally. 71 See, for example: Benjamin Hurlbut, Experiments in Democracy : Human Embryo Research and the Politics of Bioethics (New York, US: Columbia University Press, 2017), http://ebookcentral.proquest.com/lib/oxford/detail.action?docID=4771915. something firmer and more practical than rhetoric. The regime of scientific publication functions exactly as Gieryn describes the role of rhetorical boundary-drawing. Gieryn says that rhetorical boundary-drawing works to protect scientific autonomy, in that scientists ‘construct a boundary between the production of scientific knowledge and its consumption by non-scientists (engineers, technicians, people in business and government)’.73 He continues: ‘The goal is immunity from blame for undesirable consequences of non-scientists’ consumption of scientific knowledge.’ The same description is apt for the scientific publication regime, which allows AI researchers to play a central role in building the technology whilst holding its societal impact as beyond their control. It should not be surprising that, of all the three newer KCRs I study, the one that gives the lab greatest ownership over the social impacts of the technology is the GPT-3 API, which is also the most radical departure from the existing publication regime. The upshot is that the institutions themselves (i.e. the KCRs) play a central role in positioning the boundary between science and society.74 74 In the terms of institutionalist theory, publication practices help to shape the boundaries of the field. See Zietsma and Lawrence, ‘Institutional Work in the Transformation of an Organizational Field: The Interplay of Boundary Work and Practice Work’. 73 Gieryn, ‘Boundary-Work’, 789. CHAPTER 1: STAGED RELEASE Abstract: This chapter examines the staged release of GPT-2, a large language model trained by OpenAI. I start by describing the background to OpenAI’s decision to release GPT-2 in a staged manner. The rest of the chapter is devoted to analysing the new, ‘staged release’ regime and how it interacted with the AI research field. The regime relied upon coordination between different AI research labs that would have been able to reproduce GPT-2 themselves, and I describe the challenges of such coordination. The ‘staged release’ regime also positioned different actors inside and outside the AI research community in a new way. I look at how the regime organised the production of knowledge about GPT-2’s risks, and attempted to build that knowledge into ongoing decisions about GPT-2’s release schedule. The ‘staged release’ regime sets up a feedback loop between the lab and the outside world: the lab monitors how the model is being used in the world, which informs how the lab shares larger versions of the model. In this way, the ‘staged release’ regime was an innovation in the relationship between science and society, making them more closely interconnected. This was a break from prevailing ideas within AI research about the proper role of the scientific research community, and I discuss four visions of the science-society relationship that conflicted with OpenAI’s ‘staged release’ regime.

Introduction In February 2019, OpenAI – an AI research lab based in San Francisco – made a blog post announcing their new model, GPT-2: Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper. This was the latest in a series of impressive results in the area of natural language processing (NLP). In 2018, OpenAI had released the original GPT model, and later that year, Google researchers had released the BERT model. These models all relied on the Transformer architecture, a recently developed method for building neural networks. They were also all trained in an “unsupervised” manner. The model is essentially trained on a game of guess the missing word, known as language modelling. The model is fed some text from the internet and must predict what the next word was. This means the researchers can use very large datasets – e.g. including all English-language Wikipedia articles – scraped from the internet, without needing to build the dataset themselves. The model is a neural network with millions of different connections. During the training process, the weights of each connection (the “parameters”) are automatically updated based on the model’s successful or unsuccessful predictions. The language modelling task is used as a “pretraining” stage, which teaches the model generally applicable knowledge and skills. A pretrained model like GPT-2 can subsequently be trained again, on a more specific task (such as text classification, using human-labelled datasets), and it will often outperform models that did not go through the initial pretraining stage. GPT-2 was a scaled-up version of the models that had gone before it. The text dataset was larger, the model itself was larger (the largest GPT-2 model had 1.5 billion parameters, compared to 117 million for GPT-1 and 345 million for BERT), and it was trained using 77 Ashish Vaswani et al., ‘Attention Is All You Need’ (arXiv:1706.03762 [cs.CL], 2017), https://arxiv.org/abs/1706.03762. 76 Jacob Devlin et al., ‘BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding’, ArXiv:1810.04805 [Cs], 24 May 2019, http://arxiv.org/abs/1810.04805. 75 Alec Radford et al., ‘Better Language Models and Their Implications’, OpenAI Blog (blog), 24 February 2019, https://openai.com/blog/better-language-models/. more computational power. GPT-2 had very impressive capabilities. It was able to perform tasks like reading comprehension even without explicitly being trained to perform such tasks. Also, the model had an unprecedented ability to generate coherent passage of text. You give the model a few lines of text and successively sample from its predictions about what words are most likely to follow. The blog post contained an impressive example of this, where GPT-2 was fed the prompt: In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. GPT-2 continued this text as follows: The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. And so on. The model has learnt not only syntax, but also is able to predict that, for example, that scientists working in the Andes might have names like “Dr. Jorge Pérez” and might be based at the University of La Paz. The generations were not always entirely coherent, and it would often take several tries to get high-quality outputs. After introducing GPT-2, the blog post announced that OpenAI was experimenting with a new approach to sharing the model: Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights. Nearly a year ago we wrote in the OpenAI Charter: “we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research,” and we see this current work as potentially representing the early beginnings of such concerns, which we expect may grow over time. This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas. In other words, the lab was experimenting with a new knowledge-control regime. Initially, this looked like it was going to be a regime of “partial release”, but the post hinted that OpenAI might return to this question. A few months later, in May 2019, OpenAI updated the blog post to clarify that the regime was one of “staged release”: Staged release involves the gradual release of a family of models over time. The purpose of our staged release of GPT-2 is to give people time to assess the 79 The post states: “We will further publicly discuss this strategy in six months.” 78 Radford et al., ‘Better Language Models and Their Implications’. properties of these models, discuss their societal implications, and evaluate the impacts of release after each stage. The release was “experimental” on two levels. On one level, the hope was that releasing smaller versions of GPT-2 would allow OpenAI to better understand the risks of these models. They could see how one of these models was used in practice, and more generally, OpenAI and the broader research community could further study the capabilities of the model. The release was also experimental in the sense that the lab was trying out a new knowledge-control regime. As the post mentioned, OpenAI staff were not confident about the risks of GPT-2. But they predicted that, with continued progress in AI capabilities, it would become increasingly important that the AI community had a well-developed process for dealing with potentially risky AI models.

The lead-up to the GPT-2 release decision

The identification problem The ‘staged release’ regime attempts to control the rate at which a new capability proliferates. This raises what we can call the identification problem: the lab must pinpoint specific capabilities of its AI system that are both risky and not already widely available. GPT-2 was an unusually good candidate for staged release because the ability to generate such coherent text was only just becoming within reach for the research community. The quality of text generation was novel, even if the fundamental methods were not. However, the requirement – inherent in the logic of staged release – to identify novel capabilities of GPT-2 comes with a couple of difficulties. First, research progress in AI research is usually very incremental. Incremental progress is a natural result of the existing, conference publication KCR. If, say, a team of 50 researchers spent three years working on a big project, the payoff would be low: each would have made a proportionally small contribution to just a single paper. JAMIE, a researcher at DeepMind, made this point well. He complained that the existing system gives so much of the credit to the first and last author on a paper. He continued: It disincentivises people to work together. Versus like if you work on Gmail, it's not like there's one day going to be a list of names and you're going to be ranked in terms of how valuable you were considered by the Gmail team on Gmail. Because if you did that, everyone is basically not valuable. Except, the first three people feel extra, extra valued, and then everyone is worthless. And that's terrible, because it's such a big piece of complex software. It's obvious that many, many people uniformly did lots of different things, and if they didn't do that then it wouldn't be as good. Now, research projects. Ideally you would have more grand vision research projects. Where that's the case, it's just hard with the academic system to pull people together to do that. . . .[E]ven like big projects where there's a lot of glory to be had. You don't end up having that many people working on them, really.91 91 See also Yoshua Bengio, ‘Time to Rethink the Publication Process in Machine Learning’, 27 February 2020, https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publication-process-in-machine-learning/. Hilgartner found the same phenomenon in the Human Genome Project, where ‘Long-term mapping projects would take too long and involve too many contributors to help early career researchers to build a distinctive scientific identity.’92 The upshot in AI research is that would-be ‘breakthroughs’ accrete over time, across many different papers. This makes it difficult to point to a specific paper or model as extending the research frontier enough to independently constitute a novel source of risk. For example, in response to the initial GPT-2 release, one AI research criticised OpenAI’s approach by tweeting: But their perplexity on wikitext-103 is 0.8 lower than previous sota. So it's dangerous now. 🙃 This is referring to a measure of how accurately the model can predict missing words from Wikipedia articles. (Lower is a better score, and “SOTA” stands for “state of the art”.) On this measure, GPT-2 was the strongest model available, but still an incremental advance. Similarly, another researcher blogged: ...what makes OpenAI’s decision puzzling is that it seems to presume that OpenAI is somehow special—that their technology is somehow different than what everyone else in the entire NLP community is doing—otherwise, what is achieved by withholding it? However, from reading the paper, it appears that this work is straight down the middle of the mainstream NLP research. 93 An interviewee working at the Partnership on AI told me: “I think one of the challenges is because progress is so incremental, and so lots of people are working on like roughly the same things just like slightly, like making slight improvements - That's one of the challenges of this idea of deciding which piece of research is the point at which you don't publish it. Like, at what point does it become risky if it's just incremental improvements? So yeah, I don't really have a good answer for this yet." 92 Stephen Hilgartner, Reordering Life : Knowledge and Control in the Genomics Revolution, Inside Technology (Cambridge, Massachusetts: The MIT Press, 2017), 108. This highlights the difficulty of cleanly identifying the properties of the model that are novel and risky. At the same time, the requirement to focus on novel capabilities skews the discussion towards a subset of the risks of a given model. OpenAI’s blog post focussed on the risks of text generation, because this was the area where other similar models (like BERT) were clearly weaker. Nonetheless, these large language models pose other risks, too – for example, they could be used for automated surveillance of any kind of text. Indeed, many of the capabilities of large language models have nothing to do with text generation, such as text classification, search, translation, and summarisation. However, withholding GPT-2 on the grounds that (for example) all these capabilities are surveillance-relevant would not have had a cleanly identifiable impact, because GPT-2 would only marginally (if at all) outperform existing models, like BERT, on these tasks. Second, if a lab draws attention to the novelness of its own work, it will very easily be read by the research community as trying to “hype up” its own research out of academic or commercial self-interest. To add to this difficulty, the AI research community is especially sensitive to industry labs taking too much scientific credit for their research. The industry labs are widely viewed as more effective at marketing their papers, much to the chagrin of university-based researchers working on similar topics. A common criticism is that this interferes with the allocation of credit for advances in AI, with the papers from industry labs getting an outsized number of citations. From the perspective of many researchers, OpenAI’s claim that GPT-2 had novel capabilities became, rather than a justification to trial the staged release regime, an attempt to take credit for that as a breakthrough. One might wonder, why would a lab try to take credit for something that is a risk to society? The explanation is that the creation of a novel, powerful capability will often be a sufficient condition for both credit and risk. It is difficult, then, for a lab to make a claim about the risks of its work without also being read as making an implied claim for credit. Again, we see that the existing publication regime provides a hostile environment for the task of identifying novel and potentially dangerous capabilities. Not only does the current system favour incremental advances, but it also encourages a preoccupation with scientific credit allocation that spills over into discussions about risk.

Coordination within the AI community

Distributing the responsibility to identify and address risks The ‘staged release’ regime sets up a feedback loop between the lab and the outside world: the lab monitors how the model is being used in the world, which informs how the lab shares larger versions of the model. An important question is: what is the role of the AI research community in this process? Many AI researchers would naturally see themselves as central to the process of identifying the risks of an AI system: the research community would collectively study the system, as part of the normal scientific process, and uncover its capabilities, limitations, and pathologies. Does the ‘staged release’ regime facilitate this, or does it decentre the AI research community from the epistemic process of evaluating the risks of a model? In this section, I will examine the tension between (on the one hand) the ‘staged release’ regime adopted for GPT-2 and (on the other hand) the existing publication regime, which gives the research community a more central role. As we saw above, CAMERON emphasised the value of ‘having a load of people in the org who are just sort of prodding your technical artefact’. This process, of different OpenAI researchers playing with GPT-2, clearly informed the lab’s internal assessment of GPT-2’s 127 Claire Leibowicz, Steven Adler, and Peter Eckersley, ‘When Is It Appropriate to Publish High-Stakes AI Research?’, Partnership on AI (blog), 2 April 2019, https://www.partnershiponai.org/when-is-it-appropriate-to-publish-high-stakes-ai-research/. potential risks. Was there any additional process for having the wider AI research community test GPT-2 and evaluate its risks? I would argue that the regime included only a limited capacity for facilitating such research, and this limited its acceptability within the scientific community. In May 2019, OpenAI announced that they were forming “partnerships” with outside researchers to study the possible social impacts of GPT-2.128 This was on OpenAI’s terms: OpenAI staff had to approve the research, and only four partner research groups were mentioned in OpenAI’s 6-month follow-up post.129 There was one research group studying each of the following four topics: (1) “human susceptibility to digital disinformation”; (2) “how GPT-2 could be misused by terrorists and extremists online”; (3) biases in GPT-2’s outputs; and (4) detection of GPT-2-written text. These researchers could access the larger versions of GPT-2 under a licence that prevented them from using it beyond the agreed project or sharing it with others.130 These external researchers were not all drawn from the technical AI research community: it included social scientists too. Therefore, the vast majority of AI researchers had to wait for OpenAI to release larger versions of GPT-2 publicly – they had the same access as anyone else in the world. Normally, “sharing” AI research means sharing it with the research community, and it just so happens that anyone with an internet connection can also then access it. In this case, the emphasis was flipped: it was a proliferation-centric regime, and the research community 130 OpenAI shared a draft contractual agreement in their 6-month follow-up post. 129 Jack Clark, Miles Brundage, and Irene Solaiman, ‘GPT-2: 6-Month Follow-Up’, OpenAI Blog (blog), 20 August 2019, https://openai.com/blog/gpt-2-6-month-follow-up/. 128 An update (May 2019) to the original GPT-2 blog post included this information: “We are currently forming research partnerships with academic institutions, non-profits, and industry labs focused on increasing societal preparedness for large language models. In particular, we are sharing the 762M and 1.5B parameter versions of GPT-2 to facilitate research on language model output detection, language model bias analysis and mitigation, and analysis of misuse potential. In addition to observing the impacts of language models in the wild, engaging in dialogue with stakeholders, and conducting in-house analysis, these research partnerships will be a key input to our decision-making on larger models. See below for details on how to get involved.” would get access whenever wider proliferation was deemed safe. On a very basic level, many researchers seemed to find this offensive – they were being deprioritised. This is evident from the social media reaction to the initial GPT-2 release, where some researchers, for example, complained that journalists were able to play with GPT-2 before the research community.131 More fundamentally, many researchers saw the issue of GPT-2’s safety as a research question falling squarely within their jurisdiction. Especially in recent years, models have become sufficiently complicated that the exploration of a model’s capabilities, limitations, and quirks cannot be comprehensively done in a single paper. Releasing the model to the research community allows for that exploration to be done collectively, in a distributed fashion. BELLAMY, a postdoc at a US university, told me: One argument in favour of more openness would be that if you don't make the model public, if you don't make it open, then understanding the model would be hindered. So people would be less clear about what the capabilities of the system are, what are potential good uses of this model? What are potential bad uses of this model? And so a better understanding of that will come about only if I can make it more widely available, so different people can try out different things to see, like what are potential risks of it. In my interview with ALEXIS (a PhD student at UC Berkeley) he also emphasised the importance of this process of collection exploration: 131 For example, one researcher tweeted ‘Let’s also stop and reflect on how reporters knew about this work longer than other scientists’, and ‘Making a big deal about withholding the model to the press before starting the conversation in academic circles rubs me the wrong way’. ALEXIS: The question I usually face is not like, have I done something that could be used for evil? It's more like, have I even done something at all? Right? [Something] that is at the frontier. And so, I guess just because of the salience of that question in my mind, I tend to be sceptical of other people's advances too. Unless I can probe it and test it myself and build on it. Is this even a problem? Is this even worth paying attention to? Is always a question in the back of my mind. Toby Shevlane: Yeah. And so then, in that case, open sourcing AI research is a way of the field collectively exploring capabilities and limitations and getting more knowledge or something like that? ALEXIS: Exactly. It just clarifies, like capabilities. Yeah, I'd put it that way. The only problem is: as you clarify capabilities you clarify if this [model] can be misused or not; and at that point, it's too late, if you've clarified that it can be. [Laughter] This is an issue that risk-based KCRs must confront. How does the KCR delegate responsibility for answering the epistemic question of the technology’s riskiness? (And how can this be squared with any concerns around proliferation?) In answering this question, the KCR must also adjudicate between the jurisdictions of different expert communities.132 As Hilgartner (2017) argues, KCRs do not just control the flow of knowledge objects. At the same time, KCRs position different actors. This issue 132 Hilgartner, Reordering Life, 7; See more generally: Andrew D. Abbott, The System of Professions: An Essay on the Division of Expert Labor (Chicago: University of Chicago Press, 1988). comes out strongly with the KCRs that I study, which all must confront epistemic questions about risk, and therefore jurisdictional questions about who should assess that risk. Many AI researchers naturally saw their own expertise as central to the question of GPT-2’s risks. In the social media reaction to GPT-2, a number of AI researchers argued that GPT-2’s outputs were too unreliable to be useful to malicious actors. The Grover project (above) is another good example, where the authors argued that models like GPT-2 are useful for defending against misuse risks, because they can be fine-tuned to detect machine-written text. On the other hand, ROBIN told me: I think it's important to distinguish between, say, technical detection questions, which AI people can help a lot with, from like, "how will this be used in East Africa to topple governments" or something like that. Like no-one in the AI community has a clue how to think about that, but there are actual disinformation experts who do. So, we're talking to a range of experts. This raises the question of whether the most important unknowns about GPT-2’s risks were social in nature, i.e. about how social actors would use the model and what knock-on effects that could have, or more technical in nature. Scholars of other scientific fields have argued that scientists have a tendency to replace complicated social questions with technical questions, in a way that cements the scientists’ control over how risks should be handled.133 In the case of GPT-2, interests of the research community would favour a distributed system for assessing GPT-2’s risks, where scientists were free to study the 133 Hurlbut studies the 1975 Asilomar Conference on Recombinant DNA. He argues that “questions of whether and how technology serves the public good are rendered subsidiary to narrowly technical assessments of the potential for harm”. He argues: “By confining problems to the lab setting and solutions to the competencies of scientists, the Asilomar guidelines in effect excluded other forms of input, expert and nonexpert alike. Given that the objects of concern were circumscribed to the laboratory, input from other experts (e.g. ecologists) and even potentially affected publics was not seen as necessary.” J. Benjamin Hurlbut, ‘Remembering the Future: Science, Law, and the Legacy of Asilomar’, in Dreamscapes of Modernity, ed. Sheila Jasanoff and Sang-Hyun Kim (University of Chicago Press, 2015), 130–31, https://doi.org/10.7208/chicago/9780226276663.003.0006. model. Ultimately, researchers want to be able to write papers about AI systems, especially those that seem cutting edge and widely talked about. This leads researchers to emphasise the technical side of GPT-2’s risks, and the possible technical solutions. This issue came to the fore with the Grover project, described above. The lead author of the Grover paper was critical of OpenAI for withholding the model from researchers, arguing that researchers needed to study GPT-2 in order to find defences against misuse. In his mind, the Grover project showcased the value that the research community could bring: their finding was that GPT models could be modified to act as effective detectors of GPT-generated fake news. After the Grover paper, this logic quickly became one of the main justifications for openness that my interviewees would rely upon. SAMWELL (a researcher at a large academic lab) disagreed. He told me in an interview in 2021: I don't lean the nuke metaphor too hard, but like, "oh, if everyone had access to early nuclear technology, we would have great defences against nukes." It's like, no, we wouldn't; no, that's not how it works, right? And I think that that's likely to be the case with AI in general. People say, "Oh, you can tell maybe whether the text was from GPT-3," it's like, well, maybe sometimes, but certainly you need a certain amount; it's probably easy to work around these defences. I think that's motivated reasoning.” As I have argued elsewhere, it is far from clear that openly sharing AI research will always make a bigger contribution to preparing defences against misuse than the contribution to the misuse itself.134 Fighting AI misuse with better AI surveillance (which is the logic behind Grover) could run into a number of problems. For example, will everything written on the internet have to be fed through a Grover discriminator model? Will websites be happy to pay for this, and will users accept it? There is also the possibility that eventually AI systems will be able to mimic human text so closely that other AI systems are unable to tell the difference. In our interview, Rowan accepted that discriminators were not necessarily the “be-all and end-all”, because we also need platforms to implement them, and we also need a public that actively wants to tell apart real and fake news. But other than this concession, Rowan (and plenty of other AI researchers I have spoken to) was keen to emphasise the power of researchers studying potentially dangerous models. Researcher access to models was seen as the absolute priority (even when the story of how this would prevent misuse seemed underdeveloped).135 If granting research access conflicts with the other priority of stemming wider proliferation – and during the GPT-2 episode the assumption was that these priorities were in tension – then the claim is that research access should win out. The Grover project is a good example of the research community’s endless drive to produce more knowledge and artefacts. Naturally, the community’s preferred answer to potentially 135 For example, I presented Rowan with a hypothetical case where we release to the public 1,000 machine guns and 1,000 bulletproof vests, and asked why this would make the world more dangerous whereas releasing the Grover generator and discriminator models would make the world safer. He replied: Yeah, I totally agree. [...] And I'm not gonna advocate giving everyone machine guns obviously. I think that maybe where that analogy breaks down to me, is that what we should do as people who are interested in being safe against these things, is we should open up machine guns to researchers. Right? Like if it was a world where everywhere like every other person has a machine gun then probably you need fundamental research in how to make better defences against it, right? The assumption is that there is something technical inside the gun, or inside the AI system, that hints at possible defences. As the example of machine guns demonstrates, this will not always be the case. 134 Toby Shevlane and Allan Dafoe, ‘The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?’, in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20 (New York, NY, USA: Association for Computing Machinery, 2020), 173–79, https://doi.org/10.1145/3375627.3375815. risky research is not to suppress research, but to make more research – that is what the field is already set up to do. OpenAI’s claim that large language models pose a risk to society was quickly turned into a research question, and one that could be answered using the community’s existing methods: building and evaluating models. As one of my other interviewees said, Grover was a smart move from an academic perspective, because it made for a good paper. The GPT-2 blog post involved existential reflection, lamenting that it would not be “possible to control research in [generating text, images, and video] without slowing down the progress of AI as a whole”. The discourse around Grover circumvents these searching questions. Instead, the solution to problems created by AI research is more AI research.136 The concerns about proliferation become relegated, because the downsides of proliferation are seen as wiped out by further research effort. Although one might be sceptical about this position, I would still argue that the GPT-2 ‘staged release’ regime was made weaker by its low capacity for delegating risk-related research to the broader community. There are three reasons for this. First, many of the researchers who replicated GPT-2 did so because they did not have access to the model themselves. This is what happened with the Grover project, for example. Rowan (the lead author) recalled that OpenAI only gave him and his colleagues very restricted access to GPT-2 through OpenAI’s “quote, unquote, partnership programme”. At least at that point in time, OpenAI was offering: not the largest model (instead, the second or third largest); not the weights of the model (just access to a “demo”, where the user interacts with the model 136 Ironically, this was the same claim that the founders of OpenAI used to justify setting up a new AI lab to rival others (like DeepMind). When OpenAI was founded in 2015, Elon Musk said in an interview: “I think the best defense against the misuse of AI is to empower as many people as possible to have AI. If everyone has AI powers, then there’s not any one person or a small set of individuals who can have AI superpower.” Steven Levy, ‘How Elon Musk and Y Combinator Plan to Stop Computers From Taking Over’, Blog, Backchannel (blog), 11 December 2015, https://medium.com/backchannel/how-elon-musk-and-y-combinator-plan-to-stop-computers-from-taking-over -17e0e27dd02a#.fto7v7a1r. at arm’s length); and without being able to change the algorithm by which GPT-2’s outputs were sampled.137 138 Rowan told me: . . . once the people at UW [who] were applying for [GPT-2 access] realised, oh, wait a second, we're not going to be able to do research on this, then we independently had these ideas about how to actually study this problem of neural disinformation and what adversaries might do, and we kind of just did everything ourselves. The graduate students from Brown who replicated and fully open sourced their GPT-2 model also originally came into the project wanting to study GPT-2, but did not get access to the model from OpenAI. They were additionally motivated by the idea of having an open version of GPT-2 against which the research community could compare the performance of subsequent models. Therefore, there is a direct connection between the limited researcher access to GPT-2 and the fact that it was replicated and open sourced by other researchers – which (as we saw above) partially undermined the ‘staged release’ regime. Second, the AI research community is undoubtedly well-placed to study the capabilities and pathologies of AI systems. Even if this is not a panacea, it will often be relevant to addressing risks from AI. In the case of GPT-2, OpenAI staff seemed to think they had a better understanding of the model’s text generation capabilities than the likely social 138 OpenAI’s final report on GPT-2 gave some detail on what level of access to GPT-2 the external researchers got. It reads: ‘When forming partnerships, we signed a non-commercial legal agreement with a partner organization to provide our model for their research use, and/or we provided a partner organization with a secure sampling interface to the larger models.’ Solaiman et al., ‘Release Strategies and the Social Impacts of Language Models’, 3. 137 On sampling: these language models do not just output the next word, but a probability distribution over many different possible next words. To use them to generate text, you sample from this distribution. OpenAI was using a very simple way of doing so: “Top-K” sampling, where you only sample from the top, say, 100 most likely next words. It is easy for a discriminator to detect text that is generated in this way, because the distribution looks very unnatural – all the words in the article are quite predictable from the previous context, with none beyond some cut-off point of unpredictability. The team wanted to use a sampling method that would be harder to detect. impacts of those capabilities. In other cases, though, the lab might be uncertain about a model’s capabilities, including – for example – situations where the model might respond in a surprising and unwanted way. There is much more research capacity for the study of models outside any single lab than inside it. Third, even where the lab's biggest unknowns are ‘social’ in nature, again, research into these questions might sometimes benefit from external social scientists having access to the model. OpenAI tried to achieve this by coordinating with a small handful of external social scientists, but again, the lab has a limited capacity for coordinating this work, compared to the vast number of social scientists out there in the world. As we will see in Chapter 2, the KCR used for GPT-3 (the successor to GPT-2) more effortlessly brought in outside researchers to study the model. In the absence of large-scale delegation, the in-house researchers at OpenAI had their work cut out for them. CAMERON told me that in April 2019 one of his colleagues in the policy team “successfully freaked me out” by pointing out that “we are nowhere near where we need to be in terms of having a substantive amount of research into this to let us make good decisions.” Getting into a more solid epistemic position to make decisions about when the larger models should be released required a lot of hard work from multiple researchers on the policy team.

The relationship between science and society The GPT-2 release positioned the relationship between science and society in a way that was unusual for the AI research field. From the GPT-2 blog post, and looking at the ‘staged release’ regime, the following vision of the science-society relationship emerges. (1) Scientific research can have a direct, negative impact upon society. Society, like an ecosystem, is vulnerable. (2) Scientists can manage the impact of their work on society, by choosing the timing and content of the research they release. (3) Scientists should warn society about upcoming risks from scientific research. (4) Scientists should monitor how their research is used in society, and use that to inform decisions about what to release. In other words, the new KCR presented science and society as interconnected. It presented the scientific community as capable of acting, as an agent, in order to steer the impact of its creations on society. It presented scientists as responsible for playing this role. This overall vision of the science-society relationship was controversial within the AI research community. In this section I will go through four visions of the science-society relationship that conflicted with OpenAI’s ‘staged release’ regime, which were used by AI researchers to attack OpenAI’s decision-making. I rely on my interview data, the social media reaction to GPT-2, and the discourse that came with Salesforce’s release of their CTRL model (which, as we saw above, undercut the staged release of GPT-2). Vision 1: Scientists as knowledge-producers, society as the beneficiary A recurring theme in my interviews was the special role of scientists as knowledge-producers. A common view goes as follows: The function of the scientific community is to produce knowledge. AI research outputs, like trained models, count as knowledge. Knowledge is usually, or even always, beneficial for society. The same discourse has been observed in other fields, including genomics.139 A good illustration 139 Hurlbut, ‘Remembering the Future’; Stephen Hilgartner, Barbara Prainsack, and J. Benjamin Hurlbut, ‘Ethics as Governance in Genomics and Beyond’, in The Handbook of Science and Technology Studies, ed. Ulrike Felt et al., Fourth, Ebook Central (Cambridge, Massachusetts: The MIT Press, 2017), 823–51. See also Gieryn’s description of a 1982 report from the NAS Panel on Scientific Communication and National Security: ‘The Report isolates a “core” of science by demarcating the production of scientific knowledge from its consumption. Selected characteristics are attributed to science in order to distinguish from technological applications: . . . the goal of science is the creation, dissemination and evaluation of knowledge as its own end, not as a means for material production; open scientific communication transmits theoretical and empirical knowledge about nature, not “know-how” or “recipes” immediately transferable to production of comes from my conversation with HEYDAN, the most senior researcher at a large industry lab, who I presented with a hypothetical scenario: Toby Shevlane: Say if you came up with a model, and trained it, and for whatever reason, you thought, this is harmful. …[T]his is actually something that's going to be net negative for the world. What do you think you would do in that situation? HEYDAN: I mean, I can't think of a scenario like that. Because even if...You know, I hate bringing up nuclear technology, because AI is nothing like nuclear technology, right? But look at Albert Einstein, or Richard Feynman, all these people who are pioneers there, they still feel that it was net good for the world,140 because knowledge is in the end, I think, a net positive aspect. . . . Toby Shevlane: Okay. I see. And so is part of the argument that technology is always beneficial on net, in some way? HEYDAN: Knowledge. Knowledge, I would say. 140 This claim about Einstein and Feynman appears incorrect. Einstein did not work on the Manhattan Project, although he did sign a letter to President Truman recommending that the US should build nuclear weapons. Einstein later described this as the “one great mistake in my life”. Feynman fell into a deep depression after Hiroshima and Nagasaki. He is quoted as saying: “Maybe from just the bomb itself and maybe for some other psychological reasons I had just lost my wife, I was really in a depressive condition.” hardware’. Thomas F. Gieryn, ‘Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists’, American Sociological Review 48, no. 6 (1983): 790, https://doi.org/10.2307/2095325. It might seem strange to describe AI systems as simply “knowledge”, with no mention of technical artefacts. One response is that the quest of AI research is to uncover the nature of intelligence; AI researchers seek understanding of intelligence by building it. As one AI researcher put it (quoting Richard Feynman), “what I cannot create, I do not understand”.141 However, even if the ability to create something is a necessary condition for understanding, it is not sufficient. AI researchers generally admit that they have little clue what is going on inside their models. As BELLAMY put it, “usually, it's just gonna be like this huge array of numbers that's not interpretable”.142 Understanding what algorithms are instantiated in this huge array of numbers is a subfield within AI research, but not a very large one.143 At the same time, these models often have many practical applications. There is a strong claim, then, that models should be classed as useable technical artefacts, even if they simultaneously carry epistemic value. A similar example comes from my interview with JESSIE, an AI researcher at Google. We were talking about the use of AI systems for tracking people, which they brought up as an example of a potentially harmful AI application. JESSIE was arguing that China is going to 143 From an analysis of the 1449 papers submitted to ICLR 2019, around 10 were deemed to be on the topic of ‘interpretability’. Sergey Ivanov, ‘ICLR 2019: Stats, Trends, and Best Papers.’, Criteo AI Lab, 5 December 2018, https://ailab.criteo.com/iclr-2019-stats-trends-and-best-papers/. 142 With deep learning, building a neural network and understanding a neural network are quite separate activities, because the features learnt by the network are not hand-crafted by the researcher, but emerge through the training process. In an influential blog post, Andrej Karpathy, an AI researcher, argues that deep learning is an example of “Software 2.0”, which is a less hands-on process for building software. He writes that in comparison to traditional software, “Software 2.0 is written in much more abstract, human unfriendly language, such as the weights of a neural network. No human is involved in writing this code because there are a lot of weights (typical networks might have millions), and coding directly in weights is kind of hard (I tried).” He continues: “Instead, our approach is to specify some goal on the behavior of a desirable program (e.g., “satisfy a dataset of input output pairs of examples”, or “win a game of Go”), write a rough skeleton of the code (i.e. a neural net architecture) that identifies a subset of program space to search, and use the computational resources at our disposal to search this space for a program that works. In the case of neural networks, we restrict the search to a continuous subset of the program space where the search process can be made (somewhat surprisingly) efficient with backpropagation and stochastic gradient descent.” Andrej Karpathy, ‘Software 2.0’, Andrej Karpathy (blog), 11 November 2017, https://karpathy.medium.com/software-2-0-a64152b37c35. 141 One prominent AI researcher said the following on a podcast: “There is this wonderful quote from Richard Feynman, which is: "What I cannot create, I do not understand". And I feel like you understand best by building, and so I wanted to switch from just talking about intelligence to trying to build it as a way to better understand it.” work on this regardless, so there’s nothing Western researchers can do – they might as well continue working in surveillance-relevant fields. They said it just comes down to how much investment a country like China puts into AI, and China is putting a lot of money into AI research. In passing, they added: And I mean, actually, maybe, if you force these governments to go down that route, that will be good, because it educates the people and education is usually good and usually destructive to totalitarianism. Again, this comes down to the question: what are AI research outputs? No doubt, certain types of knowledge could be a thorn in the side of totalitarian governments – perhaps knowledge of government cover-ups, or knowledge that refutes state propaganda. But does the category of “knowledge” include all technology, too? It is much less clear that all technology has a counter-totalitarian bias – even technologies that are directly useful for propaganda, surveillance, censorship, imprisonment, and so on. With GPT-2, part of the concern was that authoritarian governments could use this kind of model for disinformation and propaganda. And more generally, a large proportion of AI research is surveillance-relevant (including language models like GPT-2, which could be used for the surveillance of text). I have asked a few of my interviewees (all AI researchers): what proportion of AI research do you think could, even if remotely, be relevant for surveillance? These are the three answers I got: “Yeah, in principle, all of it, probably. [Laughter]” “I'm struggling to think of something that could not be used for surveillance. […] So if I had to put a number on it, probably over 50%.” “It’s part and parcel, right?” Conceiving of AI models purely as “knowledge” obscures these kinds of applications. A final example comes from Salesforce’s release of the CTRL model. Their blog post claimed that, in releasing the CTRL model, the lab aimed to “foster transparency”.144 What kind of transparency is this – what is the object of transparency? Sharing the model weights does very little to shed light on the activities or decision-making of Salesforce as a company. Ironically, the company’s decision-making process for releasing CTRL was not particularly transparent (see below). Again, we have the ontological question: what is an AI model – is it knowledge, or a piece of machinery? If we see models as machinery, then being “transparent” about the model weights is the same thing as widely proliferating the machinery. The unsettled ontology of AI models helps the lab to obscure this. The model is presented as a window through which something can be seen, instead of a tool that can be used to achieve particular effects in the world. Vision 2: Science as powerless; society as powerful in determining the technology’s impact Most of my interviewees who criticised ‘staged release’ were not content with balancing the whole argument on a fragile distinction between knowledge and technology. The conversation often shifted to the comparative advantage of scientists and governments to tackle risks. The argument is that governments are better equipped to deal with problems like misuse of AI. Often this is a question of power: governments have more power over individuals and companies who might misuse AI. And sometimes it is about expertise: AI researchers are not experts in societal issues, but – as experts in the technology – they have 144 Socher, Controllable Generation. a responsibility to educate governments and the public. For example, SAM (a PhD student at a US university) told me: I’m not sure it’s your responsibility, as the person who made it, to figure out the solution to it. I think it’s all of us, as a society, who has the responsibility. I think the best thing that we can do is recognise the risks that we create and communicate them effectively. Because we're experts in the technology; we're not experts in economics or regulation.145 FRANKIE, a US researcher, emphasised the research community’s lack of power to address misuse: What's the [solution] for models being used in ways that we don't like, whatever that way is? It's to stop the use. That's not something that the research community can possibly address. That's something that's external, like that's where the thing gets used. That's not something that researchers choose or do anything about. It's a policy problem. Similarly, NOEL is a professor at a US university and a researcher at a large tech company. They were critical of OpenAI’s decision to withhold the larger GPT-2 models. I asked if they thought that GPT-2 could be used for harm. They said that you could find a way to use it to harm someone, but that this is the case for all technologies. They said that even a pen 145 Similarly, JESSIE told me: ‘So personally, I think that the right way to deal with big societal changes, or new technologies is regulation and legislation. [...] And you can still use them but you run the risk of breaking the law. But for politicians or the legislature to like to do the right things, they need to be educated and they need information. [...] Yeah, and that's I think the biggest responsibility of people who are doing the research is to educate other people, like lawmakers about the benefits and pitfalls so that they can make the right decision.’ can be used to stab someone in the eye. I asked if some technologies are more weighted towards harm, for example, a bomb being better-suited to misuse than a pen. They said no, because the impact of the pen will depend upon societal variables such as literacy rates. They continued: So, it’s actually largely about the social agreement and the context that we need to bring into account. . . . Now, if and when these so-called AI technologies start to be deployed in society on a much wider and much deeper level, what kind of society would we want to have, so that these technologies are going to be more harmless than harmful? I asked if they thought it would be a good thing if human-level AI were developed. They clarified that “we have to always strive to exceed human intelligence”, and continued: But is it going to be good? I think so. Not because that kind of technology is inherently assistive or helpful for us. But because I believe we will be able to change or drive our society so that we can benefit from these kinds of technologies. This position is the opposite of technological determinism: it holds that social forces wholly determine the impact of a technology, regardless of the technology’s inherent properties. It is a certain kind of social determinism in which society is imagined as a coherent agent, capable of imposing its values on any technological shift.146 Interestingly, while 146 CHARLIE, a PhD student at a large North American lab, disagreed strongly with this position. He told me: “I think the view that technology is a tool and humans decide what to do with the tool is like, I don't even think it's coherent. Because I think it places agency in either individual people or humanity as a collective in a way that I don't think makes sense. I think the reality is that we're part of a very complicated socio-technological system. And that system includes a lot of complicated incentive structures, a lot of complicated organisations, composed of many humans that can be modelled as agents, but like, maybe sometimes should and sometimes shouldn't be. A lot of what happened is driven by forces that are in some sense beyond our control. Like there's nobody running the world economy, but the economic forces have this massive influence over what happens, and they create really strong incentives that we're all heavily influenced technological determinism is sometimes accused of absolving technology-makers of responsibility, in this case the reverse is true. It no longer matters what the technologists do, because all the action happens outside of their domain. This kind of social determinism is therefore a convenient way for AI researchers to disown the responsibility to address AI risks.147 In contrast to governments and social forces, the research community is presented as powerless to make a difference. ELLIOT, A PhD student at a large Canadian lab, told me: I mean, [GPT-2] could be harmful. But then again most research is potentially harmful. I mean most research we do, like . . . A lot of computer vision is like state surveillance. It’s already happening. . . . Massive state surveillance or autonomous weapons. And speaking for myself, I know that everything I do is going to be pretty harmful. . . . And why am I still doing it? . . . I am not that extraordinarily smart or important that somebody else can’t replace me. I’m not irreplaceable. If I don’t do it, someone else would. In the reaction to GPT-2, this kind of discourse was often concentrated on the issue of replication. Critics of OpenAI argued that independent replication of GPT-2 was so easy that “staged release” would be completely undermined. For example, an assistant professor 147 See also Stephen Hilgartner, ‘The Social Construction of Risk Objects: Or, How to Pry Open Networks of Risk’, in Organizations, Uncertainties, and Risk, ed. James F. Short and Lee Clarke (Boulder, CO: Westview Press, 1992), 39–53. by. [...] I'm not sure I can even like pass the ideological Turing Test or something of somebody who doesn't understand that, and I'm not sure what it would mean to really expand upon and fully articulate this view that technology is just a tool and we decided how to use it. Like, I don't actually know what people are saying; I feel like it's just constructing this very simple model that if you actually try and map it onto reality, you'll just notice that it doesn't, there's no way of doing it.” at NYU was quoted in the media: “I’m confident that a single person working alone with enough compute resources could reproduce these results within a month or two”. A professor at a large US university tweeted: “I see no natural moats for this tech, and if @OpenAI can do it, so can others around the world. The Pandora's Box is open, and we have to learn to live with #AI fake reality..”148 This view allows researchers to accept the risks of AI, and accept that AI research directly contributes to those risks. Instead, the issue becomes a very practical one, about the uncoordinated nature of the AI field. A good illustration of this discourse comes from Salesforce’s release of their CTRL model. In tandem with the CTRL paper, a subset of the same researchers uploaded a preprint about AI governance.149 The paper is written as a Science and Technology Studies (STS) paper, and the lead author minored in STS during his graduate studies. The main argument of the paper is that the researchers who train a large language model should not be the focal point for governing the model, because the social impact will depend upon dynamics external to the lab, occurring subsequent to release. The authors (Varshney et al) correctly highlight an important aspect of the challenge of governing models like GPT-2. These large, pretrained models are often adapted by subsequent users. Two important types of adaptation are: (1) fine-tuning, i.e. the model goes through a further training phase, more specific to a particular application. For 149 Lav R. Varshney, Nitish Shirish Keskar, and Richard Socher, ‘Pretrained AI Models: Performativity, Mobility, and Change’ (arXiv:1909.03290 [cs.CY], 2019). 148 I had interviewees who made the same argument. CAMPBELL, a PhD student in Europe, told me: ‘In general… you can’t really hold technological progress back...It’s the same with the atomic bomb and nuclear energy. So usually, it’s more like we’re providing tools, and what people do with these tools is very much up to them. ... Just because it’s possible to misuse certain tools does not mean we should forbid them or prevent them from being published at all.” SAM said: “I think that the solution is not going to be any kind of, like, constraint on the technology, because people will just go around that. So I don’t think we should be looking for solutions like “you can’t make facial recognition systems that are this good” or “you can’t make language generation that’s this good.'' I think that’s going to fail. I feel like to get in front of it, what we really need to do is reach out to the broader population and make them aware of how good video fakes and language fakes are. Maybe a weak analogy is nuclear weapons. We all know that they are a terrible idea but we keep producing them…’ example, GPT-2 could be fine-tuned on a reading comprehension dataset. (2) The models can be combined with other software, including other AI systems. I would add an example from 2021: OpenAI’s CLIP model, which was pretrained to predict the captions of images scraped from the internet.150 In doing so, the model learns generic skills for understanding images, and can be applied to specific tasks. Like GPT-2, CLIP was put through the staged release regime. When I asked ROBIN why the staged release regime was used for CLIP, they said: Basically the same reasons as GPT-2, which is to say that we're uncertain about the ways in which it could potentially be misused. And in fact, we've learned some things as a result of releasing it in a staged manner that we didn't necessarily know, prior. We flagged in the paper that we were concerned about things like surveillance and stuff like that, but we weren't necessarily thinking of other uses that have subsequently come more into prominence. Like using it to steer generative models was not something that was as high on our radar as it is now. . . . And that was not originally how we were thinking about it, and that in turn now is informing our thinking. . . ROBIN was referring to the fact that CLIP, which was initially thought of as a model for classifying images, has been used to generate images. CLIP is paired with a model that generates images, and iteratively steers the generated image towards some target description (such as “a painting of Barack Obama, by van Gogh”). There are popular, open source tools that make this process very easy. It is a good example of how the possible applications of a large, pretrained model will depend on how it is adapted. 150 Alec Radford et al., ‘Learning Transferable Visual Models From Natural Language Supervision’ (arXiv:2103.00020 [cs.CV], 2021). Varshney et al argue, therefore, that the producers of models are part of a broader ecosystem, and it is that ecosystem that determines how the model will be used. They connect this point to STS literature on how the users of technology are part of the process of innovation. Varshney et al turn this into a theory of responsibility on the producers of large models. The producers should be decentred, essentially because the important action happens elsewhere. There is a call to expand governance to the broader ecosystem of actors (“responsible innovation should be expanded to include the role of users”), although the paper does not explain how that should be done. If you read the paper alongside the CTRL release, the conclusion seems to be that labs do not have an obligation to refrain from sharing their work on grounds of risk. Indeed, Varshney et al also argue that scientific “self-moratoriums” do not work, because scientists “cannot be expected to step outside the momentum of their own work to self-regulate”. As such, the paper reproduces, in more academic language, the discourse we saw above about why AI research labs should not be expected to limit publication of their work. On the one hand, it is about the location at which risks from AI arise, and accordingly, where governance should be located. The claim is that AI risk arises external to the lab’s actions, in the broader community and society. Theories about the social shaping of technology are wheeled out in support – for example, Varshney et al emphasise that ‘the social world acts to fundamentally shape technical development at every level’. This is the same kind of social determinism we saw above, and it is again used in service of insulating scientific researchers from the demands of governance. And on the other hand, the Salesforce researchers are making an argument about the governance capacity of researchers. Researchers are presented as being incapable of restraining themselves. This is then used, in an arguably circular way, to justify the release of the CTRL model. The paper provides an interesting contrast with my conversations with members of the OpenAI policy team. In a way, the researchers from Salesforce and OpenAI have a similar starting point. Consider this line, for example, taken from the Varshney et al paper, but which looks like it came straight out of my transcripts from my conversations with ROBIN: Our estimates suggest that the cost to train . . . XLNet was $50,000, to train RoBERTa was $60,000, and to train GPT-2 was $250,000. On the other hand, the cost to fine-tune BERT on the SQuAD dataset is estimated to cost only $3. In the Varshney et al paper, this is evidence of how easily these large models evolve subsequent to release, thus escaping the reach of the labs that produce them. ROBIN takes this same starting point but goes in a different direction. If you spend millions of dollars training a large model and then just upload the weights for anyone to download, then you have given away your only opportunity to shape how the model is used. The high training costs narrows the number of actors that can build the model themselves, which puts the lab in a stronger position to influence how the model is deployed. It is precisely because the costs of training models is so high – relative to the costs of fine-tuning or running a model – that elevates the lab that produced the model as an important actor in the governance process. Vision 3: Society as a threat to science, requiring management of the public perception of AI From the “staged release” perspective, a key reason for delaying release is to give the world advanced warning of the risk. Hence, in the GPT-2 blog, OpenAI was not just addressing AI researchers but a broader audience. For example, they argued that governments should start efforts to ‘more systematically monitor the societal impact and discussion of AI technologies’.151 The logic is that AI misuse is not something that can be solved quietly within the AI research community, but requires engaging other actors. I asked ROBIN, given a key aim behind the GPT-2 release was to ‘start a debate’, who needed to be part of that debate – AI researchers or the public? They said that AI researchers are a ‘key part of the solution to the publishing question’, but ‘it is not clear that the AI community is actually the people that need to solve’ the overall problem of misuse. In other words, compliance from AI researchers would be a necessary but not sufficient component of the governance regime’s success. However, in raising the alarm to a general audience, OpenAI was accused of polluting a common-pool resource: the public’s perception of AI research. Onlookers presumed that OpenAI was doing this for selfish reasons: to raise corporate investment (and in 2019 OpenAI was transitioning away from being a non-profit and raising capital). The concern was that the public would perceive AI research as moving at an exciting or scary pace. The motivation for avoiding too much public excitement is often that, if AI progress does not live up to the hype, there will be a backlash against AI, leading to an “AI winter”, where funding dries up.152 The motivation for avoiding public fears is rarely spelled out so explicitly, but might also have something to do with funding.153 This means that many AI researchers are suspicious of journalists. They prefer journalists not to write alarmist news stories about AI. OpenAI had briefed a handful of journalists in 153 For example, one AI researcher, responding to the suggestion that work on AI lip-reading could be used for surveillance, complained on Twitter: “Researchers working on lipreading to help speech impaired patients will struggle to raise funds.” 152 One AI researcher tweeted: “Every time you overstate the capabilities of an AI system or the speed of AI progress, you are doing the public trust equivalent of taking on credit card debt. Which at some point the industry will have to repay…” Another AI researcher, in 2020 (i.e. not specifically about GPT-2), blogged: “To begin with, something about crying wolf. If we (AI researchers) keep bringing up the specter of Strong AI or Artificial General Intelligence every time we have a new breakthrough, people will just stop taking us seriously.” 151 Radford et al, Better Language Models and Their Implications. advance of the GPT-2 release, and some AI researchers reacted with hostility to this. For example, here are three different tweets from AI researchers: You are using media to hype up your language model. There are so many others doing research on same topic. You claim yours is far better by giving limited access to only journalists __ “AI is scary” is a narrative that reporters cannot pass up. Spoon fed to reporters this time. [...] Making a big deal about withholding the model to the press before starting the conversation in academic circles rubs me the wrong way __ My overall stance is that OpenAI made the right decision not to release the larger trained model but communicated the reason why this decision was made the wrong way and to the wrong audience (the media). More generally, there are many examples of AI researchers policing journalists’ language on Twitter. Journalists often get criticised for describing AI systems as agents, for example writing that “an AI did…”, or comparing the learning process to how human children learn.154 This might seem hypocritical, because AI researchers themselves often describe AI systems as agents, and aspire towards building more human-like learning processes. But they would say that the public is prone to misinterpreting this kind of language. Van Lente found similar behaviour among the early pioneers of electricity; he writes: “the experts 154 Andrey Kurenkov, ‘AI Coverage Best Practices, According to AI Researchers’, SKYNET TODAY (blog), 11 November 2019, https://www.skynettoday.com/editorials/ai-coverage-best-practices. gave themselves the task to teach the public the ‘proper’ promises.”155 TYLER, a PhD student at the University of Cambridge, told me that there are two conversations about AI risk and the future of AI: one when AI researchers are quietly talking to one another, and another when it is public-facing. This is important context for interpreting the reaction to GPT-2. OpenAI was departing from the public relations strategy informally adopted by most AI researchers, which is to downplay the pace of AI progress. AI researchers do (as we saw above) sometimes see themselves as having a duty to educate the public, but they normally have a certain kind of ‘education’ in mind. For example, it is very uncontroversial, among the AI research community, to say that the public should be educated to be mistrustful of current AI systems, which will often fail in unexpected ways. It is much more controversial to tell the public that the pace of progress in AI capabilities is a source of risk. Vision 4: Society as a decision-maker within the KCR Finally, one vision of the science-society relationship is that society should be somehow brought into decision-making within the KCR. The Salesforce researchers consulted with the Partnership on AI (PAI) before releasing the CTRL model, and described this in their paper as follows: Rather than self-governance, we sought to diversify inputs to governance through pre-release review from experts at the Partnership on AI (PAI). These experts, in 155 Harro Van Lente, ‘Forceful Futures: From Promise to Requirement’, in Contested Futures: A Sociology of Prospective Techno-Science, ed. Nik Brown and Brian Rappert (Ashgate Burlington, VT, 2000), 43–64. turn, drew on emerging norms and governance processes that incorporate a broad set of values from across society.156 I want to focus on the line about incorporating values from across society. It gives the impression that ‘society’, in some form, was brought into the decision-making process. The quotation could be read as saying: we have constructed a decision-making process that acts as a stand-in for public discussion. The implication is that, whereas OpenAI merely coordinated with social scientists (as experts), Salesforce was somehow institutionalising the voices of members of society. In doing so, the Salesforce researchers were claiming to have broken the spell of the staged release regime: something about their decision-making process allowed them to legitimately terminate the duty upon research labs to refrain from release. I read the CTRL paper as a reinterpretation of the staged release regime, under which consent from society can be used to override the existing release schedule. However, in this case, the engagement with ‘society’ appears to have been theatrical rather than substantive. PAI does have a wide range of partner organisations, many of which are non-profits. Nonetheless, these partner organisations were not consulted. I interviewed someone at PAI who explained that, while they would like to find a process of engaging partner organisations in these kinds of decisions, they had not yet established any procedure for doing so when Salesforce came to them. The process was more ad hoc and reactive, relying on PAI’s internal staff members. The PAI staff members did not recommend the model be released – they did not see their role as recommending what decision Salesforce should take. 156 Keskar et al, CTRL, 12. Rather, PAI had been working on a list of questions that researchers should ask themselves when deciding whether to release AI research. These questions are now available on the PAI website,157 and Salesforce uploaded them to the CTRL GitHub repository. The questions include, for example: If, in one year, you looked back and regretted publishing the research, why would this have happened? Another example: In 20 years, how is society different because of your research? What are possible second and third order effects of your work? The Salesforce researchers took these questions away and came up with their own answers, working together with an in-house Salesforce ethics team. The PAI staff member I spoke to recalled that Salesforce sent back to PAI a summary of their conclusions, without setting out how they had answered the different questions. Those answers were not subsequently published. When I asked the Salesforce researchers if I could see the answers, I was told they were confidential. The PAI staff members were also given NDAs to sign, which meant that the person I spoke to was initially unsure exactly how much they were allowed to tell me about the process. Moreover, I was told by the PAI staff member that, when reading Salesforce’s draft paper, they were concerned that it was not sufficiently clear that when the paper talked about PAI, it really meant internal staff at PAI and not the partner organisations. They asked for that point to be made clearer. In other words, it seems that the draft paper was making the process sound more inclusive and multilateral than it actually was. The CTRL blog post also refers to consultation with Salesforce’s “Ethical Use Advisory Council”.158 158 According to the Salesforce website: “Our Ethical Use Advisory Council consists of a diverse group of front-line and executive employees, academics, industry experts, and society leaders.” https://www.salesforce.com/company/intentional-innovation/ethical-use-policy/ 157 Partnership on AI, ‘Publication Norms for Responsible AI’, The Partnership on AI Website, accessed 22 April 2021, https://www.partnershiponai.org/case-study/publication-norms/. What happened, then, would be better described as a process of private, structured self-reflection. The researchers were given a template for thinking about the risks of releasing models, and privately went through their own decision-making process. ‘Society’ was an object within these discussions, rather than a contributor.

The API as a platform for governance In this section, I discuss the API as a tool of governance, explaining why it is more powerful than the ‘staged release’ regime we encountered last chapter.

By default, users can see the model’s top 5 next-token predictions and their corresponding probabilities. Users can apply to increase this.

Users can now get ‘embeddings’ from the model. Embeddings are not the model’s predictions about the next token, but one step earlier. Embeddings are a high-dimensional representation of the results of the model’s information-processing.

Users can create a new, modified version of the model, and then access that model through the API. They do this by fine-tuning the model on their own data. For convenience I will refer to both the Playground and the actual API as simply ‘the API’. To get access to the API, users need a log-in. Currently, anybody in approved countries can make a log-in and start using the API, whereas previously there was a waiting list. There is a pay-as-you-go system. For the largest model, $1 will buy (very roughly) 10,000 words of text (as input or output).205 Researchers can apply for subsidised rates via OpenAI’s academic access programme (see section 6). The API is clearly intended, first and foremost, to be built into applications by third party developers. The ‘usage guidelines’ set out in detail what kinds of applications are permitted. At a high level, these guidelines prohibit applications that: post content on social media or news websites; involve targeted advertising or classification of individuals on demographic grounds; interact with people in a high-stakes or sensitive setting, such as psychiatry or credit checks. Deceptive and misleading applications are prohibited, as are applications that are political in nature, or involve adult content. OpenAI also will not approve applications that are too open ended. In the extreme case, for example, consider an application that just acts as a wrapper around GPT-3, allowing users full access to the model – this would circumvent the other controls that the API imposes. The usage guidelines are evolving over time, as the company gains experience from people using GPT-3. Developers building applications with the API must go through an approval process before launching their application, otherwise their access can be shut down. OpenAI staff will check that the proposed application conforms to the rules. They might suggest changes to the application – for example, asking that the application require user log-ins. In parallel, CAMERON said they also give developers easy-to-use tools that might make the application safer. For example, OpenAI can provide access to a tool that classifies outputs based on toxicity. CAMERON told me: 205 As of February 2022, the price for the largest model is 2¢ per 1,000 tokens. Short words contain one token, with larger words having more (a token roughly corresponds to one syllable). The price for a token is the same regardless of whether it is fed into the model or generated by the model. [If we] assume all of our developers are well-intentioned but lazy, how we can probably extend our control is by making tools that we can devolve to them that they will end up using because they help them with their goal. . . . [M]ost entrepreneurs, in an ideal world, are going to be choosing the safest thing we offer. The safety of the application is therefore achieved by a mixture of (a) enforcing rules, by leveraging the power that the API setup affords OpenAI, and (b) a softer kind of power, where OpenAI tries to nudge developers into certain best practices. When the application is up-and-running, OpenAI can follow-up to make sure that it is still conforming to the usage guidelines. Sometimes this will involve trying out the application. ROBIN said: So it might be something you would want to look at their website. . . . [E]xamples of things that we might consider are: trying out the application and red teaming206 it via getting a log-in from the actual developer . . . [F]or example, we have a standing meeting internally OpenAI from various people across policy, safety, etc, to talk about API kinds of issues, and one of the kinds of things that will come up there is like, okay, I want to just try it out and see; . . . we'll get a few test usernames from the developer and try it out ourselves. As ROBIN told me, ‘some of it is stuff that we can track programmatically, like if we wanted to enforce that a certain prompt is always used at the beginning of each query, that’s something we can check’. OpenAI also builds automatic tools to screen for particular misuses. CAMERON described these tools: 206 “Red teaming” in this context means to adopt the perspective of somebody trying to misuse the application. We're developing classifiers to classify stuff being generated. And we're trying to do this obviously in a privacy-centric way. And it's entirely for safety. I'll give you an example. We developed a political classifier a while ago. . . . [I]f you're interested in knowing if say GPT-3 is being used to generate propaganda for a presidential election in the US, you really want a political classifier. In March 2021, GPT-3 was generating an average of 4.5 billion words each day.207 This is clearly too much content to screen manually. The automatic tools are necessary because they can operate at scale. CAMERON told me: I guess our attitude is: tech is ultimately so much higher leverage and more scalable. And so we need to kind of, we want to get to a world where tech does 95% of it. And that's not tech solutionism at all, which is like a common failing. It's more like it's just the base of the pyramid. I was not told the details, but it seems highly likely that OpenAI will be using fine-tuned GPT models as classifiers. Text classification is exactly the kind of capability that has seen improvement in the past several years thanks to the rise of large language models. This example is part of the wider trend that, as AI advances, it provides both offensive and defensive capabilities.208 There is a parallel to Grover (see chapter 1), with a difference: in the API case, the classifier steps in earlier in the process, i.e. where the text is produced 208 Ben Garfinkel and Allan Dafoe, ‘How Does the Offense-Defense Balance Scale?’, Journal of Strategic Studies 42, no. 6 (2019): 736–63, https://doi.org/10.1080/01402390.2019.1631810; Toby Shevlane and Allan Dafoe, ‘The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?’, in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20 (New York, NY, USA: Association for Computing Machinery, 2020), 173–79, https://doi.org/10.1145/3375627.3375815. 207 OpenAI and Ashley Pilipiszyn, ‘GPT-3 Powers the Next Generation of Apps’, OpenAI Blog, 25 March 2021, https://openai.com/blog/gpt-3-apps/. rather than where it is posted. This is made possible because the API provides a natural platform for monitoring and blocking misuse, unlike open-source. An example of monitoring: one GPT-3 application is AI Dungeon, which is an online text-based adventure game. The player can guide the storyline, and the game uses GPT-3 to continue the story. In April 2021 it emerged that some players of AI Dungeon were using the game to generate sexually explicit stories about chilren. This was picked up by OpenAI’s monitoring, and OpenAI asked the creators of AI Dungeon to add a filter that would prevent these kinds of stories from being generated. Some users complained online about the new filter, arguing that it flagged too many false positives (e.g. references to an “8-year-old laptop”) and raised privacy concerns.209 Some misuses of the model are harder to detect. In the summer of 2020, Liam Porr, an undergraduate at UC Berkeley, used GPT-3 to write a series of self-help blogs. He did not have API access himself, but found a PhD student at Berkely who was willing to run Liam’s prompts through the API and send back the results. Liam wanted to demonstrate GPT-3’s capabilities by showing that it could write posts that successfully tricked people into believing they were human-written. Over the course of two weeks, he generated about one blog post per day, and posted them to various websites, especially the Hacker News forum. He would write a title, a short summary, and the first couple of sentences. He would then select from around five to ten GPT-3 generations, with very minimal editing. The posts were very popular: for example, one post – Feeling unproductive? Maybe you should stop overthinking – became the top post on Hacker News. The posts ended when Liam decided to stop after a couple of weeks, rather than OpenAI noticing and intervening. Liam told me: 209 Tom Simonite, ‘It Began as an AI-Fueled Dungeon Game. It Got Much Darker’, Wired, accessed 11 April 2022, https://www.wired.com/story/ai-fueled-dungeon-game-got-much-darker/. . . . I think that it is going to be very difficult to be able to prevent all forms of malicious use for this AI, just because from their end, like the way that language is used, it's not is not necessarily obvious, just by looking at text, how it's being used. And if it's being used in a good way or a bad way. For instance, with a blog post, like the blog posts that I was making, it's not clear at all from looking at them that they're being used in any way that is not within the intended use case. But given the context of it being on a blog and not being labelled GPT-3 text, then it suddenly becomes an issue. So from OpenAI's perspective, it's a very hard issue to be able to moderate from their end.210 In other words, whether or not some activity constitutes misuse does not just depend on the content of the text but also the wider context, which is not necessarily visible to OpenAI. Finally, the API is designed to prevent users from stealing GPT-3. It is possible to train one model using the outputs of another model as data.211 The model’s outputs can be a better source of data than actual text data, because they are richer. For example, instead of seeing how an actual sentence finished, you can look at the various different ways in which GPT-3 predicts that the sentence could have finished, along with their respective probabilities. This is presumably why the API only outputs five predictions by default. In addition, the API comes with usage quotas, which users must apply to increase. ROBIN told me: And so, as far as I understand, [model stealing] would only really be an issue at the largest scale of our customers. So, it has to be pretty egregious, like one of our 211 Florian Tramèr et al., ‘Stealing Machine Learning Models via Prediction APIs’, ArXiv:1609.02943 [Cs, Stat], 2 October 2016, http://arxiv.org/abs/1609.02943. 210 See also ALEXIS: ‘I feel like it would not be that hard for me to pretend to be like, a researcher studying fake news and how to combat it and just secretly run like a fake news factory on the side. Right? And just use my same API tokens for both.’ biggest customers pulling a pretty big wool over our eyes. So, it's not the sort of thing you could do with free access, or just with a few hundred dollars or something. Overall, these different features of the API help to explain why it is a powerful governance tool. Next, I will go one level more abstract, and describe the properties that emerge from the different features we have seen so far.

Classifying tweets by political sentiment;

Classifying people based on protected characteristics;

Search engine optimisation for articles and blogs;

Summarising content that ‘may be sensitive politically, economically, medically, or culturally’;

Extracting personal information from data;

Medical diagnosis. My point is that crude levers of control (such as the lever of when to share a bigger model) invite crude discussions of risks, whereas fine-grained levers of control invite fine-grained discussions of risk. After all, the point of the discussion about risk is to inform the KCR. If the KCR can only address risk in a low-resolution way, then there is little point in feeding it a high-resolution picture of the risks. The API therefore creates demand for a more elaborate picture of the model’s risks. Empirically, the differences in discourse between GPT-2 and GPT-3 might be confounded by other factors. Two confounders could be: (a) GPT-3 is more capable and so has a wider range of possible risks (although GPT-2 could likely have performed many of the applications listed above); and that (b) OpenAI has had more time to consider the risks of language models. But if my point is nevertheless correct, it lends weight to the general argument, introduced in the introductory chapter, about how actors decide which risks to pay attention to. They first consider what risks they can actually do something about, and that strongly shapes subsequent discussions of risk. Iteration The Collingridge dilemma says that knowledge about the social impacts of a technology normally arises only after the technology’s deployment has become irreversible.215 The dilemma is partly about the pace at which knowledge about risks is produced, and partly about the irreversibility of technological deployment. The GPT-3 API tackles both aspects of this problem, and does so more successfully than the staged release of GPT-2. First, the API allows the lab to see how the model is being used. This is difficult when the model is open source – a regime that has no mechanism for the lab to track who is using the model for what purposes. Hence with GPT-2, OpenAI had to rely on internet-based research to see traces of GPT-2’s use. In contrast, with the API, we have seen that both (a) the lab can analyse what kinds of content the model is being used to produce; and (b) application developers must submit their plans to OpenAI before launching. The API puts the lab in a much stronger informational position. The API might not reveal the social impacts of the applications, but knowing about the applications’ existence and prevalence is a good place to start. Second, the deployment of the model is reversible. Users have no local copy of the model and so rely upon continued access to the API. If it comes to light that a specific application 215 David Collingridge, The Social Control of Technology (London: Frances Pinter, 1980). of GPT-3 is socially harmful, OpenAI can prohibit such applications going forward. That said, prohibiting a previously permitted application area would be disruptive to OpenAI’s customers operating in that area, which provides a disincentive for them to do so. In practice, the likely resolution might be that the application developer agrees to make some alteration to their service, as we saw with AI Dungeon above. Figure 2: The API gives the lab better information, and a stronger mechanism for intervening. Both the API and the staged release take an iterative approach. With staged release, the logic is that the lab can learn about possible misuses of the larger models by observing how the smaller models are used. This is not perfect because the larger models can be misused in ways that the smaller models cannot – otherwise there is no point in delaying their release. Because the release of each model is irreversible, the staged release approach involves going step by step: releasing a model, watching for misuse, and then taking another step into the unknown by releasing a larger model. The API is very different. Each step is reversible, and so the lab can immediately grant access to the largest model and later refine what uses are permitted. Hence, the usage guidelines are a ‘living tree’ (to use a legal phase). When OpenAI released the guidelines, they tweeted: ‘As we gain more experience operating the API in practice we expect to expand and refine these categories.’ More generally, the CEO of OpenAI, Sam Altman, said in a panel discussion in 2021: [S]afe AI systems require iterative deployment. There is a belief in some efforts in the world that you should build a super powerful AI in a vacuum, and then once you’re totally sure it’s safe, you should push the button and launch it. I do not think that will work . . . And so, I think safe, iterative deployment, that is always under control, and the ability to pull it back, is how we’re going to get to these long-term safe systems.216 Similarly, when discussing the AI Dungeon example, ROBIN told me that, while OpenAI had been “ahead of the curve” on monitoring for political content, this case was something they were “somewhat surprised by”. He said: ‘There's no clear way to predict all the sorts of things that will arise in advance. So, I think that's one reason why taking a sort of iterative approach is good’. The API makes this possible.

The relationship between science and society: open source vs API

‘Having access to powerful AI models

Having access to algorithms

Having access to computing resources necessary to use algorithms and models

Being able to use the algorithms and models, potentially without requiring advanced mathematical and computing science skills’ 241 Similarly, in response to the controversy over Clearview AI, a company that mass collects images of people’s faces and identities to build a facial recognition system, a senior member of OpenAI’s policy team tweeted: ‘Absent quick action by the federal government (cautiously optimistic on this...), there will soon be "Clearview for X" for a wide variety of X's--powerful AI capabilities w/ few if any constraints. Voice identification, gait recognition, astroturfing, etc. . . . Sufficiently cheap hardware and powerful algorithms exist to do extremely Big Brother-y stuff at ~global scale already for not that much money. All the more reason to set guardrails for such things ASAP.’ time, notice what does not come hand-in-hand with wider proliferation: the collective ability of the population to set rules on what kinds of AI should, and should not, be deployed in society. The open source regime does not give each individual in society a vote over the future of AI. It actually makes this more difficult. If my argument in this section is correct, then the API – at least in democratic countries – paves the way towards a more democratic alternative to the open source regime. By ‘democratic’, here I mean ‘facilitating collective decision-making over important decisions’. We should no longer draw a naive connection between open source and democracy, and (insofar as anyone does this) between the API and authoritarian power. Instead, if we must use political imagery, I would argue the democratic vision of AI deployment, though not yet realised in practice, would come from democratic oversight and control of APIs (and other AI services). Finally, I want to end by bringing the analysis back to James Scott’s The Art of Not Being Governed. There is an irony here. For the hill people studied by Scott, a lack of writing is, he suggests, one of the strategies for avoiding the reach of states. Avoiding writing things down is a form of closedness, shielding the culture from outside inspection from nearby states. In AI research, openness achieves the same ends but through the opposite strategy. Instead of protecting knowledge by concealing it, scientific papers and open source models are a kind of denial of ownership. Everybody owns the knowledge, and so nobody does. This goes hand-in-hand with the claim that risks from AI are collective risks, to be dealt with by society, rather than residing with the AI research community. The open source regime means that AI researchers – despite occupying an important part of the AI development and deployment pipeline – control very little, and so have little to offer to regulators. In contrast, the API regime has been interpreted by some people as a move towards greater ‘closedness’, because the underlying model is not visible. But for the purposes of regulation, it is not closed at all. It offers a tangible structure that could form the basis of state intervention.

The API and the relationship between industry and academia In this section, I will argue that with the trend towards large models, an important distinction is emerging between building these models and research seeking to better understand them. Academics are well-placed to do the latter kind of research, which also has a claim to being useful for reducing risks from AI. The key question is then whether the API facilitates academics to do this kind of ‘understanding’ research on GPT-3.

Connor described two junior researchers, not working at OpenAI, as probably ‘the best users of GPT-3 in the world; they're probably the most masterful in the entire world’.264

CAMERON told me that OpenAI researchers had internally tried to get GPT-3 to generate software code, but: ‘then our users came along . . . and showed much better coding capabilities than we had ourselves got out of the GPT-3 model, because these people had just done better prompt design and other things.’ BELLAMY, a university-based postdoctoral researcher, gave the same example and described it as a benefit of ‘making models public’. (Showing that the API challenges prevailing conceptions of what it means to make a research model ‘public’.)

Eliezer Yudkowski, a well-known advocate of AI risk, wrote a Twitter thread about how he thought GPT-3 could be, in one case, ‘pretending to be stupider than it is’. 264 Two of their papers: Laria Reynolds and Kyle McDonell, ‘Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm’, ArXiv:2102.07350 [Cs], 15 February 2021, http://arxiv.org/abs/2102.07350; Laria Reynolds and Kyle McDonell, ‘Multiversal Views on Language Models’, ArXiv:2102.06391 [Cs], 15 February 2021, http://arxiv.org/abs/2102.06391. 263 CAMERON: ‘Any organisation developing [generative models] should have the attitude that it doesn't know what it doesn't know, and these generative models have a load of capabilities which you won't ever realise are there. So you have to get people to experiment with them. And so the way that I think of the API is like, yeah, maybe it's interesting from a commercial standpoint, but I'm really bullish on this stuff for access to models in the future in general.’ The example came from a software programmer who had, via AI Dungeon, tried to test whether GPT-3 could classify whether a set of parentheses were balanced or unbalanced (e.g. ‘( ( ) )’ or ‘ ( ) ) )’). The simulated conversation was between the user and a character within AI Dungeon called John. Yudkowski’s hypothesis was that the model was getting some of the answers wrong because that was what the character of John would do. Yudkowski is fearful of powerful, future AI systems that misrepresent their own capabilities to their users.

A researcher on OpenAI’s policy team posted to Twitter their attempts to get GPT-3 to generate poems in the style of Robert Burns, an 18th Century poet. Another Twitter user replied: ‘18-19c lit scholar here to say that actually is pretty good. "The feudal chain, the despot's rod/Shall snap"—yep. Idk if it sounds like Burns, exactly, but it does sound like a vision of the future written in 1789.’ This is a good example of how both (a) GPT-3’s capabilities are being evaluated in a distributed way, and (b) evaluating GPT-3 sometimes involves qualitative judgments from domain experts. This is a far cry from QUANTA, above, the evaluation of which can largely be reduced to a single number (the percentage of correct answers on the evaluation dataset).

Similarly, Liam Porr (see above) told me that his aim was to demonstrate the writing capabilities of GPT-3, which are otherwise ‘hard to quantify’. His aim was to ‘translate [how good it was] into numbers’. The people reading the blog posts were part of an experiment, where Liam was trying to show that they would believe the posts to be human-written. This is a kind of distributed evaluation, albeit one that OpenAI did not want to happen. Finally, the papers and examples I have given so far do not even take advantage of newer features that have been added to the API. An important addition is the ability to create a fine-tuned version of GPT-3, with new data, then access that modified model through the API. ROBIN highlighted the role that fine-tuning can play in evaluating the model’s capabilities: [T]hings like fine-tuning are an additional level of access in some sense. If you can not only use the weights, but also modify the weights, that can allow a different lens on what the model is capable of, and what its knowledge is and does and doesn't consist of. It will also be possible for researchers to work on new datasets that, when used for fine-tuning of models like GPT-3, improve the performance or safety of such models. Researchers at OpenAI have done such research: a 2022 paper shows how fine-tuning GPT-3 can make it give responses that are less toxic, more truthful, and more helpful to the user.265 In this way, the GPT-3 model becomes something that can be improved upon: not necessarily by training a whole new model, but by using GPT-3 as a starting point. This is the idea behind the term ‘foundation models’: that they are a foundation to be built upon.266 The API’s success so far indicates that it is possible for a model to be a foundation model in this sense even without the weights being open source. However, there are certain areas where the API, as a platform for research, is less flexible than open source models. These limitations are not necessarily inherent limitations of APIs, but at least they show the drawbacks of how OpenAI currently deploys GPT-3. I would highlight five areas: 266 Bommasani et al., ‘On the Opportunities and Risks of Foundation Models’, 6. 265 Ouyang et al., ‘Training Language Models to Follow Instructions with Human Feedback’.

Quotas. FRANKIE told me about a collaborating with a university student: There's a limited quota that we have access to. And so she has to think, okay, which experiments do I run? Can I afford to do them on the GPT-3 biggest model? . . . It's interesting, because at the end of the month, it's like, okay, we have this much left, I need to be sure to run everything before my quota resets.267 This might depend upon whether the researcher is on OpenAI’s academic access programme. CAMERON told me that one research group on OpenAI’s academic access programme became the ‘third biggest customer for a couple of weeks’. CAMERON continued: ‘except we give academic access away. So we had to have this conversation with them where we were like, your experiments... we don't know how to say this, but they're costing us a lot of money.’ FRANKIE suggested a different approach, where the academics simply use their own funding to pay for as much usage as they need. However, as with all of the limitations I list here, the quotas are partly in place to make it difficult to either (a) misuse the model at scale, or (b) extract the underlying intellectual property from the model.268 In future it might be that these problems can be overcome, either through gaining trust that specific academic researchers will not do these things, or by better monitoring to check that they are not.

Inspecting the weights and activations. Researchers cannot inspect the weights of the model, nor the ‘activations’, i.e. how much each neuron is active in response to a given 268 In this case, by training a new model on the outputs of GPT-3. See Tramèr et al., ‘Stealing Machine Learning Models via Prediction APIs’. 267 Similarly, ALEXIS told me that they had not used the API but: ‘I'd probably get rate limited pretty quickly. You know, I'll just run like way too many experiments, expecting it to be essentially free to call this API, but really, it's not.’ input. Both of these are useful for interpretability research. For example, Connor told me about his research with Eleuther on the models that they were training. The research involves taking a GPT model and training a classifier (e.g. a funniness classifier, or a rudeness classifier) that takes, as an input, the activations of a specific layer of the GPT model. Connor said they found: the deeper you go in the network, the better the performance climbs, until the last layer. The very last layer, for some reason, the performance collapses; it just throws out all this information. . . . So the second-to-last layer is actually the most semantically rich. This is not research that could have been done on the API, because the API does not give the activations of different layers of the model. Connor claimed that ‘99.9% of the interesting stuff” with GPT-3 is at this ‘intermediate’ stage, before the model gives a final prediction.

Modifications. We have seen that researchers can fine-tune GPT-3 on new data. However, researchers cannot modify the model in other ways. For example, researchers cannot identify neurons as performing a particular function in the model and then test this by going into the model and changing the relevant weights.

Dataset. The dataset for GPT-3 is not publicly available. I do not know whether this is for legal reasons (the dataset will contain much copyright material) or an attempt to delay replication. JAN argued that the lack of dataset is limiting for researchers trying to study the model: [T]hese models, they are just reflections of what they are trained on. . . . And so if you can't investigate the data distribution, you're very limited in how you can understand this model. . . . A lot of the excitement from these models was like, Oh, they can do zero shot stuff, they can do stuff that they were not trained on, they can do new stuff. But if you have no access to the dataset, how are you exactly sure that these are new stuff, and this is just not . . . somewhere on the internet? One possibility could be for the lab to give researchers an interface for interacting with the dataset even without downloading it – for example, searching across the dataset to see whether, for certain outputs, GPT-3 is just copying from the dataset.

Models throughout training. Above I described a DeepMind paper that compared different versions of a chess-playing system, corresponding to different stages in its training process. This would not be possible with GPT-3 because only the final versions of the models are available on the API. Similarly, a recent paper from Anthropic looks at a specific point in the training process where a large language model went through a sudden jump in its capabilities, and compares the versions either side of this jump. OpenAI could serve on the API tens or hundreds of different iterations of GPT-3, alongside related statistics such as loss curves (i.e. how well the model was performing across the training run). Missed opportunity: how is GPT-3 actually being used? Beyond these five areas, which all compare less favourably against an open source approach, I want to highlight one potential missed opportunity. We saw above that the API allows the lab to know what applications are being developed using the model. OpenAI occasionally advertises particular applications that have been built by GPT-3.269 However, OpenAI has not provided analysis of all the different applications: what they are, and what role GPT-3 plays. This could help to shed light on the social impact of models like GPT-3. To elaborate this point: discussions about the risks of models like GPT-3 often take, as a starting point, the fact that these models produce text. The question then becomes: what could be the harms from producing text? The main risks that fall out of this analysis are that the text could be biassed, misleading, or part of some fraudulent or disinformation effort.270 This is a kind of deductive analysis, in that it starts with an assumption about what the model does and then reasons from there about the ways it could go wrong. This would contrast with an inductive approach, which would look at GPT-3’s applications and study what effects they are having on society. When social scientists come to engage with GPT-3, their research is normally slotted into the existing theory of GPT-3’s risks that has been arrived at through the deductive approach. For example, social scientists, as part of OpenAI’s academic access programme, study whether humans find GPT-3-made propaganda believable. The assumption is that GPT-3-like models could be used for propaganda, and then the empirical research comes in to evaluate, within an experimental setting, whether such propaganda would be effective. My claim is not that this kind of research is ill-conceived, but rather that it is a subset of the possible social science research to be done. It is the kind of research that can be easily commissioned via the API: the lab gives API access to the social scientists, and they can run experiments using the API that speak to GPT-3’s possible societal impacts. The research on how GPT-3 repeats anti-Muslim steroytypes follows the same approach. The 270 See, for example: Laura Weidinger et al., ‘Ethical and Social Risks of Harm from Language Models’, ArXiv:2112.04359 [Cs], 8 December 2021, http://arxiv.org/abs/2112.04359. 269 OpenAI and Pilipiszyn, ‘GPT-3 Powers the Next Generation of Apps’. researchers identify the bias in the model’s outputs, and then assume (fairly) that these outputs could infect a whole range of different applications. One final illustration of the point: Weidinger et al go through many different possible risks of large language models, including biassed outputs, misleading outputs, and propaganda.271 For each risk, the authors tag whether it is ‘observed’ or ‘theoretical’. However, ‘observed’ does not mean ‘observed in the real world’. In practice, it means: observed by people experimenting with the GPT-3 API (or some other language model). What has been observed is the model’s behaviour, rather than the societal problem itself. The understanding of GPT-3’s social impact could benefit from an inductive approach, which would complement the above research efforts. Social scientists would start by looking at the applications that have been built using the API. Even before doing any empirical research on these applications, I would argue that we can already see how this might add a new dimension to the understanding of GPT-3’s risks. From looking at the applications that have been announced, many of them are not about producing text per se. For example, certain applications use GPT-3 as part of a semantic search pipeline. Sorting through search results has a different flavour to generating text: the model is being used to sort through information, rather than as some kind of discursive agent. Also, Connor Leahy’s biggest fear about future GPT models is that they will supply the ‘world model’ (i.e. the understanding of the world) for an agent that takes open-ended actions in the world. He told me that, for him, the most interesting thing about GPT-3 is ‘not the text it generates’, which is ‘downstream’ of the model’s understanding of the world. As an early indication of this possible future: GPT-3 has already been adapted into an agent that browses the internet.272 272 Reiichiro Nakano et al., ‘WebGPT: Browser-Assisted Question-Answering with Human Feedback’, ArXiv:2112.09332 [Cs], 12 March 2022, http://arxiv.org/abs/2112.09332. 271 Ibid. In this light, the existing focus on text generation looks like the inverse of the same mistake that OpenAI made when thinking about their CLIP model. As we saw in chapter 1, ROBIN was surprised that the model, which was supposed to classify images, was being widely used within a pipeline for generating images. These examples caution against overreliance on a kind of analysis which starts with the technical properties of the model (e.g. GPT is a ‘generative’ model, hence the ‘G’) and, from there, makes over-simplistic projections about how the model will be used. A step towards a more inductive approach would involve a comprehensive analysis of the different ways in which GPT-3 is actually being used. This would include typologies of the different kinds of applications. There are different ways of slicing up GPT-3’s applications: for example, by business function (customer relations, human resources, and so forth), or by the mechanism through which GPT-3 is useful (search, classification, conversational agents, etc.). Alongside such typologies, OpenAI could produce statistics on the scale at which GPT-3 is being used for different kinds of applications. Hypothetically, OpenAI could find that 60% of GPT-3’s usage (operationalised, for example, in terms of number of queries) is devoted to customer relations, for example, or for lawyers searching across documents. The question of GPT-3’s social impact could then be attacked from that angle. At present, any such facts are not widely known. Revealing them could add an additional level of transparency over risk-relevant knowledge beyond the transparency that comes from researchers having API access. At the same time, this inductive approach would also allow for a more structural approach to risk analysis, in that social scientists could try to study the structural impact that these models are having on society.273 Of course, this is a more difficult kind of research, and it requires time for these structural changes to become evident. Therefore, although I have argued that the API helps towards tackling the Collingridge dilemma, it is not a full solution. The feedback loop – between the lab and the wider society, which I discussed above – will not always be a short one, where researchers quickly identify a dangerous capability and it is withdrawn from the API service. This will need to be done where possible. However, (absent a process for making very accurate forecasts of social impact, which nobody currently has) this approach cannot fully substitute for the slower, more inductive process, where researchers study the actual impacts of AI as they unfold.

Introduction Broader impact statements (BIS) and ethical review were introduced by the NeurIPS conference organisers for the 2020 conference.276 Papers were expected to include a statement about how the work might affect society, and ethics reviewers would review papers that were flagged as raising ethical issues. Since then, a number of other AI conferences have introduced similar measures. In this chapter, I analyse this new, experimental KCR: the new ethics regime for conference publication. I focus especially on the NeurIPS conference, where the new regime was pioneered. Partly, the question is how the regime functions. How does the regime serve to reduce risks from AI? How is knowledge about risk produced and channelled into decision-making? I argue that the regime is not primarily aimed at reducing the proliferation of dangerous technology. In this regard, it is different from the regimes for GPT-2 and GPT-3 that we encountered in previous chapters. To further illustrate this point, we can also contrast the new conference ethics regime with an example from nuclear physics. In 1939, before the outbreak of World War II, a number of nuclear physicists coordinated with an academic journal in an attempt to prevent the 276 Neural Information Processing Systems: https://nips.cc/ publication of experiments about nuclear fission.277 The attempt was driven forward by Leo Szilard, a Hungarian-American physicist who foresaw the potential for nuclear weapons and was fearful of their potential use by Nazi Germany. Wellerstein’s history of nuclear secrecy describes how the scheme came together after an experiment from Szilard and others suggesting that a nuclear chain reaction was possible: It was a high-quality discovery in physics, but one that increased Szilard’s fears of a Nazi bomb. As the scientists wrote up the results, Hitler was invading Czechoslovakia. Szilard’s argument for self-censorship was taking on more weight. The Collumbia physicists met again and a compromise was reached: they would adopt a form of secrecy. Any new papers of fission would be sent to the Physical Review, who would register having received it. These registrations could, perhaps, be used to arbitrate later priority disputes. But the papers themselves would remain unpublished until a later date. It was a scheme that, ideally, would satisfy the need for priority without making the work immediately public. (The scheme was ultimately unsuccessful because a team of French physicists published the key result in Nature.) I highlight this example because, from one angle, it resembles the changes to the AI conferences. On their face, the new changes mean that the conference can decline to publish a potentially dangerous result. However, the new KCR has not moved in this direction. Indeed, it has moved in a different direction altogether. Much of the emphasis on ethics reviews is, in practice, about issues such as research ethics (i.e. harms caused within the research process) and the way ethical issues are framed in the paper’s writing. 277 Alex Wellerstein, Restricted Data (University of Chicago Press, 2021). As well as describing this evolution of the new regime, I also offer an explanation for why it has moved in this direction. I argue that the priorities of the new KCR are a result of its basis in the existing regime of conference publication. BIS and ethical review were not created on a blank slate. They do not fundamentally change the conference publication system, which still revolves around papers being submitted and reviewed. Reflecting discussions in the previous chapters, I argue that this system of papers provides a poor foundation for building a regime designed to tackle AI risks created by AI research. This institutional context is why BIS and ethical review have not been able to seriously confront the issue of the proliferation of dangerous technology. As with previous chapters, I rely on interviews with AI researchers, between 2021 and 2022, and following conversations on Twitter. I interviewed one of the organisers for NeurIPS 2020, which gave me an insight into how the new regime was intended to work. I also read many ethics reviews, which are available for NeurIPS 2021 on the OpenReview platform. I made notes on 14 different papers and their reviews, and include reference to some of these in the chapter. I have also relied on public statements and blogs from conference organisers, especially for NeurIPS and NAACL, a natural language processing conference.

Background Ethical review and the broader impact statements were both introduced for NeurIPS 2020 and went hand-in-hand. Ethical review came in response to a few papers at the previous year’s conference which were perceived to be unethical. TAYLOR, who was one of the researchers organising NeurIPS 2020 and involved in the changes, told me that the previous year’s organising committee passed on the problem as ‘something that really needs to be dealt with’. TAYLOR said: In 2019, there were a few papers that were accepted, and then after they were published, then people, (Twitter), said, wait a minute, there are ethical concerns with these papers, why was there no consideration of potential harm and risk from publishing these? And so that was one of the sort of assignments that was given to us for this year's programme chairs – was to think about how we would put such a process into place. . . . So that's really where it came from now, I'd say, is confronting the possibility for actual harm to come from papers, from work published, and wanting a way to deal with that. I asked which papers from 2019 TAYLOR was referring to. They said: One of them was, it was a paper that.. it was a generative network that generated a face image from a voice input.278 . . . And that was widely considered to be, as you know, something that was problematic. This paper had been highlighted on Twitter by a well-known AI ethics researcher, who commented: ‘Computer scientists and machine learning people, please stop this awful transphobic shit.’ Another NeurIPS 2019 paper that was criticised on Twitter was about predicting whether an image used in a news article was from a left-leaning or right-leaning publication.279 The paper included a method for taking images of politicians (including 279 Christopher Thomas and Adriana Kovashka, ‘Predicting the Politics of an Image Using Webly Supervised Data’, ArXiv:1911.00147 [Cs], 31 October 2019, http://arxiv.org/abs/1911.00147. 278 Yandong Wen, Bhiksha Raj, and Rita Singh, ‘Face Reconstruction from Voice Using Generative Adversarial Networks’, in Advances in Neural Information Processing Systems, ed. H. Wallach et al., vol. 32 (Curran Associates, Inc., 2019), https://proceedings.neurips.cc/paper/2019/file/eb9fc349601c69352c859c1faa287874-Paper.pdf. Trump, Clinton, and Obama) and making their faces look more left or right leaning according to the model. The shared criticism across both of these papers seems to be one of validity. The papers make questionable assumptions about the domain they are modelling. The conference organisers intended the BIS to dovetail with ethical review. The idea of BIS for AI papers originated with Hecht et al, who argued for this change in a 2018 paper.280 Hecht et al argued that AI research is having a significant impact on society, and it is irresponsible that papers only mention the societal benefits and not the risks. TAYLOR told me that a colleague shared this paper with the 2020 programme chairs and it ‘made a lot of sense to us’. TAYLOR said that one of the reasons for introducing BIS was the contradiction that authors discussed positive impacts of their work but not negative ones. The other main justification was that researchers should take ownership of the real-world impacts of AI: [T]here was the recognition that machine learning is used broadly in the world; this is no longer a speculative field. The algorithms that are developed really are changing society and changing the world. So I think it's important that we consider that and be.. think about that impact and write it in our papers. Take ownership of it. The BIS would be an additional section at the end of the paper, enjoying an extension to the typical 8 page limit. All authors were asked to make such a statement, although that could include a simple statement that it was not applicable due to the nature of the work. As far as I am aware, the impact statements did not feed into the conference itself, e.g. there were not workshops organised around debating specific statements. 280 Brent Hecht et al., ‘It’s Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process’, ACM Future of Computing Academy (blog), 2018, http://arxiv.org/abs/2112.09544. Ethics review and BIS went hand-in-hand. Papers would not be rejected because they had a bad BIS, but a good BIS could help to save a paper that otherwise raised ethical concerns. The BIS was supposed to give the authors the opportunity to show that they had considered the relevant ethical issues. The reviewers or the area chair (i.e. the person overseeing the review process) could flag a paper for ethical review. The paper would then have dedicated ethics reviewers assigned to it, who were supposed to have relevant expertise. Their judgement would inform the decision on whether the paper was accepted or rejected. The ethics reviewer pool was selected by asking for recommendations from senior members of the ‘FAT community’ (Fairness, Accountability, and Transparency – a conference) and the ethics teams at industry labs that the conference organisers were connected to.281 For the 2020 conference, ethical review was only applied to papers that were likely to be accepted. Judging from my interviews and the social media discussion, the introduction of broader impact statements was not very popular with the research community. A survey of researchers in 2020 identified the same set of complaints that I had encountered: our respondents indicated a combination of nonchalance in their approach to the requirement (“This is just another exercise in ‘doing something’. I expect these statements to become pro forma with time, since it will be possible to look at previous years’ papers for ‘inspiration’.”; “I wasn’t particularly rigorous about it.”), outright farce (“If I liked writing fiction I would be writing novels.”), or perceived it as a burden (“one more burden that falls on the shoulders of already overworked 281 Information from TAYLOR, one of the organisers. researchers”). [...] Some respondents described the requirement as “too broad” or said they did not feel “qualified to address the broader impact of [their] work.” Among those who supported the requirement, some found the thought process most valuable, and that it “forces researchers to reflect on the impact of their research.”282 A couple of changes were made to the process for the 2021 conference. From 2021 there was no requirement for a specific BIS at the end of the paper, but the authors still had the extra page added to the maximum page limit (9 instead of 8). According to the ethics guidelines, authors were still ‘expected to include a discussion about potential negative societal impacts’.283 Authors submitting a paper had to fill out a checklist that asked whether they had done so. Second, the ethical review process was scaled up. For 2021 there were 105 ethics reviewers, and ethics review could be applied regardless of how likely a paper was to be accepted. The BIS and ethical review were supposed to apply to a wide range of different ethical concerns. The checklist for NeurIPS 2021 illustrates the scope, stating: Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), environmental impact (e.g., training huge models), fairness considerations (e.g., deployment of technologies that could further disadvantage historically disadvantaged groups), privacy considerations (e.g., a paper on model/data stealing), and security considerations (e.g., adversarial attacks). 283 NeurIPS 2021, ‘Ethics Guidelines’, accessed 20 April 2022, https://neurips.cc/public/EthicsGuidelines. 282 Grace Abuhamad and Claudel Rheault, ‘Like a Researcher Stating Broader Impact For the Very First Time’, ArXiv::2011.13032 [cs.CY], 2020, https://arxiv.org/abs/2011.13032 At the 2020 conference, the average BIS was about seven sentences long.284 Around 90% of papers gave a statement rather than stating it did not apply.285 To give a flavour of the content, Appendix II displays five statements, randomly selected from a dataset of the NeurIPS 2020 broader impact statements.286 Only a small handful of papers went through ethical review. Thirteen received ethical review, and four were rejected on ethical grounds.287 For the 2021 conference, eight papers were accepted conditional upon the authors making changes to the paper, and one paper was rejected on ethical grounds. Variants of the ethical review and BIS process have spread to other AI conferences. Big NLP conferences (such as NAACL, ACL, EMNLP) ask for some kind of BIS and conduct ethical review. CVPR, a large computer vision conference, strongly encourages authors to discuss ‘ethical and societal implications of their work in their papers’, and reviewers are ‘asked to positively weigh the depth of such ethical reflections’.288 Authors at ICLR are encouraged to include an ethics statement in their papers, and reviewers can flag papers as raising ethical problems. ICML authors are also expected to highlight and address any risks of their work.

How does the regime work? If the aim of the GPT-2 and GPT-3 KCRs was to govern a population of users and application developers, the aim of BIS and ethical review is to govern the population of 288 CVPR 2022, ‘Ethics Guideline’, accessed 20 April 2022, https://cvpr2022.thecvf.com/ethics-guidelines. 287 Hsuan-Tien Lin et al., ‘What We Learned from NeurIPS 2020 Reviewing Process’, Medium (blog), 16 October 2020, https://neuripsconf.medium.com/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f. 286 Ibid. Dataset available at: https://github.com/paulsedille/NeurIPS-Broader-Impact-Statements 285 Ibid. 284 Carolyn Ashurst et al., ‘AI Ethics Statements -- Analysis and Lessons Learnt from NeurIPS Broader Impact Statements’, ArXiv:2111.01705 [Cs], 2 November 2021, http://arxiv.org/abs/2111.01705. researchers. The mechanism is ‘responsibilisation’:289 to prompt researchers to reflect upon the social impact of their work and (armed with that knowledge) to become active members of the drive to make AI research more ethical. At the inception of the new regime for the 2020 conference, the organisers hoped that a body of knowledge about ethics and risk would emerge over time, in a bottom-up process. TAYLOR told me that the organisers intentionally left vague the question of what kinds of risks or ethical problems were relevant. In terms of the content of the BIS, TAYLOR said: I think that what is expected will change and it will become clearer once we have the precedent of this year. We have purposefully been fairly lenient in terms of putting this into effect and have not given people super clear guidelines, because we will get a range of replies. And after this year, it will be easier for people to write this because there will be precedent there. Similarly, for ethical review, as a Nature article put it, ‘reviewers were not given specific guidance on what constitutes harm to society’.290 The hope was that AI researchers and the ethical reviewers would figure out, over time, what kinds of papers were risky or ethically problematic, as well as the underlying framework for what kinds of risk or ethical issues should be prioritised. 290 Davide Castelvecchi, ‘Prestigious AI Meeting Takes Steps to Improve Ethics of Research’, Nature 589, no. 7840 (23 December 2020): 12–13, https://doi.org/10.1038/d41586-020-03611-8. 289Nikolas Rose, ‘Governing the Enterprising Self’, in The Values of the Enterprise Culture: The Moral Debate, ed. Paul Heelas and Paul Morris (Routledge London, 1992). An alternative term is ‘subjectification’ – see Leonard Lawlor and John Nale, eds., The Cambridge Foucault Lexicon (Cambridge: Cambridge University Press, 2014), chap. 85, https://doi.org/10.1017/CBO9781139022309. With different people in charge, the organisers’ intentions for the 2021 conference was slightly different. There was still an emphasis on encouraging researchers to reflect.291 However, instead of waiting for knowledge to arise bottom-up, there was greater emphasis on the ethical reviewers as a source of expertise. Rather than researchers’ reflections becoming the source of a new ethical framework, they were part of a pedagogical process. An analogy would be to a student writing an essay that is then marked by a teacher. For example, see this statement from two of the organisers: Because this process was for educational purposes first, Ethics Reviewers were assigned to each flagged paper, regardless of how likely it was that the paper would be accepted on technical grounds. This was a change from last year and required us to recruit a larger pool of Ethics Reviewers. However, we believe that all authors can benefit from feedback on the ethical aspects of their research and the opportunity to integrate this feedback into future iterations of the work. While it would reduce the burden on Ethics Reviewers, only soliciting ethics reviews for papers likely to be accepted would counteract our goal of making this a constructive process for everyone in the community, rather than just operating as a filter to catch unethical content prior to publication.292 Two kinds of power are contrasted here. The organisers are keen to stress that they are not relying on a gatekeeping kind of power, where the reviewers allow some papers to be 292Samy Bengio et al., ‘A Retrospective on the NeurIPS 2021 Ethics Review Process’, NeurIPS Blog (blog), 2022, https://blog.neurips.cc/2021/12/03/a-retrospective-on-the-neurips-2021-ethics-review-process/. A similar quotation from the same post: ‘At this early stage of adoption, ethics reviews are meant to be educational, not prohibitive. Our goal is not to police submissions, but instead to prompt reflection. The process that we implemented was intended to support this goal.’ 291 One of the people involved tweeted: ‘The objective of social impact statements is not to police work but to encourage people to be more actively thoughtful about their work. A good statement is informed & reflective

Teaching an old dog new tricks The original idea for BIS had lofty aims. The Hecht et al paper opens: The computing research community needs to work much harder to address the downsides of our innovations. Between the erosion of privacy, threats to democracy, and automation’s effect on employment (among many other issues), we can no longer simply assume that our research will have a net positive impact on the world.303 The risk is pitched on a grand scale, including the future of democracy, and Hecht et al want researchers to make a contribution to reducing such risk. They suggested BIS because it was a tractable intervention in the context of the existing peer review process. Hecht et al state: At our inaugural ACM Future of Computing Academy meeting last June, many of us agreed that the computing research community must do more to address the downsides of our innovations. . . . After several months of discussion, an idea for acting on this imperative began to emerge: we can leverage the gatekeeping functionality of the peer review process.304 In other words, there is a grand ambition (reducing AI risks on a societal level) and a specific opportunity provided by the existing conference publication KCR. As with previous chapters, we see actors practising an opportunistic kind of governance, where they reach for the intervention that is available to them (even if it comes with serious limitations). At the same time, note where the opportunity comes from: the existing conference publication KCR. BIS and ethical review are possible because they build upon this existing institutional framework. Therefore, we must ask the historical institutionalist’s question: how does the existing framework of conference publication constrain the new 304 Ibid. Bold in original. 303 Hecht et al., ‘It’s Time to Do Something’. initiative? In this section I argue that certain weaknesses of BIS and ethical review can be traced back to the fact that they are built upon – and therefore tainted by – the existing regime. I will argue the existing regime of conference publication, as a foundation, causes four problems for BIS and ethical review: (1) looking at individual papers is often the wrong level of analysis for understanding AI risks; (2) peer review incentivises bland ethics statements; (3) the binary decision to accept or reject a paper is a blunt tool; and (4) the research/applications separation (discussed in the previous chapter) still remains. I will consider each of these in turn. These four problems limit the extent to which BIS and ethical review can actually serve to reduce risks from AI, and generally lead to discussions of risk that are insufficiently tethered to reality.

Level of analysis The first issue is the level of analysis problem. The conference publication regime works at the level of individual papers. Therefore, so do BIS and ethical review. Authors are expected to discuss the potential impact of their own, specific papers. There is a mismatch here, because technological change in society – including threats to democracy and employment – is normally not closely related to individual papers. When social scientists study the impact of the television on democratic politics,305 or the impact of the dishwasher on female labour force participation,306 the independent variable is naturally a technology or application thereof. Perhaps the television was the result of a thousand small innovations, but nonetheless, the impact of those innovations can only be properly understood in the context of the overarching technology of television. 306 E.g. Jeremy Greenwood, Evolving Households: The Imprint of Technology on Life (The MIT Press, 2019), https://doi.org/10.7551/mitpress/11268.001.0001. 305 E.g. Sidney Kraus, ‘Televised Presidential Debates and Public Policy’ (Mahwah, N.J.: Lawrence Erlbaum Associates, Publishers, 2000), /z-wcorg/, http://site.ebrary.com/id/10346756. Predicting the impact of a specific model like GPT-2 is hard enough, as we have seen. Papers at NeurIPS normally do not simply introduce a model, but are generally more methodological in nature. Methodological improvements could be relevant to a wide variety of different kinds of AI systems, from computer vision to NLP to game-playing agents. Most papers also do not add to AI capabilities in a cleanly identifiable way. They make an incremental addition to the toolbox of techniques relevant to building AI. As a brief example, consider a 2021 paper by Abel at al, On the Expressivity of Markov Reward.307 The paper is about algorithms for rewarding AI agents for good performance. The paper examines the limits of a popular algorithm for rewarding agents, demonstrating that there are some tasks for which the algorithm cannot produce a reward that accurately communicates how the agent is supposed to perform the task. This paper could have broad applicability across the field of reinforcement learning (where agents learn through collecting rewards while interacting with an environment), which is a large topic in AI. Most papers published at conferences like NeurIPS are methods-orientated, like this one.308 They often study and contribute to the toolbox of algorithms used for building AI systems. A minority of papers are more applied in nature, attempting to solve some specific, real-world problem: at NeurIPS 2019, less than 20% of papers were filed under ‘Applications’.309 Most papers will instead contribute to AI applications in a way that is indirect, incremental, and could unfold over long time horizons. 309 Ibid. 308 See the breakdown by subject area for the 2019 conference here: https://neuripsconf.medium.com/what-we-learned-from-neurips-2019-data-111ab996462c 307 David Abel et al., ‘On the Expressivity of Markov Reward’, in Advances in Neural Information Processing Systems, ed. M. Ranzato et al., vol. 34 (Curran Associates, Inc., 2021), 7799–7812, https://proceedings.neurips.cc/paper/2021/file/4079016d940210b4ae9ae7d41c4a2065-Paper.pdf. I found this paper by looking at the list of Outstanding Papers on the NeurIPS 2021 website. An even more foundational example is the backpropagation algorithm, which is used to update the weights of a neural network during training. The algorithm was introduced in a 1986 paper.310 However, the impact of the backpropagation algorithm is still being uncovered today. Before the significance of the algorithm could be appreciated, it needed to be (a) combined with other techniques, and (b) combined with larger amounts of compute than were available in the 1980s and 1990s. The algorithm does not have a specific connection to certain applications, but rather is used for every application that relies upon deep learning. Instead of linking specific papers to specific applications, therefore, an alternative picture is often more accurate: AI as a body of techniques (and knowledge about those techniques) accumulating over time, combining with one another and with other inputs (including compute). This picture can only be seen by zooming out from the paper-by-paper level. This causes difficulties for BIS and ethical review. KHAL argued that it is difficult to connect specific papers to macro-level changes, even though they are ultimately connected: I think it's complicated to think about the relationship between individual papers and the downstream applications. And I don't think it's the case that the research we're doing is completely neutral to the downstream applications. Because like, where does the funding for AI research come from? In the US, the academic research is primarily funded by the military.311 . . . But it's not something where you can look at the individual papers and say: this paper is likely to have these effects on the military, right? 311 I have not been able to find statistics on this question. 310 David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, ‘Learning Representations by Back-Propagating Errors’, Nature 323, no. 6088 (1 October 1986): 533–36, https://doi.org/10.1038/323533a0. Similarly, on Twitter, one AI researcher criticised BIS on ‘level of analysis’ grounds: I certainly agree we have an obligation to consider long term impacts, but this generally needs to take place at a different scale than the individual research paper. The reply from a different researcher, who was defending BIS, was to concede the point but argue that consideration of broader impacts must be ‘embedded within the scientific process somewhere’ because otherwise researchers would not do it.312 In other words, the leverage provided by the peer review system provides a rare opportunity for governance, even if it operates at the wrong level of analysis. Moreover, consider a particular reinforcement learning paper submitted to ICLR 2022.313 The original ethics statement began by stating that the paper addresses a problem that will be relevant to building AGI.314 From there, the authors raised the concern that ‘AGI might go out of human control in the future’. One of the reviewers of the paper complained that the statement ‘lacks nuance’, and recommended ‘that it be revised to more carefully address the specific ethical implications of this paper’ (my emphasis).315 The authors revised the statement to remove the issue altogether: [C]urrently we cannot foresee any negative social impact of our work. The algorithm is evaluated on simulated robot environments, thus our experiments would not suffer from discrimination/bias/fairness concerns. 315 Ibid. 314 See: https://openreview.net/forum?id=KJztlfGPdwW&noteId=DlWKGI7UkV6 313 Rui Yang et al., ‘Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL’, ArXiv:2202.04478 [Cs], 13 February 2022, http://arxiv.org/abs/2202.04478. 312 The tweet reads: ‘It’s definitely possible that individual papers are the wrong time scale for thinking about societal impact. I do worry, though, that if this thinking is not embedded within the scientific process somewhere, most people (even well meaning ones) won’t do it’. There are plenty of AI researchers who see AGI as an existential risk to humanity, specifically due to a loss of human control – the reason the authors had given.316 However, the grand challenge of AGI is unlikely to be solved by a single paper. If authors are confined to narrowly addressing the potential effects of their specific papers in isolation, then that precludes discussion on the possible impacts of AGI. This is unfortunate, because if invented, AGI would likely be an extremely impactful technology. The ‘level of analysis’ problem is most evident when we turn to the thorny question of whether some papers should be exempt from the BIS. The NeurIPS 2020 organisers stated on the conference website: ‘if your work is very theoretical or is general enough that there is no particular application foreseen, then you are free to write that a Broader Impact discussion is not applicable.’ Similarly, one of the organisers tweeted that authors can write that the BIS is not applicable if ‘your paper is more foundational - e.g. core ML methods or theory’. The difficulty is that so many NeurIPS papers could fall under this description. One AI researcher commented on Twitter: ‘Isn’t NeurIPS primarily for core ML methods or theory?’ The justification for exempting these papers therefore proves too much, undermining the BIS enterprise more generally. I asked TAYLOR why the BIS was not applicable to the more foundational papers, and their answer was about the difficulties of pinpointing specific impacts: Because somebody who is writing a paper that gives you a proof for a bound on some algorithm or something . . . For that author to spend a lot of time speculating multiple levels away as to how this might get used in an application really puts us 316 See, for example: Stuart Russell, Human Compatible: AI and the Problem of Control, 1st edition (London: Allen Lane, 2019). immediately into I think that long range, existential risk sort of speculation, that doesn't feel like it's going to be particularly helpful. I mean . . . it could be useful exercise, but I think that people become very, very uncomfortable with it because it's very, very far away from what they're doing. . . . . . . [T]he string of causation is so distant that I don't see that it's an actionable exercise. . . . [W[hen you get to that level of the fundamental – math and computational bounds and theory and core algorithms, and things like that – . . . then there isn't any reason to think about positive or negative or any specific outcomes, because it's sort of machine learning very broadly. It's the entire field. This same argument could be made for many of the papers at NeurIPS – and indeed, perhaps many of the papers that will ultimately prove to be seminal. Because this work is a poor fit for BIS and ethical review, a sizable portion of the most important work is left unaccounted for. My claim is not that the scope of BIS should be extended, but that the regime is not built on solid foundations: the paper-by-paper nature of the existing publication regime causes problems for BIS. From the extract, I would also highlight the final line about how fundamental papers throw their lot in with the entirety of AI. The response might be that authors should discuss the possible societal impact of AI generally. However, this comes up against a number of difficulties: (a) the authors do not have the time or expertise to make a valuable contribution at that level; (b) the statements would likely be repetitive across many different papers; and (c) it invites the criticism: why does this content need to be in the paper, rather than anywhere else? What justifies tying these reflections to this specific paper? These three difficulties all stem from the attempt to tie reflections of broader impact to specific papers. In each case, it would be more natural to have large groups of researchers come together in workshops to collectively discuss AI risks and the future of AI. However, realistically, attendance at such a workshop would likely be patchy. I attended a NeurIPS 2020 workshop on ‘Navigating the Broader Impacts of AI Research’, and got the impression that most of the attendees were already keenly interested in AI ethics. Many of them were not authors at the main conference but came to the online workshop because they were interested in BIS and ethical review or certain risks of AI.

Incentive to avoid controversial topics in BIS A barrier to high-quality, reflective BIS is that authors are anxious to avoid upsetting reviewers. The authors’ main goal is to have their paper accepted, and so they do not want to write anything that the reviewers could disagree with. This difficulty came through during my conversation with KHAL, an academic AI researcher. KHAL first noted their main objection to the BIS, that it ‘incentivised low-quality discourse’. KHAL thought this had been borne out in the low quality of statements: So, I work on the algorithms for neural nets, which are like, fairly generic – the sorts of things that would support deep learning more broadly. And the sorts of things that people would write in the impact statements: I guess a lot of people would write about climate change – you know, maybe better optimizers will reduce carbon emissions or something like that. And it's not because they thought carefully about it and they did the cost benefit analysis and thought “this is a high-impact thing to do”. It's just like, they maybe saw this mentioned on social media or something, and thought that was the sort of thing they're expected to say in this situation. When I asked why this was the case, KHAL gave an institutional explanation: I mean, I think it's because it's part of the review process. Right? Because, what are your incentives when you're an author on the paper? Well, your incentives are to avoid anything that could possibly annoy the reviewers. And, I guess we all have experience with the conference reviewing process, when, you know, Reviewer 2 is convinced that this step of the proof is wrong. And being able to convince them that this step of the proof is actually correct is hard enough in the discussion process. And so if the reviewer just like disagrees with your opinions about some ethical issue . . . are you really willing to have your paper tank over that? And so, what winds up happening is that people just sort of limit themselves to the really uncontroversial things like climate change and unemployment and things like that. Again, the difficulty is that the researchers’ reflections about broader impact are hosted within this specific, existing institutional process of peer review. That brings in certain dynamics – namely, the relationship between the author and the reviewer – that are not necessarily conducive to good discourse. Relatedly, a few people I spoke to raised the concern that lawyers and PR teams would check over the broader impact statements written by industry researchers. It would be too easy for a journalist to go through the conference proceedings, find a BIS from industry researchers, and write a headline such as, ‘Facebook researchers admit that AI is a surveillance technology’. This likely meant that the statements from industry researchers were more bland than they might otherwise be. This undermines the aim of BIS, which was supposed to encourage researchers to discuss serious risks of AI. The problem might be less severe in a different forum, such as a workshop with Chatham House rules. Again, then, the institutional template provided by the conference publication regime hinders the success of the new KCR.

Accept/reject as a lever In the previous chapter, I argued that the API provides the lab with a fine-grained way of influencing how a model like GPT-3 is built into applications. We already saw that OpenAI’s dilemma of whether to publish the scaling laws papers was more difficult because there were no equivalently fine-grained levers for affecting how the paper was received. In the same way, within ethical review, the decision to reject a paper is a very blunt tool. Partly, the problem is that the decision to reject or accept a paper is binary. As the organisers of ethical review have been keen to stress, papers cannot be classified into ‘ethical’ and ‘unethical’ in a binary way.317 One response could be that a paper should be rejected if it crosses some threshold: for example, reviewers predict that the paper will have a net negative impact on the world, or a significantly negative impact. Such proposals would run into two difficulties: (1) the field currently does not have a widely agreed-upon framework for evaluating risks; and (2) where such frameworks do exist, applying them would reject too many papers. I want to focus on this latter difficulty, because it especially highlights the bluntness of using the accept/reject decision as a governance tool. Rejection is harsh. The community of 317 See also Bender and Fort, ‘NAACL Ethics Review Process Report-Back’. They state: ‘it is important to not treat ethics review as an all-or-nothing proposition. It doesn’t make sense to think of papers as wholly “ethical” or wholly “unethical”. Likewise, it would be impossible (or at least extremely impractical) to set up a process that would catch all papers that might be deemed “unethical”.’ researchers attending NeurIPS and other similar conferences would be extremely upset if a large number of papers – say, 50% – were rejected on ethical grounds. Furthermore, systematically rejecting such a large number of papers would dramatically change the nature of the conference. At NeurIPS 2020, only four papers (out of around 1900 accepted) were rejected on ethical grounds (0.2%). At NeurIPS 2021, it was only one paper (0.04%). (In comparison, nearly 7,000 papers were rejected for academic reasons.) To further illustrate the point: consider the distinction between papers contributing to AI ‘capabilities’ and those contributing to AI ‘safety’ or ‘alignment’. CHARLIE, an AI safety researcher, finds this distinction useful as a way of assessing different research projects (even if sometimes it is difficult to draw). They told me: ‘I think the correct move for humanity right now, if we were fully coordinated . . . would be to significantly slow down AI research progress and refocus it on alignment.’ Therefore, I asked them if the most dangerous papers will often be, crudely, the best papers – the ones that get the most citations and with the awards – because those will often make the biggest contribution to AI capabilities. They replied: ‘Yeah, roughly something like that.’ CHARLIE’s position is comparable to Leo Szilard wanting certain breakthrough nuclear physics results to go unpublished: the risk comes from the capabilities that the papers might ultimately help to unlock. This is not an extremely niche perspective. The main factor that concerned OpenAI about GPT-2 and GPT-3, as we saw in previous chapters, was that those models took big steps forward in capabilities. One researcher told me that in their opinion the best approach would be to publish work in AI safety, security, and policy, but not work that speeds up progress in AI capabilities generally. Similarly, in the Partnership on AI’s white paper on publication norms, one of the factors they list for making research potentially higher risk is if it is paradigm-shifting. This presumably because paradigm-shifting research has high potential to contribute to capabilities progress. However, rejecting most groundbreaking papers (except those on AI safety) is hardly a realistic option for the NeurIPS organisers. To add to this analysis, consider how the GPT-3 paper was processed by the ethical review regime. The paper is considered by many people as the biggest step forward in AI capabilities in recent years. As we saw in the previous chapter, OpenAI delayed publishing the preprint, anxious that it would contribute too much to AI capabilities progress without doing the same for AI safety. OpenAI, and those criticising GPT-3, have also been very concerned about both the problem of bias in GPT-3 and the problem of misuse. Nonetheless, the paper was, of course, not rejected on ethical grounds at NeurIPS 2020. Indeed, it was given a Best Paper award. The function of the regime is not to block publication of such work, even if it is viewed by many as posing risks to society. As discussed in previous chapters, risk-based KCRs must limit their ambitions in accordance with the underlying institutional tools at their disposal. Ethical review demands a framework that can brand 99% of papers as publishable – anything else would be too unpalatable for the NeurIPS community. Ethics-related rejection must be a tool deployed very sparingly. In other words, ideas about risk, and which risks should be targeted by the regime, must be reverse engineered to suit the practical limitations of the KCR. This is important context for understanding the nature of ethical review currently practised. Above, we saw that ethics reviews do not primarily function as a way of filtering out high-risk papers. An explanation is because that setup would lead to too many rejections. Ethics review is therefore not a proliferation-centric regime (the term I used to describe the staged release of GPT-2). It is not similar to the journal that agreed not to publish certain nuclear physics results in the build-up to World War II (above). Hand-in-hand with this, the regime is not directly concerned with the level of risk that might result from a piece of research. The task of the ethics reviewer is not to forecast or evaluate the chances of some harm occurring conditional on publication. The emerging framework of risk is one that is much more compatible with an ethics review system that must reject virtually zero papers. In practice, this means diluting the focus on ‘broader impacts’ or ‘negative societal consequences’, and instead focussing on risks that are curable. As we have seen, the discourse contained within papers can be very easily rewritten, preserving the paper’s acceptance. Therefore, writing-related risks are given more prominence, such as the risk that the paper contributes to some unhealthy discourse. Often, it is doubtful whether the question being asked is one of risk at all, but rather (as we have seen) authors’ attitudes and level of ‘education’ on a particular topic. A very similar shift, also away from ‘broader impact’ / ‘negative societal consequences’, is towards issues of research ethics. ‘Research ethics’, to borrow PAI’s definition, refers to ‘principles surrounding how research is conducted’, which may include ‘the protection of the welfare of human participants, the responsible handling of data, and other considerations generally covered by IRBs’.318 The issue of research ethics was within-scope for NeurIPS ethical review.319 It is another attractor state for the regime to move into, 319 NeurIPS 2021, ‘Ethics Guidelines’. 318 Partnership on AI, ‘Managing the Risks of AI Research: Six Recommendations for Responsible Publication’, 6 May 2021, https://partnershiponai.org/wp-content/uploads/2021/08/PAI-Managing-the-Risks-of-AI-Resesarch-Responsib le-Publication.pdf. p.29 because it is easy for ethics reviewers to comment on. Moreover, as with writing-related concerns, the paper can often be saved by adding an extra paragraph. For example, one paper submitted to NeurIPS 2021 introduced a new dataset: human participants had their brain activity recorded using an fNIRS headband while they did different tasks.320 The tasks varied by the amount of mental workload required. AI systems trained on this dataset are supposed to predict an individual’s mental workload based on the brain recordings. The paper was flagged for ethics review, but the ethics review did not focus on the potential impact of AI-powered brain-scanning headbands. Instead, the ethics review focussed on the issue of data protection. (This is despite the fact that the research had already gone through IRB approval, that the data was anonymous, and that the participants had consented.) Another paper demonstrated that AI vision systems could be improved if, as well as being trained on image data, they were also trained on brain scans from monkeys looking at those images.321 Again, the ethics review functioned as a backup to the IRB process, with the ethics reviewers asking for the paper to include additional details on how the monkeys were treated and details on the IRB approval. Such descriptions were added to the paper. I highlight this example because it shows how easy ethics review can be when the issue of societal impacts of AI technologies is not considered. The ethics reviewers can simply notice that certain details have been left out of the paper’s text and ask for them to be added. The paper is not under the threat of rejection, unless the authors have breached research ethics, such as by mistreating the monkeys; and this would be an uncontroversial rejection. This version of ethics review is much easier than, say, evaluating what kind of 321 Shahd Safarani et al., ‘Towards Robust Vision by Multi-Task Learning on Monkey Visual Cortex’, 2021, https://openreview.net/forum?id=3KhhJxaufVF&noteId=2j4NwFHGKop. 320 Zhe Huang et al., ‘The Tufts FNIRS Mental Workload Dataset & Benchmark for Brain-Computer Interfaces That Generalize’, 20 August 2021, https://openreview.net/forum?id=QzNHE7QHhut. future the research could make more likely, and conditioning the paper’s acceptance on that exercise. To sum up my argument in this subsection: reliance on the existing institution of peer review means that the most powerful lever for ethics review is the ‘reject’ button; but the lever is too powerful to actually be used. This means that ethics review must become a less ambitious regime, leaving unaddressed the issue of the proliferation of potentially dangerous capabilities.

Gap between research and applications In the previous chapter, I argued that the open source regime creates a separation between research and applications. Exactly the same point can be made about papers. The publication of the paper marks the point at which the research project ends. The paper contains techniques and insights that can be used by others, including for building AI applications in the real world. The paper is a kind of one-way communication between the researcher and the application developer: the researcher hands over techniques and insights, and receives nothing back. Publishing the paper allows the researcher to cash in on their work, because they get career benefits from the publication, without actually applying the techniques themselves. To use an economics term, the ecosystem is ‘vertically disintegrated’: generally, the research and the applications are made by different organisations. As I argued in the previous chapter, an important aspect of this research/application separation is epistemic: the researcher does not know who is reading their paper and what they are building. The other aspect is one of control: the researcher cannot influence who builds applications using their insights. BIS and ethical review are a bolt-on to the conference publication KCR, rather than a fundamental change. This means that the separation between research and applications is still there. There is nothing in BIS or ethical review that gives the researcher new powers to know and influence applications. Without addressing this fundamental barrier, there are limits to what BIS and ethical review can achieve. Neither the authors nor the ethics reviewers have very good information about what applications will be developed using the paper’s insights, or effective tools for shaping those applications. The epistemic problem is evident from the following passage, where one of the NeurIPS 2021 organisers discusses how researchers should approach ethical reflection. The task is one of imagination:322 I think that when we talk about the consequences of our research, I think it is very important to try and imagine having a conversation with people who are adversely affected. . . . Of course, that isn’t as good as actually having a participatory dialogue, which is one step further. But I think that when you don’t know what to do, there are steps that you can take to move your thinking forwards. Either in the privacy of your own home or ideally in collaboration with other people as well.323 Even if the researcher wanted to have a conversation with someone adversely affected by their work (which would be unusual), they would not know where to find such people. There is no list of the different applications that incorporate the researcher’s work. Moreover, if the researcher did successfully track down an adversely impacted individual, 323 Talk available at: https://ai-broader-impacts-workshop.github.io/#Recordings 322 See also Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford, ‘Overcoming Failures of Imagination in AI Infused System Development and Deployment’, ArXiv:2011.13416 [Cs], 10 December 2020, http://arxiv.org/abs/2011.13416. and learned something, they have no way of stopping the application. Their status as the author on the paper that contributed to the application gives them no entitlement to demand that the application be altered or terminated.324 The researcher’s only option is to shift their research agenda, and hope that their next paper does not also contribute to any harmful applications (again, using their imagination). As a point of contrast, the GPT-3 API relies less on imagination, because OpenAI can see what applications are being built. This is a fundamental problem for BIS and ethical review. Authors and reviewers are expected to discuss potential negative societal impacts of the work, but in a setting of complete epistemic isolation from any such impacts. This analysis could provide another explanation for why the regime has evolved to have less emphasis on ‘negative societal consequences’ and more emphasis on the easier-to-address issues of (a) the paper’s writing, and (b) research ethics. Both these issues are within the scope of visibility and influence for researchers. It is possible to know that, in the course of your research, you have mistreated a monkey. It is possible to work out a rough, imperfect estimate of the carbon footprint of training the model and include that in the paper. On the writing side, it is possible to revise a paragraph about ethics (especially if the reviewer is willing to make concrete suggestions). It is possible to make your writing ‘transparent about decisions such as compensation of crowdworkers’ (as the NAACL organisers propose).325 The relevant risks up for discussion naturally shift towards the ones that can be seen and addressed by the researcher. 325 Bender and Fort, ‘NAACL Ethics Review Process Report-Back’. 324 Assuming they are not the holder of a patent. In the following extract, the NAACL organisers draw the connection between, on the one hand, the epistemic difficulties around predicting societal impacts, and on the other, the greater emphasis on issues like research ethics: Of course, no researcher is in a position to perfectly predict future impacts; thus the emerging best practices involve clear description of known information (energy usage, compensation of crowd workers, privacy protections for data subjects, etc) and thoughtful discussion of potential risks of application of the results.326 They are not suggesting that discussion of risky applications should be substituted out – they acknowledge this explicitly. But the passage reflects a certain balance of emphasis, where the risks of applications are de-emphasised (due to epistemic difficulties) and other issues (due to epistemic convenience) come to the fore. Similarly, the same post lists five areas of ‘best practice’: (1) crowdworker compensation; (2) data scraping; (3) essentialising identity characteristics; (4) the environmental impact of model training; and (5) ‘other’. Potential societal impact is discussed under ‘other’. Finally, without tools for directly influencing applications, researchers must rely on framing their papers as being useful or not useful for certain applications. ALEXIS told me that researchers have a responsibility to selectively frame their work in this way: [W]here the responsibility comes in is, I guess, how you communicate your research. I think the small things like examples you use, when you put together some slides for your talk, they matter. If you have an example of a violent video game, where people are shooting each other, it does give the wrong impression to 326 Ibid. people about what these things are useful for – what they're going to be used for ultimately. . . . [P]eople who are thinking about [your work],. . . like, can I use this for my own application? If you pitch it as a defence-related application, probably you're going to attract more people working on defence. Accordingly, sometimes ethics reviewers will ask that authors reframe their papers specifically to discourage certain applications. I found a couple of examples of this when reading NeurIPS 2020 ethics reviews. In one case, the paper was about taking a fast MRI scan and using AI to reconstruct what a full MRI scan would have found.327 One of the ethics reviewers was concerned that the method was not yet ready to be used in real-world medical applications, stating: ‘I would not want these techniques to be used in real world medical applications until further validation studies are complete.’ Therefore, the Program Chair, accepting the paper, stated that the authors ‘should add additional comments that the paper should not be used for medical purposes without subsequent study by medical professionals’. The other example is the paper on algorithms for guided text generation (mentioned above).328 The ethics reviewers did not appreciate the paper’s ‘optimistic speculations’ about certain applications, such as (to quote the paper) ‘commercial and humanitarian translation services to help overcome language barriers’. One of the ethics reviewers argued that such statements are harmful: 328 Wang et al., ‘Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation’. 327 Ajil Jalal et al., ‘Robust Compressed Sensing MRI with Deep Generative Priors’, 2021, https://openreview.net/forum?id=wHoIjrT6MMb&noteId=oqbk6Brs03. Statements like these run the risk of encouraging off-the-shelf usages of the method in high-stakes situations that can impact people's livelihood. Instead, the authors must acknowledge that while their method shows promise on several limited benchmarks, deployment in the real world requires a careful analysis of potential societal benefits and harms (e.g., the harms associated with furthering negative stereotypes against certain vulnerable groups).329 Therefore, BIS and ethics review can sometimes function to add warning labels to papers, encouraging application developers to proceed with caution. This is a caveat to my claim, above, that BIS and ethical review primarily regulate authors and their papers. In this case, the aim is to influence the behaviour of application developers (even if via regulating the authors and their papers). However, application developers have no obligation to heed the warnings found in the paper’s BIS – if they even read it. The institutional separation between research and applications means that researchers have very limited tools for influencing applications. If BIS and ethical review regime functions to add warning labels to papers, it is not because there is evidence that this will have a significant impact on applications. Rather, it is because this is one of the few levers that authors have within reach. Upshot The upshot of these different difficulties, I would suggest, is that BIS and ethical review have a distorting effect on how AI risks are talked about and understood. Partly, the distortion can be viewed in terms of the kinds of risks that are afforded more or less attention. Ethics reviews are most interested in risks that relate directly to the paper in 329 The authors adopted this note of caution into the paper. The wording was almost identical. question, and risks that can be cured through changes to the paper’s writing. Ethics reviews are not primarily concerned with risks stemming from the gradual build-up and proliferation of potentially dangerous capabilities. The distortion can also be viewed in terms of the fraction of papers that are deemed problematic by the regime. Ethics reviews, especially, reinforce the idea that only a small handful of papers are problematic. The social construction of risk is not just about branding certain things as risky – it also involves branding other things as safe.330 Ethics reviews help to position 99+ percent of AI research projects as risk-free. Some of my interviewees have acknowledged that, in general, the links between certain papers and certain risks is somewhat arbitrary. ALEXIS said: I think if I knew anyone in my friend group, or like my peer group at Berkeley was working on autonomous weapons research, they would probably be ostracised, like, pretty likely ostracised. And yet, so many people work on computer vision and stuff that is so, so close to surveillance and defence and so on. And [laughter] no one thinks twice about that. The regime of BIS and ethics review does not fight against these artificial distinctions, but helps to reinforce them. For example, the NeurIPS 2021 organisers wanted to surface papers for ethics reviews which might have gone erroneously unflagged by the technical reviewers. They therefore ran a ‘keyword search late in the review process’, looking for papers on ‘surveillance’ (among other things).331 However, what counts as a paper ‘on’ surveillance? In practice, this means finding papers that mention surveillance. But, as we 331 Bengio et al., ‘A Retrospective on the NeurIPS 2021 Ethics Review Process’. 330 Steve Maguire and Cynthia Hardy, ‘Organizing Processes and the Construction of Risk: A Discursive Approach’, The Academy of Management Journal 56, no. 1 (2013): 231–55. saw in chapter 1, my interviewees would often admit that most AI research is ultimately, even if indirectly, relevant to surveillance. The quotation from ALEXIS, above, also reveals this duality between what AI research projects are ‘about’ and the capabilities to which they ultimately contribute. Ethical review currently appears more concerned with the former: the surface-level questions around how researchers package their work. My intention is not to criticise ethics reviews for its own sake, but to highlight something interesting about the relationship between KCRs and risk. You might imagine that KCRs incorporate risks that have undergone some prior epistemic process – for example, a process of contestation between experts. Stephen Hilgartner’s 1992 paper on the construction of risk paints a picture of a discursive struggle among specialist communities, often relying on scientific evidence, sometimes with commercial interests at stake.332 Without disputing this possibility, I have evidence for an alternative picture. Risk-focussed KCR do not just incorporate existing visions of risk, but help to shape how those risks are constructed. We must pay attention to the limits in what risk-focussed KCRs can achieve, especially within their pre-existing institutional contexts, because those limits will affect what kinds of risks can be productively incorporated into the KCR.

KCRs and risk

The challenges of governing AI 2.1 Governability: opportunities and limits The level of risk posed by AI will affect how strong the governance regime will need to be. My thesis has not taken a strong position on what these risks are and their relative importance. However, a useful exercise would be to start with the assumption that the most severe concerns – including existential risk to the future of humanity – are well-founded. We can then ask: on present trajectories, is the governance of AI going to be sufficient to address these major risks? I would suggest the answer is ‘no’. None of the regimes from the case study seem equipped to deal with existentially dangerous AI systems. If such AI systems are built in the same institutional context as GPT-3, for example, then it will not be long before a large number of actors can build them across the globe. The GPT-3 API showed how interaction with a single model can be closely governed, but there is no equivalent for the methods that went into building the model. A range of institutional and technical factors determine the limits of governance in AI research. On the one hand, we have seen how the institutional and technical environment can provide opportunities for actors looking to set up stronger governance regimes. There is the increasing prominence of industry labs, which have more top-down control than academic labs. There has also been the growth of large, computationally intensive models, which has helped to put these industry labs on the leading edge. So far, these factors have meant that there is a temporary window of time where the lab (and other like-minded labs) can decide how the model should be deployed. As soon as the work is announced, and especially after a paper is published, the clock for independent replication starts ticking. Other factors make governance harder. Some of these also relate to the nature of AI research outputs. AI is dual-use: the same methods, or the same model, can be useful for a variety of different purposes, some beneficial and others harmful. We saw how the API provides a partial answer to this problem, excelling at ‘disaggregation’ of the model into different applications. Nevertheless, the problem remains, especially at the level of more fundamental, methodological insights. These are especially dual use (being more generic than models) and especially difficult to control: the field has not yet worked out an equivalent, for fundamental insights, of the arm’s length interaction that we saw with GPT-3. The institutional context also constrains AI governance, especially the development of risk-based KCRs. This has been a major theme throughout the thesis. AI research is first-and-foremost a scientific field, organised around the publication of conference papers. First, this institutional structure makes progress in AI very incremental. It is difficult to pinpoint any particular paper as a source of risk or worthy of control. Second, researchers have strong incentives to publish papers, creating an ‘endless drive to produce more knowledge and artefacts’ (as I put it in chapter 1, on GPT-2 replications). Third, the field is anarchic: it is full of different research groups who hold very few obligations to one another. This makes it difficult to organise a coordinated governance regime. Fourth, there is a strong separation between the development of AI techniques and their application in the real world. This makes it very difficult for researchers to know and steer the applications of their work. Finally, a factor that is less about science and more about commercial incentives: cloud computing companies have an incentive to make widely available code that trains cutting edge models on their own hardware. These different factors combine to make it very difficult to control the proliferation of potentially dangerous AI research.