# superintelligenc a superintelligence is a hypothetical agent that possesses intelligence surpassing that of the most gifted human minds. philosopher nick bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest technological researchers disagree about how likely present-day human intelligence is to be surpassed. some argue that advances in artificial intelligence (ai) will probably result in general reasoning systems that lack human cognitive limitations. others believe that humans will evolve or directly modify their biology to achieve radically greater intelligence. several future study scenarios combine elements from both of these possibilities suggesting that humans are likely to interface with computers or upload their minds to computers in a way that enables substantial intelligence amplification. the hypothetical creation of the first superintelligence may or may not result from an intelligence explosion or a technological singularit some researchers believe that superintelligence will likely follow shortly after the development of artificial general intelligence. the first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability including the capacity of perfect recall a vastly superior knowledge base and the ability to multitask in ways not possible to biological entitie several scientists and forecasters hav been arguing for prioritizing early research into the possible benefits and risks of human and machine cognitive enhancement because of the potential social impact of such technologie ## artificial superintelligenc philosopher david chalmers argues that artificial general intelligence is a very likely path to artificial superintelligence (asi). chalmers breaks this claim down into an argument that ai can achieve equivalence to human intelligence that it can be extended to surpass human intelligence and that it can be further amplified to completely dominate humans across arbitrary task concerning human-level equivalence chalmers argues that the human brain is a mechanical system and therefore ought to be emulatable by synthetic materials. he also notes that human intelligence was able to biologically evolve making it more likely that human engineers will be able to recapitulate this invention. evolutionary algorithms in particular should be able to produce human-level ai. concerning intelligence extension and amplification chalmers argues that new ai technologies can generally be improved on and that this is particularly likely when the invention can assist in designing new technologie an ai system capable of self-improvement could enhance its own intelligence thereby becoming more efficient at improving itself. this cycle of "recursive self-improvement" might cause an intelligence explosion resulting in the creation of a superintelligenc computer components already greatly surpass human performance in speed. bostrom writes "biological neurons operate at a peak speed of about 200hz a full seven orders of magnitude slower than a modern microprocessor (~2ghz)." moreover neurons transmit spike signals across axons at no greater than 120m/s "whereas existing electronic processing cores can communicate optically at the speed of light". thus the simplest example of a superintelligence may be an emulated human mind running on much faster hardware than the brain. a human-like reasoner who could think millions of times faster than current humans would hav a dominant advantage in most reasoning tasks particularly ones that require haste or long strings of action another advantage of computers is modularity that is their size or computational capacity can be increased. a non-human (or modified human) brain could become much larger than a present-day human brain like many supercomputers. bostrom also raises the possibility of collective superintelligence: a large enough number of separate reasoning systems if they communicated and coordinated well enough could act in aggregate with far greater capabilities than any sub-agen humans outperform non-human animals in large part because of new or enhanced reasoning capacities such as long-term planning and language use. (see evolution of human intelligence and primate cognition.) if there are other possible improvements to reasoning that would hav a similarly large impact this makes it more likely that an agent can be built that outperforms humans in the same fashion humans outperform chimpanzee the above advantages hold for artificial superintelligence but it is not clear how many hold for biological superintelligence. physiological constraints limit the speed and size of biological brains in many ways that are inapplicable to machine intelligence. as such writers on superintelligence hav devoted much more attention to superintelligent ai scenario in 2024 ilya sutskever left openai to cofound the startup safe superintelligence which focuses solely on creating a superintelligence that is safe by design while avoiding "distraction by management overhead or product cycles". despite still offering no product the startup became valued at $30 billion in february 202 in 2025 meta created meta superintelligence labs a new ai division led by alexandr wan ## biological superintelligenc carl sagan suggested that the advent of caesarean sections and in vitro fertilization may permit humans to evolve larger heads resulting in improvements via natural selection in the heritable component of human intelligence. by contrast gerald crabtree has argued that decreased selection pressure is resulting in a slow centuries-long reduction in human intelligence and that this process instead is likely to continue. there is no scientific consensus concerning either possibility and in both cases the biological change would be slow especially relative to rates of cultural chang selective breeding nootropics epigenetic modulation and genetic engineering could improve human intelligence more rapidly. bostrom writes that if we come to understand the genetic component of intelligence pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of iq gain (if one embryo is selected out of two) or with larger gains (e.g. up to 24.3 iq points gained if one embryo is selected out of 1000). if this process is iterated over many generations the gains could be an order of magnitude improvement. bostrom suggests that deriving new garmetes from embryonic stem cells could be used to iterate the selection process rapidly. a well-organized society of high-intelligence humans of this sort could potentially achieve collective superintelligenc alternatively collective intelligence might be constructed by better organizing humans at present levels of individual intelligence. several writers hav suggested that human civilization or some aspect of it (e.g. the internet or the economy) is coming to function like a global brain with capacities far exceeding its component agents. a prediction market is sometimes considered as an example of a working collective intelligence system consisting of humans only (assuming algorithms are not used to inform decisions a final method of intelligence amplification would be to directly enhance individual humans as opposed to enhancing their social or reproductive dynamics. this could be achieved using nootropics somatic gene therapy or brain−computer interfaces. however bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent cyborg interface is an ai-complete proble most surveyed ai researchers expect machines to eventually be able to rival humans in intelligence though there is little consensus on when this will likely happe in a 2022 survey the median year by which respondents expected "high-level machine intelligence" with 50% confidence is 2061. the survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human worker in 2023 openai leaders sam altman greg brockman and ilya sutskever published recommendations for the governance of superintelligence which they believe may happen in less than 10 year in 2025 the forecast scenario ai 2027 led by daniel kokotajlo predicted rapid progress in the automation of coding and ai research followed by asi. in september 2025 a review of surveys of scientists and industry experts from the last 15 years reported that most agreed that artificial general intelligence (agi) a level well below technological singularity will occur before the year 2100. a more recent analysis by aimultiple reported that “current surveys of ai researchers are predicting agi around 2040 ## design consideration exploring the potential motivations of an artificial superintelligence bostrom distinguishes final goals and instrumental goals. from the point of view of an agent final goals are intrinsically valuable whereas instrumental goals are only useful for attaining final goals. he proposed the "orthogonality thesis" which postulates that in principle virtually any final goal can be combined with virtually any level of intelligence. bostrom also introduced the concept of instrumental convergence which postulates that certain instrumental goals (such as self-preservation resource acquisition or cognitive enhancement) increase the probability of achieving final goals in a wide range of situations and would thus likely be pursued by a broad spectrum of intelligent agent william macaskill argued that aligning superintelligence with current human values could be catastrophic if those values are permanently locked in and humanity still has moral blind spots like slavery in the pas several proposals for an asi's final goals hav been put forward **+** coherent extrapolated volition (cev) - the ai should hav the values upon which humans would converge if they were more knowledgeable and rational **+** moral rightness (mr) - the ai should be programmed to do what is morally right relying on its superior cognitive abilities to determine ethical actions **+** moral permissibility (mp) - the ai should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to cev bostrom elaborates on these concepts > instead of implementing humanity's coherent extrapolated volition one could try to build an ai to do what is morally right relying on the ai's superior cognitive capacities to figure out just which actions fit that description. we can call this proposal "moral rightness" (mr).. > > mr would also appear to hav some disadvantages. it relies on the notion of "morally right" a notoriously difficult concept one with which philosophers hav grappled since antiquity without yet attaining consensus as to its analysis. picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong.. > > one might try to preserve the basic idea of the mr model while reducing its demandingness by focusing on moral permissibility: the idea being that we could let the ai pursue humanity's cev so long as it did not act in morally impermissible way ## potential threat to humanit the development of artificial superintelligence (asi) has raised concerns about potential existential risks to humanity. researchers hav proposed various scenarios in which an asi could pose a significant threat ## intelligence explosion and control proble some researchers argue that through recursive self-improvement an asi could rapidly become so powerful as to be beyond human control. this concept known as an "intelligence explosion" was first proposed by i. j. good in 1965 > let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. since the design of machines is one of these intellectual activities an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion' and the intelligence of man would be left far behind. thus the first ultraintelligent machine is the last invention that man need ever make provided that the machine is docile enough to tell us how to keep it under contro this scenario presents the ai control problem: how to create an asi that will benefit humanity while avoiding unintended harmful consequences. eliezer yudkowsky argues that solving this problem is crucial before asi is developed as a superintelligent system might be able to thwart any subsequent attempts at contro ## unintended consequences and goal misalignmen even with benign intentions an asi could potentially cause harm due to misaligned goals or unexpected interpretations of its objectives. stuart russell provides an example > a system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances rather than to improve the external worl this aligns with bostrom's earlier example of an asi turning the solar system into a calculating device to solve a mathematical problem inadvertently killing humanity in the proces ## potential mitigation strategie researchers hav proposed various approaches to mitigate risks associated with asi **+** capability control - limiting an asi's ability to influence the world such as through physical isolation or restricted access to resources **+** motivational control - designing asis with goals that are fundamentally aligned with human values **+** ethical ai - incorporating ethical principles and decision-making frameworks into asi systems **+** oversight and governance - developing robust international frameworks for the development and deployment of asi technologie despite these proposed strategies some experts such as roman yampolskiy argue that the challenge of controlling a superintelligent ai might be fundamentally unsolvable emphasizing the need for extreme caution in asi developmen ## debate and skepticis not all researchers agree on the likelihood or severity of asi-related existential risks. some like rodney brooks argue that fears of superintelligent ai are overblown and based on unrealistic assumptions about the nature of intelligence and technological progress. others such as joanna bryson contend that anthropomorphizing ai systems leads to misplaced concerns about their potential threat ## recent developments and current perspective the rapid advancement of llms and other ai technologies has intensified debates about the proximity and potential risks of asi. while there is no scientific consensus some researchers and ai practitioners argue that current ai systems may already be approaching agi or even asi capabilitie **+** llm capabilities - recent llms like gpt-4 hav demonstrated unexpected abilities in areas such as reasoning problem-solving and multi-modal understanding leading some to speculate about their potential path to asi **+** emergent behaviors - studies hav shown that as ai models increase in size and complexity they can exhibit emergent capabilities not present in smaller models potentially indicating a trend towards more general intelligence **+** rapid progress - the pace of ai advancement has led some to argue that we may be closer to asi than previously thought with potential implications for existential ris as of 2024 ai skeptics such as gary marcus caution against premature claims of agi or asi arguing that current ai systems despite their impressive capabilities still lack true understanding and general intelligence. they emphasize the significant challenges that remain in achieving human-level intelligence let alone superintelligenc the debate surrounding the current state and trajectory of ai development underscores the importance of continued research into ai safety and ethics as well as the need for robust governance frameworks to manage potential risks as ai capabilities continue to advanc **+** ai takeove **+** artificial brai **+** artificial intelligence arms rac **+** effective altruis **+** ethics of artificial intelligenc **+** existential ris **+** friendly artificial intelligenc **+** future of humanity institut **+** intelligent agen **+** machine ethic **+** machine intelligence research institut **+** machine learnin **+** neural scaling law– statistical law in machine learnin **+** noosphere– philosophical concept of biosphere successor via humankind's rational activitie **+** outline of artificial intelligenc **+** posthumanis **+** robotic **+** self-replicatio **+** self-replicating machin **+** superintelligence: paths dangers strategie 1. ↑ mucci tim; stryker cole (2023-12-14). "what is artificial superintelligence?". ibm. retrieved 2025-04-17 2. ↑ bostrom 2014 chapter 2 3. ↑ pearce david (2012) "the biointelligence explosion: how recursively self-improving organic robots will modify their own source code and bootstrap our way to full-spectrum superintelligence" in eden amnon h.; moor james h.; søraker johnny h.; steinhart eric (eds.) singularity hypotheses the frontiers collection berlin germany and heidelberg germany: springer berlin heidelberg pp.199–238 doi: 10.1007/978-3-642-32560-111 isbn978-3-642-32559-5 retrieved 2022-01-1 4. ↑ gouveia steven s. ed. (2020). "ch. 4 "humans and intelligent machines: co-evolution fusion or replacement?" david pearce". the age of artificial intelligence: an exploration. vernon press. isbn978-1-62273-872-4 5. ↑ legg 2008 pp.135–137 6. ↑ chalmers 2010 p.7 7. ↑ chalmers 2010 pp.7–9 8. ↑ chalmers 2010 pp.10–11 9. ↑ chalmers 2010 pp.11–13 10. ↑ "clever cogs". the economist. issn0013-0613. retrieved 2023-08-10 11. ↑ bostrom 2014 p.59 12. ↑ bostrom 2014 pp.56–57 13. ↑ bostrom 2014 pp.52 59–61 14. ↑ vance ashlee (june 19 2024). "ilya sutskever has a new plan for safe superintelligence". bloomberg. retrieved 2024-06-19 15. ↑ "there's something very weird about this $30 billion ai startup by a man who said neural networks may already be conscious". futurism. 2025-02-24. retrieved 2025-04-27 16. ↑ tan eli (2025-12-10). "meta's new a.i. superstars are chafing against the rest of the company". the new york times. issn0362-4331. retrieved 2026-02-02 17. ↑ sagan carl (1977). the dragons of eden. random house 18. ↑ bostrom 2014 pp.37–39 19. ↑ bostrom 2014 p.39 20. ↑ bostrom 2014 pp.48–49 21. ↑ watkins jennifer h. (2007) prediction markets as an aggregation mechanism for collective intelligenc 22. ↑ bostrom 2014 pp.36–37 42 47 23. ↑ roser max (2023-02-07). "ai timelines: what do experts in artificial intelligence expect for the future?". our world in data 24. ↑ roser max (7 february 2023). "ai timelines: what do experts in artificial intelligence expect for the future?". our world in data. retrieved 2023-08-09 25. ↑ "governance of superintelligence". openai.com. retrieved 2023-05-30 26. ↑ roose kevin (2025-04-03). "this a.i. forecast predicts storms ahead". the new york times. issn0362-4331. retrieved 2025-04-27 27. jump up to: 1 2 orf darren (october 2025). "humanity may achieve the singularity within the next 3 months scientists suggest". www.msn.com. popular mechanics. retrieved 2025-10-02 28. ↑ bostrom 2014 chapter 7: the superintelligent will 29. ↑ piper kelsey (november 2022). "review: what we owe the future". asterisk magazine. retrieved 2026-02-16 30. jump up to: 1 2 bostrom 2014 pp.209–221 31. ↑ good i. j. (1965). "speculations concerning the first ultraintelligent machine". advances in computers 32. ↑ russell 2019 pp.137–160 33. ↑ yudkowsky eliezer (2008). "artificial intelligence as a positive and negative factor in global risk" (pdf). global catastrophic risks. doi: 10.1093/oso/9780198570509.003.0021. isbn978-0-19-857050-9 34. ↑ russell 2019 p.136 35. ↑ bostrom 2002 36. ↑ bostrom 2014 pp.129–136 37. ↑ bostrom 2014 pp.136–143 38. ↑ wallach wendell; allen colin (2008-11-19). moral machines: teaching robots right from wrong. oxford university press. isbn978-0-19-970596-2 39. jump up to: 1 2 dafoe allan (august 27 2018). "ai governance: a research agenda" (pdf). center for the governance of ai 40. ↑ yampolskiy roman v. (july 18 2020). "on controllability of artificial intelligence" (pdf). arxiv: 2008.04071 41. ↑ brooks rodney (october 6 2017). "the seven deadly sins of ai predictions". mit technology review. retrieved 2024-10-23 42. ↑ bryson joanna j (2019). "the past decade and future of ai's impact on society". towards a new enlightenment? a transcendent decade. 11. isbn978-84-17141-21-9 43. ↑ bubeck sébastien; chandrasekaran varun; eldan ronen; gehrke johannes; horvitz eric; kamar ece; lee peter; yin tat lee; li yuanzhi; lundberg scott; nori harsha; palangi hamid; marco tulio ribeiro; zhang yi (april 2023). "sparks of artificial general intelligence: early experiments with gpt-4". arxiv: 2303.12712 44. ↑ wei jason; tay yi; bommasani rishi; raffel colin; zoph barret; borgeaud sebastian; yogatama dani; bosma maarten; zhou denny; metzler donald; chi ed h.; hashimoto tatsunori; vinyals oriol; liang percy; dean jeff; fedus william (2022-06-26). "emergent abilities of large language models". transactions on machine learning research. arxiv: 2206.07682. issn2835-8856 45. ↑ ord toby (2020). the precipice: existential risk and the future of humanity. london new york (n.y.): bloomsbury academic. isbn978-1-5266-0023-3 46. ↑ "how and why gary marcus became ai's leading critic > marcus says generative ai like chatgpt poses immediate dangers". ieee spectrum. 17 september 202 **+** bostrom nick (2002) "existential risks" journal of evolution and technology 9 retrieved 2007-08-07 **+** chalmers david (2010). "the singularity: a philosophical analysis" (pdf). journal of consciousness studies. 17: 7–65 **+** legg shane (2008). machine super intelligence (pdf) (phd). department of informatics university of lugano. retrieved september 19 2014 **+** müller vincent c.; bostrom nick (2016). "future progress in artificial intelligence: a survey of expert opinion". in müller vincent c. (ed.). fundamental issues of artificial intelligence. springer. pp.553–571 **+** santos-lang christopher (2014). "our responsibility to manage evaluative diversity" (pdf). acm sigcas computers and society. 44 (2): 16–19. doi: 10.1145/2656870.2656874. s2cid5649158. archived from the original on july 29 201 **+** hibbard bill (2002). super-intelligent machines. kluwer academic/plenum publishers **+** bostrom nick (2014). superintelligence: paths dangers strategies. oxford university press **+** tegmark max (2018). life 3.0: being human in the age of artificial intelligence. london england. isbn978-0-14-198180-2. oclc1018461467.`{{cite book}}`: cs1 maint: location missing publisher (link **+** russell stuart j. (2019). human compatible: artificial intelligence and the problem of control. new york. isbn978-0-525-55861-3. oclc1113410915.`{{cite book}}`: cs1 maint: location missing publisher (link **+** sanders nada r. (2020). the humachine: humankind machines and the future of enterprise. john d. wood (firsted.). new york new york. isbn978-0-429-00117-8. oclc1119391268.`{{cite book}}`: cs1 maint: location missing publisher (link)