BLUF: The concept of “existential risks” from AGI is controversial and tied to ideologies promoting a techno-utopian future, where preventing these risks is seen as vital to achieving a blissful and valuable universe. However, this vision may be impractical and potentially harmful, as it overlooks the catastrophic consequences for many humans and disregards the complexities of achieving such a utopia.
Here’s the transformed article:
Artificial general intelligence (AGI) is believed by some to pose an extreme danger to humanity. AGI doomers argue that a misaligned AGI could annihilate humans simply because we are made of atoms that it can repurpose. This perspective gained traction after the release of ChatGPT and has prompted calls for AGI research regulation. But are these existential risks truly plausible, and what do they entail?
The colloquial definition of “existential risk” refers to the complete extinction of Homo sapiens, eradicating humanity forever. However, the canonical definition, associated with influential Silicon Valley ideologies, encompasses ambitious goals like creating superior posthumans, colonizing space, and generating immense value over countless years. Essentially, it imagines a technological utopia teeming with digital posthumans in surpassing bliss.
Many TESCREALists argue that these existential risks must be averted at any cost to safeguard utopian aspirations and astronomical value. Some even propose risking a thermonuclear war to prevent the development of dangerous AGI. They believe that a thermonuclear war would likely spare enough survivors to rebuild society, whereas an AGI apocalypse would eliminate all life, erasing the envisioned techno-utopian future.
However, skepticism arises regarding the canonical definition of existential risks. The focus on achieving a utopian vision seems unrealistic and potentially hazardous. The significance of 10^58 digital individuals residing in vast computer simulations spread across the universe is questionable and might not justify the associated moral catastrophe as believed by many TESCREALists.
Moreover, the pursuit and realization of this utopia could result in catastrophic consequences for a majority of humanity. Take OpenAI, founded by TESCREALists and currently led by one. While the company claims to ensure AGI’s benefits for all of humanity, its large language models (LLMs), including ChatGPT and GPT-4, inherently possess limitations and unintended biases, raising concerns about their potential impact.
In conclusion, the notion of existential risks from AGI remains controversial, heavily influenced by ideologies promoting a techno-utopian future. While the preservation of this vision is regarded as vital by some, it disregards the complexity of achieving such a utopia and overlooks the potential harm it may cause to a significant portion of humanity.