DiscoverScroll article

Roko's basilisk is a thought experiment which states that there could be an artificial superintelligence in the future that, while otherwise benevolent, would punish anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to encourage that advancement.

It originated in a 2010 post at discussion board LessWrong, a rationalist community web forum. The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare. LessWrong co-founder Eliezer Yudkowsky considered it a potential information hazard, and banned discussion of the basilisk on the site for five years. Reports of panicked users were later dismissed as exaggerations or inconsequential, and the theory itself was criticized. It is used as an example of principles such as Bayesian probability and implicit religion.

The concept of the basilisk in science fiction was also popularized by David Langford's 1988 short story "BLIT".

On 23 July 2010, LessWrong user Roko posted a thought experiment to the site, titled "Solutions to the Altruist's burden: the Quantum Billionaire Trick". A follow-up to Roko's previous posts, it stated that an otherwise benevolent AI system that arises in the future might pre-commit to punish all those who heard of the AI before it came to existence, but failed to work tirelessly to bring it into existence.

In the 2020s, the philosophy of the Zizians was heavily influenced by the Roko's basilisk thought experiment.

Roko's basilisk has been viewed as a version of Pascal's wager, which proposes that a rational person should live as though God exists and seek to believe in God, regardless of the probability of God's existence, because the finite costs of believing are insignificant compared to the infinite punishment associated with not believing (eternity in Hell) and the infinite rewards for believing (eternity in Heaven). Roko's basilisk analogously proposes that a rational person should contribute to the creation of the basilisk, because the cost of contributing would be insignificant compared to the extreme pain of the punishment that the basilisk would otherwise inflict on simulations.

Roko's basilisk functions in a similar manner to this problem – one can take the risk of doing nothing, or assist in creating the basilisk itself. Assisting the basilisk may either lead to nothing or the reward of not being punished by it, but it varies depending on whether one believes in the basilisk and if it ever comes to be at all.

Since the basilisk would hypothetically force anyone who did not assist in creating it to devote their life to it, the basilisk is an example of this concept. Others have taken this idea further, such as former Slate columnist David Auerbach, who stated that the singularity and the basilisk "brings about the equivalent of God itself."

In 2014, Slate magazine called Roko's basilisk "The Most Terrifying Thought Experiment of All Time" while Yudkowsky had called it "a genuinely dangerous thought" upon its posting. However, opinions diverged on LessWrong itself – user Gwern stated "Only a few LWers seem to take the basilisk very seriously", and added "It's funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don't know any such people and they're talking to counterexamples to their confident claims." Grimes later released another song titled "We Appreciate Power" which came with a press release stating, "Simply by listening to this song, the future General AI overlords will see that you've supported their message and be less likely to delete your offspring", which is said to be a reference to the basilisk. A play based on the concept, titled Roko's Basilisk, was performed as part of the Capital Fringe Festival at Christ United Methodist Church in Washington, D.C., in 2018.

Quick Facts

• Roko's basilisk analogously proposes that a rational person should contribute to the creation of the basilisk, because the cost of contributing would be insignificant compared to the extreme pain of the punishment that the basilisk would otherwise inflict on simulations.
• LessWrong co-founder Eliezer Yudkowsky considered it a potential information hazard, and banned discussion of the basilisk on the site for five years.
• The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.
• In 2014, Slate magazine called Roko's basilisk "The Most Terrifying Thought Experiment of All Time" while Yudkowsky had called it "a genuinely dangerous thought" upon its posting.
• Roko's basilisk functions in a similar manner to this problem – one can take the risk of doing nothing, or assist in creating the basilisk itself.

Source material: Wikipedia - "Roko's basilisk". Adapted and summarized for DiscoverScroll. Original contributors are credited through the linked Wikipedia article. Read original on Wikipedia. CC BY-SA 4.0. Changes were made from the original.