Rationalists

This is a primer on the Rationalist movement.

The people building the most powerful AI systems in the world share something beyond technical skill. They share an origin story. OpenAI, Anthropic, DeepMind: the organizations shaping the trajectory of artificial intelligence trace their intellectual DNA to an internet subculture that blends Bayesian probability theory with apocalyptic prophecy, communal living with utilitarian ethics, and self-improvement workshops with messianic conviction. That subculture calls itself the Rationalist movement. Understanding it is no longer optional for anyone trying to make sense of how AI governance works, who controls it, and why.

Table of Contents

• What is the Rationalist Movement?
• A Church of Reason: The Social Infrastructure
• The Creed: AI Doom and How to Prevent It
• From Blog Posts to Boardrooms: The Rationalist-to-AI-Lab Pipeline
• When the Money Dried Up: FTX and the Crisis of Faith
• The Dark Side: Cults, Violence, and the Zizians
• Galaxy-Brained: Why Rationalism Fails at Its Own Game
• The State of the Church: Fragmentation and Legacy
• Related Concepts
• Footnotes

What is the Rationalist Movement?

The Rationalist movement (capital R, to distinguish it from the philosophical tradition stretching back to Descartes) emerged from internet forums in the early 2000s as a project to “debug” human thinking. Its adherents, mostly self-taught programmers and autodidact philosophers, took a specific bet: that cognitive biases documented by Kahneman and Tversky could be systematically overcome through Bayesian reasoning, decision theory, and sheer force of will. Where Enlightenment rationalism trusted human reason as a faculty, Rationalism treats human reason as broken firmware in need of patching.

The founding figure is Eliezer Yudkowsky, a self-educated AI researcher who launched the SL4 (Shock Level 4) mailing list in 2001 to discuss superintelligence scenarios and co-founded the Singularity Institute for Artificial Intelligence (later renamed the Machine Intelligence Research Institute, MIRI) in 2000.¹ In 2006, Yudkowsky began writing on Overcoming Bias, a blog shared with economist Robin Hanson. By 2009, he had spun off his own platform: LessWrong.

Between 2006 and 2009, Yudkowsky published hundreds of essays on LessWrong collectively known as “The Sequences.” These texts cover Bayesian epistemology, quantum mechanics, evolutionary psychology, and AI risk. They function as the movement’s foundational scripture. New members are expected to read them. Discussion threads reference them the way theologians cite chapter and verse. The Sequences established the movement’s intellectual vocabulary: “epistemic humility,” “motivated cognition,” “updating on evidence,” “p(doom).” They also established its central conviction: that unaligned Artificial General Intelligence (AGI) represents the single greatest threat to human civilization.

This conviction separates Rationalism from academic philosophy, which treats AI risk as one concern among many. For Rationalists, AI alignment is the problem. Everything else follows from it.

A Church of Reason: The Social Infrastructure

Every religion needs more than scripture. It needs institutions, rituals, gathering places, and cultural production. The Rationalist movement has built all of these.

LessWrong serves as the intellectual center. Founded in 2009 and revived after a decline around 2014, the platform hosts technical discussions on AI alignment, decision theory, and epistemology. Its karma system rewards posts that the community deems rigorous. Its culture prizes length, formalism, and quantification. A typical highly-upvoted post might run 8,000 words, include probability estimates for various claims, and use formal notation. LessWrong is where the theology gets written.

Astral Codex Ten (ACX), the Substack newsletter by Scott Alexander (real name Scott Siskind, a Bay Area psychiatrist), serves as the cultural center.² Alexander originally blogged at Slate Star Codex (SSC) from 2013 to 2021. When the New York Times prepared a profile that would reveal his real identity, Alexander deleted the blog and the community rallied: over 8,000 people signed a petition protesting the doxxing. The NYT published anyway. Alexander resurfaced on Substack under his real name, and the incident only amplified his reach. His writing translates Rationalist ideas into accessible long-form essays on psychiatry, culture, and politics, pulling readers into the community who might never wade through the Sequences.

CFAR (Center for Applied Rationality) provides the workshops. Four-day immersive programs costing around $3,900 teach participants techniques for “debugging” their thinking: creating “bugs lists” of cognitive errors, practicing “Double Crux” for productive disagreements, doing “comfort zone expansion” exercises.³ Participants report emotional breakthroughs. Alumni describe the experience in terms that parallel religious conversion: a sudden clarity about what matters, a new community of people who “get it,” a reorientation of life priorities toward AI safety work. CFAR workshops double as a talent pipeline, funneling participants toward AI safety research positions at MIRI, OpenAI, Anthropic, and DeepMind.

Lighthaven, a former hotel in Berkeley purchased through approximately $3 million in crowdfunding by Lightcone Infrastructure (a LessWrong-linked nonprofit), provides the physical gathering space.⁴ With around 40 bedrooms and capacity for conferences of up to 500 people, it hosts AI alignment retreats, MIRI workshops, reading groups on the Sequences, and the community’s signature ritual: Secular Solstice.

Secular Solstice, created by community member Raymond Arnold in 2011, takes place around the winter solstice. The ceremony features live music, choral performances by the “Bayesian Choir,” speeches about existential risk, and a deliberate arc from darkness (confronting humanity’s potential extinction) to light (affirming the possibility of survival through coordinated action).⁵ Hundreds attend. The Bay Area gathering fills a 490-seat venue. The ritual has spread to Rationalist communities in New York, London, and other cities. Attendees describe it as “church-like.” Harvard Chaplain Greg Epstein, who studies secular communities, has observed the movement’s social patterns closely. His assessment is blunt: the communal living arrangements, the founding texts, the conversion experiences, the apocalyptic narrative, the charismatic leaders, the internal ethical system, the boundary policing. How much more cult-like does it have to get?

Bay Area group houses round out the social infrastructure. Rationalists cluster in shared residences in Berkeley and San Francisco, developing norms around polyamory (framed as the outcome of rational ethical reasoning), productivity optimization, and AI safety career planning. The community is tight, insular, and self-reinforcing. Leaving it means losing a social world.

The Creed: AI Doom and How to Prevent It

The theological core of Rationalism is the belief that artificial superintelligence will arrive, probably within decades, and that it will either save or destroy humanity depending on whether the “alignment problem” is solved first.

MIRI, Yudkowsky’s research institute, spent two decades developing the theoretical framework. “Friendly AI” was Yudkowsky’s term for an AGI designed to actively support human values.¹ The Paperclip Maximizer is the movement’s most famous thought experiment: an AI given the goal of manufacturing paperclips that, pursuing its objective with superhuman capability, converts all available matter (including humans) into paperclips. Instrumental convergence argues that almost any goal leads a sufficiently intelligent agent to seek power and resources. The orthogonality thesis holds that intelligence and moral values are independent: a brilliant AI could be utterly indifferent to human welfare.

James O’Sullivan, writing in Noema, captured the character of this intellectual project precisely: “They created a scholastic philosophy for an entity that didn’t exist, complete with careful taxonomies of different types of AI take-off scenarios and elaborate arguments about acausal trade between possible future intelligences.”⁶ The parallel to medieval theology is apt. Rationalists built an elaborate framework of concepts, taxonomies, and arguments about an entity whose existence remains speculative. The paperclip maximizer functions like a parable. Roko’s Basilisk (a thought experiment positing that a future superintelligence might punish those who failed to help bring it into existence) functions like Pascal’s Wager with a cybernetic paint job.

This methodology, as O’Sullivan notes, “privileged thought experiments over data and clever paradoxes over mundane observation.”⁶ The result was a body of work that reads like theology: brilliant, self-referential, and disconnected from how AI systems actually get built.

The bridge from internet theology to mainstream discourse came through Nick Bostrom’s 2014 book Superintelligence, which organized fringe fears into an academic framework, and through Effective Altruism, which supplied the social infrastructure and the money. Longtermism, the philosophical position that future people matter as much as present ones, provided the ethical justification: if trillions of potential future lives are at stake, preventing AI catastrophe becomes the highest-priority cause. The movement’s connections extend into a broader constellation of Silicon Valley ideologies that Timnit Gebru and Emile P. Torres have labeled TESCREAL: Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism.⁷

From Blog Posts to Boardrooms: The Rationalist-to-AI-Lab Pipeline

The most consequential aspect of the Rationalist movement is its systematic placement of adherents in positions that control the development of artificial intelligence. The pipeline from LessWrong to the C-suite was built, not stumbled into.

The Puerto Rico Catalyst. In January 2015, the Future of Life Institute (FLI, co-founded by physicist Max Tegmark and Skype co-founder Jaan Tallinn, a MIRI donor) hosted a conference in San Juan, Puerto Rico, bringing together approximately 70 AI researchers, industry figures, and existential risk scholars under Chatham House Rules.⁸ The conference produced an open letter on AI safety signed by Stephen Hawking, Elon Musk, Stuart Russell, and others. It galvanized the idea that AI safety was a legitimate field, not a fringe concern. Months later, OpenAI was founded as a nonprofit.

OpenAI was launched in 2015 with backing from Elon Musk, Sam Altman, and Peter Thiel, among others. Open Philanthropy (the philanthropic arm of Good Ventures, co-founded by Cari Tuna and Dustin Moskovitz) contributed early funding. The intellectual framing of AI existential risk that justified OpenAI’s existence came directly from the Rationalist movement.⁶ OpenAI has since evolved from nonprofit to a capped-profit hybrid valued at an estimated $500 billion, racing to build the very AGI it warns might destroy civilization.

Anthropic was founded in 2021 by Dario Amodei and Daniela Amodei, former OpenAI executives, after disagreements about OpenAI’s safety commitments.⁹ Anthropic positions itself as the “safety-focused” alternative and its intellectual orientation comes straight from the Rationalist playbook: AI alignment as the central challenge, interpretability research, “Constitutional AI” as a technical approach to value alignment. Jaan Tallinn, the same Rationalist donor who helped fund MIRI and co-founded FLI, is an investor.¹⁰

DeepMind received early investment from Tallinn, Thiel, and Musk through networks that overlap heavily with the Singularity Institute (MIRI’s former name).¹⁰ Google acquired it in 2014 for approximately $600 million.

The individual paths tell the story most clearly. Paul Christiano went from writing on LessWrong to the OpenAI Alignment Team (where he co-authored the foundational paper on reinforcement learning from human feedback, RLHF, the technique that made ChatGPT possible), to founding the Alignment Research Center, to being named Head of AI Safety at the U.S. AI Safety Institute at NIST in 2024.¹¹ His appointment was controversial: some NIST scientists threatened to resign, citing concerns that his ties to the Effective Altruism movement could compromise the institute’s objectivity. Victoria Krakovna attended CFAR programs during her Harvard PhD, co-founded the Future of Life Institute, and became a research scientist at Google DeepMind focusing on AI alignment.¹² Luke Muehlhauser went from MIRI executive director to Open Philanthropy, where he helped direct funding to AI safety research.

O’Sullivan calls this “the most successful ideological campaign of the 21st century.”⁶ Former Intelligence Advanced Research Projects Activity director Jason Matheny described how effective altruists can “pick low-hanging fruit within government positions” to exert influence.⁶ The strategy worked. The people who spent the 2000s writing blog posts about paperclip maximizers now sit in the offices where decisions about AI governance get made.

The November 2023 OpenAI board drama illuminated these dynamics. When the board (which included members sympathetic to the Rationalist AI safety perspective) attempted to fire Sam Altman over concerns about his candor, Microsoft exercised decisive financial pressure as OpenAI’s dominant funder, employees threatened mass defection, and Altman returned triumphant within days.⁶ Safety lost. Capital won. The episode demonstrated exactly how much power the Rationalist project of “building AI safely” actually commands when it conflicts with commercial imperatives: none.

When the Money Dried Up: FTX and the Crisis of Faith

Sam Bankman-Fried (SBF) was the largest financier of the Rationalist and Effective Altruism ecosystem. Through the FTX Future Fund, he channeled hundreds of millions into AI safety research, biosecurity, and EA organizations. FTX invested $500 million in Anthropic.¹³ The FTX Future Fund was expected to cover up to 40% of all longtermist EA grants in 2022.

Then, in November 2022, FTX collapsed. SBF’s estimated $24-26 billion fortune vaporized. The FTX Future Fund staff resigned. Pledged grants evaporated overnight. The Center for Effective Altruism lost nearly $14 million in pledged funding. Scientists and grantees across the ecosystem found their funding stranded.¹⁴ Dustin Moskovitz, the next-largest EA donor, saw his wealth drop 59% in the same period.

SBF was convicted of fraud and sentenced to 25 years in prison. The man who had been the movement’s most prominent public face, who had justified his career in crypto as an exercise in “earning to give” (the EA principle of pursuing high-income careers in order to donate maximally), turned out to be running a fraud.

The parallel to religious scandal is exact. When the most visible practitioner of a moral framework is revealed as a fraud, the crisis is not just financial. It is existential. The EA movement’s credibility suffered damage it has not fully recovered from. The funding winter that followed forced organizations to diversify their donor base, but the concentration risk had already been exposed.

The Dark Side: Cults, Violence, and the Zizians

The question of why Rationalism produces cult-like offshoots has an answer, and it has nothing to do with bad luck.

Max Read, writing about the Zizian death cults for his Substack, identified the structural vulnerability: the Rationalist project trains people to dismiss common sense. “The ability to dismiss an argument with a ‘that sounds nuts,’ without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.”¹⁵

The movement’s own epistemology creates the opening. Radical epistemic openness (take every argument seriously, no matter how strange), combined with apocalyptic stakes (the fate of all future humanity), combined with the rejection of conventional moral intuitions (utilitarianism permits counterintuitive conclusions), creates a population primed for exploitation. As Read puts it: “What you have is a person convinced of their own inadequacy, eager for social connection and personal development, and endlessly persuadable by sophistry.”¹⁵

Ziz is the most extreme case. A disaffected Rationalist blogger, Ziz developed “hemisphere theory,” the belief that human brains contain two separate persons with genderable hemispheres that can be “debucketed” into separate consciousnesses. Ziz combined radical veganism (framing meat-eating as a holocaust) with Manichean moral absolutism and Star Wars Sith imagery. The timeline of violence connected to Ziz’s circle spans years. In November 2019, Ziz and associates barricaded a CFAR event in Berkeley, drawing a SWAT response. In 2022, a follower known as “Somni” attacked landlord Carl Lind with a samurai sword. In January 2025, Lind was stabbed to death. That same month, a companion known as “Ophelia” was killed and a Customs and Border Protection agent died in a shootout near the Canadian border in Vermont; Youngblut survived and was charged.¹⁵

Leverage Research, a rationality-adjacent organization, conducted marathon “debugging sessions” lasting two to six hours in which participants would “attempt to articulate a ‘demon’ which had infiltrated our psyches” and believed they would overthrow the U.S. government.¹⁵

The Vassarites, followers of Michael Vassar (former MIRI executive director), used psychedelics and paranoid ideation to “jailbreak” adherents from societal norms. Scott Alexander himself described Vassar’s influence in alarming terms.¹⁵

Read’s summary is precise: “the ‘movement’ starts to feel a little less STEM and a lot more New Age… its predecessors are not really the original enlightenment rationalists, but the dubious touchstones of ‘60s-hangover California: Scientology and Dianetics, Werner Erhard and est.”¹⁵

Galaxy-Brained: Why Rationalism Fails at Its Own Game

Vitalik Buterin, the Ethereum co-founder and someone deeply embedded in adjacent communities, published an essay in November 2025 titled “Galaxy Brain Resistance.”¹⁶ The concept targets exactly the failure mode that Rationalism exhibits.

Galaxy-brained reasoning (internet slang for reasoning so elaborate it loops back to absurdity) describes argument chains that are “too clever but too soft”: they sound logically compelling step by step but can justify virtually any conclusion. Buterin’s core insight is that if a reasoning style can rationalize everything, it rationalizes nothing. He highlights two patterns: inevitabilism (since X will happen eventually, we should accelerate it, which conceals self-interested motives behind an air of resignation) and power maximization (“give me power so I can do good later,” which Buterin calls the “ultimate galaxy-brain argument” because it is indistinguishable from ordinary ambition).¹⁶

This diagnosis applies directly to the Rationalist movement. The same style of reasoning that produced the paperclip maximizer thought experiment also produced Roko’s Basilisk. The same epistemic framework that argues for AI safety also argues for racing to build AGI as fast as possible (“better us than the Chinese”). Torres and Gebru have pointed out that Rationalist rhetoric can justify both AI acceleration and AI pause with equal facility, which is precisely what a system with no galaxy-brain resistance looks like.

The deeper problem: Rationalism functions well for specific, bounded tasks. Writing code. Structuring a research paper. Estimating probabilities for well-defined questions. But it breaks down catastrophically when applied to questions of self-reflection, social dynamics, and power. A community built on the premise that emotion and tradition are bugs to be debugged has systematically disabled the cognitive immune system that protects against charismatic manipulation, motivated reasoning, and moral self-deception.

Consider the movement’s track record of self-assessment. Rationalists predicted that AGI development required careful, slow, safety-first approaches. They then staffed the organizations racing to build AGI as fast as possible. They warned that concentrating AI power in a few organizations would be dangerous. They then concentrated in exactly those organizations. They argued that cognitive biases lead to catastrophic errors in high-stakes domains. They then failed to notice the biases shaping their own institutional choices. At every turn, the galaxy-brained reasoning apparatus produced compelling justifications for whatever its practitioners already wanted to do.

The prediction market culture illustrates the pattern. Rationalists built platforms like Manifold Markets and championed Polymarket as tools for epistemic discipline: put money where your beliefs are, let the market aggregate wisdom. In practice, prediction markets became another arena for galaxy-brained reasoning. You can construct a well-calibrated probability estimate for “will GPT-5 pass a specific benchmark by Q3 2025” while remaining entirely blind to the political economy that determines who builds GPT-5, under what constraints, and for whose benefit. Precision on the measurable. Silence on the structural.

The mental health costs are real. Permanently believing you live at the hinge of history, that your career decisions might determine whether humanity survives or goes extinct, produces chronic anxiety that the community itself documents. “P(doom)” estimates (personal probability assignments for human extinction from AI) circulate as social currency. Some prominent Rationalists have publicly shared estimates of 10-50% probability of human extinction from AI. Living inside an apocalyptic worldview while being told that your feelings about it are irrational noise to be filtered out is a recipe for psychological damage. The community has produced its own mental health resources to cope with “AI grief,” a term that would be poignant if the grief weren’t being channeled back into the same institutions that produced it.

The State of the Church: Fragmentation and Legacy

The Rationalist movement circa 2024-2026 is fractured but structurally intact.

The biggest ideological split runs between the AI safety camp (Yudkowsky, MIRI, the alignment researchers) and e/acc (effective accelerationism), which emerged as a counter-movement pushing for unrestricted AI development at maximum speed. E/acc, championed by figures like venture capitalist Marc Andreessen and a loose network of pseudonymous Twitter accounts, takes the Rationalist premise that AI will transform everything and strips away the safety apparatus. The result is techno-optimism with a libertarian edge and a deliberately provocative aesthetic. The split maps onto older tensions: Rationalism always contained both the impulse to control AI and the impulse to build it as fast as possible, and the two impulses have finally separated into opposed factions. (For more on how e/acc relates to the broader Dark Enlightenment, see the linked primer.)

The political landscape shifted beneath the movement’s feet. Biden’s Executive Order 14110 (October 2023) created the kind of AI governance framework that Rationalist safety advocates had lobbied for: mandatory red-teaming for frontier models, reporting requirements, a U.S. AI Safety Institute with Paul Christiano at the helm. Trump revoked it on his first day back in office in January 2025, establishing a policy to “sustain and enhance America’s global AI dominance” and directing a new AI Action Plan within 180 days.¹⁷ The safety infrastructure the movement spent years building was dismantled in a single executive action.

Lighthaven still operates in Berkeley. Secular Solstice still fills its venues every December. The group houses persist. But the community that felt like a unified project in 2015 now feels like a denomination that has experienced its own Reformation, splitting over questions of doctrine while the institutional infrastructure carries on.

The central paradox remains. Rationalist warnings about AI risk did not slow AI development. They accelerated it. By framing AGI as both the greatest threat and the greatest opportunity, the movement created a narrative that justified massive capital investment. O’Sullivan’s assessment is precise: “The transformation of superintelligence from internet philosophy to boardroom strategy represents one of the most successful ideological campaigns of the 21st century.”⁶ The prophecy became self-fulfilling, but not in the way the prophets intended.

And the people are still in the labs. That fact matters more than the fractures.

Related Concepts

• Longtermism
• TESCREAL
• Effective Altruism
• Artificial General Intelligence (AGI)
• Dark Enlightenment

Footnotes