The Rationalist Canon - Pope v Caliph - Round One

History Can Be Cruel

[audio version of this post available here, courtesy of AskWho Cast AI]

Now that I’m in the Bay Area, I’m attending the weekly Less Wrong Sequence Readings at Lighthaven. Our last reading contained two posts on the same subject from the two most prominent Rationalist writers: Eliezer Yudkowsky and Scott Alexander. This allows us to compare their styles, offer commentary on the legacy of the posts, and adjust our own writing based on these insights (if we wish).

Eliezer wrote the foundational text of Bayesian Rationalism (the Less Wrong Sequences) and is often jokingly called the “Pope” of Rationality.

Scott is the most popular and influential writer in Rationalism of the past 10 years, and is sometimes jokingly called the “Rightful Caliph” of Rationality.

The topic is how we (humans) group concepts in our minds.

Eliezer Yudkowsky

For the purposes of the reading group, Eliezer’s primary post on this topic is The Cluster Structure of Thingspace¹.

The first thing that jumps out is how precisely clarifying Eliezer’s original LW posts were. They would often take nebulous concepts that we “knew” in some way, but never in a rigorous way, and made them legible to our “System 2” brain. Eliezer took nebulous latent knowledge and crystalized it into something manipulable, with shocking efficiency. Often he offered convenient labels to give one handles with which to talk about it thereafter (eg “clusters in thingspace”). The ability to do this is both so rare and so valuable that he’s recognized as among the most influential writers of his time.

The second thing that jumps out is how much these posts demand of their readers. This is brought to attention by contrast with the invitation… the invitation to the weekly readings states each post will take ~5-6 minutes to read. This is a lie². It takes 5-6 minutes to run your eyes across all the words. To actually read a post is a minimum of 15 minutes. The post requires active engagement to understand, you must do work to get anything from it. [EDIT: official wording is now 15-20 min to reread 4 posts and 30-60 min for a first-time read. Partial victory! I still think this is optimistic for many posts, but maybe I’m at the lowest edge of ability to grok the Sequences and it takes me extra time to catch up to the true target audience]

This was easy to do when there was one post per day (or less) and each was eagerly anticipated. It’s daunting if one finds the sequences now and sees the vast edifice of work before them. It is, however, worth it. The engagement is part of what makes the solidification of the knowledge within your mind possible, and ensures the concept will be retained longer and on a deeper level. The Sequences wouldn’t work any other way, and it is good that Eliezer knows his target audience, doesn’t make excuses for them, and doesn’t destroy the value by trying to pander to everyone.

To expand on this point: it’s not that Eliezer is forcing you to do cognitive work because there is “virtue” to working hard. When working out, lifting many heavy things repeatedly isn’t virtuous because it is hard. It is literally the mechanism that builds muscle. Eliezer could write a simpler explanation with more words that requires less effort. This would be similar to a schoolchild being given the explanation for what multiplication is, why it works, and how it works, and then moving on to a different topic. One could do this in a single afternoon. It would leave the child without the foundation needed for more complicated math. Drilling multiplication tables is literally the mechanism that makes the recall effortless (except for the late 7s. grrrrr!) and makes building higher possible. The brain must be engaged like the muscles are engaged for there to be enough foundation to go harder.

The final thing that jumps out, highlighted by the contrast with Scott Alexander’s post, is how politically-agnostic the post is. Eliezer took his own advice and stayed the hell away from any political or personal agenda. This denies him the opportunity to grab high-valence attention hooks inside his audience, and prevents the post from being seared in via emotional resonance. Plus he doesn’t get to influence culture in an advantageous direction.

On the other hand, it means the post has aged very gracefully. It is as strong and clarifying as the day he wrote it. It is clearly about just one thing - epistemics. It wants to help you think better, and give you tools for that, and nothing else. I can enthusiastically recommend this post to literally any human that wants to think better, regardless of any specific beliefs or dogmas they may hold. I think this will remain the case for as long as humans-as-currently-exist are still around. That’s incredibly valuable. I didn’t realize how valuable at the time, and I’m grateful now for Eliezer’s foresight. More on this later in the post.

Scott Alexander

For the purposes of the reading group, Scott’s primary post on this topic is The Categories Were Made For Man, Not Man For The Categories.

The first thing that jumps out is how delightful Scott’s writing is. Eliezer’s posts are a mental gym. Scott’s are a playground. He ends up covering the same concepts, but his writing invites you to play. It’s like joking and chatting with your friends. You will laugh, usually more than once. You will be charmed by his wit and insight. There are stand-up comedians who would kill for this level of personable relating.

Vitally, he does this not just to entertain, but to get across a solid intellectual concept. Like “clusters in thingspace.” Because the stories are so relatable they are also easy to remember. Who’s going to forget King Solomon pointing out you didn’t know the word dag ten minutes ago while he’s been using it his whole life so maybe you aren’t the authority you think you are?

Thus, even though the wordcount of Scott’s posts are many times those of Eliezer’s (this single post of Scott’s was longer than all three Yudkowsky posts combined), they are a lot easier to digest. The words spent on joking and story telling make everything go down easier. These posts don’t demand that I slow down, churn over the thoughts and concepts, and try to integrate them manually. Rather, I find that I want to keep reading, eager to get to the next line and next point. It’s just plain fun, and that’s awesome. (And on a purely video-game-trained level, it feels better to have spent 20 minutes reading 5000 words than 20 minutes reading 1200 words.)

This also makes it very easy to recommend. I’m not telling someone “here’s a very insightful task you’ll be happy you worked through afterwards,” I’m telling them “here’s something fun!” It’s more approachable.

Perhaps more importantly, this makes it much easier to personally repeat Scott’s argument when it comes up in conversation. This is an story about King Solomon that’s fun to relate, and easy to remember. One can explain the insight to their friends at a party! Trying to guide someone through a Less Wrong workout during a party is a hard ask for both the speaker and the listener — frequently it’ll be a lossy summary instead, which doesn’t transfer the knowledge very well (if at all). Thus the common refrain of “You should read the post” (or “you should read the Sequences”). A Scott post embeds the work in the narrative. :)

Scott, however, isn’t writing this post just to teach people about epistemology. The epistemology lesson is there because it’s necessary for the ultimate point of the post, which is an application of that lesson on an emotional topic that is important to Scott. This is in theory fine on at least two different levels.

The first is that this actually helps with conveying the concept. It grabs readily-available attention-hooks that are already primed to latch onto related arguments. It makes the lesson emotionally important immediately, in addition to being entertaining and education. Additionally, it allows the lesson to be directly applicable to something in the reader’s present daily experience. This is valuable in its own right!

Secondly, one is allowed to make good arguments for things that are important to oneself! Making good arguments for stuff is a valuable service for all humanity regardless of the subject of the argument or the object-level position of the arguer. If one has to teach some epistemology along the way to make that point, that’s just pure added bonus. Now we have one more good argument and some more people know how to think about reality better. There are many people that never read Eliezer so having that core lesson, taught in a different style so that it can reach more people, is a great boon.

The major downside is that while rules of good thinking don’t change, political realities do, and these changes can have unfortunate consequences. More on that later in this post.

Major Divergence - Simple Boundaries

The biggest point of divergence between Eliezer and Scott is their opinion on simple boundaries. Eliezer believes simple boundaries are not just important, but absolutely crucial to clean epistemology. In Superexponential Conceptspace, And Simple Words he warns the boundaries of clusters in thingspace must be simple—

Otherwise you would just gerrymander Thingspace. You would create really odd noncontiguous boundaries that collected the observed examples, examples that couldn't be described in any shorter message than your observations themselves

and

If you don't draw simple boundaries around your experiences, you can't do inference with them. […] the probability distribution comes from drawing the boundaries, not the other way around

Inference itself depends on simple boundaries, and the more convoluted and gerrymandered you carve up boundaries, the less any inference is meaningful. This tool for understanding the universe depends on good epistemics at a fundamental level.

Scott on the other hand makes a case for gerrymandering category boundaries. He spent half of this post speaking eloquently about how the ways that we humans cluster things is a decision we make for practical purposes. His example being that for many purposes, clustering fish as swimming-animals-of-the-water is the best cluster, and allows us to make the most powerful/fast inferences. Since humans determine where to draw boundaries between clusters based on what is useful, we should gerrymander boundaries when it is more useful to do so. He gives quite a few examples, many humorous and memorable, and usually with the very powerful consequence of preventing millions of humans from slaughtering each other in wars. A powerful argument to be sure.

He extends this to mental health treatments, and argues for gerrymandering thingspace in a way that he feels would maximize health outcomes across the population. This part… it ended up aging poorly.

History is Cruel

While re-reading Scott’s Categories post I was surprised to find how dishonest it felt³. This was shocking both because I remembered it being a great post, and because I view Scott as one of the most scrupulously honest people on the internet. What the heck was going on here?

I checked the date of the post. 2014. A decade in the past. It can be hard to remember how long ago I first encountered an idea, and just as hard to recall how much reality has shifted in ten years.

In Categories, Scott presents a woman that has to be institutionalized to prevent self-mutilation due to suicidal levels of body dysphoria as a central example of a trans person. Scott compares the level of debilitation to schizophrenia and asks us to “imagine if we could ameliorate schizophrenia with one safe simple surgery,” and rightly points out that discovery would win Noble prizes.

Back in 2014, this was likely quite close to a central example of tranness, especially for someone working in psychiatry. But in October 2024 this is a very non-central example, especially to an average member of the public. Someone treating it as a central example to a general audience would be rightly accused of making The Worst Argument In The World.

In the post Scott asks us to “imagine that we could make a serious dent in bipolar disorder just by calling people different pronouns. I’m pretty sure the entire mental health field would join together in bludgeoning anybody who refused to do that.” And he’s right! Bipolar is life-destroying! An intervention this simple, especially when “compared to the alternative of “nothing else works”” would be revolutionary.

Again, in 2014 it’s possible this was an accurate representation of the state of the populace. I certainly agreed with him on every point back then. In fact, I still agree that in the cases presented we should do everything he says⁴. But in 2024 it’s laughable to suggest that the median person who demands non-fitting pronouns is asking for treatment akin to lithium for bipolar. The median such person in 2024 is not helped by this in any meaningful way. The primary effect of their demands is poisoning the groundwater of public trust so that the trans people of the 2014-Categories-Variety are viewed with apprehension at best and malice at worst.

Scott didn’t know any of this would happen from back in 2014. An argument that was presented in order to make the world better now feels like a slight-of-hand used to sneak in harmful behavior under the guise of epistemic hygiene. If a reader doesn’t look at the date and remind themselves how different the world looked a decade ago they would come away with a far worse view of both Scott and concern about epistemology itself.

Regardless of how words and movements get hijacked, that is the real danger. Tying concern for epistemology to the shifting sands of political struggles leaves the two linked. The more skilled and effective the writer is, the stronger the tie is. This tie can become a noose if the political movement degrades into something awful. Epistemology itself can become suspect if this is what its use is associated with.

Scott tried to do a good thing in 2014. This wasn’t his fault. The future is impossible to predict. Any valid reading must recognize times change, everything written is an artifact of it’s era, we can only strive to do the best with the knowledge we have in an uncertain world, and we best be darn careful not to judge past actions outside of their context.

But it is now clear that breaking epistemics just a little bit, just to do a very good thing that would help a lot of very suffering people was in fact a big mistake. The tool of inference was irrevocably damaged in the given domain. Trust in systematized reasoning has been damaged. The even the actual net-benefits may now be outweighed by the legit harms on the object level. It’s not a Fauci-level mistake, but it’s the much higher-level version of that same mistake that a much-smarter Fauci would have made, in a much more interesting way.

It really cements in my mind the wisdom of a 20-something Eliezer to intentionally eschew political and cultural commentary to focus purely on epistemology.⁵ He wouldn’t risk tying the legacy of his core work on the vagaries of future political dogfights, no matter the temptation to do good. It’s legit awe-inspiring. Never compromise on the basic fundamentals of truth and epistemology. No matter how good the cause, it’s never worth it.

To me the value of unblemished epistemology outweighs accessibility. This would be a harder call if history hadn’t broken in the unfortunate direction that it has. But with that value thrown into stark relief, my call in that this round goes to Eliezer.

1. We also read How An Algorithm Feels From The Inside and Superexponential Conceptspace, And Simple Words. In addition, all three posts draw on concepts that he previously outlined and then further developed in preceding posts.

2. So far I’ve pointed this out to one LH-adjacent person in each of my two attendances, I hope to keep doing this until someone who can change this comes to agree that lying to the invitees is net-negative in the long run and changes it. Yes, an hour of reading is a lot, but reality is what it is.

3. I initially said “I was surprised by how dishonest it was.” But due to having just read How An Algorithm Feels From The Inside I was able to immediately realize that this was an artifact of my perception. It felt dishonest from the inside of my algorithm. This realization is what prompted me to look for explanations of why I felt this way, and looking at the publication date.

4. Heck, I’m firmly of the position that adults can do anything to their bodies that they want to. They can and should be allowed to buy any hormones over the counter without questions, and the age of majority is currently way too high so in an ideal world people would be allowed to do this years earlier than presently.

5. Yes, I’m aware of the irony that (AFAIK) he agrees with Scott on the object level here