10 min read7th Jan 201874 comments

75

tldr: a Demon Thread is a discussion where everything is subtly warping towards aggression and confusion (i.e. as if people are under demonic influence), even if people are well intentioned and on the same 'side.' You can see a demon thread coming in advance, but it's still hard to do anything about.

("Flame Wars" are similar but I felt the connotation was more like "everything has already gone to hell, and people aren't even pretending to be on the same side")


I kept wanting to reference this post when discussing internet discussion policy, and I kept forgetting that nobody has written it yet. So here it is.

Suggested Background Reading:

If someone in the future linked you to this post, it's probably because a giant sprawling mess of angry, confused comments is happening - or is about to happen - and it's going to waste a lot of time, make people upset, and probably less likely to listen to each other about whatever the conversation ostensibly is about.

I have some ideas on what to do instead, which I discuss in this followup post.

But for now, this post is meant to open a discussion, explore the mechanics of how demon threads work, and then in the comments brainstorm solutions about how to handle them.

Wrong On the Internet

I find “Someone Is Wrong On the Internet” to be a weird, specific feeling.

It's distinct from someone being factually wrong - people can be wrong, point it out, and hash out their disagreements without a problem. But a common pattern I’ve witnessed (and experienced) is to notice someone being wrong in a way that feels distinctly bad, like if you don’t correct them, something precious will get trampled over.

This is when people seem most prone to jump into the comments, and it’s when I think people should be most careful.

Sometimes there actually is an important thing at stake.

There usually isn’t.

It often feels like there is, because our social intuitions were honed for tribes of a hundred or two, instead of a world of 7 billion. We live in a different world now. If you actually want to have an impact on society, yelling at each other on the internet is almost certainly not the best way to do so.

When there actually is something important at stake, I think there are usually better plans than “get into a giant internet argument.” Think about what your goals are. Devise a plan that actually seems like it might help.

Different situations call for different plans. For now, I want to talk about the common anti-pattern that often happens instead.

Demon Threads are explosive, frustrating, many-tentacled conversations that draw people in regardless of how important they are. They come in two forms:

  • Benign Demon Threads are mostly time wasting. Nobody gets that angry, it's just a frustrated mess of "you're wrong" "no you're wrong" and then people spend loads of digital ink arguing about something that doesn't matter much.
  • Malignant Demon Threads feed upon emotions of defensiveness, anger, tribal affiliation and righteousness - and inflame those emotions, drawing more people into the fire.

(A malignant demon thread is cousin to the flame war - people hurling pure insults at each other. What makes a malignant demon thread insideous is the way it can warp discussion even among people who are earnestly trying to communicate, seek truth and solve problems)

If you find yourself in a malignant demon thread, I think it's likely you are not only not helping, but are actually hurting your cause.

The Demon Seed

How to write so that people will comment [disclaimer: not necessarily good advice]

1. Be wrong
2. Be controversial
3. Write things people feel qualified to have opinions on.
4. Invoke social reality.
- Writing That Provokes Comments

In the comments on YouTube, or the worst parts of Facebook or tumblr, demon threads are not surprising. People write comments that inflame ideological warfare all the time. Internets be internets. People be people. What can you do?

The surprising thing is how this works in places where everyone should really know better. The powers of demons are devious and subtle.

There's an experiment — insert obligatory replication crisis disclaimer — where one participant is told to gently poke another participant. The second participant is told to poke the first participant the same amount the first person poked them.

It turns out people tend to poke back slightly harder than they were first poked.

Repeat.

A few iterations later, they are striking each other really hard.

I think something like this is at work in the mechanics of demon threads.

The Demon Seed is the first comment in what will soon become a demon thread. It might look pretty innocuous. Maybe it feels slightly rude, or slightly oblivious, or pushing a conversation that should be about concrete empirical facts slightly towards being about social consensus (or, vice versa?).

It feels 1% outside the bound of a reasonable comment.

And then someone waters the demon seed. They don't want to let the point stand, so they respond with what seems like a fair rebuke.

Maybe they're self-aware that they're feeling annoyed, so they intentionally dial back the aggression of their response. "Ah, actually this probably comes across as too hostile, so I'll tweak my wording to reduce hostility by 4%." But, actually, the words were 6% more hostile than they thought, and now they've escalated 2%.

Repeat 2-3 times. The demon seed is watered. Latent underlying disagreements about how to think properly... or ideal social norms... or which coalitions should be highest status... or pure, simple you're insulting me and I'm angry

They have festered and now they are ready to explode.

Then someone makes a comment that pushes things over the edge, and a demon thread is born.

(It is, of course, possible to skip steps 1-4 and just write a blatantly rude, incendiary comment. I'm trying to describe how this happens even when everyone is well intentioned and mostly trusts each other)

From there, if you're lucky it's contained to two people. But often, well meaning bystanders will wander by and think "Ah! People are being wrong on the internet! Wrong about things I am qualified to have opinions on! I can help!"

And it grows.

Then people start linking it from elsewhere, or FB algorithms start sharing it because people are commenting so the thread must be important.

It grows further.

And it consumes days of people's attention and emotional energy. More importantly, it often entrenches people's current opinions, and it burns people's good will that they might have been willing to spend on honest, cooperative discourse.

Why Demon Threads are Bad

I think demon threads are not just a bad plan - I think they are often net negative plan.

The reason is best expressed in Conor Moreton's Idea Innoculation and Inferential Distance. [Edit: the full article is no longer available]

Inferential distance is the gap between [your hypotheses and world model], and [my hypotheses and world model]. It's just how far out we have to reach to one another in order to understand each other.
If you share political and intellectual and cultural foundations, it's (relatively) easy. If you have completely different values and assumptions, (say you get dropped off in the 15th century and need to argue with Christopher Columbus) it may be nigh impossible.
It's right in the name—inferential distance. It's not about the "what" so much as it is about the "how"—how you infer new conclusions from a given set of information. When there's a large inferential distance between you and someone else, you don't just disagree on the object level, you also often disagree about what counts as evidence, what counts as logic, and what counts as self-evident truth.
What makes this really bad is idea inoculation.
When a person is exposed to a weak, badly-argued, or uncanny-valley version of an idea, they afterwards are inoculated against stronger, better versions of that idea. The analogy to vaccines is extremely apt—your brain is attempting to conserve energy and distill patterns of inference, and once it gets the shape of an idea and attaches the flag "bullshit" to it, it's ever after going to lean toward attaching that same flag to any idea with a similar shape.
When you combine idea inoculation with inferential distance, you get a recipe for disaster—if your first attempt to bridge the gap fails, your second attempt will also have to overcome the person's rapidly developing resistance.
You might think that each successive attempt will bring you closer to the day that you finally establish common ground and start communicating, but alas—often, each attempt is just increasing their resistance to the core concept, as they build up a library of all the times they saw something like this defeated, proven wrong, made to look silly and naive.

A demon thread is a recipe for bad attempts at communicating. Lots of people are yelling at once. Their defenses are raised. There's a sense that if you give in, you or your people look like losers or villains.

This'll make people worse at listening and communicating.

Why the Internet Worse

"Demon threads" can happen in person, but they're worse online.

One obvious reason is that the internet is more anonymous. This reduces consequences to the person writing a comment, and makes the target of the comment easier to round off to a bad stereotype or an abstract representation of The Enemy.

Other things people do:

A. People end up writing long winded monologues without anyone interrupting them to correct basic, wrong assumptions.

i.e. "you're just wrong because you think X, therefore... [complicated argument]", without providing opportunity for someone to respond "no I don't actually think X at all". And then, having written out [complicated argument] you're already invested in it, despite it being built on faulty premises.

B. Lots of people are writing. Especially as the demon thread grows. After 24 hours of its existence, the thread will have so much content it's a huge investment to actually read everything that's been said.

C. The comments aren't necessarily displayed in order. Or, if they are, people aren't reading them in order, they're reading whatever it's largest or most interesting.

D. The internet is full of lots of other content competing for attention.

This all means that:

E. People are skimming. This is most true when lots of people are writing lengthy monologues, but even when the thread first begins, people's eyes may be bouncing around to different tabs or different threads within a page so they aren't even reading what's being said, not with the intentionality and empathy they would when confronted with a real person in front of them.

And they might first be reading the most explosive, recent parts of a thread rather than piecing together the actual order of escalation, which may make people look less reasonable than they were.

This all adds up to giant threads being a uniquely bad way to resolve nuanced, emotionally fraught issues.

Containment?

Demon threads are like wildfires. Maybe you can put them out, with coordinated effort. You can also try to ignore them and hope they burn themselves out.

But if you wanted to actually stop it, the best bet is to do so is before they're erupted in the first place.

I've developed a sense of what seeds look like. I'll see a comment, think "god, this is going to become a demon thread in like two hours", and then sure enough, two hours later people are yelling at each other and everything is awful and everyone involved seems really sure that they are helping somehow.

Some flags that a demon thread might be about to happen:

Flags Regarding: Tension and Latent Hostility

  • When you look a comment and want to respond, you feel a visceral sense of "you're wrong", or "ugh, those people [from group that annoy me]" or "important principle must be defended!" or “I am literally under attack.”
  • You feel physiological defensiveness or anger - you notice the hairs on the back of your neck or arms standing on end, or a tightness in your chest, or however those emotions manifest in your body.
  • People in the thread seem to be talking past each other.
  • For whatever reason, tensions seem to be escalating.

Flags Regarding: Social Stakes

  • The argument seems like it's about who should be high or low status, which people or groups are virtuous and which are not, etc.
  • The argument is about social norms (in particular if at stake is whether some people will end up feeling unwelcome or uncomfortable in a given community/space that is important to them - this is extremely threatening)
  • More generally - the argument touches in some way on social reality, in ways that might have ramifications beyond the immediate conversation (or that people are afraid might have such ramifications).

If some of the above seem true (in particular, at least one of the first group and at least one of the second), then I think it's worth stepping back and being very careful about how you engage, even if no comment seems especially bad yet.

Potential Solutions

The first line of defense is to notice what’s happening - recognize if you’re feeling defensive or angry or talking past each other. Brienne's Noticing Sequence is pretty good for this (as well as her particular posts on training the skills of Empathy and handling Defensiveness - these may not work for everyone but I found the underlying thought process useful).

But while noticing is necessary, it’s not sufficient.

Rather than list my first guesses here, I’ll be discussing them in the comments and following this up with a “best-seeming of the potential solutions” post.

Meanwhile, some factors to consider as you decide what to do:

  • How are you involved?
    • Are you one of the people initially arguing, or a bystander?
    • How much do you normally trust the people involved?
    • Is it possible to take the conversation private?
  • Are we on the demon seed or demon thread stage? Is there common knowledge about either?
  • What are the actual stakes?
  • What are the moderation tools available to you?
  • Are you in a venue where you have the ability to shape conversational norms?
    • Do you directly control them (i.e. personal blog or feed?)
    • Does anyone have direct ownership of the venue? (either technically, or culturally)
    • Is there anything you can do unilaterally to make the conversation better, or will it require help from others?
  • Are you building a site where you get to develop entire new tools to deal with this class of problem?

With that in mind…

In whatever venues you most find yourself demon-thread-prone, what sort of plans can you actually think of that might actually help?


Note: I have since written a followup post with a working example of what I think people should usually do instead of demon threads.

New to LessWrong?

New Comment
74 comments, sorted by Click to highlight new comments since: Today at 11:23 AM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

One idea to attempt to short-circuit demon threads:

Step 1. Make it easier and normalized to take a conversation private if someone is feeling annoyed/threatened/angry (and it seems like the conversation is actually important).

Step 2. In private chat, they do their best to: communicate honestly, to notice when they are defensive, to productively find the truth as best they can. (I think this is much easier 1-on-1 than in public)

Step 3. Someone writes a short summary of whatever progress they were able to make (and any major outstanding disagreements that remain), focusing primarily on what they learned and rather than "who's right."

The summary should be something both parties endorse. Ideally they'd both sign off on it. If that trivial inconvenience would prevent you from actually writing the post, and you both generally trust each other, I think it's fine to make a good-faith effort to summarize and then correct each other if they missed some points.

Writing such a summary needs to get you as much kudos / feel-good as winning an argument does.

Step 4. The public conversation continues, with the benefit of whatever progress they made in private.

Ideally, this m... (read more)

3Raemon6y
I am curious - this has a lot of upvotes and few comments. I've thought about this a bit and obviously think the suggestion is good, but I expect to run into hidden gotchas. I'm interested if anyone has thoughts on "what are particular snags you'd expect such a policy to run into, both if it were naively implemented by individuals, and if implemented as a widescale policy (either on LW or elsewhere)
1Erfeyah6y
If the people that are discussing do not follow the convention of returning to the comment thread with a summary, or in order to continue the discussion, we will end up with comment threads ending abruptly. On the other hand, this could be seen as addresed by your "if you are not willing to do this work etc." comment. Could be funny though. Maybe, in these cases, the system can add an automated comment stating that "unfortunately the two parties never returned from their private chat..." :P
3Erfeyah6y
This is such a great suggestion. I have even noticed this dynamic in verbal conversations where I will have a perfectly civil and productive conversation with a person until we are part of a larger group. Another interesting thing is that the reverse can happen. A person that disagrees with me in private will support the same point when I defend it to another person in a group setting! Such a clear indication that the person's goal was not learning but getting high on the emotion of winning! Meta: It is not possible to 'move to private' in LW is it?
2Raemon6y
Currently, the PM and notification system isn't working, but getting them working is an upcoming priority, and I think it'd probably be valuable if it was designed in such as way as to make the above suggestion work seamlessly.
0Erfeyah6y
Maybe depending on a threshold number of back and forth comments between two users a check can be made to detect if they are currently logged in. If they are then a chat option can appear next to the reply that directs to a chat window like the one you are using for feedback. Alternatively, the check could even happen automatically when the preson attempts yet another reply, informing them of the etiquette to follow. That is if we get convinced that it is a worthwhile methodology. I have no idea if this would be succesful in practice but it is such a novel idea that it might be worth a test during the beta. Not sure about the implementation complexity though...

My current plan for next time I get into a demon thread (or other drawn-out arguments) is to say something like: "this doesn't feel like a good discussion to keep having, so unless that changes, I'm going to limit myself to two more posts in this thread".

It's hard to just stop outright, for various reasons. Among them: do I reply to their most recent points or not? If not, it looks like I'm giving up as soon as I can't continue; they might feel like I've wasted their time. If I do, I make them choose between "don't reply, appear to have no response" and "do reply, appear to be attacking someone who can't defend themself". I don't like when people tap out so abruptly on me, whether they reply or not, even if I'm glad the discussion is over.

But if I allow myself a couple more replies, I avoid looking like I'm giving up out of weakness, and I let them choose how much more time they want to invest, and if they have something they want a reply to they can still say it and get a reply. We can work together to bring the thread to a conclusion, if that's what they want. (And if they don't, perhap... (read more)

7Raemon6y
Huh - my reaction to your first sentence was "it seems better to just rip the bandaid off", but by the time I finished reading you had me pretty sold (at least as a useful tool to have in my toolkit)
Demon Threads are explosive, frustrating, many-tentacled conversations that feel important but aren’t.

I want to object to this framing, particularly the "but aren't." It's far from clear to me that demon threads are unimportant. It may seem like nothing much happened afterwards, but that could be due to everyone in the thread successfully canceling out everyone else's damage. If that's true it means that no one side can unilaterally back down in a demon thread without the thing they're protecting potentially getting damaged, even while the actual observed outcome of demon threads is that nobody apparently benefited.

(I have a particular example in mind as I write this where I think that I personally partially canceled out potential damage from a demon thread, both on the thread and later in a RL conversation, but I guess it would be in bad taste to go into specifics.)

In this frame the appropriate response to demon seeds is to delete them, so nobody bears the burden of backing down. That might be a little too extreme though.

8Raemon6y
Actually, on second thought, I'm doubling down on "doesn't matter." (Or, "the degree to which it matters is horribly un-coupled from your intuitions about it.") The discussions that first crystalized "demon thread" for me were related to the Effective Altruism world - people were making controversial claims, people and organization's relative status was at stake. And I felt compelled to log in and slog through the comments myself.... ...and then I looked at the effective-altruism.com front page, and all the other threads that were not about juicy social drama... but where the thing at stake was "which intervention will actually save lives / bring about great value to the world if people donated money to it, in a community that is about donating money or taking actions on things that matter." And on one hand, it would have been a lot of less-fun effort to think about the actual "effective altruism" discussions. And there's some case to be made that my marginal contribution wouldn't have affected anything. But, on the other hand... the social stakes in the Drama Thread didn't actually affect me, and in some cases, didn't affect anyone I cared about. There were some people for whom the drama was object-level relevant, and maybe it was right for them to wade in. But it was clearly a waste of my time. I just got swept up into it because of an illusion of mattering. There are different degrees of mattering, and not mattering. There's: * "Literally the President or Congress or Leaders of the industry would have to be paying attention for you for this internet argument to matter." They won't, so it doesn't matter. At all. * "Literally millions of people would need to work in tandem for this social norm to matter." But you aren't strategically engaging millions of people or working in-concert with organizers who are, so it won't matter. At all. * A smallish-dunbar-ish number of people will in fact be affected by this thing, but not anyone you directly care about.
5Qiaochu_Yuan6y
I think I still disagree but I don't know how to productively explain my disagreement without going into object-level examples and details.
3Raemon6y
[Note: Qiaochu and I eventually talked in private, and I wrote up a summary of my takeaways here]
5Raemon6y
This does sound reasonable. I had some thoughts I was planning to write up later on "when are Demon Threads in fact a useful tool you should use on purpose?" (partly because not acknowledging that would be dishonest, and partly because it's actually useful, and partly to highlight that I think most situations are not useful much of the time) Quick thoughts for now: * My goal with this post is to move towards a world where we successively prevent demon threads in the first place, not one where we try to stop them or unilaterally disarm after the fact. This is only possible in places with some minimum threshold of... well, I'll just call it "civility". I think LW can be such a place (and probably you can carve out sections of FB/tumblr to be that, with more effort). * Note that in my suggested-solution-comment, the ideal execution is to double-crux on the issue before it explodes, and then do a joint-post that explains whatever you were able to agree on and/or how to constructively discuss the issue further. * I think the second best solution is, after the seed has exploded into a thread, find the people who are the loudest/highest-profile (or highest-profile-who-are-in-the-stratosphere-of-people-who'd-listen-to-you), double crux with them, and then do a collective effortpost. * I think one of the most useful things demon threads offer in a Civil World is a threat that you will escalate to them, and cause a major bruhaha. If you do so, everyone has to spend a much of time, and the overton window moves only slightly. So just like politics can be cheaper/more efficient than war, a single cooperative effort by people on opposite sides of a dispute might be cheaper/more efficient than a giant controversy (as well as preserving a status quo where people try earnestly to seek truth / not think tribally, which there is tremendous value to)
2Raemon6y
I do want to acknowledge - resolving disputes about the overton window needs to happen somehow in Civil World. I think at least some of those disputes can dissolve in an Archipelago-esque lens. I think a lot of things-that-cause-demon-threads are actually just pointless. (i.e. demon threads about economic policy seem less useful than demon threads about social norms, since the latter actually affect your social group. The former are only useful for tribal signaling, which is probably also important but I think can probably be refactored a bit).
3Raemon6y
(Man, I'm not happy with the connotations of "Civil World", in particular because I'm the one who linked to Civility is Never Neutral at the beginning. Not sure what else to call it. "Rational World" feels, well, differently loaded. "Idealized/Platonic Rational World?")

If the demon thread has two to three participants who know each other, I wonder about the effectiveness of making repair attempts. If one participant says something like "I'm sorry, let me try to say that better," or "I agree with part of what you're saying," or (I don't know) links to a cat picture or something, does it tend to deescalate the situation? I'm not actually sure but I think it's worth trying.

I've found that certain topics predictably degenerate into demon threads (I had an example, but then reconsidered the wisdom of giving it). On my blog, when I'm writing about a topic tangentially related to one of those topics, I will often put up a commenting note like "no discussion of [TOPIC]," which nips that in the bud.

Another reason demon threads sometimes escalate is that there are antisocial persons like myself who really enjoy participating in demon threads. I am not sure what to do about us in the general case. Ideally there'd be some site where we could all argue with each other about extremely unimportant topics.

If people enjoy demon threads, it may not be strictly true that the 'Someone is wrong on the internet' feeling (noticeably) feels bad.

When reading the OP, I thought, "I recognise that feeling, but my main (noticed) 'someone is wrong on the internet'-response is a positive, inspired motivational one."

Perhaps these feelings do get jumbled, and distinguishing how much is 'inspired' vs 'this is wrong' is part of the skill of avoiding demon threads.

I still sense that there's two different feelings here:

Type 1. Clearly negative – "This can't stand" or "That person needs to be corrected" or "If other people see that person's post, they will become wrong too – I need to save them."

Type 2. Positive(?) –"There's some interesting ideas to be corrected" or "Wow, this person thinks really differently from me, how did that happen?"

The second type might have shock and incredulity, but the core feels like surprised curiosity.

The first type feels more uncomfortable, as if tribal honour has been breached.

3Raemon6y
Presumably the exact feelings vary a lot from person to person. I think the potential bad outcome of the positive version is something like "ah, I can explain this!", coupled with a misunderstanding of what the other person is about. (i.e. you think you're making a simple correction, but you're actually telling them that some deep seated part of their identity is wrong)
3ultimape6y
For me, fighting this feeling is really hard without a lot of mindfulness about it. Its interesting to think that one person's Demon Thread is another person's playground? It also suggests there may be a secondary "infferential distance" effect going on, but at an emotional response level vs the cognative model. I'd be curious how you'd describe the enjoyment feeling. I know for me it almost feels like an adrenaline rush when I'm in a heated argument, and it combines with a certain kind of single-minded lucidity that also crops up when I'm in the middle of being in a "flow state". It's not really an anger feeling, but a kind of thrill like doing really well in a sport, or excelling in a compettive video game? An example in the article: Is actually what I experience in these states, but I dont feel anger or rage. Its more like Frisson (and I think my eyes dialate too) Thinking about it, this result may due to being more attuned to my frustraition reflex. I have noticed that if I'm in that frission state and feel frustraited that it expresses itself as anger. But growing up I didn't pay attention to that transition as much. It wasn't until I started practicing a form of mindfulness to help with anxiety that I was able to differentiate them. (as a result, the frustration/anger is the point where I'd tune out of the conversation).
3Raemon6y
I think you can, but difficulty scales with how involved/explosive the thread currently is. (also, thanks for the repair article)
3Raemon6y
Yeah - proactive explicit moderation is a pretty useful tool. Worth noting that I enjoy participating in demon threads, but I try to be deliberate about when and where I'm doing it, and to try to do it for things that don't actually matter that much when I want to blow off steam on the internet. (I also try to do them in spaces that are already Basically Hell, so additional demon threads aren't making things worse) (Facebook and tumblr occupy a space where they contain people with varying degrees of "bothering at this present moment to have a serious conversation", so you can sometimes have real conversation and sometimes have Suddenly Demons Everywhere. I don't know tumblr that well, but on FB there's usually some subtle clues about whether the particular space you're in is a "let's have a cathartic demon thread" place, or a "guys we're trying to have a real conversation" place.)

Often in Demon threads, people are trying to reinforce or disrupt the placement of a belief into a shared narrative as common knowledge. In other words, the argument is often not just about which thing is true, but which side bears the burden of proof. For such cases, recommendations that one side go out of their way to resolve the object-level disagreement amicably (e.g. changing venues and letting the other side's public comments stand uncorrected) are by default recommendations to concede the actual conflict. It's not hard to see why this solution might not be appealing.

6Qiaochu_Yuan6y
Agreed. I think the thing my social instincts are tracking during demon threads is precisely what is or is not entering common knowledge, which I think is in fact important. (Edit: things entering common knowledge include changes in the social status of individuals or groups.)
2ShardPhoenix6y
Agreed, and I think this is why "This will be downvoted but.../This is an unpopular opinion but..." can actually work, because they let you broadcast your opinion without others worrying that it will become common knowledge.
2Raemon6y
Basically agreed, see my response to Qiaochu elsethread.

Obviously a demon thread is a lousy way to get work done, in the sense of "improving the world." But surely people get into these things largely for entertainment value?

When I see a big clusterfuck of a demon thread, my emotional response is "Yay! I can get lots of stimulation from this! People will talk to me!" They will not necessarily talk intelligently and neither will I, but talking will happen and continue vigorously for a while and that is super fun. There are lots of "hooks" for someone to speak. It's wonderful in that sense.

The downside is that there's so much conflict that someone's likely to get hurt in earnest.

There are non-confrontational ways to get conversational stimulation, though. The "poll" or "ask meme" thread where you ask everyone to chime in with their personal experience or preference in response to a prompt is my favorite example. Personality typologies also work for this -- "Which Hogwarts house are you" etc. The mood is friendly but the format has the same property -- everyone has something to say and you can talk for ages. Prompts for people to talk about themselves can explode into huge multi-day threads just like demon threads do, but people are less likely to get hurt or regret the experience.

If you want to throw a big social media party and invite EVERYONE to chime in, might I suggest poll/opinion/personality-type threads?

9Raemon6y
Huh, I thought I'd originally touched on this motivation (that I experience myself), but it looks like I ended up editing it out of the OP. (Although it looks like I ended up saying it in a comment to Ozy) (I have some thoughts about when Demon Threads are the right tool for the job. I actually designed this post in a fashion intended to prompt a controlled, carefully summoned benign demon thread that would get engagement without [much] hurt feelings) I do like your suggestions here of "when you just want the engagement and don't have particular other goals in mind, here are social media party things you can do instead".

In my experience the evolution of demon threads is moderately dependent on the mechanics of commenting, and (to extend the demonic metaphor) "exorcism comments" work differently depending on the mechanical position of new comments.

No matter how commenting works, a comment that "fixes" the bulk of the demon aspects of the larger conversation needs to have clean and coherent insight into whatever the issue is. You shouldn't worry too much about writing such a post unless you are moderately confident that you could pass an ideological... (read more)

4Matt Goldenberg6y
Meta: There's a really cool point in here about HOW TO EXORCISE DEMONS FROM THREADS, without ending the thread. But people may miss it because the bolded text and first few sentences mostly seem to be about technical ideas on commenting. Recommend reading this comment if you skimmed it previously. Non-Meta: I too have noticed a certain tone that's factual, friendly, and non-combative that can seem to take the wind out of demon threads because it somehow disables everyone's defensiveness centers. I think this is probably the best solution to demon threads, and also reflectively useful in that if people get great at this tone, demon threads are less likely to happen in the first place.
4Raemon6y
Agreed with the tone-thing. Re: meta: Oh, I too had read the first three things, and I think read the paragraph after "reddit style commenting" and then sort of thought I was done reading and apparently stopped. I think having a fourth bold title highlighting the exorcising comment concept would have helped.
2JenniferRM6y
Thank you both for the feedback. I've taken the liberty of adding underlining in a second pass edit.

I promoted this to featured for:

  • Useful ideas.
  • Precise writing.
  • Important topic.

Meta thread for commentary on the OP's approach/style. I'd like this to be a good post I can reference a lot. Ideally it'd make it's points more succinctly (and/or be written in such a way that someone skimming it will come away with the right idea, even if they're currently stressed out and pressed for time)

Any feedback on that is appreciated.

I would like for us to stop using this term. I don't think "demon thread" really says much that "terrible thread" doesn't, and while I think some of the observations you've made about these threads are helpful, the introduction of "demon thread" as yet another jargon term is IMO becoming annoying.

Doesn't nearly this entire concept lean toward violating the concept brought up in the third citation, "Civility Is Never Neutral"?

To me, it sounds like you're opening up a path to mark any conversation a fair number of people don't want to have as "a demon seed/thread" and then raising the bar higher on anyone being allowed to discuss such a thing at all. The ultimate outcome is just a chilling effect on controversial speech and (likely) an entrenchment of existing biases.

2Raemon6y
An earlier version of this post went into more details about how to apply the Civility is Never Neutral concept. (That version of the post in general was heavier on "propose solutions", and I cut a bunch of stuff after realizing I didn't understand the problem well enough to confidently propose anything concrete. I posted some of it as a top level comment here, which got heavily upvoted but hasn't had much discussion) I have some half-finished writing which may either become a comment here, or a followup post, which explores the Civility is Never Neutral issue. Meanwhile I wanted people to know that I at least considered it important background reading. In short: there's definitely tension between naive "prevent demon threads" and naive "make sure everyone has the right to be heard." Ignoring the issue doesn't make it go away though. Demon threads are bad. Chilling effects are bad. In some situations, one is worse than the other, and deciding which is which requires judgment calls. My top-level-comment-solution here, is aiming for something like "anti-demon-thread measures don't mean 'don't talk about an issue'. Instead, 'provide avenues to talk about it that don't involve demon-thread dynamics.'" (i.e. talk privately, then summarize the outcome of the conversation for others) For that to work, you need a lot of trust, the trust need to be deserved (i.e. no only are people willing to talk in good faith, they need to be actually be willing to put in the time to do so effectively. If the rule is "hash emotionally-fraught things out in private" but then one party doesn't want to put in the effort, that would indeed result in the same chilling effect. My current goals are to try to flesh out and operationalize how we can create a world (at least on LW) with as low friction as possible to resolving sensitive issues in a healthy way.

This is dispiriting. It implies that if I want to practice writing blog posts without spending too much time on them I should write about topics that aren't too important to me so I don't waste the opportunity to explain something I actually care about in a way that inoculates people against better explanations in the future.

2Qiaochu_Yuan6y
Huh, I don't know what happened, this was supposed to be a comment on Idea Inoculation and Inferential Distance and I don't know how it ended up here.
1habryka6y
Maybe multiple tabs open and you accidentally posted it in into the wrong one? I haven’t experienced anything in this domain, so I will be In the lookout for similar things happening, and see whether it might be some hidden bug.
2Qiaochu_Yuan6y
Just happened to me again and I didn't have multiple tabs open, so I think it's a bug. Both times the page I saw after submitting was the post I'd intended to comment on, but my comment wasn't there.
2Raemon6y
I think this may be a thing where LW stores your comment in local memory (so if your computer crashes or the site reloads due to a new version deploy, you don't lose your comment). I think comments that are parented under a comment get stored properly as a reply to that comment, but maybe global comments get stored as "truly global" rather than parented under a post. And then maybe when you go to a new post, it keeps the old half-written comment, making it easier to accidentally po? (Except that I just tried to replicate this on purpose and couldn't get anything to happen, so maybe this is just wrong. But maybe points in a useful direction?)
1habryka6y
I am actually just debugging a similar problem for private messages. Will try to replicate it for comments. (Basic idea is that we persist the new-comment form between page transitions, and there is a bug where some of the properties of the form don't properly propagate, and this might sometimes result in you being on a new page, with a form that still has the properties of the old page)
1habryka6y
Yep, replicated it. Fixing it right now. Sorry for that happening.
2Raemon6y
Hmm. I'd frame it more as "be aware of the cost of doing so" than "don't do it." My takeaway is more like "if you expect the piece you are writing to be controversial, it's probably better to put it in a place where people who are the right inferential steps away will run into it." (On FB, this would mean maybe filtering it to particular friend groups. On LW2.0, it might mean making it a personal blog or feed post until it's been through a round of comments to iron out the kinks) Bear in mind there's a whole different set of concerns like "value of you having the opportunity to improve as a writer" and "value of one set of people learning from the ideas even if another set of people get innoculated against them." It does involve judgment calls, though.
  1. I think I'm prone to falling into demon threads. Some of the attributes feel very familiar (such as consumed energy, ever growing rudeness, terrible miscommunication, eventual lack of progress) but some feel quite foreign (such as everything in your text involving the substring "soc" or more than two people). Apparently we've all been in such threads, but I wonder if we're all talking about the same things. I wonder if there is meaningful classification of different kinds of demon threads. I wish people actually shared the daemon t
... (read more)
5Benquo6y
3 seems like it would sneak unjustified premises into everything, and make it prohibitively expensive to challenge them. Philosophers already tried this, and all we got was analytic philosophy, which is not very interesting, and still doesn't do anything to solve problems of emphasis, which are another way in which words can be wrong (your schema might treat rare events as central cases and common ones as exceptions, for instance).
1zulupineapple6y
Obviously, I wouldn't know how to do formal reasoning correctly, even if I seriously tried to do it. I'm sure there are many problems with the idea that don't have known solutions. I believe that complete and correct formal reasoning is easier than full AI, but not by much. Having that in mind, it's hard to make claims about what this reasoning would look like. I'm not sure what you mean by unjustified premises and problems of emphasis, so I'll make a guess. You might worry that some people would dedicate a lot of time and effort into constructing increasingly convoluted proofs showing how, e.g. flat earth, is consistent with various observations and experiments. Such proofs might be admitted into the Long List of True Statements. However, as long as these proofs lead to no implications about what NASA should be working on, they are not a problem. Another possibility is that the proofs are of the form "if lizard people run NASA, then it's most likely that the earth is flat". Again, if you don't share the assumptions, there is no harm from such proofs, they might even be beneficial in some ways (e.g. displaying our sensitivity to bad priors). In this framework, building perverse proofs for "the outgroup is stupid" might actually be a productive activity.
2Benquo6y
I'm worried about what happens before people start putting time and effort into proofs. Related: 37 Ways That Words Can Be Wrong
1zulupineapple6y
Well, that's a long list, but I don't see why formal logic would make any of those problems worse, and it seems many could be solved. Do you have some specific worries?
3Raemon6y
Social-related-stuff isn't the only cause of demon threads, it just is a more-reliable-way-than-average to cause them. (I think it's possible that what I'd call malignant demon threads require some element of social stakes, even if it's just two people devolving into personal attacks) I have vague plans (I'm not sure if this will turn out to be a terrible idea), about attempting to have a followup post that walks through specific examples while treading very carefully to avoid re-opening the original threads in question.
1zulupineapple6y
Is that even worth worrying about? One one hand this seems to assume very little faith in LW users, and on another hand, if it did reignite conflict, would that really be bad?

(Coming here from the Duncan-and-Said discussion)

I love the term "demon thread". Feels like a good example of what Duncan calls a "sazen", as in a word for a concept that I've had in mind for a while (discussion threads that naturally escalate despite the best efforts of everyone involved), but having a word for it makes the concept a lot more clear in my mind.

4Raemon1y
I feel obligated to point out that Duncan dislikes the term demon-thread, for being too opinionatedly rhetorically forceful against a conversation pattern that's not necessarily that bad. (I thought about trying to call this out more in the other post when I referenced the term, but it felt like too many extra clauses in the point I was trying to make at the time)
1PoignardAzur1y
I mean, seeing some of those discussions thread Duncan and others were involved in... I'd say it's pretty bad? To me at least, it felt like the threads were incredibly toxic given how non-toxic this community usually is.

My objection is that it doesn't distinguish between [unpleasant fights that really should in fact be had] from [unpleasant fights that shouldn't]. It's a very handy term for delegitimizing any protracted conflict, which is a boon to those who'd like to get away with really shitty behavior by hijacking politeness norms.

"There was a Muggle once named Mohandas Gandhi," Harry said to the floor. "He thought the government of Muggle Britain shouldn't rule over his country. And he refused to fight. He convinced his whole country not to fight. Instead he told his people to walk up to the British soldiers and let themselves be struck down, without resisting, and when Britain couldn't stand doing that any more, we freed his country. I thought it was a very beautiful thing, when I read about it, I thought it was something higher than all the wars that anyone had ever fought with guns or swords. That they'd really done that, and that it had actually worked." Harry drew another breath. "Only then I found out that Gandhi told his people, during World War II, that if the Nazis invaded they should use nonviolent resistance against them, too. But the Nazis would've just shot everyone in sight. And

... (read more)
7PoignardAzur1y
I think this is a subject where we'd probably need to hash out a dozen intermediary points (the whole "inferential distance" thing) before we could come close to a common understanding. Anyway, yeah, I get the whole not-backing-down-to-bullies thing; and I get being willing to do something personally costly to avoid giving someone an incentive to walk over you. But I do think you can reach a stage in a conversation, the kind that inspired the "someone's wrong on the internet" meme, where all that game theory logic stops making sense and the only winning move is to stop playing. Like, after a dozen back-and-forths between a few stubborn people who absolutely refuse to cede any ground, especially people who don't think they're wrong or see themselves as bullies... what do you really win by continuing the thread? Do you really impart outside observers with a feeling that "Duncan sure seems right in his counter-counter-counter-counter-rebuttal, I should emulate him" if you engage the other person point-by-point? Would you really encourage a culture of bullying and using-politeness-norms-to-impose-bad-behavior if you instead said "I don't think this conversation is productive, I'll stop now"? It's like... if you play an iterated prisoner's dilemma, and every player's strategy is "tit-for-tat, always, no forgiveness", and there's any non-zero likelihood that someone presses the "defect" button by accident, then over a sufficient period of time the steady state will always be "everybody defects, forever". (The analogy isn't perfect, but it's an example of how game theory changes when you play the same game over lots of iterations) (And yes, I do understand that forgiveness can be exploited in an iterated prisoner's dilemma.) Again, I don't think I have a sufficiently short inferential distance to convince you of anything, but my general vibe is that, as a debate gets longer, the line between the two starts to disappear. It's like... Okay, another crappy metaphor is,

Where did the term "demon thread" come from? A lot more people are going to be familiar with the term "flame war". (A search engine will confirm it is a common phrase, and that it is essentially synonymous with what you are calling "demon thread".). If you want to distinguish one from the other, you should have a good reason to do so, and you should tell the reader. Otherwise I'd just say "flame war".

EDIT: Maybe "malignant demon thread" = "flame war"?

https://www.urbandictionary.com/define.php?term=flame%20war

Short answer: I made up the term because it seemed to best describe what was happening.

I feel like the connotations are a bit different. Flame Wars feel like something where there's no good will at all, nothing but pure angry yelling. The most important new bits of information in this article are about the Demon Seed - how a well intentioned discussion among intellectual truthseekers can nonetheless turn into some weird monstrosity, often while people are still earnestly trying to communicate and accomplish something good.

"Flame" connotes pure warfare.

"Demon" connotes some kind of malevolent force that's twisting good intentions into evil.

I should probably at least reference flame wars in the post (partly to distinguish them and maybe just for the SEO)

[Edit: added a note about it]

[Edit 2: it turns out the current SEO for "Demon Thread" mostly refers to Java processes, which isn't great. It may be that I want to change the jargon to something more standard, but I do still feel Flame War doesn't quite capture it)]

9Ben Pace6y
Agreement that 'flame war' seems to put the blame on the individuals more than 'demon thread' which suggests there was some sort of attractor pulling the humans toward a bad place.
2Qiaochu_Yuan6y
How about "demon war" (he said, without putting much thought into it)?

There's an experiment — insert obligatory replication crisis disclaimer — where one participant is told to gently poke another participant. The second participant is told to poke the first participant the same amount the first person poked them.

It turns out people tend to poke back slightly harder than they were first poked.

Repeat.

A few iterations later, they are striking each other really hard.

Do you know where I could read this study? I was unable to find it online with keywords like "poking", "escalation", etc.

There were too many comments to get through, so I'm not sure if I'm repeating ideas here, but some of the thoughts that initially occured to me on reading this article were about how to build better social media to combat this sort of thing. My brief thoughts were: 1)limit messages to long enough to express a simple thought but not long enough to go on rants (maybe 300 charact... (read more)

2Raemon6y
These are interestomg ideas, and I don’t think any were mentioned yet. Several of them feel like they’d have bad externalities but point towards something that feels worth exploring more. Will think more on though.
if the government decides to increase the tax on gasoline to "fight Global Warming" this will impact the status of a lot of people.

That's an indirect impact, which I don't think is a plausible motivator. Like, it's a tragedy of the commons, because each individual would be better off letting others jump in to defend their side, and free-riding off their efforts. It may feel like the real reason we jump into demon threads, but I think that's a post-hoc rationalization, because we don't actually feel twice as strong an imp... (read more)

I think I have a TAP that's something like, "Notice I'm in a Demon Thread, bow out of conversation." The way I bow out is something like "It doesn't feel to me like there's anything useful coming out of this discussion, so I won't be replying further".

This seems potentially less useful than the norm of "take it to private". But it does seem to reliably end the demon threads. Not sure if it also has a chilling effect.

2Raemon6y
Yeah, the take-it-private part was specifically a subset of "if the thread seems actually important."

Maybe this is discussed in one of the linked articles (I haven't read them). But interestingly, the following examples of demon topics all have one thing in common:

Latent underlying disagreements about how to think properly... or ideal social norms... or which coalitions should be highest status... or pure, simple you're insulting me and I'm angry

While it's possible to discuss most things without also making status implications, it's not possible with these issues. Like, even when discussing IQ, race, or gender, it's usually ... (read more)

2Raemon6y
This was basically my initial guess (I was conscious of it as I described those things), although zulupineapple's comment was a reminder that this isn't always the case.
1zulupineapple6y
Can you give some examples of topics that are definitely not about status? It seems to me that every topic that people care about can be said to be about status, which would make your theory have very little explanatory power.
2MakerOfErrors6y
Good point. I dono, maybe almost everything really is about status. But some things seem to have a much stronger influence on status than others, and some are perceived as much larger threats than others, regardless of whether those perceptions are accurate outside of our evolutionary environment. Even if everything has a nonzero status component, so long as there is variation we'd need a theory to explain the various sources of that variation. I was trying to gesture at situations where the status loss was large (high severity) and would inevitably happen to at least one side (large scope, relative to audience size). Change My View (the source I thought might make a good proxy for LW with politics) has a list of common topics. I think they span both the scope and severity range. * Abortion/Legal Parental Surrender: Small scope, high severity. If discussed in the abstract, think mostly only people who've had abortions are likely to loose status if they let a statement stand that infers that they made a bad decision. If the discussion branches out to body autonomy, though, this would be a threat to anyone who might be prevented from having one or have tribal members who would be prevented. * Climate Change: Low scope, low severity. Maybe some climatologists will always loose status by letting false statements stand, but most other people's status is about as divorced from the topic as it's possible to be. Maybe there's a tiny status hit from the inference that you're a bad person if you're not helping, which motivates a defensive desire to deny it's really a problem. But both scope of people with status ties and the severity of status losses are about zero. * Donald Trump: If I say "Donald Trump is 1.88m tall" no one looses any status, so that topic is low-scope, low-severity. that's defining him as a topic overly narrowly, though. There certainly are a surprisingly large number of extremely inflammatory topics immediately adjacent. The topic of whether he's do
3zulupineapple6y
So you've listed a few topics. How likely is each of them to result in demon threads? I can easily see people furiously arguing about any of those, I doubt there is much variation between them. The fact that many people happen to have opinions on these topics (i.e. that they are common in CMV) seems more relevant than any reality-based measure of their importance. Consider also more niche topics such as "best programming language" that sometimes also result in demon threads (though, to be fair, I haven't personally seen any recently), while having objectively no impact on the real world. No, it's not that everything is "about status", it's that "about status" is just an obtuse way to say "people care". Every interaction between two people is by definition social, and LW is very happy to reduce all social interactions to status comparisons. But what exactly does that explain?
1MakerOfErrors6y
My prediction is that almost no discussion that starts about whether Donald Trump is 1.88m tall should turn into a demon thread, unless someone first changes the topic to something else. Similarly, the details of climate change itself should start fewer object-level arguments. I would first expect to see a transition to (admitedly closely related) topics like climate change deniers and/or gullible liberals. Sure, people may then pull out the charts and links on the object level issue, but the subtext is then "...and therefore the outgroup are idiots/the ingroup isn't dumb". We could test this by seeing whether strict and immediate moderator action prevents demon threads if it's done as soon as discussion drifts into inherently-about-status topics. I think if so, we could safely discuss status-adjacent topics without anywhere near as many incidents. (Although I don't actually think there's much value in such discussions most of the time, so I wouldn't advocate for a rule change to allow them.)
1zulupineapple6y
Trump's height is definitely not why "Donald Trump" is a common topic in CMV, so I don't see how that's relevant. On the other hand, such trivial fact based topics can easily become demonic - consider birtherism. If there was a subset of population that believed "Trump is actually 1.87m tall", this could easily lead to demon threads. There is nothing inherent about height that prevents it from being a demon topic. Object level disagreements about climate change are definitely a big part of why it's a common topic and why it might cause demon threads. Of course, the argument eventually involves insulting the outgroup, but that's hardly a topic. This is based on the assumption that some topics really are inherently about status. My claim is that topic popularity is a decent predictor of demonic threads, and that your status related evaluations add very little to that.