Turning the Technical Crank

A few months ago, Vaniver wrote a really long post speculating about potential futures for Less Wrong, with a focus on the idea that the spread of the Less Wrong diaspora has left the site weak and fragmented. I wasn't here for our high water mark, so I don't really have an informed opinion on what has socially changed since then. But a number of complaints are technical, and as an IT person, I thought I had some useful things to say.

I argued at the time that many of the technical challenges of the diaspora were solved problems, and that the solution was NNTP -- an ancient, yet still extant, discussion protocol. I am something of a crank on the subject and didn't expect much of a reception. I was pleasantly surprised by the 18 karma it generated, and tried to write up a full post arguing the point.

I failed. I was trying to write a manifesto, didn't really know how to do it right, and kept running into a vast inferential distance I couldn't seem to cross. I'm a product of a prior age of the Internet, from before the http prefix assumed its imperial crown; I kept wanting to say things that I knew would make no sense to anyone who came of age this millennium. I got bogged down in irrelevant technical minutia about how to implement features X, Y, and Z. Eventually I decided I was attacking the wrong problem; I was thinking about 'how do I promote NNTP', when really I should have been going after 'what would an ideal discussion platform look like and how does NNTP get us there, if it does?'

So I'm going to go after that first, and work on the inferential distance problem, and then I'm going to talk about NNTP, and see where that goes and what could be done better. I still believe it's the closest thing to a good, available technological schelling point, but it's going to take a lot of words to get there from here, and I might change my mind under persuasive argument. We'll see.

Fortunately, this is Less Wrong, and sequences are a thing here. This is the first post in an intended sequence on mechanisms of discussion. I know it's a bit off the beaten track of Less Wrong subject matter. I posit that it's both relevant to our difficulties and probably more useful and/or interesting than most of what comes through these days. I just took the 2016 survey and it has a couple of sections on the effects of the diaspora, so I'm guessing it's on topic for meta purposes if not for site-subject purposes.

Less Than Ideal Discussion

To solve a problem you must first define it. Looking at the LessWrong 2.0 post, I see the following technical problems, at a minimum; I'll edit this with suggestions from comments.

  1. Aggregation of posts. Our best authors have formed their own fiefdoms and their work is not terribly visible here. We currently have limited support for this via the sidebar, but that's it.
  2. Aggregation of comments. You can see diaspora authors in the sidebar, but you can't comment from here.
  3. Aggregation of community. This sounds like a social problem but it isn't. You can start a new blog, but unless you plan on also going out of your way to market it then your chances of starting a discussion boil down to "hope it catches the attention of Yvain or someone else similarly prominent in the community." Non-prominent individuals can theoretically post here; yet this is the place we are decrying as moribund.
  4. Incomplete and poor curation. We currently do this via Promoted, badly, and via the diaspora sidebar, also badly.
  5. Pitiful interface feature set. This is not so much a Less Wrong-specific problem as a 2010s-internet problem; people who inhabit SSC have probably seen me respond to feature complaints with "they had something that did that in the 90s, but nobody uses it." (my own bugbear is searching for comments by author-plus-content).
  6. Changes are hamstrung by the existing architecture, which gets you volunteer reactions like this one.

I see these meta-technical problems:

  1. Expertise is scarce. Few people are in a position to technically improve the site, and those that are, have other demands on their time.
  2. The Trivial Inconvenience Problem limits the scope of proposed changes to those that are not inconvenient to commenters or authors.
  3. Getting cooperation from diaspora authors is a coordination problem. Are we better than average at handling those? I don't know.

Slightly Less Horrible Discussion

"Solving" community maintenance is a hard problem, but to the extent that pieces of it can be solved technologically, the solution might include these ultra-high-level elements:

  1. Centralized from the user perspective. A reader should be able to interact with the entire community in one place, and it should be recognizable as a community.
  2. Decentralized from the author perspective. Diaspora authors seem to like having their own fiefdoms, and the social problem of "all the best posters went elsewhere" can't be solved without their cooperation. Therefore any technical solution must allow for it.
  3. Proper division of labor. Scott Alexander probably should not have to concern himself with user feature requests; that's not his comparative advantage and I'd rather he spend his time inventing moral cosmologies. I suspect he would prefer the same. The same goes for Eliezer Yudkowski or any of our still-writing-elsewhere folks.
  4. Really good moderation tools.
  5. Easy entrance. New users should be able to join the discussion without a lot of hassle. Old authors that want to return should be able to do so and, preferably, bring their existing content with them.
  6. Easy exit. Authors who don't like the way the community is heading should be able to jump ship -- and, crucially, bring their content with them to their new ship. Conveniently. This is essentially what has happened, except old content is hostage here.
  7. Separate policy and mechanism within the site architecture. Let this one pass for now if you don't know what it means; it's the first big inferential hurdle I need to cross and I'll be starting soon enough.

As with the previous, I'll update this from the comments if necessary.

Getting There From Here

As I said at the start, I feel on firmer ground talking about technical issues than social ones. But I have to acknowledge one strong social opinion: I believe the greatest factor in Less Wrong's decline is the departure of our best authors for personal blogs. Any plan for revitalization has to provide an improved substitute for a personal blog, because that's where everyone seems to end up going. You need something that looks and behaves like a blog to the author or casual readers, but integrates seamlessly into a community discussion gateway.

I argue that this can be achieved. I argue that the technical challenges are solvable and the inherent coordination problem is also solvable, provided the people involved still have an interest in solving it.

And I argue that it can be done -- and done better than what we have now -- using technology that has existed since the '90s.

I don't argue that this actually will be achieved in anything like the way I think it ought to be. As mentioned up top, I am a crank, and I have no access whatsoever to anybody with any community pull. My odds of pushing through this agenda are basically nil. But we're all about crazy thought experiments, right?

This topic is something I've wanted to write about for a long time. Since it's not typical Less Wrong fare, I'll take the karma on this post as a referendum on whether the community would like to see it here.

Assuming there's interest, the sequence will look something like this (subject to reorganization as I go along, since I'm pulling this from some lengthy but horribly disorganized notes; in particular I might swap subsequences 2 and 3):

  1. Technical Architecture
    1. Your Web Browser Is Not Your Client
    2. Specialized Protocols: or, NNTP and its Bastard Children
    3. Moderation, Personal Gardens, and Public Parks
    4. Content, Presentation, and the Division of Labor
    5. The Proper Placement of User Features
    6. Hard Things that are Suddenly Easy: or, what does client control gain us?
    7. Your Web Browser Is Still Not Your Client (but you don't need to know that)
  2. Meta-Technical Conflicts (or, obstacles to adoption)
    1. Never Bet Against Convenience
    2. Conflicting Commenter, Author, and Admin Preferences
    3. Lipstick on the Configuration Pig
    4. Incremental Implementation and the Coordination Problem.
    5. Lowering Barriers to Entry and Exit
  3. Technical and Social Interoperability
    1. Benefits and Drawbacks of Standards
    2. Input Formats and Quoting Conventions
    3. Faking Functionality
    4. Why Reddit Makes Me Cry
    5. What NNTP Can't Do
  4. Implementation of Nonstandard Features
    1. Some desirable feature #1
    2. Some desirable feature #2
    3. ...etc. This subsequence is only necessary if someone actually wants to try and do what I'm arguing for, which I think unlikely.

(Meta-meta: This post was written in Markdown, converted to HTML for posting using Pandoc, and took around four hours to write. I can often be found lurking on #lesswrong or #slatestarcodex on workday afternoons if anyone wants to discuss it, but I don't promise to answer quickly because, well, workday)

[Edited to add: At +10/92% karma I figure continuing is probably worth it. After reading comments I'm going to try to slim it down a lot from the outline above, though. I still want to hit all those points but they probably don't all need a full post's space. Note that I'm not Scott or Eliezer, I write like I bleed, so what I do post will likely be spaced out]

Comments

sorted by
magical algorithm
Highlighting new comments since Today at 2:48 AM
Select new highlight date
Rendering 50/134 comments  show more

Whatever your solution ends up looking like, a key feature has to be "I can post a link on Facebook or whatever that people can click on and read in their web browser." If you can't be linked to it's no good.

I (too?) am nostalgic for the good ol' days of Usenet. I'm very very unconvinced that an NNTP-based system could realistically replace Less Wrong. I'm interested in what you have to say, but wonder whether there's value in some kind of brief overview post along the lines of "Here's the one-paragraph summary of why I think this is probably a good idea. Here are one-sentence summaries of the strongest five objections and why they don't change my mind."

(But maybe not; the effect might be to put off people who could have been persuaded with a gentler run-up.)

I do wonder whether you can really need twenty posts to make your case. Perhaps much of the material will be useful in other ways (e.g., to inform future attempts at community-building, collaboration, etc.)?

I do wonder whether you can really need twenty posts to make your case.

The perennial cry is "LessWrong needs more high-quality content!" And when such is offered you go "Eh, condense it into a tl;dr"?

Nope, that's not what I intended to say. (My apologies for any lack of clarity.) Rather, I think that

  • 20 posts of high-quality content on a single rather peripheral topic arriving in rapid succession would not necessarily be an improvement;
  • if there are going to be 20 such posts, a TL;DR as well might be helpful at the outset.

There's not necessarily a one-to-one bullet-point-to-post correspondence in that list; I won't know exactly how much space it takes to make each point until I've done it. It seems excessively long to me too, but it's how my notes map out.

A summary post is more or less what I tried to start with and failed. It kept coming out in arguments that only 'work' for people that already know where I'm coming from, e.g. it assumed the superiority of specialized protocols and the Rule of Separation, neither of which means anything to nontechnical users or even most power users. Our base is tech savvy but it's mostly post-2000 tech-savvy, I think.

[edit: I might add a bullet-point summary to the bottom of this post without justifications after I've had a chance to see the comments and which objections people actually raise]

[edit: I might add a bullet-point summary to the bottom of this post without justifications after I've had a chance to see the comments and which objections people actually raise]

I assumed you didn't want people to raise technical objections to this post, and let you present your argument first. But if you want them now, here are some objections that gjm didn't mention:

  1. Our goal is to make something better than the existing LW software / UX. But we must also allow free linking in, to posts and comments, from ordinary blogs and other sites. These links will be the natural gateway for users new to the community, and lurkers. They must also have a UX at least as good as today's LW, exposing the features of the new / non-web solution, or else this whole endeavor will be a regression and doomed to fail.
  2. People won't like a plaintext-oriented interface; they want rich text, inline images and tables, which in practice (in the world of NNTP) translates to HTML. But HTML is far too rich (and unsuitable for human editing). We need an equivalent to comments' markdown support (or something better that would also be usable for posts). With NNTP, this would be client dependent, so at best it would vary by poster and at worst it would simply not be supported (or not out of the box).
  3. Editing of posts and comments is a crucial feature which NNTP doesn't provide.
  4. Other features we need or are used to: RSS feed; tagging; user management and direct messaging without the trivial inconvenience of creating a new pseudonymous email account on a different site; server-based state (e.g. 'unread' message state in user inbox) for those with multiple client devices; .....
  5. Any proposal that isn't for a gradual change to the existing site will need to be run on a new site. The old lesswrong.com won't be shut down until the new one has clearly succeeded. So you'll need to directly compete with lesswrong.com to convince users to switch. You can make an LW -> NNTP gateway, but not an NNTP -> LW gateway; that is, lesswrong.com can't automatically publish content (comments) posted via NNTP (or any other protocol, really). So during this period of competition, even if users cross-post, discussion threads will be separate. The new software will have to be clearly superior to convince LW users to switch, let alone diaspora blog authors.

I didn't, but I was assuming people would anyway. I was actually hoping for higher-level objections like the problems I listed in the post, but, well. I'll answer you anyway and maybe edit the post later. Most of these fall under 3.2 and 3.3 in the outline.

  1. is actually the only part of the problem for which no off-the-shelf solution exists. The short version is that the site UX, instead of talking to a database directly on the backend, would talk to NNTP on the backend. Links to arbitrary posts (as a non-exhaustive example) could be as simple as https://newlesswrong/message-id. Top level posts could support some subject-generated shorthand, perhaps.
  2. I actually do want a plaintext-oriented interface, but I know I'm in the minority. You've expressed the solution to the markup problem yourself, though: Markdown is already the effectively-default format on Usenet and all extant NNTP clients. In fact it's the rise of lightweight markup in general and Markdown in particular that convinced me this could ever be more than a pipe dream. There are more powerful forms of lightweight markup that could be used; the key is that the input format must be readable as plaintext for interoperability between the web and native clients to be possible.
  3. I believe that cancels and supersedes can be kludged to support something that looks like editing even if it isn't. I may be wrong about this; in particular I'm not sure how supersedes interact with reply chains, because they're unusable for that purpose on Usenet proper.
  4. User management has an established solution, and DMing can be implemented as a LW mailbox (that only takes local messages) or a forwarding address, or both. RSS is probably easy. Server based state is easy for the 'default' website (really an in-browser client, covered in 1.1 and 1.7) but may be hard for native clients (but anyone using a native client presumably knows what they're getting into). Tagging may be hard, I'm not sure. Karma is definitely hard, but may be unnecessary.
  5. The short version of this is that, designed correctly, any author or site can adopt the network without being any worse off than they currently are; that is, cooperate-defect leaves you no worse off than defect-defect. The long version is...pretty much all of section 2, actually.

(ETA: I am answering this in moderate detail not to encourage technical back-and-forth but to demonstrate that I have thought this through)

This is a subject that strongly matters to me. I too would love to see a return to non-proprietary, open communication protocols, open source software and decentralized hosting - everywhere on the Internet, not just on Less Wrong. This is one of the few capital-C Causes in my area of professional competence that I would happily donate a lot of labor and/or money to, if only I knew of a way to promote it. But I don't, and I don't know of anyone who does.

To argue that the problem can be solved in the LW microcosm would need to either take advantage of LW-specific community features, or explicitly not solve the general problem (e.g. by not scaling, or by admitting that some things would always remain Web-only and non-interoperable). If either one is the case, please mention that explicitly.

Like gjm, I immediately want to jump the inferential distance to the usual unsolved problems. (E.g., how do you handle 'graceful degradation' for people who encounter a necessarily web/http link for the first time, so the community can grow and people with regular blogs can link to it?)

It might help if you add explicit disclaimers saying "please don't bring up issue X, that's for a future post". Are there things you don't want to talk about before a certain point? Is your sequence planned out enough (and short enough in practice) that I should refrain from anticipating certain issues, even in separate posts?

I fear that my comment(s) might appear negative, focusing on problems that I don't know how to solve before you even posted about them. I very much want this conversation (and the wider LW 2.0 one) to be constructive! If you think there's a better way for me to engage with it, please don't hesitate to tell me so. And thank you for taking the time to advocate a solution to a problem I deeply care about.

ETA: also, I would very much enjoy myself writing posts on subjects like "Your Web Browser Is Not Your Client" (or as it's sometimes known, The Web Is Not The Net), "The Proper Placement of User Features (is at the clientside)" aka "separation of protocols from implementation", and so on. I just didn't think it was on-topic for LW. But if you make it on-topic, then I might just join in.

Talking about optimizing a widely-used system seems very on-topic for Less Wrong. At any rate, it doesn't seem any more off-topic than things like fibromyalgia. I probably couldn't contribute anything of value, but I'd be fascinated by those hypothetical posts.

The only LW-specific community feature my proposal takes advantage of is our cultural applause of "I cooperate in Prisoner's Dilemmas."

It scales in the specific sense that it allows for incoming users and authors to be added incrementally. It fails to scale in the sense that it can never be an open system in the same sense that Usenet is an open system. However, that is no worse than our existing situation.

Graceful degredation is a hard problem. Please don't bring up graceful degredation, that's for a future post. :-P

I am waffling on whether to encourage people to anticipate issues. On the one hand, it's helpful for me to know what I need to address along the way. On the other, I really don't want the comment threads bogged down by material that only makes sense to our technical contingent.

I love it that you jumped to the correct interpretation of The Proper Placement of User Features.

I think any proposal based on actual NNTP is probably doomed.

I think any proposal that asks the user to use a client that isn't The Web is doomed (but it looks like you are addressing that in 1.7.)

BUT, I think the notion of redesigning this system around something that is morally just like NNTP is a hugely interesting and not-totally-crazy one; AND even if you completely fail, I think there will be hugely valuable ideas in this sequence for people like me who also think about this kind of thing.

So please write this sequence!

What do you think about the following alternative approach?

  1. Expose all LW features via a convenient, well designed, documented/stable API. (I don't know how much work that would be, but let's ignore that for a moment.)
  2. You're now free to write and use a non-web client for LW. You can add whatever features you want that make sense as clientside-only features. Perhaps you can add support to an existing email/usenet/... client instead of writing one from scratch.
  3. You can also add support for other sites, such as wordpress or tumblr blogs. Naturally, the client will have to enable for each site only the 'shared' features that site supports, like voting on LW. Then you can use that client to improve your personal experience on all sites, to the degree that's possible without changing each site's server software.
  4. In fact, you will naturally create (or adopt, or adapt) a middleman protocol which clients will talk with per-server plugins. If you're sufficiently masochistic, you can even use NNTP here.
  5. If many people like your client and start using it, we can at that much later date consider issues like encouraging more newcomers to use it, or otherwise making it the 'main' client for some sites.

This would be very reminiscent of the multi-protocol, interoperable, and open-standard IM scene of the 90s and early 2000s, before the big providers (Google, Yahoo, Facebook, et al) all killed off their Jabber support and became closed gardens. And if such a protocol or client ever comes close to succeeding on a world-wide scale, I expect it would be killed in the same manner. In practice, of course, it would fail much sooner: the HTTP traffic of a typical website isn't meant to be an API and can't be easily reverse engineered to behave like one, never mind stability guarantees. But if we only want it for a few friendly sites, then it's not technologically problematic.

This would be a second-best approach. The main benefit that the use of NNTP has over such an approach is the ability to leverage the huge existing library of NNTP server and client software. The only from-scratch development required would be a forumesque in-browser client -- which might already exist, though I am aware of no good ones.

What you describe would be very similar to designing an NNTP 2, a goal that I find laudable but that I really do think is socially (not technically) impossible. If it were possible, I wouldn't recommend implementing it on top of HTTP. "Cram the round peg of semantic information over http no matter how badly it fits that square hole" is my major beef with the entire direction of software development over the last ten years.

The comparison to Jabber is apt, and I hate the death of jabber for reasons very similar to my hate for the death of nntp. Mechanism should not be a closed garden. Individual communities, sure; and, as you say, what I want it for here is a few friendly sites. But mechanism, never.

I, for one, am very interested in this. I don't see this working because people are resisting change even when presented with clearly better alternatives, and also legacy community.
But, as you say, this is LessWrong, it's worth a try.

I also loved usenet! It fell apart when all the spammers and trolls and idiots turned up. (I think the binary groups are a distraction, plenty of usenet servers just didn't carry them. )

Those are just words for people whose opinions you'd rather not read, so we need some sort of moderation system.

Trusted moderators don't have the time or energy to do that, and don't scale, so we need some sort of group voting system.

Reddit was brilliant for a while, then Hacker News, then Less Wrong.

All three seem to have gone downhill in different ways. What can we learn from these three? (It's actually possible that Hacker News is still excellent, but it no longer posts the sort of things that interest me often enough that I ever look anymore.)

Stack Overflow (and related sites) seem to have stayed consistently excellent for a long time now. They're also laudably open. But Stack Overflow has a consistent problem with interesting questions getting deleted by fascist moderators, and it seems to only be good for Questions and Answers. It's not a place to pontificate about fibromyalgia, for instance.

What can we learn from Stack Overflow? Can we make something like that, for people we like to post essays and comment on them, without driving our best away?

But Stack Overflow has a consistent problem with interesting questions getting deleted by fascist moderators, and it seems to only be good for Questions and Answers. It's not a place to pontificate about fibromyalgia, for instance.

That's not a bug, that's a feature. SO was designed and restricted for a Q&A format; it deliberately omits features like comment trees. Its creators believe that it succeeded because it was restricted, since a more specific problem is easier to solve. They've gone on to try and solve the more general discussion forum "problem" with Discourse, but it's still very lean on features - because it's explicitly designed for mobile touch interfaces and for reading over writing.

Personally, I'm not convinced that SO - as a community - couldn't have succeeded just as well, or better, with quite different software features. But that's what they believe.

Stack Overflow is different from the other websites you mentioned, because it is, to some degree, timeless.

I suspect that the importance of time when posting on news sites contributes to the deterioration. Some people spend more time on the website, some people spend less. The people who spend more time get a bonus in the system.

Problem is that "spending a lot of time debating online" can correlate with some undesired traits, such as "doesn't do research", "writes without thinking", "doesn't read the whole article before commenting", etc. If these traits are turned up to eleven, of course those people get banned. But within the acceptable range, those on the bad side of the range get an advantage, and the 'Overton window' will gradually shift in their favor.

Note that despite my cynicism about this as a way to save/improve/fix LessWrong specifically, I would absolutely love a solid reddit-to-nntp gateway, that allowed me to use trn or whatever to read and respond to posts and comments, with similar threading and better state management.

I'd use it on LW and a bunch of other similar places.

That suddenly reminds me of Urbit. Wouldn't it be funny... X-)

1) I loved Usenet prior to Eternal September, and used it through much of the 90s as well. It's not coming back.

I'm part of another group which tried replacing their disfunctional mailing lists with NNTP, and probably a dozen of us used it for a month or two before we realized that nobody else was coming and went back to the main group.

2) Running code trumps theoretical arguments. Don't write a series of posts, set up your system and see if it works.

Re 2: running code that would take a lot of effort to integrate into an existing system, and no one else is interested in it, least of all the site admins, doesn't trump anything.

Easy entrance is how September happened, both on LessWrong and on Usenet.

My personal bias here is that I see little hope for most of the application level network protocols built in the 80s and 90s, but have high hope for future federated protocols. Urbit in particular since a certain subtribe of the LW diaspora will already be moving there as soon as it's ready.

From the position of author, the important difference between posting an article here and posting an article on my personal webpage is the control over the discussion.

Posting here is convenient: the whole website is already set up and maintained, I just need to write the text. My article will immediately get many readers, and it will be approximately the kind of readers I want. Even the moderation by crowd is provided for free.

On the other hand, the cost of the convenience is my freedom to make different choices. If I have opinions on the website functionality or design, it's not my choice. I have to think whether my topic is appropriate for the website; while on my own blog I can post anything. If I disagree with the moderation, too bad, I am just one among many voters. On my own blog I can make the ultimate decisions, block the users I don't like, and keep the debate nice according to my criteria of niceness.

One of the interesting things about NNTP's structure is that the moderator and the host don't need to be the same entity or even use the same software. The same goes for UX elements. It would be entirely possible to run something-that-looks-like-a-blog on your own site, have it use hypothetical-lesswrong-hosted NNTP for hosting its content (buying you native-client support for users who want it), and still have ultimate control over who can post what. I'll be describing how that works at some point.

It would rely on goodwill from the LW hosts, of course; but the worst they could do is stop hosting you -- and they could not hold your content hostage as long as someone, somewhere, has kept a local cache of it. You could even self-host and still interoperate with the site, because the system was designed to be decentralized even though it doesn't have to be used that way.

Is this a good summary of your argument?

NNTP was a great solution to a lot of the problems caused by mailing lists. The main ones being:

  • content duplication - mailing lists are bad because everyone gets their own copy of each article.
  • reduced content accessibility - mailing lists suck because you miss out on great articles if they were sent before you were part of the mailing list.

We are facing similar problems now. A lot of people have their own sites where they host their own content. We either miss out on great content if we don't trawl through a ton of different sites or we try to make lesswrong a central source for content and face problems with:

  • content duplication - through needing to cross post content (essentially duplicating it)
  • harder content accessibility - the alternative to cross posting is providing a link, but this is an annoying solution that can be jarring as you need to go to an entirely different site to access the content you want.

NNTP would solve the problems we have now in a similar way to how it solved the problems with mailing lists. That is, it would provide a central repository for content and a way to access this content.


I am currently thinking that the best way to think about the last point is that it means that we should set up a Web API similar to the Blogger Web API. Discussing NNTP, at least to me, is making the solution appear a lot more complicated than it needs to be. Although, I don't know much about NNTP, so I could be overlooking something very important and am interested about what your future posts will explore.

With a Less Wrong Web API, websites could be created that act like views in a database. They would show only the content from a particular group or author. This content would, of course, be styled according to the style rules on the website.

These websites could be free, dns name and web development costs aside, using services like github pages. This is because there should be no need for a back-end as the content and user information is all hosted on Less Wrong. You post, retrieve content and vote using the API. It should also be fairly easy to create more complicated websites that could aggregate and show posts based on user preferences or even to create mobile applications.

Go for it. If we listened to cranks more, we could have finished that Tower of Babel.