[r0bbbo]

Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry

[jallen_dot_dev]

Wild guess: visiting a private sub requires an extra call to a service/db to check if the user can view it. Normally there are only a small number of these checks because private subs were usually small communities. Now, many large subs having switched private is causing some poor mircoservice somewhere to get hammered.

[bakuninsbart]

I never worked at this scale, but could it also be that different subs are horizontally scaled and with so many people reverting to the subs that are still open the load is unevenly balanced?

[q7xvh97o2pDhNrh]

Good question! And few people get to work at this scale, so it's not an unreasonable guess. I'll join you in speculating wildly about this, since, hey, it's kind of fun.

IMHO sharding traffic by subreddit doesn't pass the smell-test, though. Different subreddits have very different traffic patterns, so the system would likely end up with hotspots continuously popping up, and it'd probably be toilsome to constantly babysit hot shards and rebalance. (Consider some of the more academic subreddits vs. some of the more meme-driven subreddits — and then consider what happens when e.g. a particular subreddit takes off, or goes cold.)

Sharding on a dimension that has a more random/uniform distribution is usually the way to go. Off the top of my head (still speculating wildly and basically doing a 5-minute version of the system-design question just for fun), I'd be curious to shard by a hash of the post ID, or something like that. The trick is always to have a hashing algorithm that's stable when it's time to grow the number of shards (otherwise you're kicking off a whole re-balancing every time), and of course I'm too lazy to sort that out in this comment. I vaguely remember the Instagram team had a really cool sharding approach that they blogged about in this vein. (This would've been pre-acquisition, so ancient history by Silicon Valley standards.)

As for subreddit metadata (public, private, whatever), I'd really expect all of that to be in a global cache at this point. It's read-often/write-rarely data, and close-to-realtime cache-invalidation when it does change is a straightforward and solved problem.

[adra]

For really really large sets, you'll still eventually want to reduce read compute costs by limiting specific tenants to specific shards in order to reduce request fan out for every single read request. If say I get a super quiet forum, would it make sense to query 2 shards or 6000? Clearly there's a loss of performance when all read requests have infinite fan out.

[dbanon9]

A good, but wrong, assumption is to assume Reddit's engineers know what they're doing.

The founding Reddit team was non-technical (Even Spez. I've been to UVA; it's not an engineering school; and spez had never done any real engineering before coming onto Reddit). They ran a skeleton crew of some 10ish people for a long time (none from any exceptional backgrounds. One of them was from Sun & Oracle, after their prime).

Same group that started with a node front-end, python back-end monolith, with a document-oriented database structure (i.e., they had two un-normalized tables in Postgres to hold everything). Later they switched to Cassandra and kept that monstrosity -- partly, because back then no one knew anything about databases except sysadmins.

Back then they were running a cache for listing retrieval. Every "reddit" (subreddit and the various pages, like front-page, hot, top, controversial, etc.) listing is kept in a memcache. Inside, you have your "top-level" listing information (title, link, comments, id, etc.). The asinine thing is that cache invalidation has always been a problem. They originally handled it using RabbitMQ queues: votes come in, they're processed, and then the cache is updated. Those things always got backed up, because no one thought about batching updates on a timer (or how to use lock-free) (and no one knew how to do vertical scaling, and when they tried, it made things even harder to reason about). You know what genius plan they had next to solve this? Make more queues. Fix throughput issues by making more queues, instead of fixing the root cause of the back-pressure. Later, they did "shard"/partition things more cleanly (and tried lock-free) -- but they never did any real research into fixing the aforementioned problem (how to handle millions of simple "events" a day... which is laughable thinking back to it now).

That's just for listings. The comment trees are another big bad piece. Again, stored un-normalized -- but this time actually batched (not properly, but it is a step up). One great thing about un-normalized databases and trees, is that there are no constraints on vertices. So a common issue was that you could back-up your queue (again) for computing the comment trees (because they would never get processed properly) (and you could slow the entire site to a crawl because your message broker was wasting its time on erroneous messages).

Later, they had the bright idea to move from a data center to AWS -- break everything up into microservices. Autoscaling there has always been bungled.

There was no strong tech talent, and no strong engineering culture -- ever.

-------

My 2 cents: it's the listing caches. The architecture around it was never designed to handle checking so many "isHidden" subreddits (despite that they're still getting updates to their internal listings) -- and it's coming undone.

[onemiketwelve]

I read this as a pretty scathing dressing down of incompetent engineering at reddit. But after having breakfast, what I'm realizing again is that perfect code and engineering are not required to make something hugely successful.

I've been parts of 2 different engineering teams that wrote crap that I would cuss out but were super succesfuly, and most recently I joined a team that was super anal about every little detail. I think business success only gets hindered on the extremes now. If you're somewhere in the middle, it's fine. I'd rather have a buggy product that people use than a perfect codebase that only exists on github.

[eloquenceN]

Agreed. In fact, I believe success stories actually skew the other way. Those that actually build something that gets off the ground and is successful will in many cases not have the time to write perfect code.

[andreareina]

https://en.m.wikipedia.org/wiki/Consistent_hashing

[award_]

Yep, this is definitely just speculation, but I think this is it. Code/queries that worked fine at small load for private subs just doesn't work at scale when tons of subs are private.

[CodeCompost]

Isn't the source code of Reddit on GitHub? At least it used to be...

[FractalParadigm]

They stopped updating/supporting the code about 6 months before the big redesign (last push Oct '17, redesign launched Apr '18). Call me a bit of a conspiracy-theorist, but they just happened to raise $200 million on their Series C less than 3 months prior to abandoning their commitment to open-source.

[geekthattweaks]

They have two older versions up but nothing recent.

[core-utility]

Wouldn't want the recent stuff anyways (except to pontificate)

[raldi]

It could also be that the cache-hit rate is like 1/10th normal with the would-be front page full of smaller subreddits today.

[burnte]

My personal guess is it's down on purpose so they can say only 5% of subs who said they would go dark actually did, we just happened to have a service outage that day so they can push their own narrative t investors. Spez doesn't care anymore, he's focused on that IPO payday and to tell with everyone else. He's a liability to the company now, but the board isn't acting.

[cyral]

Here's a site which tracks how many subs have been made private. It's actually crazy that it's up to 93% of them now: https://reddark.untone.uk/

[thesh4d0w]

The site is decieving, it's 93% of the ones that committed to going dark. It's not 93% of all subreddits.

If you search for r/news for example, it's nowhere on that page.

[mlyle]

And most of the ones that "haven't" have restricted submissions... e.g. no new posts.

[readthenotes1]

I've tried a few listed as public that are actually private. reddit is darker than the report (as they mention)

[growt]

It's funny that /r/therewasanattempt and /r/whatcouldgowrong are on that list but still public (or public again?)

[wellthisisgreat]

/r/amitheasshole staying public is funny too

[vitorgrs]

Subs that are supposed to go down. This don't list all sub. A own a small sub that is not there, for example.

[paulmd]

Yup, this seems super plausible. Even things like the frontpage feed and user comment history probably work on the assumption that most of the data they're pulling is probably visible (which leads to the "just filter it in the backend" approach), but also every external link or bookmark into a now-private sub will trigger the same kind of check.

This is likely shifting load in very unpredictable ways... I'm sure a sibling comment is right that it's probably less load overall in general, but it'll be going down codepaths that aren't normally being exercised by >95% of requests and aren't working on the assumption that virtually all content is being hidden.

There's probably some microservice instances that are currently melting themselves into a puddle until they can deploy additional instances, additional DB shards, or roll out patches to fix dumb shit that wasn't dumb until the usage assumptions were suddenly inverted. Meanwhile there's tons of other instances that are probably sitting idle.

[sleight42]

Having worked at this scale, this is a fine guess! This scenario would have been a distant edge case for them. They likely didn't optimize for it. BOOM.

[munk-a]

Personally I'd like to believe that the servers themselves are standing in solidarity with the blackout over the API changes. I, for one, welcome our new robot overlords.

[HeavyStorm]

Came here just to say this.

Speculation, but having major subs private change the load profile which may result in the outage. Reddit certainly wasn't optimized for this.

[lillesvin]

I would imagine that a normal visit generates more backend traffic, given that it needs to fetch posts, thumbnails, etc. whereas a visit to a private sub wouldn't need to check more than authorization.

I could easily be wrong though, I haven't done web development for years.

[mfer]

They use a microservice architecture. Some services could scale well in servicing all those assets. What handles checking access to private subs may not.

You can’t treat the scaling as a binary feature, that it does or doesn’t

[jallen_dot_dev]

Sure, but that type of traffic is expected and they can handle it with things like caching and autoscaling. I'm suggesting that a part of the system that usually doesn't get a lot of requests wasn't designed to handle a huge influx of requests.

[bamfly]

All that other stuff's easy to cache. Authorization's cacheable, too, kinda, with some trade-offs, but they may not have bothered if it'd never been a problem before. Or this particular check my have been bypassing that caching, and it'd never cause a problem before because there weren't that many of those checks happening.

You start getting a lot more DB queries than usual, bypassing cache, the DB's remote, it's clustered or whatever, now you've got a ton more internal network traffic and open sockets, latency goes up, resources start to get exhausted...

[HankB99]

Not so wild IMO. I frequently have trouble loading pages on Reddit so I suspect any additional pressure could push it over the edge. It might be as simple as more users checking in to see if their favorite Reddit has gone private or shut down.

[cranekam]

If a DB check is needed to see if a sub is private or not it has to happen for every request. You can’t just limit the check to private subs because it’s not known if they are private or not at the time.

Reddit goes wrong often so I expect this outage could have any number of causes.

[jallen_dot_dev]

> You can’t just limit the check to private subs because it’s not known if they are private or not at the time.

That's not necessarily true. Perhaps the status of subreddits is cached (because there's no reason to hit the DB 100 times/second to check if r/funny is private or public). But for a given request to a private sub, it would need to check each user.

[taneq]

You don’t reckon it’s just disgruntled users running “set all my past comments to Boo Reddit Boo” scripts? I don’t imagine it’d take a huge proportion of users doing that simultaneously to slag the servers.

[skhr0680]

A meme pointing out that u/spez was a mod of r/jailbait in 2008 was the top post on r/all a few hours ago

[aw1621107]

It's worth noting that a very old user (/u/andrewsmith1986 IIRC) responded that at that time it was possible to add arbitrary users as mods without needing interaction/feedback/etc. from the user in question. If that was the case, then any user being a mod on any particular sub at that time doesn't really mean much.

Obviously I can't reference the comment in question right now, but I'll try to remember to circle back and add a reference when(/if?) Reddit comes back up.

Edit: /u/andrewsmith1986's comment can be found at https://old.reddit.com/r/dankmemes/comments/1477psa/all_3_ar...

[Zak]

I remember the change to requiring people to accept an invite. Adding people as moderators of distasteful subreddits without their consent was a common form of abuse just prior.

[roncesvalles]

I mean it's also worth noting that spez was CEO of Reddit and subreddits like that were allowed to exist for many, many more years.

[dionidium]

And it would in turn be worth noting that the creators of reddit had a philosophical and political commitment to free speech that drove their light-touch approach to moderation. It's not like the existence of that subreddit is evidence of an endorsement on their part.

[rchaud]

> a philosophical and political commitment to free speech

Surely this is a meme by now? Any CEO that has ever said this about a website they control is just pandering to the crowd. Musk's Twitter has complied with more government takedown requests than the previous regime had.

[thatguy0900]

https://en.m.wikipedia.org/wiki/Aaron_Swartz

If anyone would say it and mean it, it was probably this guy. He was quite the activist before he committed suicide.

[callalex]

It is now, but it was very different at the time. The old guard of the internet supporting absolute freedom of speech didn’t used to be associated with Nazis. Some good examples that still exists today are the Electronic Frontier Foundation and to some extent the American Civil Liberties Union.

[roncesvalles]

Lascivious pictures of clothed children count as CSAM in many jurisdictions. Free speech is fine but did they not have lawyers until 2015?

[Ajedi32]

This is partly why I now prefer the term "free exchange of ideas" over "freedom of speech". I believe it more closely captures the essence of what makes free speech worth protecting, while also conveniently excluding stuff like this (among other things, like spam or antisocial behavior).

[WindyLakeReturn]

At the time reddit was not some unknown back corner of the internet and had already begun working with law enforcement to enforce anti-CSAM laws due to material being treaded in private messages and private subreddits. That it took a media exposure take down the specific subreddit indicated that it was likely on the legal side of the line, through it was going close enough to the line to make others uncomfortable. If the material was actually illegal, wouldn't it have made reddit the largest clear-net site containing CSAM? In such a case, I find it hard to conceive that media exposure and not legal actions is why led to it shutting down, and with no admins being arrested it seems the most reasonable assumption left is that it was on the technically legal side.

This would likely be like the use of underage subjects in nudist art, painted, drawn, or photographed. Such art is generally not considered pornographic and are legally protected, and some even displayed in museums and the like, yet websites will still ban the material to not have any relationship to it and to not have users trying to push the boundary. I'm speaking to the extent that rules are enforced, many websites have issues with enforcing rules in general due to the amount of user generated content being much larger than the amount of moderation available, so there will be something slipping through moderation from time to time.

[Zak]

They didn't have image or video hosting at the time, just links and comments. Their legal exposure was pretty limited.

[tredre3]

Can you give us one example of a jurisdiction where a photo of a teenager in bikini would count as CSAM?

[Quarrelsome]

I thought part of the reason the US is so blessed with a thriving startup community is the light-touch the law has towards its startups. For example; didn't the billion dollar acquisition YouTube gain a lot of its initial growth through copyright infringement?

[cmh89]

>And it would in turn be worth noting that the creators of reddit had a philosophical and political commitment to free speech that drove their light-touch approach to moderation

That's nonsense. The Sears debacle showed that reddits leadership team was fine with deleting posts if it was going to cost them money to not delete them.

That 'political commitment to free speech' sure disappeared quickly when r/jailbait and u/violetacrez hit the main stream media.

spez was fine with hosting a community of child predators because it was one of the most popular subs. It was the top recommended result when you searched for reddit on google.

You can support free speech without actively providing a community for predators

[nicolas_t]

What was the Sears debacle?

[wpietri]

Hi! I've done a bunch of trust and safety work and I see this trope a fair bit. Please help me understand what the difference is between, say, platforming racist harassment because of a "political commitment to free speech" and platforming racist harassment because you just kinda like racist harassment?

I get that it might be different in the heads of the people who have worked very hard to create those platforms. I'm just not seeing any different in its effects on the world or on the targets of the racial harassment.

[jnky]

> Please help me understand what the difference is between, say, platforming racist harassment because of a "political commitment to free speech" and platforming racist harassment because you just kinda like racist harassment?

The difference is intent. Intent matters. Intent is the difference between murder and manslaughter, or between a conspiracy and mere speech.

[dionidium]

The argument goes like this:

There is probably a line. But you don't know where it is and neither do I. You and I might agree that X is to one side of that line, but if we ban that behavior, then we have initiated a process that we might call line-discovery -- the search for the line that X was to one side of -- and line-discovery is highly prone to outcomes that result in bans on content from the other side of that line. So we don't want to engage in line-discovery, even though there are obvious examples of things to one or the other side of the line.

You may think you can ban the obvious things without ultimately engaging in line-discovery, but, the argument goes, you are mistaken. You will ultimately find yourself doing line-discovery.

You start out with obvious-sounding prohibitions on racism and hate speech, but eventually you're arguing about, say, whether it's racist to report on polling showing that violent protests are unpopular. [0]

And that's because banning any speech always leads to line-discovery.

So it comes down to a question of which scenario is worse:

A. You ban obviously bad stuff while accepting some risk of banning things that aren't actually over the line.

B. You privilege all content to avoid that outcome.

Some people are outraged by this framing and think it's obvious that you would want to risk banning some behavior to the right side of the line if it means eliminating the most obnoxious speech. But, basically, that is not obvious to everyone, no matter how many times they are reminded that there is some really bad stuff out there. [1]

[0] https://reason.com/2020/08/27/protests-violence-david-shor-k...

[1] Interestingly, this is really not so different from the argument about evidentiary standards for punishing criminal behavior, except in that case the politics are flipped. There conservatives would rather risk punishing some innocent people if it means the absolute worst actors are guaranteed harsh punishment, but liberals think it's worth risking some amount of literal rape and murder in order to prevent punishing the innocent. So I think, actually, both sides are entirely capable of seeing this from the other side; they just don't want to.

[moomin]

Many people have said this better than me, but there are plenty of people who have thought they can do better than the current status-quo regarding user-generated content on the internet.

They end up conforming or losing money. There’s no one reason for this. You try to run a website visible to the world, you’re gonna be subject to a world full of reasons.

[KennyBlanken]

> the creators of reddit had a philosophical and political commitment to free speech that drove their light-touch approach to moderation

The notion that reddit ever was a bastion of free speech is absurd. They didn't "light touch" on upskirt, revenge, and kiddy porn because of "philosophical and political commitment to free speech", they did it because they didn't want to accept any more responsibility for content than they absolutely had to, and that's because it is not financially viable to moderate large communities using paid labor. That is why you see so many social media companies pushing against rules for online content; not because they're champions of free speech.

If it were about "a commitment to free speech", they wouldn't allow completely unaccountable and anonymous members to delete content, silently mute users, and ban users....including employing automatic tools that would ban people preemptively based on subreddits they posted in, or automated tools for powermods to ban someone across all the subreddits they moderated.

If you pissed off a powermod, your account could end up getting banned from nearly all the major, common subreddits - not just from theirs, but they'd communicate in private channels to other powermods that they wanted someone to be banned elsewhere.

Oh, and they were happy to moderate, severely, anyone who revealed any personal details about a reddit user. Which conveniently helps protect people doing stuff like upskirting and posting revenge porn.

"philosophical and political commitment to free speech", my ass.

[Red_Leaves_Flyy]

>It's not like the existence of that subreddit is evidence of an endorsement on their part.

It is though. 230 be damned. These were not small or hidden communities. They were frequently on the front page. Generally, and especially in this case silence is violence. The optics of that sub and the frequenters thereof are terrible. Do you want to try and justify their inaction further or concede this point? It should have never been allowed in the first place. Spez/Reddit at al should continue to be shamed for their long-standing tacit approval of these communities. Earning respectability requires public contrition for bad decisions that affect the public and non participants. As is typical, the communities were only shuttered when the victims cries grew loud enough to affect their brand image. Cf fph, wpd, fappening, t_d, Boston bomber fiasco, all the racist subs, and countless other controversies that spez/Reddit fumbled. Reddit deserves to close. The management team is evidently not competent or mature enough for the task and has repeatedly proven that their inability to learn from their mistakes and failure to become the proactive stewards needed will result in preventable harm to people who do not even use the platform.

[freejazz]

> light-touch approach to moderation

Heh. So what did they moderate? things besides jailbait? How free!

[wahnfrieden]

dionidium, dedicated defender of promoted jailbait access and hosting

[scubbo]

I mean, define "endorsement". Permitting something to exist when you have the power to do otherwise is a mild form of endorsement. A commitment to free speech is, to an extent, an endorsement of all the speech that results.

[capableweb]

> A commitment to free speech is, to an extent, an endorsement of all the speech that results.

Absolutely not. I'd argue that anyone should be free to talk with others about their opinion, but that doesn't mean I agree with that opinion. And letting then speak without shutting them down doesn't mean I agree either, just means I agree that they should be able to speak freely.

What kind of dystopian viewpoint is that? You go around stopping everyone from saying stuff you disagree with?

[dangond]

Enabling and endorsement are two different things, no need to conflate the two. If I'm a dentist and tell my patient that they could use any toothpaste they want but that I don't recommend the specific brand that they use, how is that an endorsement in any way? I'm allowing them the choice without endorsing in this case.

[wwtrv]

> Permitting something to exist when you have the power to do otherwise

That's an exceptionally broad definition and would sound pretty scary with no context.

[wwtrv]

> Permitting something to exist when you have the power to do otherwise

That's an excewptionally broad definition and would sound pretty scary with no context.

[oxygen_crisis]

It's not just that he allowed them to exist, he created a special one-of-a-kind "Pimp Daddy" trophy to award to the moderator of r/jailbait and r/creepshots.

https://old.reddit.com/r/TrophyWiki/comments/mohrlg/reddit_t...

There is some whitewash in the comments there: "[violentacrez] received the trophy because all the work he did to moderate the site..." as if he got the award for keeping things clean, but consider that he contributed the vast majority of those subreddits' content himself by cruising social media for salacious pictures of minors to share while he was in his 40's, and the award is named "Pimp Daddy."

[livejamie]

IIRC Violentacerz modded like 50 different porn subreddits, and he did a goob job by moderation standards so he was appreciated by the admins for being the overseer of the porny side of reddit.

[aw1621107]

Sure, and one can absolutely criticize him for that, but I think if one wants to criticize how /r/jailbait and similar subs were handled it's better to do that directly rather than making a more nebulous insinuation that stands on weaker ground.

[predictabl3]

Anyone trying to lose weight and skip a meal today? Open a reddit client and do a subreddit search with "teen" as a prefix. Or don't, you know. Silly me for assuming that this had been solved after any of the various sketchy-porn related subreddit purges.

I'm not naive, user content is hard to moderate. But is it hard to say "any subreddits with these keywords go on a list for review"?

[veave]

It's easy to prove, this is the code change that added the moderator invitation system: https://github.com/reddit-archive/reddit/commit/db7b531a7180...

[chromakode]

I added that invitation flow in response to widespread abuse of the 1-way add moderator button. As an aside, the invitation message felt too bland and clinical during dev, so I added "gadzooks!" to the beginning, which became a meme for a while.

[bee_rider]

It seems like a funny distinction to make in the first place. Was he an admin of the site when it was hosting that sort of stuff? Anyone who was an admin at the time is responsible for the policies that allowed it to exist on the site…

[the_gipsy]

Other users apart from u/andrewsmith1986 also confirmed this, in that thread. Sounds plausible.

[rodgerd]

He was also part of the team that gave the owner of the sub awards. It's not really credible to claim that spez wasn't somehow very enthusiastic about it.

[JoyfulTurkey]

Damn, just looked at this account signup date and mine is older than it. Feeling ancient on a Monday morning :)

[tiffanyg]

Hahaha, please, son ...

Remember when Phrack was added to the LOC ... and was over 7 years old already (http://phrack.org/issues/42/1.html)?

More seriously, you really start to feel "ancient" when your body goes from "new year, pretty much the same as old year" ... to ... "new year, who removed vital organs and bodily fluids while I slept last night?! The bastards!!!"

Of course, it is a classic trope nevertheless ... and I did start to feel ancient even 15+ years ago ... but once you start noticing real changes, then you REALLY get that feel (and you know more is coming, lol): https://youtu.be/MqBNSMbEzI0

[adhvaryu]

https://www.reddit.com/r/dankmemes/comments/1477psa/all_3_ar...

>Back in the day, you used to be able to add anyone as a moderator and it auto accepted.

>People would make shitty subs and add people, take a screenshot, shut down the sun or make private, then use that screenshot to start a witch hunt. Violentacrez could have added you as a mod of the sub and you'd be in the same situation.

>TL:DR I used to mod a sub with Barack Obama and Snoop Dogg.

[kaiken1987]

Oh man that's a username I have seen in a log time. Wasn't there a theory that they were multiple people using a single account?

[wahnfrieden]

This counter-narrative is also misleading

Consider: spez was the voluntary "mod" of the entire platform as CEO, and he maintained that subreddit.

Excusing him for the unsolicited mod invite is just optics management. It would be like saying lowtax had no responsibility as co-signer for the existence of subforums that his paid or unpaid staff maintained.

[invokestatic]

Back when Obama hosted an AMA on Reddit, a bunch of users added his account as a moderator to a bunch of subreddits, including some pretty objectionable ones. This prompted a change that moderators would be invited instead.

[trompetenaccoun]

Doesn't seem that unbelievable when you look into some of the other stuff he's done. For example he secretly used his admin powers to edit user comments from users he didn't like or who criticized him.

[mmastrac]

I don't like what he's done one bit, nor him personally, but this is a really big stretch and not really the level of comment HN should strive for.

EDIT: to clarify -- a big stretch to connect editing comments (which was well-known to have happened) with other actions.

[mrgalaxy]

Parent commenter shouldn't have said secretly. That was something he openly admitted to doing.

https://www.theverge.com/2016/11/23/13739026/reddit-ceo-stev...

[AlecSchueler]

I believe the issue the commenter above was taking was that just because someone commits, for lack of a better term, comment fraud, we shouldn't jump to suggesting he's also a paedophile.

[unshavedyak]

Oh, i took that comment to suggest that the downtime could (mostly jokingly, i assume) have something to do with Spez dealing with a post he didn't like.

[b112]

That was something he openly admitted to doing.

Not true! He did not openly admit to doing this. Instead:

* he did it, and told no one at all

* was caught doing it, after people figured it out

* then, admitted he did it

This is absolutely not the same as "openly admitting it".

He did secretly edit comments.

[DANmode]

It was secret/nontransparent until he admitted to it.

Still misleading, I agree.

[MattGaiser]

He admitted to doing it.

https://techcrunch.com/2016/11/23/reddit-huffman-trump/

[danudey]

"Openly admitting" has a very different connotation than "fessing up after you get caught".

[croes]

I think it's about

> was a mod of r/jailbait in 2008

not the comment editing

[tipsysquid]

the accusation of editing comments I believe to be confirmed as fact, everything else I am not sure of.

[koprulusector]

This is something I’ve seen repeated on Reddit often. Virtually any meta thread on Reddit about Reddit will have several comments containing these allegations. I can’t imagine how much time someone might need to spend to dig through all the noise to get to the truth, but it rings of something that might have a kernel of truth, given the prevalence and uniformity of such accusations.

In either case, it’s easily as speculative as the parent comment above, maybe slightly more so, since the parent came from Twitter.

[drak0n1c]

>“Yep. I messed with the “fuck u/spez” comments, replacing “spez” with r/the_donald mods for about an hour,” Huffman, who co-founded Reddit with Alexis Ohanian in 2005, wrote.

He did not admit to it until evidence was compiled and hit the front page, and let it appear for part of the week to participants and onlookers as if there was massive internal strife.

https://techcrunch.com/2016/11/23/reddit-huffman-trump/

[brazzledazzle]

Completely out of line to connect admin abuse with pedophilia.

[EMCymatics]

It is really really wild for other comments to try to pretend the comment above is about something other than the admin (spez) using his powers to edit reddit comments of people he didn't like or political opponents.

[EMCymatics]

This didn't get enough exposure

[chomp]

This isn't that surprising, you used to be able to add anyone on the entire site as a moderator and it'd autoaccept. It's doubtful he actually moderated it in any capacity. He's still moderator of some random subreddits.

[gpm]

"Pointing out"

Fabricating? Or do you have some reason to believe this is true.

The wayback machine doesn't list him as a mod (SFW): https://web.archive.org/web/20100420183707/http://nl.reddit....

PS. Wow, the sidebar makes that sub seem even scummier than I assumed it was.

[karaterobot]

What's funny about this to me is that the actual moderators of r/jailbait thought "I know how I can insult u/spez, I'll make him a moderator of my sub, so he'll look like a scumbag, like I am"

[satvikpendem]

I'm not sure if this is true, but if I were a creator and admin of a site, I'd assume I'm automatically a mod of every subreddit or subforum. It doesn't necessarily mean spez was specifically moderating that sub.

[3836293648]

There's a difference between an admin having mod access to every subforum and being listed in that specific subforum's list of moderators

[EamonnMR]

I'm surprised he didn't get more flack for this when he last courted controversy by changing posts critical of him.

[mort96]

What surprises me is that he's not ousted by the board yet. Surely at this point he's causing more trouble than he's worth?

[mjr00]

Part of spez's job is to be the lightning rod for controversial decisions. The board I'm sure is pushing for the same things (increases in pricing, driving users to the official app) in order to boost metrics before the IPO. Aside from the somewhat pointless AMA where his frustrations came out a bit too much, if you assume that the effective removal of API access had to happen, what do you think he's done wrong during this?

[TheNewsIsHere]

Many take exception to his handling of Reddit’s relationship with Apollo and Christian Selig specifically.

Steve Huffman has reportedly told employees that Selig threatened Reddit. Selig posted a (perfectly legally recorded and disclosed) call recording showing the alleged “threat” was a misunderstanding over which the Reddit employee on the call apologized immediately.

Huffman serves in an official capacity at the Anti-Defamation League. People are (rightly, I think) critical of his handling partly in light of that.

[baud147258]

Maybe he knows where all the skeletons are buried, so the board removing him would be to difficult (or they're not ready to pay his golden parachute/buy his silence)

[chakintosh]

Back in the day you could create a subreddit and invite anyone to mod it. The invite would be automatically approved. I suspect this is what happened here.

[meghan_rain]

A meme? So was he or was he not?

[nilespotter]

Nobody likes spez and reddit is the worst of the worst in terms of what the Internet could be and what it is. Reddit is where know nothings pontificate as if they were experts on anything and everything while congratulating each other. It used to be a great site, now I'm ashamed I ever used it and I'm embarrassed for people who mention they use it.

That said, that meme is propaganda and defamation. Anyone could add anyone as a mod anywhere for a time, and violentacrez added spez "for the lulz" or however people like that think. I'd be beside myself it there's evidence that he was an active moderator working on that sub. Although, given the prevailing winds of the modern left, in a few years he'll probably be trying to claim that he was.

[paxys]

Pointless speculation. Reddit has constant outages of this exact nature almost weekly. There's nothing special about this one.

[flutas]

Reddit 100% has stated that it was because of the sub blackout.

> According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge.

https://www.theverge.com/2023/6/12/23758002/reddit-crashing-...

[ilyt]

Them "anticipating" "expected" issues with... people visiting reddit less sounds a bit weird...

[Dylan16807]

Making a ton of subs private does a lot more than change view counts.

[SkyMarshal]

Reddit definitely doesn’t have outages like this almost weekly. Maybe 10+ years ago, but it’s rare now.

[nullityrofl]

You can't be using Reddit very often if you think these are a relic from 10+ years ago. I would say I experience a Reddit outage at least weekly. My friends and I have a running joke about how often it's down.

The official Reddit Status Twitter account proves that it happens very, very regularly: https://twitter.com/redditstatus

[29083011397778]

To be clear: You mean you have timeouts and failures using Reddit's own "new and improved" web UI and mobile client? Because using RedReader, old.reddit.com, and other third-party apps, I don't actually recall the last time Reddit didn't load for me.

[nullityrofl]

No, I only ever use old.reddit and BaconReader. I've never used the new UI or the official app.

The official Reddit Status Twitter confirms this is a pretty common occurrence: https://twitter.com/redditstatus

The "elevated error rates" always presents as an "oops, you broke reddit!" landing even on old.reddit. I imagine since it is an "elevated error rate" rather than a total outage that it might be localized to geo or some other kind of shard. I'm on the US West Coast, though, so I can't imagine I'm in a minority.

(Which is confirmed by the number of people responding to GP.)

[29083011397778]

Little late, but I think I see how we have such different experiences. Assuming other comments are right, and Reddit's pulling pretty much entirely from cache, you probably just scroll longer than I do - long enough to run out of the first ~1000 (cached) posts, and hit uncached items.

You'd get timeouts, and I'd never see them - despite being West Coast (Canada) as well. Or at least, that's my best guess so far.

[nullfield]

Well, BaconReader is likely dead at the end of the month with their API changes (along with Apollo, Sync, Reddit Is Fun, etc.)

[yreg]

E.g. https://www.reddit.com/r/RedditEng/comments/11xx5o0/you_brok...

[SkyMarshal]

I use reddit daily, am constantly refreshing certain subreddits. Fwiw I use new reddit but I have all fancy settings disabled so it looks and works like old.reddit. I also use the iOS app daily. I’m also on the US West coast fwiw. And no reddit premium or anything like that. I literally never have outages or “You broke reddit” or stuff like that.

Edit: I wonder if it’s because all the subreddits I’m on are low or medium traffic. I’ve unsubscribed from the front page and /r/all and tend to only read niche subs.

[nullityrofl]

Yeah, I'm also US West Coast and I only ever use old.reddit or BaconReader. The Reddit Status Twitter and number of people responding to you confirm this isn't an isolated incident, though.

[SkyMarshal]

Your reddit status link is showing outages about twice monthly fwiw.

But yeah clearly others have a different experience with this. I wonder why.

[jeremy_k]

Not true at all in regards to 10+ years ago. Once at week, if not more, Reddit fails to load for me. I refresh the page and it appears I've been logged out. Continue refreshing the page until my session is revived and things are back to normal. I'll often open a post and the header will load but Reddit will fail to load the comments with a click to retry loading comments.

[cleandreams]

This has literally never happened to me. I started using it almost daily 3 years ago.

[atchoo]

It's often not working for me in the UK e.g. "You've broken reddit" and "Can't reach CDN".

Maybe it is a regional thing?

[Merad]

Eastern US here. I will get those errors a few times a week but I'd say 80% of the time it works on a refresh and most of the rest of the time it's back within 30s.

[lamontcg]

Also very common for when you post a comment that it appears to not have posted it and returns an error, then you retry multiple times and get multiple comments posted. And even if you don't get hit by one of those outages, they tend to get saved in the comments from everyone else having hit them.

[unaindz]

For me in the EU it fails once a year or so and only on the few occasions I use reddit's own webpage instead of a third party app or web like libreddit.

[SkyMarshal]

Might be, I’m US west coast fwiw.

[fullspectrumdev]

I see outages browsing Reddit almost daily - either the “can’t reach cdn” error or the generic “you broke it” error.

I always just assume the uptimes absolute dogshit.

[static_motion]

I see those every now and again but it's usually a one-off, it'll load properly after a refresh. This time it was down or extremely slow for 10-15 minutes which is definitely uncommon.

[babypuncher]

It's still weekly, the outages just don't last as long.

Source: I'm finally breaking a 13 year Reddit addiction.

[vitorgrs]

I use Reddit daily and I get the outages pretty frequently (twice a month?)... From Brazil.

[davidw]

I see stuff like that all the time. It feels pretty common to me.

[jonathantf2]

Does it? I usually use Reddit on my lunch about this time every day and I can't remember the last time it was down.

[philote]

I run into issues on occasion, usually with loading comments. I use the web interface.

Interestingly, HN is being really slow for me right now and also gave an error when I first loaded it. Maybe something more global is going on, like network or cdn issues?

[gonzo41]

I'm also seeing HN being sluggish. But I have crap Australian internet so YMMV.

[pdimitar]

I have a gigabit internet, am in Eastern Europe, most US services have 40ms or so latency -- pretty strong link.

I had to reload this page 5 page until I was allowed to post a reply. Something is definitely going on.

[piva00]

It is sluggish and throwing some errors. I have a 1Gbps internet connection via fibre in Sweden, so definitely it's not an issue with your connection.

[ktm5j]

Me too on the HN issues. East Coast US here

[yreg]

Or some of the people who would use reddit right now came here. I usually read hacker news when I try to reach reddit and it is down.

[static_motion]

Same thing here, Western Europe on a 500Mbps connection. Maybe higher traffic from people looking for Reddit alternatives?

[sebzim4500]

Presumably you are using old.reddit.com

[riffic]

outages are incredibly common everywhere

[pyeri]

But this time it's different because most high traffic subs have already gone dark, so the traffic must be minimal as there will be very few posters relative to what the server can handle. And yet, I'm getting "You broke reddit" when I try to visit which is quite ironical.

[treis]

Probably a massive shift of traffic to still open subs. That'd probably take it down due to caches suddenly having all the wrong data. No inside info, though.

HN seems to be groaning under the load too and might go down with Reddit.

[rozap]

Maybe it's expensive to generate the 403 for a private subreddit.

A place I once worked had a 404 handler that was extremely expensive, but nobody noticed for a while because 404s are relatively rare. One time a vulnerability scanner took down the site because it was just hitting known vulnerability paths which all 404'd. The code that executed during a 404 was n^2.

[rspoerri]

As long as n=1 it‘s no problem :-)

[rozap]

That's exactly why the problem didn't reveal itself for a while :)

[ljm]

80% of requests to Reddit via their web UI are basically 500s or CDN errors.

Hyperbolic I know, but that’s what it feels like.

[nprateem]

I've been a long-time user of Reddit (I still remember when WSB had only 12m users) and I can't remember an outage.

[activiation]

Probably DDoS more likely today

[b3orn]

Did they DDoS themselves trying to fake numbers to show that the blackout has no effect on daily active user counts?

[activiation]

That would be funny

[SkyMarshal]

Reddit definitely doesn’t have outages like this almost weekly. Maybe ten+ years ago, but it’s rare now.

[Capricorn2481]

No, it doesn't

[BoxOfRain]

Maybe the protest was so successful at driving traffic away from Reddit it looked like something was broken to their monitoring which tried to failover unsuccessfully? If that happened it might be the first case of a site being brought down by the polar opposite of a DDoS.

[cfors]

Could also make a guess that the blackout has changed how their normal traffic patterns operate, causing some issues with autoscaling/hot partitions.

[mrzimmerman]

Yeah, my assumption was that something in some layer of their application isn’t well optimized when asked to return posts from a subreddit that has “gone dark” in whatever fashions the mods chose to do that.

For example, maybe it causes reads from the database take a lot longer than they normally would, locking up the database or causing the process the crash (again, that’s just pure speculation).

[paulmd]

one I've been wondering about is user overview pages. People use those a lot (it's actually my bookmark for getting onto reddit) and yesterday I noticed that a post I made wasn't in my overview, and it's because that sub had gone dark early.

What happens when a user has 99% of their posting in subs that are now hidden, and the API is programmed to produce a fixed 30 comments of history on the overview page? The answer is extremely deep database pulls... you might pull a year of comment history to get 30 comments that aren't hidden. And depending on how they do that, it may actually pull the whole comment history for that timespan, since most of the time posts aren't hidden like this.

I worked at a backend team at work with some very overburdened legacy tables in mongo, and this is the kind of thing we'd think about. Yeah you can use an index, but then you have to maintain the index for every record, and change it every time a sub goes private/public (and we literally were hitting practical limits on how many indexes we could keep, we finally instituted a 1-in-1-out rule). And how often does that happen? Even deleted comments are overall probably a minority such that indexes don't matter, but, this is relational data, you have to know which subreddits are closed before you can filter their results, and mongo sucks at joins. And the mongo instance can become a hotspot, so, just filter it in the application instead for those "rare" instances. Even if they are doing it in mongo, the index/collection they're joining may suddenly be 100x the size, which could blow stuff up anyway.

edit: for me, one overview page is now taking me back one month in comment history. And I comment a lot on subs that are currently closed, so it could easily be throwing away 5-10 comments for every comment it displays.

[ilyt]

I'm guessing hit on the open subreddit mostly goes directly out of caching layer while hit on private one incurs DB hit to check whether user belongs there

[teaearlgraycold]

That’s what Deimos thinks: https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...

[nicce]

An alternative conspiracy theory: put Reddit down for two days to hide the existence of the protests from major users.

E.g. many Google searches lead into private communities now. Word is spreading.

[hn_go_brrrrr]

"You can't take us down! We'll take _ourselves_ down, so there."

[bombcar]

If people see the dead reddit dude they just assume it broke and come back later.

If they see a "this sub is private because reddit is dumb and also evil" they may come back later, they may not.

[marcosdumay]

Honestly, looks as reasonable as anything else about this thing.

[rangledangle]

This is my guess. Looks better on the record, since people are committing only 48 hours anyways (for whatever reason, should be permanent).

[Algent]

I wouldn't be surprised if so many big subreddits being dark is causing issues around denied API calls.

As for the forced reopening, beside the conspiracy this is something that could happen. It's a private company, moderator on strike are a loss of business, they would be 100% in their right to remove all the "traitors" (I'm not saying this would be a smart move, simply that if they really plan to go down this self destructive path it's the best time to do this and prove potential investor they still have control).

[brightlancer]

> It's a private company, moderator on strike are a loss of business, they would be 100% in their right to

Legally, of course. Morally, it is completely unacceptable. This isn't "oh they're jerks"; this is "the system is broken".

A meatspace analogy:

You host a weekly gathering at a restaurant. You decide to temporarily boycott the restaurant to protest some behavior of theirs -- your actions are a loss of business, _so the restaurant decides to host your weekly gathering without you_.

We'd never accept that in the real world, but for some reason we do online -- we fall back to the legal argument that It's A Private Business (which is true) and completely ignore that Reddit doesn't own the community, that the community doesn't _belong_ to Reddit. They own the platform (the restaurant); they don't own the community.

[Algent]

Oh I completely agree this would be the most stupid thing they could do. I wrote it because this is something I can picture happening at some point (I do expect the protest to multiply), you say they don't own the community but I'm pretty sure they think they can control it.

[abluecloud]

where the hell are they going to fine thousands of willing users to become mods of thousands of communities, overnight?

[brightlancer]

I think there are millions of folks on Reddit who would jump at the chance to be a petty dictator of a subreddit.

[Deestan]

That's easy.

Less if you add "competent" to the qualifications list :)

[commandlinefan]

> a failed attempt to re-open all closed subreddits and instate their own moderators

My first thought this morning was "if I was reddit, I would re-open all closed subreddits and instate my own moderators"... I'm kind of surprised they didn't. I'd have to imagine that if they wanted to, the attempt wouldn't fail - this seems like it could be done almost trivially.

[Sebb767]

They'd just replace their unpaid workers by paid ones and annoy the userbase even more in the process. Doesn't sound like a great plan.

[humansubjects]

I think you are underestimating the amount of unpaid labor that keeps reddit functioning

[ryanstorm]

Attempting this to be more of a discussion prompt than speculation, but as an advertiser or other consumer analytics customer of reddit, would this sort of protest and the potential fallout be concerning? I'm not sure if all this is a drop in the bucket for those customers or if it's significant. If it was significant, would reddit be scrambling to save face for these advertising customers? An outage like this would certainly skew the difference in their analytics for today's relative traffic.

[FireBeyond]

I can't imagine it either, though it wouldn't be outside the realm of some of the stupidity that has gone on. But it would almost certainly be the death knell for the site - those subreddits who didn't "go dark" now would.

[hgsgm]

They don't have people to be moderators of all those subs. That's why subs have volunteer mods.

[irthomasthomas]

HN getting hugged, too.

[tester457]

> (I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much)

> I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process.

https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...

[jackson1442]

definitely not the case, reddit has stated that they would not be doing that, at least for the two-day protest. would be incredibly bad PR at this scale, especially after repeatedly and directly making this statement.

however, they have stated that may do this if the protest extends beyond the 48h mark.

[ben7799]

Wild. Probably 50% of the subs I subscribed to are gone, and some of the others have deleted all the content and locked posting to the sub.

Others look like they got the hostile takeover and a new mod.

[lsaferite]

Could you point out some that appear to have had a hostile takeover? I'd love to check it out.

[nonstopdev]

That is an interesting point. Would assume your subreddit isn’t yours so at any point you could grow and generate content and users and Reddit could take it over and add in their own moderators?