Whenever I see a product this large having an outage (a lot recently, especially with Facebook, GCP, and AWS), I can only think of how stressful it must be for whoever needs to fix it. I've been on the other side of the outage before, albeit with a much smaller product, but lord was that stressful. Thinking about the random engineers that are stressed out and thinking that they could be fired for this, even if the cause wasn't their fault (since management at large companies can be pretty thick) is very upsetting to me. Say what you want about a large company having a large outage, but it's normal engineers that are trying to fix it at the end of the day, and I can sympathize.
I can also sympathize with the engineers trying to fix this, but I hope that they wouldn't be thinking that they could be fired for this. Successful teams that I have worked on, even at big companies with high-usage products, have always promoted a culture of "systems break - let's improve the system, not blame a person". Any 'mistake' by an employee is actually a sign of a problem with the system. Any resilient system should account for human errors - those always happen. I wouldn't want to work for a team or company that would consider firing somebody for causing an outage rather than addressing the root cause.
Response for this incident went by the book, as described in Brent's talk above. Incident Management programs like these ensure that incidents can be resolved while also minimizing stress and chaos for engineers and other responders.
PagerDuty has a good Incident Responder and Incident Commander training courses, if you are interested in setting up a program similar to Slack's:
stressing out and panicing is the worst thing you can do in this situation.
usually, keep your head cool and focus on the problem at hand.
This is also the reason why engineering/operations should be seperate from customer communications. The people who are fixing the issue should not also be the ones doing the communication with the outside world.
The cool thing about Slack (and Discord for that matter) is that it's essentially federated on a technical level. SlackHQ controls all the instances, but the UX is very achievable by an open source alternative. I'm bullish on federated chat in the next few years.
It would be awesome to have essentially the same experience as using Slack, but also a server running on our lab's hardware that we can reboot/debug ourselves.
They've got to nail the integrations. Anytime moving to slack alternatives is brought up with people I work with, it always comes down to everything integrating with slack.
We aren't even particularly dependent on slack, but the convenience of a few basic integrations like google calendar make moving to Discord or anything else a non starter.
The integrations, and chat itself needs an open protocol, otherwise it's just going to be more and more of slack in the future.
I understand that's one of at least several, and that if Slack isn't actively making trouble, looking for alternatives is a switching cost that may itself be prohibitive -- but if Slack is becoming really troublesome, it might be easier than you think.
That's interesting. We've played with a couple integrations but don't use any. What sort of org are you in? We're an academic bioinformatics lab with 15-20 people.
Not OP, but we have numerous integrations. Technical, corporate, and fun. GitHub, Jira, Status Hero, Outlook Calendar, Asana, Conference Booking, polls, "water cooler" type apps, and more. I actually can't get a comprehensive list right now because Slack is down...
We could make do without many of these integrations if we had to but the integrations and webhooks are THE killer feature to me. Chat is easy to solve.
FWIW, some of the hosted Slack alternatives, like Zulip, support at least versions of some of these -- see their integrations page here: https://zulip.com/integrations/
What would be ideal from this perspective would be if they could make the integrations somehow code-compatible with Slack, but that's tough. For example, Zulip's topic system is, IMHO, vastly superior to Slack's threads, but every public message is under some topic (within a stream -- there are two levels, not one), everything posting a public message has to supply one, and that alone might make it difficult to reuse a Slack integration unaltered.
I think "first comer's advantage" will win over for the really big outfits like apple, ibm, intel, etc because they've gone through integration pains with other services and don't want to go through that pain organization wide. However, I think small to mid-size businesses could find some migration from slack to matrix pretty advantageous.
I'm pretty familiar with the various alternatives. Mattermost isn't federated, right? Anything without a strong inter-instance collaboration story can't replicate the experience of Slack/Discord.
Tried with my friends a few years ago. We almost achieved escape velocity, but it wasn't ready. I've been keeping an eye on it, and my feeling is while it may be ready for what we needed then, it's not ready for what we need now, because now it has to be good enough to not just work but assuage my friends' bad feelings from the last time.
I think I'll probably try again once most people have switched to a server written in Go or Rust.
One fun thing we discovered an outage or two ago: Zoom, which is likely already on your computer too, has a very Slack-like Chat feature. Open up the Zoom app, then click the Chat icon in the top header. You probably missed it this entire time. You can make rooms, DMs, animated gifs, the works :)
It's strange but Zoom as chat was actually how I first used it. A whole year prior to the pandemic the place I worked (fairly large) they purchased Zoom as their internal chat tool.
Slack does a good job with threading. The side panel allows you to watch a thread while you continue another conversation. It also puts together an above average notification system, where at a glance you can see if there is no new content, new content, or your name mentioned. It allows you to ignore slack without risk of missing somebody shouting for you.
On the contrary, I find Slack threads to be very annoying for discovering/stumbling upon conversations.
Your whole team could be having a new discussion within a thread and you'd never know unless you participated in the thread previously or somebody @'s you. Compare that to more IRC-like discussion where each message is posted to the channel. Sure, you may end up with contextual issues reminiscent of the IRC days, but at least everything is discoverable.
I've used Slack competitor Flowdock in the past which did the more "elevated" approach to threads. Rather than "hide" them from the main branch of a channel, the main branch had every message in the top level and you could collapse to a thread view if you wanted to read it in isolation.
In my experience, the main problem here was that people regularly lost sight that a conversation even was in a thread. So they'd post their response unthreaded right in line. Now the people in the thread view can't see it, but if you were in the main channel it still looked just fine (after all they were one after the other). If it was a nascent threaded conversation (like 2 or so posts only), often the thread was just abandoned and everything was done top level. For bigger threads where you caught after the fact it was a threaded, people would either delete and repost or would just duplicate their post over into the thread again (which means it appears twice in the main channel). The worst offenders were people who refused to thread so even if it was humming along fine they'd just post their response inline always (and repeatedly).
Don't know if I have a particular point there other than both had tradeoffs. I do find that the Slack thread updates disappearing seems to be less frequent of an issue and the threading problems in the alternative were near constant. I think Slack's issue would be largely a nonissue if viewing a thread "subscribed" you to it so you could see updates without needing to post. Maybe reactions are a workaround too. It's fairly easy to unsubscribe if one's blowing you up. Subscribing to all threads regardless seems to reduce some of their usefulness and that's the current solution I believe.
> Your whole team could be having a new discussion within a thread and you'd never know unless you participated in the thread previously or somebody @'s you.
That's kind of the point? You shouldn't be bothered unless you want to be. To me, a thread isn't necessarily supposed to be geared toward discoverability. You'll see the main topic hit the channel, and then you can choose to follow the thread if you like.
That spyware thankfully isn't no mine, and if you're forced to use it (if you for whatever reason can't use Jitsi/Teams) then I strongly encourage you not to install the app and instead use the web version.
Please don't encourage the use of Zoom, they already have enough market share they don't deserve after lying through their teeth to the public on multiple occasions.
What alternatives do you recommend? As far as I can tell none of the major video conferencing solutions are made by companies I think are any more trustworthy than Zoom at this point. Plus Google and others ran a focused assassination campaign against Zoom at the start of the pandemic to claw some of their market share.
Google is certainly more secure than Zoom to outside attacks.
Regardless of what Google chooses to do with your data, they do want to protect that data from everyone else, on your behalf. At least as much as any company can (government subpoena affects all companies).
I don’t understand why zoom is so much more popular than Google meet. I never used zoom before my current company who uses Google meets for daily meetings and then zoom meeting for the biweekly review. No idea what advantages zoom is supposed to have.
I think Zoom just had a better sales team and went after corporate clients earlier. Now that no one is in the office, it seems silly, because Meet is just straight up a nicer experience on all fronts, but in the Old Days, no one ever got fired for putting Zoom hardware in a conference room.
It seems like the popularity of Zoom is related to the singularity of the product's purpose. Zoom is a video conferencing service. Google is everything and the kitchen sink. So when the pandemic started, that worked in Zoom's favor, sadly.
I know it’s not seen as a professional service, but Discord has great quality video chat (better than Zoom, for example). I don’t know if its privacy/security would hold up for larger companies but we use it in my tiny company with no issues at all.
Any recommendations on getting Discord notifications to reliably show up in a reasonable time on mobile? That's the one big issue I have with Discord. Otherwise I find it very useful.
If you have the desktop client up, you don't get mobile notifications. They've always worked fine for me so long as I don't set my status as do not disturb
> In its complaint, the FTC alleged that, since at least 2016, Zoom misled users by touting that it offered “end-to-end, 256-bit encryption” to secure users’ communications, when in fact it provided a lower level of security... In reality, the FTC alleges, Zoom maintained the cryptographic keys that could allow Zoom to access the content of its customers’ meetings, and secured its Zoom Meetings, in part, with a lower level of encryption than promised.
Wikipedia has a few paragraphs dedicated to their lack of privacy, security, and other unethical telemetry practices. I believe it was a couple years ago when these stories maintained headline positions for a few weeks.
Things like this remind me how dependent I am on IM for doing any work, which is weird because it doesn't feel like that should be the case. If you were to measure the raw number of seconds I spend sending IMs, it would be a relatively low percentage of my day, but I always forget how vital that percentage actually is.
As it stands, while I'm not fully blocked from being productive, I am blocked for a lot of work that needs to get done.
Yep. Especially if your development flow has integrated Slack alerts.
Coupled with COVID work-from-home constraints, there's no practical way right now to get code reviewed for merge into the main codebase 'round these parts.
Just set this up real fast as a quick backup, since we already use Gmail and everybody is already in it. Having a non-AWS hosted solution seems important to have a Slack backup for me, as well. Oh hey would you look at that, they support webhooks too...
We switched from Slack to Teams for various reasons about two years ago. I'm still mad about it. Teams blows for chat. The video meetings are fine though.
The MS Teams UX is pretty painful, especially for infrequent, communication-heavy users. Not painful enough to outright prevent work, but clearly more painful than the repercussions of missing a few hours of work.
This seems at best a funny tweet. They're both just tools and frankly terrible in terms of "getting work done". This seems like a great time to go heads down on a deep task and if you really need to contact someone send an email or even phone them.
Surprised no one is talking about how awesome slack is. Think about it in user stories:
* I want to find someone by name => ctrl+K
* I want to search with sane keywords => search, "from:me" "to:<channel name>" work.
* I want to remember someone I talked to recently => people you chat with show up in the list on the left
* I want to keep com channels organized as the ground changes => easily rename channels, favorites UI works well, channel grouping works right.
* I want to give a public emotional response to someone's statement => reactions
* I want to continue discussion of a point someone made, which may not be relevant to everyone in the channel => threads
* I want to edit a typo I just made => hit up on keyboard
It's got that quality Factorio has, where you can let yourself imagine it is the ideal product, and start expecting features you need to be there, rather than not bothering to explore because you think they won't be.
It is AMAZINGLY good at solving actual user stories around communication. I have lots of respect for their PMs.
Sorry, but dropping Markdown support in favor of an input UX that is to this day atrocious gives me little faith in Slack's user stories moving forward. I'm fine if they want to make the default mode more intuitive for a broader audience, but I have yet to hear a good argument for not having a little button to enable raw Markdown mode.
EDIT: See comments below for instructions on re-enabling Markdown mode.
This was broken when I tried it out during the Markdown-->WYSIWYG switchover a while back. I updated my OP to show they have actually fixed this (go Slack!). I just switched back to Markdown again.
I still have my pitchfork from all the bugs I encountered when they released the new formatting (still works wonky depending on the workspace for me). That said, I believe this is an option since they changed things:
I have yet to hear a good argument for not having a little button to enable raw Markdown mode.
That toggle is buried in the preferences/settings under "advanced."
Format messages with markup
The text formatting toolbar won’t show in the composer.
* I used to be able to type a message containing "@channel" or "@here" and just hit Enter without looking, and 100% of the time it would be parsed correctly. Now it fails to detect the "@channel" or "@here" often enough that I have to look to see if it failed, then go back and retype that part of the message until it lights up. Once a week or so, I see a message posted by someone else that contains "@channel" in black, and they expected the message to notify everyone but it didn't.
* Recently Slack pushed everyone to switch from usernames to full names with spaces, and eliminated the entire concept of unique usernames. There were many annoying consequences to this, one of which is that when I type "@name", now I have to look to see if it highlighted, or look to interact with the drop-down menu, instead of just typing and knowing it will work.
* Searching also got worse for the same reason. When I type "@name" in the search bar, it NEVER lights up. For 90% of searches this means I can no longer type in a search query and just press Enter. I always have to look through the drop-down menu and either mouse-click or press down-arrow repeatedly to get to the thing I want.
Ah but ctrl-v while highlighting text with a URL on the clipboard creating a link is such a nice experience. I honestly wish I could do that everywhere.
I'm curious about how many users are on your organization's Slack. I consider it one of my go-to sources for my org that has had between 250-300 people for the past 3 years. I can see it being suboptimal for a bigger company, though.
I'm in an organization with many thousands of people and find search to work very well.
Does the pricing plan influence the amount of resources your organization can use? e.g. slow search on the free plan, but much better on Enterprise Grid.
> A configuration change inadvertently lead to a sudden increase in activity on our database infrastructure. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. We introduced tighter rate limits on connection requests to reduce the load on the system. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected.
> Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. However, we moved too quickly and the increased activity affected the system again. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.
I wonder if their bottleneck is at vitess vttablet or at mysql.
I run an IRC network in my spare time that has had 13 minutes of actual downtime in the last 15 years (and about 20-25 additional minutes of degraded availability).
The degraded availability thing is because I run multiple servers for latency, normally an IRC outage is super clear, except with netsplits where IRC servers themselves unlink from each other. (I average about 3 of those a year and they last about 30s roughly).
It's not video though, so that doesn't answer your question thoroughly.
At first, I thought it was just my internet connection making fun of me. Then, I asked my co-workers if they experienced the same thing, and they did. It's a bummer that it temporarily made our communications quite hard and unstable.
Slack was down to me, and now that it is back, it is begging me to subscribe to use paid features, with the nag screens blocking my view of important stuff.
I guess it was down so they could do this crappy update.
Most of my company have spotty connection. Down for Windows and Linux alike. Interestingly (and several have commented this already), the mobile app seems to be somewhat working.
Something’s gone awry and we’re having trouble loading your workspace.
Sorry we can’t be more specific – this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us.
In all seriousness, since I'm sure a lot of people don't know this, use SHIFT-CTRL-r. That instructs the browser to reload everything, ignoring cached resources. This can occasionally "restart" a web page that even a reload or loading the URL in the URL bar again is not sufficient for. Slack is one of those things I've seen it work on.