[toddmorey]

I've been part of a response team on a security incident and I really feel for them. However, this initial communication is terrible.

Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?

The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.

The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.

I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.

[birdsongs]

Seriously. Why am I reading about this here and not via an email? I've been a paying customer for over a year now. My online news aggregator informs me before the actual company itself does?

[shimman]

Please remember that this is the same company that couldn't figure out how to authorize 3rd party middleware and had, with what should be a company ending, critical vulnerability .

Oh and the owner likes to proudly remind people about his work on Google AMP, a product that has done major damage to the open web.

This is who they are: a bunch of incompetent engineers that play with pension funds + gulf money.

[throwanem]

This industry's favored idiot children.

[1970-01-01]

I just deleted my account. Their laid-back notice just is not worth it anymore. I will hold them accountable using my cash. You can get out with me. Let their apologies hit your spam filter. They need to be better prepared to react to the storm of insanity that comes with a breach or they lose my info (lose it twice, I guess..)

[salomonk_mur]

Says they emailed affected customers...

[btown]

Via the incident page:

> Environment variables marked as "sensitive" in Vercel are stored in a manner that prevents them from being read, and we currently do not have evidence that those values were accessed. However, if any of your environment variables contain secrets (API keys, tokens, database credentials, signing keys) that were not marked as sensitive, those values should be treated as potentially exposed and rotated as a priority.

https://vercel.com/kb/bulletin/vercel-april-2026-security-in... as of 4:22p ET

[aziaziazi]

The “sensitive” toggle is off by default. I’m curious about the rationale, what's the benefit of this default for users and/or Vercel?

https://vercel.com/docs/environment-variables/sensitive-envi...

[throw03172019]

Simpler for vibe coders.

[aziaziazi]

Ok but it's not the original intent: that default exists since at least 2020: https://web.archive.org/web/20201130022511/https://vercel.co...

[loloquwowndueo]

Sensitive environment variables are environment variables whose values are non-readable once created.

So they are harder to introspect and review once set.

It’s probably good practice to put non-secret-material in non-sensitive variables.

(Pure speculation, I’ve never used Vercel)

[_heimdall]

I have used Vercel though prefer other hosts.

There are cases where I want env variables to be considered non-secure and fine to be read later, I have one in a current project that defines the email address used as the From address for automated emails for example.

In my opinion the lack of security should be opt-in rather than opt-out though. Meaning it should be considered secure by default with an option to make it readable.

[jtchang]

How does the app read the variable if it can't be read after you input it? Or do they mean you can't view it after providing the variable value to the UI?

[ctmnt]

They mean the latter. Very unclear how that translates to meaningful security.

[btown]

You could have a meaningful wall between administrative/deployment interface backends and the customer server backends - only the latter get access to services that have the private keys to decrypt the at-rest storage of secure variables, and this may be fully isolated to different control planes. So it becomes write-but-not-read.

But that's just a bare-minimum defense-in-depth. The fact that an attacker was able to access the insecure variables, and likely the names of secure variables, is still horrifying.

[ctmnt]

I agree / hope that’s what they meant. It seems disingenuous, though, to describe it as unreadable, since obviously something has to read it to bake it into the deploy. And given their apparent lack of effective security boundaries in one area, why should we assume that they’ve got the deploy system adequately locked down?

It’s not like I had a ton of trust in them before, but now they’ve lost almost all credibility.

[gherkinnn]

Last year Vercel bungled the security response to a vulnerability in Next's middleware. This is nothing new.

https://news.ycombinator.com/item?id=43448723

https://xcancel.com/javasquip/status/1903480443158298994

[tcp_handshaker]

Security is hard and there are only three vendors I trust: AWS, Google and IBM ( yes IBM ). Anything else is just asking for trouble.

[esseph]

Having worked both public and private, I can agree with this.

Google in particular has been staggeringly good, and don't sleep on IBM when they Actually Care.

[dd_xplore]

Oracle too

[gustavus]

Oracle? Oracle?

The Oracle that published an announcement that said "we didn't get hacked" when the hackers had private customer info?

The Oracle that does not allow you to do any security testing on their software unless you use one of their approved vendors?

The Oracle that one of my customers uses where they have to turn off the HR portal for 2 weeks before annual performance evaluations because there is no way to prevent people from seeing things?

The only reason Oracle isn't having nightmarish security problems published every other week is because they threaten to sue anyone that does find an issue.

Oracle is a joke in every conceivable way and I despise them on a personal level.

[warmedcookie]

I love a good cathartic rant

[0xmattf]

> The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.

This and because it's so convenient to click some buttons and have your application running. I've stopped being lazy, though. Moved everything from Render to linode. I was paying render $50+/month. Now I'm paying $3-5.

I would never use one of those hosting providers again.

[nightski]

Looking at linode, those prices get you an instance with 1Gb of ram and a mediocre CPU. So you are running all of your applications on that?

[0xmattf]

Personal projects/MVPs/small projects? Absolutely. For what I'm running, there's no reason to need anything beyond that.

The point is, I used to just throw everything up on a PaaS. Heroku/Render, etc. and pay way more than I needed to, even if I had 0 users, lol.

[lelanthran]

> Looking at linode, those prices get you an instance with 1Gb of ram and a mediocre CPU. So you are running all of your applications on that?

I ran a LoB webapp for multiple companies on a similar setup. Turns out 1GB of RAM is insufficient to run even the most trivial Java webapps, like Jenkins, but is more than sufficient for even non-trivial things using Go + PostgreSQL.

Your stack may be slow, not the machine.

[Orygin]

Most of my services run with 1vCPU and 512Mb of ram. You don't need huge specs for most normal applications.

[adhamsalama]

For $3.5, Hetzner gives 2 vCPU, 4GB RAM, 40 GB SSD, and 10 TB of bandwidth.

[eatery1234]

Pretty oversold iirc, but then again, that's the same for Linode

[normie3000]

Do you mean these are shared instances, and the stated resources are not actually available?

[skeeter2020]

how much work should the GP do to migrate if Linode is good enough, to potentially save up to $1.50/month (or spend 50 cents more)?

[cleaning]

If you're only paying $3-5 on Linode then your level of usage would probably be comfortably at $0 on Vercel.

[0xmattf]

It could be $0 on Render too, but then there's going to be a 3 minute load time for a landing page to become visible, lol. So if you don't want your server to sleep, you're going to have to pay $20/month.

Does Vercel do the same?

[somewhatgoated]

No, I run several small websites on Vercel for free for years, always served static pages very quickly

[0xmattf]

Static pages, sure. But what do you do if you want a contact form or something? Yeah, you can use services like formspree, but then you may end up paying $20/month for that alone. Perhaps I'm just ignorant.

[anurag]

Render offers free static sites that are served via a CDN and load instantly: https://render.com/docs/static-sites

[0xmattf]

When I said landing page, I had contact forms and more in mind, not documentation sites.

But that is news to me. Interesting. Although for static sites, I always use Netlify or even GitHub pages.

[cleaning]

No.

[arch-choot]

Repeating a prior comment I've made about this[0]: I run a rust webserver on a €4 VPS from hetzner that serves 300M (million) requests a day.

From what I can figure out, Vercel charges "$0.60 per million invocations" [1], which would cost me $180 per day.

[0] https://news.ycombinator.com/item?id=47611454 [1] https://vercel.com/docs/functions/usage-and-pricing#invocati...

[mmastrac]

I run a Rust webserver on a literal Pi3 in my basement and I think I managed to bench it up >1000 rps for standard loads. And that includes a bunch of tanvity querying as well.

I suspect I could do 3000+ rps with some tuning and a more modern CPU or hetzner VPS, but there's some fun cachet from running on an old Pi while there's still headroom.

[esseph]

Makes sense considering the quality of Vercel's security response and customer communication.

[00deadbeef]

What if they have an actual back-end with long-running processes and scheduled tasks?

[p_stuart82]

exactly people paid the premium so somebody else's OAuth screwup wouldn't become their Sunday. and here we are.

[rybosome]

Completely agreed. At minimum they should be advising secret rotation.

The only possibility for that not being a reasonable starting point is if they think the malicious actors still have access and will just exfiltrate rotated secrets as well. Otherwise this is deflection in an attempt to salvage credibility.

[lo1tuma]

Yeah, given there insane pricing I think the expectations can be higher. Although I know it is impossible to provide 100% secure system, but if something like that happens, then the communication should at least be better. Don’t wait until you have talked to the lawyers... inform your customers first, ideally without this cooperate BS speak, most vercel customers are probably developers, so they understand that incidents like this can happen, just be transparent about it

[elmo2you]

Welcome to the show.

While a different kind of incident (in hindsight), the other week Webflow had a serious operational incident.

Sites across the globe going down (no clue if all or just a part of them). They posted plenty of messages, I think for about 12 hours, but mostly with the same content/message: "working on fixing this with an upstream provider" (paraphrased). No meaningful info about what was the actual problem or impact.

Only the next day did somebody write about what happened. Essentially a database running out of storage space. How that became a single point of failure, to at least plenty of customers: no clue. Sounds like bad architecture to me though. But what personally rubbed me the wrong way most of all, was the insistence on their "dashboard" having indicated anything wrong with their database deployment, as it allegedly had misrepresented the used/allocated storage. I don't who this upstream service provider of Webflow is, but I know plenty about server maintenance.

Either that upstream provider didn't provide a crucial metric (on-disk storage use) on their "dashboard", or Webflow was throwing this provider under the bus for what may have been their own ignorant/incompetent database server management. I guess it all depends to which extend this database was a managed service or something Webflow had more direct control over. Either way, with any clue about the provider or service missing from their post-mortem, customers can only guess as to who was to blame for the outage.

I have a feeling that we probably aren't the only customer they lost over this. Which in our case would probably not have happened, if they had communicated things in a different way. For context: I personally would never need nor recommend something like Webflow, but I do understand why it might be the right fit for people in a different position. That is, as long as it doesn't break down like it did. I still can't quite wrap my head around that apparent single point of failure for a company the size of Webflow though.

/anecdote