[cogman10]

> Now, all of this might have been fine had Beyond Trust not written a feature which allowed users to directly, programmatically interact with psql (the postgres command line interface).

That's the buried lede.

Yes, there was a vulnerability in psql... but that's so much less a problem than the huge gaping hole of allowing users to directly interact with psql.

No DB can be safe if you are turning untrusted user commands into psql executions. It'd be like giving untrusted users ssh access and then complaining when they find a privilege elevation exploit.

[gowld]

Let's be clear: Beyond Trust is not a company that wrote a database-backed web app and made the all-too-common mistake of writing insecure code that tickled a bug in the database that allowed privilege escalation. Beyond Trust's is a company whose entire contribution is adding a security layer to prevent privilege escalation, and their solution here was to bypass Postgres's standard functionality and use this weird `psql` hack instead.

They had one job, and they failed at it. This amateur-level mistake should sink the entire company.

[pjc50]

Didn't happen when Crowdstrike broke all their customers.

The problem is that getting information security right is a matter of process control, which everyone hates, and so CEOs are absolute suckers for being sold a product which magically "adds on" security. This is like trying to buy "anti-lead-paint" rather than actually remove all your existing lead paint.

[1oooqooq]

but they got the right sales people to get to the IRS and they hired the "right" (wrong) certification company. so that's three jobs at least.

we now have one job to ask for accountability and will not do it.

[somat]

There is an interesting project https://github.com/Abstrct/Schemaverse where the whole game takes place entirely within a postgres database that the players connect directly to.

The author discovered quite a few... well lets just call them policy errors in postgres, but had a hard time filing bug reports, mainly because the response was usually an incredulous "why on earth would you even do that in the first place?.

But the author has fun with this, There is a trophy in the game that you can only get by putting your name in the trophy table.

[0xbadcafebee]

The only reason BeyondTrust implemented that was it wasn't untrusted user commands. They sanitized the data, so it should have been fine. The unfortunate problem was that the sanitizer didn't sanitize.

Systems are built on a set of expectations. Undermine the expectations and you undermine the system.

[cogman10]

> They sanitized the data, so it should have been fine.

This is a 101 rookie level approach to SQL or injection defense.

It's dumb for exactly the same reason why this is dumb

    "SELECT * FROM foo WHERE bar=" + sanitize(userInput)

The correct way to do something like this will always be parameterized input which looks something like this

    "SELECT * FROM foo WHERE bar=?"
    bindParameter(1, userInput);

Why? Because that the postgres protocol splits out the command and the data for the command in a way that can't be injected. Something that should be viewed as impossible to do when data and command are merged into 1 String.

IF this company wanted to build dynamic queries, then the only correct way to do that is to limit input to only valid variables. IE "isValidColumnName(userInput)" before sending the request. And even then, you'd not use psql to do that.

You simply can't use a generalized sanitizer and expect good results.

[voxic11]

This is the article I link my developers whenever I see them making this mistake "Don’t try to sanitize input. Escape output." https://benhoyt.com/writings/dont-sanitize-do-escape/

Its been fairly effective at making them realize the fundamental mistake they are making. Quoting the key part:

> The only code that knows what characters are dangerous is the code that’s outputting in a given context.

> So the better approach is to store whatever name the user enters verbatim, and then have the template system HTML-escape when outputting HTML, or properly escape JSON when outputting JSON and JavaScript.

> And of course use your SQL engine’s parameterized query features so it properly escapes variables when building SQL

[TeMPOraL]

> and then have the template system HTML-escape when outputting HTML, or properly escape JSON when outputting JSON and JavaScript

Or, stop using stringly template systems, and treat the data as what it is: a structured language, with well-defined grammar.

One of these days I need to write an article titled "Don't play with escaping strings. Serialize output.". Core idea being, "escaping your output" still looks too much like "sanitizing input"[0], and one tiny mistake is all it takes to give an attacker ability to inject arbitrary code into the page (or give an unlucky user ability to brick the page for themselves) - so instead of working in "string space", work in whatever semantics your output is, and treat the string form as a serialization problem. In case of HTML, that means constructing tree of tags as data structure, and then serializing them. Then, bugs in serializer notwithstanding, the whole class of injection problem disappears - you can't do "<h1>$text</h1>" -> "<h1></h1><script .... </h1>", when your "template" is made of data structures like [:h1, $text], because $text can't possibly alter the structure here. Etc.

In some sense, "Don't escape, serialize instead" is the complement of "Parse, don't validate".

(See also: make invalid states unrepresentable.)

--

[0] - Who ever sanitizes input? I've only ever seen this kind of sanitization the article describes happen in the output, within string-gluing templates.

[oefrha]

The more you work in software the more you should realize the developers writing security-critical software (in this case the one writing that sanitizer) are often/usually as clueless as you are. The solution? Hard to say.

[namaria]

I have had a developer look me dead in the eye and say "this is not a security concern, because I can't see how this can be exploited".

Security by obscurity from self. It was very hard to explain to that person what was wrong with that line of reasoning.

[immibis]

But... You have to see things that way or else literally everything becomes a security concern.. Extra whitespace before a semicolon? I don't see how it can be exploited, but with the mindset you imply, I have to treat it as a security concern. But removing the whitespace is also a security concern.

[namaria]

Yes, general computers are fundamentally unsafe. We should always think about threat models, vulnerabilities, blast radii, defense in depth.

What we should never do is dismiss something as a non-concern because we don't know how it could be a problem. Especially when someone is trying to point out something we're doing is extensively documented as a security concern. In that case it would be quite obtuse to claim in a public discussion that the person pointing it out is wrong because you don't understand the issue, and yet I have lived through that.

[D-Coder]

> It was very hard to explain to that person what was wrong with that line of reasoning.

"So... you're the smartest person in the world?"

[lazide]

It’s very hard to argue with someone who asserts ‘what I don’t know can’t hurt me’, because usually they’ll refuse to know anything that will hurt them. Like that there are things they don’t know, that can hurt them.

[thaumasiotes]

> The correct way to do something like this will always be parameterized input which looks something like this

> Why? Because [] the postgres protocol splits out the command and the data for the command in a way that can't be injected.

I'm not sure I'm comfortable with this. You can create a prepared statement and then pass user input to it as parameters, sure. https://www.postgresql.org/docs/17/sql-prepare.html

But who says that means your statement can't be injected? It would have to be true that the handling for EXECUTE statements is bug-free. Maybe it is bug-free. Or maybe it isn't. Maybe I can figure out just the right username to cause your prepared statement to have some undesirable side effects.

That wouldn't be SQL injection in the sense of putting hostile values into an SQL query in order to form a different SQL query, but it would be SQL injection in the sense of putting hostile values into an SQL query in order to accomplish nefarious goals via the database. The only real qualm I'd have about calling it "SQL injection" is that it wouldn't be portable across different database implementations; it would be more accurately described as "PostgreSQL injection".

If we feel entitled to assume that PostgreSQL's provided functionality to interpret strings that are provided to EXECUTE statements has no bugs, why aren't we also entitled to assume that PostgreSQL's provided functionality to interpret strings that are provided to the string escaper has no bugs? I don't really see the conceptual difference.

[TheDong]

The postgres escape function actually worked fine before this "CVE". It was documented as escaping something for use as part of a postgres query.

BeyondTrust used it as input to the 'psql' tool, which is an interactive tool you're not really supposed to programmatically invoke, and the documentation for the postgres escape function didn't say it escaped input for psql. Even though postgres was fine calling it a CVE and fixing it, I think this is 100% on BeyondTrust for assuming that escaping a string for a postgres query meant it was safe for psql.

If BeyondTrust had just used it as part of a postgres query string, the escape function would have been sufficient.

.... and that's also exactly the reason that using parameterized queries is better. With parameterized queries, the escaping and the query parsing are done in the same place, so there's no chance of confusion for the programming language's string library to get in the way, or for the psql tool's input parsing to re-interpret and alter the escaped string before sending it over the wire.

[oefrha]

> If BeyondTrust had just used it as part of a postgres query string, the escape function would have been sufficient.

That's completely false. The following (pseudo code because working with C strings is verbose and beside the point) with nothing to do with psql should be fine:

  PQexec(conn, "SELECT * FROM user WHERE nickname = '" + PQescapeString(user_input) + "';")

but thanks to the vulnerable PQescapeString(), the following user_input

  "\xc0'; DROP TABLE user"

would fuck it up. That's just the failed escape function leading to a classic SQL injection. Using psql makes it worse because psql can execute additional non-SQL commands, but this escape function is not "fine" at all with or without psql.

> With parameterized queries, the escaping and the query parsing are done in the same place

Again, wrong. For parametrized queries, params don't go through serialization because they don't need to hit the parser, there's no "escaping" whatsoever.

[TheDong]

> but thanks to the vulnerable PQescapeString(), the following user_input would fuck it up

Nope. To quote: https://www.rapid7.com/blog/post/2025/02/13/cve-2025-1094-po...

> Because of how PostgreSQL string escaping routines handle invalid UTF-8 characters, in combination with how invalid byte sequences within the invalid UTF-8 characters are processed by psql

If you just pass it to postgres over a normal query, postgres will reject an invalid byte sequence in the query with an error, refuse to even parse the query, and thus you won't get a SQL injection. It's just that psql didn't hard-error on invalid utf-8, even though postgres did.

That's why the escape function was suitable for postgres, both the escape function and postgres's query parser assume invalid byte sequences are, you know, invalid.

> For parametrized queries, params don't go through serialization because they don't need to hit the parser, there's no "escaping" whatsoever.

You're right of course, I used imprecise language that everyone understands, and you're choosing to read critically in order to be combative.

You don't have to be so combative.

[thaumasiotes]

> The postgres escape function actually worked fine before this "CVE". It was documented as escaping something for use as part of a postgres query.

> BeyondTrust used it as input to the 'psql' tool, which is an interactive tool you're not really supposed to programmatically invoke, and the documentation for the postgres escape function didn't say it escaped input for psql.

But this is the documentation for psql:

> psql is a terminal-based front-end to PostgreSQL. It enables you to type in queries interactively, issue them to PostgreSQL, and see the query results. Alternatively, input can be from a file or from command line arguments. In addition, psql provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.

We can learn two things from this:

1. You are definitely supposed to be able to invoke psql programmatically, or else the suggestion to "write scripts and automate a wide variety of tasks" would make no sense.

2. The input to psql is documented as a "query", and it seems fine to assume that a "query" for psql is the same thing as a "postgres query".

[furstenheim]

100%.

Btw, even using psql directly allows binding parameters https://www.postgresql.org/docs/current/app-psql.html

[gcr]

I’m starting to think that string-based languages like SQL which mix structure and content are a mistake.

Maybe future database systems will only accept queries serialized from protobuf, or JSON (output by a proper serializer)

[immibis]

JSON is a string-based language. So are Java, C, Rust, and so on.

[Nijikokun]

His argument is that you are building on the assumption that this is safe:

    "SELECT * FROM foo WHERE bar=?"
    bindParameter(1, userInput);

But what happens when it turns out that isn't safe?

[jerf]

Then you, or someone else, fix the underlying library.

This being Postgres, that process was likely completed decades ago.

Anticipating the next question, "but what if it is still unsafe?", the answer is that there is no fundamental reason why this can't be safe code. Security exploits like this require a cooperation between the receiving code and the incoming data, and it is in fact perfectly possible for the receiving code to be completely safe. There is no such thing as data just so amazingly hackerish that it can blast through all protections. There has to be a hole, and this is among the best-tested and most scrutinized code paths in the world.

[btown]

There's some nuance here. Even the most battle-tested system, with distinct slots for executable code and primitive values, might have a way in which an untrusted primitive input can overrun a buffer, or be split in an unsafe way, and cause unexpected behavior. But there's a vast difference in attack surface between that, and "just give us a string, don't worry we'll sanitize it on our end."

It's all about defense in depth. Even a system that's tested all the way through with Coq or similar is still at the mercy of bugs in the specification or in underlying system libraries. But intentional API design can make it materially less likely that a security issue will arise, and that's worth a heck of a lot.

[Dylan16807]

> But there's a vast difference in attack surface between that, and "just give us a string, don't worry we'll sanitize it on our end."

If you have a type system that distinguishes between sanitized and unsanitized strings then it's not a very big difference in attack surface.

The main difference between the two methods is the risk that you can forget to sanitize. But that's not what happened here, so calling them dumb for having that risk is not a useful way to analyze the problem.

Parameters are not an extra layer of defense. For anything other than forgetting to sanitize, parameters are a sidegrade, not defense in depth.

[cogman10]

That is not what's happening. You can't bind parameters in psql.

What beyond trust did was in fact what I said, constructing the entire query as one string and sending that on to psql. The sanitization method failed, but that's not what you should use anyways when dealing with user input.

If you are using a postgres client, then the message breakdown to postgres when you bind looks something like this (not the actual message stream format, just the jist of what it ends up looking like)

    SELECT * FROM foo WHERE bar=$1
    BIND 1 escape(userInput)
    ENDBIND

There isn't the same opportunity to create malformed data that can cause an injection attack in the Postgres message stream. There are far fewer things that need to be escaped in order to put in the full message. (I skimmed through the protocol, one improvement I'd make to it is adding a length to the bind instead of having a termination like it appears to do. That would, however, preclude streaming data)

These yahoos took a different route to running a command than what everyone does and should do and they were bit by it.

EDIT Actually, yes you can bind parameters with psql. However, it's there mostly as a way to test postgres and not something users are expected to use.

[graphememes]

They aren't saying that's what is happening but the underlying assumption is similar to that.

[zoeysmithe]

Just a guess but this looks like politically powerful dev culture overwriting cybersecurity culture, demanding, thus getting an exception from management for 'productivity' and 'being agile.'

I dont think we appreciate how much of a wild west things are with the incredible mix of hugely complex and powerful tools available trivially to developers and the concept of "move fast, break things."

Especially as corporate sees devs like they see salesmen (big moneymakers who deserve perks, exceptions) and top-down security culture as a cost center.

The other buried ledes are that postgres allows emojis (not sure if that's intended but it works) and that you can just run system commands and scripts directly from postgres cli. I imagine a lot of eyes are going to be on new hardening guidelines for postgres now.

I also imagine the first high performance enterprise friendly drop-in db written in something like rust is going to one day be a big deal.

[Terr_]

Hey now, a large portion of developers are seen as cost-centers too! Not everybody has the skill of flattering managers into approving greenfield projects, and then transferring away before they break horribly. :p

[randmeerkat]

> Especially as corporate sees devs like they see salesmen…

You’re onto something here. People perceive the world through the lens of their education and environment. Sales, legal, finance, are all easy constructs for a business leader to view the rest of the world through. The secret of the game isn’t to have the best tech or to code the most, it’s to “outsell” your competing business unit.

[singpolyma3]

Giving untrusted users ssh access... You mean like every shared hosting company or shell provider?

[charcircuit]

Which is why shared hosting companies get hacked relatively more.