[exceptione]

  > You'd still return a multiplicative amount of rows, even if those rows contained only a reference

Sure, but a pointer is still a massive win over records, and I think a further cartesian product wire protocol extension would not be worth the hassle.

  >  `array_agg` in postgres avoids this

That one is tracked here: https://github.com/npgsql/efcore.pg/issues/2633

[CodesInChaos]

I don't think pointers (beyond a simple "same as in previous row" marker) will be a huge improvement, since you still get a multiplicative number of rows. And it comes with the cost of keeping all that data in memory. This approach also competes with using cheap compression (e.g. LZ4).

Some kind of "product" operator on the other hand reduces the cost to additive (just like `array_agg`).

> That one is tracked here: https://github.com/npgsql/efcore.pg/issues/2633

That issue is only about supporting `array_agg` as a function on tuples, not as an implementation strategy for `Include`s of collections.

[exceptione]

  > I don't think pointers (beyond a simple "same as in previous row" marker) will be a huge improvement, since you still get a multiplicative number of rows

A marker as "byte offset in response data" is very efficient. Consider this:

  SELECT BlogPost bp LEFT JOIN Comment c WHERE bp.id=101 AND c.blogPostId=bp.id

If the BlogPost is 1KiB in size and has 500 comments, doing it naively will return `500 * 1KiB` for the BlogPost part. To contrast, suppose your format needs 3 bytes for pointers, you will use `1 * 1KiB + 500 * 3B` for the BlogPost part.

Compression and decompression takes CPU-time, I have a hunch this brings in an unacceptable penalty. At least this route hasn't been chosen while it would be the easiest to implement.

[to11mtm]

Agree that Compression is not a good option for this; you'd wind up needing to decompress the rows anyway (or spend a good amount of time on trying to use the compressed symbols to detect a dupe, which I feel like would be very dependent on picking the right algo or maybe even a custom one.)

As far as your idea, I think where a lot of the challenge lies in that it complicates a lot of the up-front protocol.

Some of these DB protocols are downright ugly [0], and need to have a very long tail for supporting systems built on older versions. And they can be very finicky thus people are afraid to touch both the DB side and the Driver side.

So, at best you'd need to have a special connect string and then work with driver teams and/or vendors to understand the new format, deal with all of the teething issues, and will have to continue to support the old protocol in your DB code for at least another decade (probably longer for any commercial product.)

[0] - TDS comes to mind, I'm betting TTC/OPI is not fun either.