[unwind]

Reading serialized data and then casting it into a structure type is ill-defined, very bad practice, and there's nothing in C that tells you it's a good or safe thing to be doing. So, please don't do that.

The exact layout of a structure in memory is up to the compiler, and can easily change with compiler options, even if the compiler itself and the target architecture and platform remain the same.

An externally-visible serialization format should of course be stable, and not depend on the compiler used to build the code, or the hardware platform which might have ideas of how various fields should be aligned (or even how to order the bytes in integers larger than 8 bits).

Serializing data is something you do, just blindly copying the run-time representation that you have to an external medium is not serialization. You should do it field by field, with a well-defined format chosen for each field.

[viraptor]

That's the theory, sure. But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?

[joosters]

Also, if you're passing data between two processes (e.g. your code fork()d a child), you can be sure that the data format is going to match, no packing required. (still need to ensure alignment in memory though)

[mikeash]

This is not necessarily true. Lots of systems support running 32-bit and 64-bit binaries simultaneously, and structs won't necessarily be portable between the two. Some versions of Mac OS X even supported running 32-bit PowerPC binaries (in an emulator, but supported directly by the OS) next to both 32-bit and 64-bit x86 binaries, so you could not only get size mismatches, but also endian mismatches.

This does not apply to simply forking, of course, but it does apply to the "two processes" case in general.

[joosters]

Oh yes, I was a bit unclear but I meant two processes (from the same binary) communicating.

Without that condition, you've got all the original problems, as listed. You can also have incompatibilities between compilers, or even compiler versions - e.g. gcc has ABI changes between some versions.

[raverbashing]

On the same machine, it's going to be rare

But the most common occurrence of this I can think are online games.

[mikeash]

Depends on what system and context you're talking about. I'd assume it's rare on Linux where most things are built from source, but it's pretty common on e.g. Apple platforms where binary compatibility means 32-bit processes keep being supported for years, and they still need to IPC to various system services. I've seen one highly amusing bug in Apple's iOS frameworks where they didn't quite handle the 32->64 IPC case properly because of an implicit assumption that data types remained identical.

[mikeash]

The fact that you can't know all of your future targets will support that is a problem.

Endian issues are also a big problem here. The need to byte-swap everything eliminates a lot of the convenience.

It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code.

Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible.

[raverbashing]

"The fact that you can't know all of your future targets will support that is a problem."

I don't see this as much of a problem as endianness, but yeah, maybe, x86 made us accustomed. And you can use portable types, defined on headers (linux does that, like u8, u16, etc)

But sometimes you know your target won't change (for a long time)

"It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code."

In the same way you can break your software doing any change. You can mark it with a naming convention but the easiest way is seeing the "pack" directives on it. And unit tests

"Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible. "

Well, it's not five minutes. And it makes your program slower (for the most convoluted data types). A simple example:

http://en.wikipedia.org/wiki/BMP_file_format

And yes, GIMP reads field by field https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp...

[mikeash]

Does it really make your program slower? That padding you have to eliminate for this technique is added for speed, after all. The initial read may be faster, but every access to the data is going to be slower.

I'm not quite sure what the BMP file format is supposed to be an example of, especially since file formats are separate from techniques used to read or write them.

[asveikau]

Some years ago I had a discussion with a then-colleague who pointed out something interesting that stuck with me: the world has mostly settled on little endian. If you stay outside of certain niches you are unlikely to ever see a big endian CPU.

Time was you might have to run on a SPARC or PowerPC or whatever. But x86 and little-endian ARM have pretty much won for most people.

I don't think my friend was right that this means you can totally ignore the issues, but I have to admit he was right that from a strictly practical perspective it's not the huge deal it was in the 90s. I agree with you that you don't know how the future will break you, but my guess is the momentum of binary compatibility will keep it so for a while.

[mikeash]

That's a good point, and he's largely right. On the other hand, I've been burned before (Macs were always going to be 68k until they went PPC, they were always going to be PPC until they went Intel, they were always going to be Intel until Apple made these tiny mini-Macs that were ARM) and I'm wary of making assumptions anymore.

[asveikau]

This is usually my kind of instinct too, which is why I was attributing the other viewpoint to someone else. But it was interesting to understand where someone like that is coming from. You can spend a lot of time adding the right swaps etc. but in the end if you don't own hardware that works that way, aren't testing it regularly, from a certain perspective you may be wasting your time.

OTOH I recall that insightful article from Rob Pike about how the "right" way to do it in a testable fashion is to not think in terms of swapping at all, and just do shifts that are portable regardless of architecture. http://commandcenter.blogspot.com/2012/04/byte-order-fallacy...

(By the way, in your 68k -> PPC -> Intel -> ARM example, the endianness only changes once. This was actually part of my friend's original argument, that endianness changes are even more costly than the instruction set, and platform vendors would be unwise to change it. "Modern" CPUs support both endianness types, so those shipping ARM platforms are in effect consciously deciding to be the same as Intel.)

[mikeash]

For what it's worth, I completely agree with Pike there. It makes a lot more sense to me, and refrains from completely falling afoul of the C standard.

[viraptor]

> The fact that you can't know all of your future targets will support that is a problem.

To be honest I've even seen compilation issues coming up between ubuntu 10.x and 11.x. I don't worry about future targets that much. Until they're tested, I assume they will not work.

[weland]

> But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?

You know that at this moment they all support the right __attribute__((packed)) et co.. Bonus points for endianness crap.

[vidarh]

All of that is true unless it is, say, cache data that you can afford to ditch if the format has changed, and you validate it after reading (say with a carefully structured "magic" value).