[copper_think]

At Microsoft the compiler team (Visual C++) and the Windows team are joined at the hip. I'm sure the same was true at Sun. This can lead to good decisions about undefined behavior that I hope would make Linus smile.

I recently learned of one such good engineering decision (I hope I'm remembering it correctly). Let's say you have a struct with an int32 and a byte in it. That's 5 bytes, right? But the platform alignment is a multiple of 4 bytes, so there's 3 bytes of padding (sizeof the struct is 8 bytes). If we stack-allocate an array of 11 of these and zero-initialize with = { 0 }, what would you expect to see in memory after initialization?

It turns out the answer was that the first element of the array would have its 5 bytes zeroed, but the 3 bytes of padding would be left uninitialized. Then, the remaining 10 elements of the array would be zeroed with a memset that actually zeroed all 80 remaining bytes. It sounds weird but this is a legal thing to do from the standard's perspective. All they're obligated to zero out are the non-padding bytes. This UB was leading to disclosure of little bits of kernel memory back into user mode because Windows engineers assumed that = { 0 } was the same as leaving the variable uninitialized and then memsetting the whole thing to zero. Nope!

The compiler team fixed this by always zeroing out padding too. Problem solved. There are some cases where it's not quite as fast. But it's the right engineering decision by the compiler team for their customers, both internal and external.

[hermitdev]

Minor nitpick, the zeroing (or lack thereof) of the padding is not undefined behavior, it's unspecified behavior. Undefined behavior and unspecified behavior often look and perhaps behave the same to the programmer, but have semantic differences. In the face of undefined behavior, the compiler is allowed to do pretty much anything it wants (including formatting your hard drive and/or launching the nukes). With unspecified behavior, the compiler implementer must make a conscious decision on what the behavior will be and document the behavior it will follow.

[jcranmer]

> With unspecified behavior, the compiler implementer must make a conscious decision on what the behavior will be and document the behavior it will follow.

No, what you described is implementation-defined behavior.

It may be confusing, but here's the breakdown of different kinds of behavior in the C standard:

* Well-defined: there is a set of semantics that is defined by the C abstract machine that every implementation must (appear to) execute exactly. Example: the result of a[b].

* Implementation-defined: the compiler has a choice of what it may implement for semantics, and it must document the choice it makes. Example: the size (in bits and chars) of 'int', the signedness of 'char'.

* Unspecified: the compiler has a choice of what it may implement for semantics, but the compiler is not required to document the choice, nor is it required to make the same choice in all circumstances. Example: the order of evaluation of a + b.

* Undefined: the compiler is not required to maintain any observable semantics of a program that executes undefined behavior (key point: undefined behavior is a dynamic property related to an execution trace, not a static property of the source code). Example: dereferencing a null pointer.

[susam]

Nice comment! Here are the excerpts from n1570.pdf[1] with some punctuation added by me to compensate for the limited formatting support on this forum:

§3.4.0: behavior: external appearance or action

§3.4.1: implementation-defined behavior: unspecified behavior where each implementation documents how the choice is made. EXAMPLE: An example of implementation-defined behavior is the propagation of the high-order bit when a signed integer is shifted right.

§3.4.2: locale-specific behavior: behavior that depends on local conventions of nationality, culture, and language that each implementation documents. EXAMPLE: An example of locale-specific behavior is whether the islower function returns true for characters other than the 26 lowercase Latin letters.

§3.4.3: undefined behavior: behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements. NOTE: Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message). EXAMPLE: An example of undefined behavior is the behavior on integer overflow.

§3.4.4: unspecified behavior: use of an unspecified value, or other behavior where this International Standard provides two or more possibilities and imposes no further requirements on which is chosen in any instance. EXAMPLE: An example of unspecified behavior is the order in which the arguments to a function are evaluated.

[1]: WG14 working paper for the C11 standard: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf

[billforsternz]

Thank you kind sir, you win the prize for most coherent and informative technical snippet in the world. On this day anyway.

[loup-vaillant]

Accessing uninitialised memory is undefined. As in Nasal Demon undefined. If you don't zero out the padding, I'm sure there's a clever way to access those bytes in ways that invoke undefined behaviour.

[caf]

The padding has unspecified values, which is distinct from being uninitialised.

If it were otherwise, you couldn't `memcpy()` structures around.

[loup-vaillant]

I believe there's an exception for the likes of `memcpy()`. Something along the lines of "type punning and reading indeterminate values is undefined except when we're reading through a `char*` or something.

I'll check the unbelievably thick book that tries to specify C11 (I've printed it, it's over 2 pounds).

[caf]

The only way to access the padding without otherwise falling into undefined behaviour is by using a char * anyway (including indirectly using a char *, through memcpy()).

[iainmerrick]

You say that as if it’s entirely the responsibility of the programmer to avoid these bear traps that have been left lying around everywhere.

Why not just have the compiler zero the memory, and thereby remove the trap? Seems very sensible to me. Do you think it’s a bad idea, and if so, why?

[kazinator]

There is a concern for performance. But that's no reason. Zero initing could be default behavior that can be declared away. E.g.

As a type qualifier keyword:

  { int x, y; /* x and y are zero */ }

  { int noinit x, y; /* x is indeterminate, y is zero */ }

Or as a declaration specifier:

  { noinit x, y; /* both x, y indeterminately-valued */ }

Or a special constant for suppressing zero initialization:

  { int x, y = noinit; /* x zero, y indeterminate */ }

Similarly, unspecified order of evaluation could be supported by explicit request:

  decl (unspec_order) { /* comma-separated list of decl items */
     a[i] = i++; /* UB */
  }

  a[i] = i++; /* well-defined */

[rurban]

zero initing is default behavior for static values or structs.

    static int x, y; /* x and y are zero */

[sam_dam_gai]

Good idea!

[loup-vaillant]

> You say that as if it’s entirely the responsibility of the programmer to avoid these bear traps that have been left lying around everywhere.

Oh no no no, I was picking on hermitdev's characterisation of the behaviour. Sure, what the compiler does with the padding bytes is unspecified. But it can still lead to undefined behaviour, if some unwary user ever reads them. A misinterpretation like that, and poof you get a security vulnerability. The C standard is insane.

[iainmerrick]

Aha, sorry, I misunderstood! Glad to hear you’re on the side of sanity. :)

[speedplane]

It's more than a logical or performance issue, it's a cultural one. In C culture, there's a sense that the programmer has direct control over the hardware. They can literally write or read to any memory address as they see fit, and have very fine-grained control over what the machine is doing.

Modern processors, with their out-of-order execution, complex caching algorithms, deep pipeliens and multi-threaded hardware often render this sense of control more illusory than factual, but the C culture remains wedded to the idea that the programmer is in control.

Accordingly, it's pretty provocative to suggest that a compiler or runtime would zero out memory without you specifically saying so. Any C programmer can overload malloc (doesn't zero memory) to calloc (which does). Whether it's a good or bad idea to do so is up to the programmer. The overall idea is don't do anything unless I say so.

[xorblurb]

Sane compilers should do that. The standard should eventually specify that. But before it does, you can not write portable code that expect that (but hopefully once enough compilers are sane but before the standard is updated, you can write code that targets only the compilers, and don't give a fuck about the other broken garbage that try to trap the world)

[MichaelMoser123]

if you reference a null pointer then this behavior is 'undefined' from the perspective of the C compiler - however it is very well defined by the operating system (if you are doing a user space application).

No nukes get involved here - only core dumps.

[MichaelMoser123]

If you are working on top of an operating system that launches nukes upon null pointer access, then you should consider to switch vendors.

[ythn]

> and document the behavior it will follow.

So it's unspecified in terms of the standard, but specified by the implementation

[jexah]

Right! Basically it's up to the compiler programmer to pick a path and follow it... assuming you're talking about implementation-defined.

[bigcheesegs]

The problem with this kind of approach is over time it removes the ability to use any other implementation. You are no longer using C, you are using <Implementation>C.

This becomes a problem when a different implementation adds amazing tools for finding bugs (the sanitizer suite, for instance), and you can't use them because your code doesn't build with any other implementation.

[8_hours_ago]

Large and complex pieces of software, and operating systems in particular, tend to be tightly tied to their compilers. It is never easy and in some cases practically impossible to port to a different compiler. I expect that Microsoft has come to terms with the fact that Windows will only support being compiled by their compiler.

When a different toolchain introduces a new feature for finding bugs that would be useful for Windows, the Microsoft compiler team can add that functionality to their own tools instead of porting Windows. An advantage of this is that they can customize the feature for exactly their use case. Yes, this is the definition of NIH syndrome, but that’s how large companies work.

[xorblurb]

Large pieces of sw have been able to switch some plateform specific code to other compilers (chrome for windows comes to mind).

This is probably way smaller than the whole Windows, but I would not be surprised if some MS dev are already internally compiling some of their components with clang for their own dev/testing (even if just for extra warnings, etc.)

And a major part of the work of the MSVC team today seems to be about standard compliance.

But yes, I do not really expect that they switch, and actually they probably don't even have the beginning of a serious reason to do so. This is not even a case of NIH. Their compiler derives from an ancient codebase and has been continuously maintained for several decades. They "invented" it. The only modern serious competition (that cares enough about Windows compat and some of their specific techs) has been started way after... They probably also have all kind of patents and whatnot about some security mitigations that are implemented by collaboration between the generated code and (the most modern versions of) low level parts of the Windows platform.

[pnathan]

> . You are no longer using C, you are using <Implementation>C.

If you go deep and gnarly enough with your system, this is always the case, I'm afraid. There's a famous talk at Stanford by a Coverity... founder? consultant? about this.

[taneq]

> You are no longer using C, you are using <Implementation>C.

This is always the case because the C standard is only a partial spec. There's no such thing as "a program written in C(++)", there's only "a program written in <Implementation> C(++)". If you compile your program with a different implementation, then it's a different program. It may work, but it may not.

What I'd really like to see is a modified version of the C/C++ standards which keeps the language the same "in spirit" but removes all undefined and implementation-dependent behaviour. This would give compiler writers a stationary (or at least slower-moving) target to aim for and make it possible for C to be portable in theory as well as (sorta) in practice.

[Gibbon1]

However in this case 'Standard C' is broken and <Implementation C> is not. The proper thing is to fix the standard to require padding bytes be zero'd.

[BeeOnRope]

It's easy to see both sides here. The overwhelming majority of code is not written to be run a in a kernel at the kernel-user security boundary, so for most code paying something to initialize padding might not be a great tradeoff. That is, the vast majority of code is running within a single security domain and doesn't need to protect itself from itself.

Still, avoiding initializing padding is probably not a great example of a performance win through standards exploitation: in the example given it's not clear why you'd not just do one 8 byte zeroing write to cover the whole structure, rather than apparently splitting it into a 4-byte and a 1-byte write. Perhaps this was 32-bit code, where 8-byte write are slightly trickier, but even two 4-byte writes are likely to perform just as well or better. Probably it's just bad codegen to treat the initialization of the struct[11] in two parts: a single struct and a 10-member array.

[iforgotpassword]

Luckily clang does a good job at being GCC compatible, so I don't worry too much about using GCC extensions. It's quite unlikely one of them will go away anytime soon, and theory pretty much cover all architectures/platforms that have ever existed.

[pjmlp]

Usually true, unless you want to target embedded, mainframes or some industrial OSes.

[saagarjha]

It doesn't support everything; for example, Clang doesn't do nested functions.

[pjmlp]

This was pretty much how writting C code between K&R, ANSI C89 and subsets e.g. Small-C used to look like.

[EpicEng]

I fail to see how implementing an unspecified part of the standard in a way which _doesn't_ leak kernel memory could ever be a problem.

[BeeOnRope]

It's not a problem for the compiler, of course.

The problem is the C code that relies on it: effectively you are using a dialect of C which gives stronger guarantees, so you lose the ability to use any other implementation which doesn't provide those guarantees.

[EpicEng]

Yeah, I (somehow...) missed the part where the GP explained that the kernel dev's fix was to just rely on the now updated unspecified behavior. I assumed the changed the compiler, but he also memset the structure before sending.

[coldtea]

>The problem is the C code that relies on it: effectively you are using a dialect of C which gives stronger guarantees, so you lose the ability to use any other implementation which doesn't provide those guarantees.

How do you lose it in this case? You shouldn't been reading values from those padding bits anyway...

[boomlinde]

Tell that to a hacker. The problem here isn't that the code is functionally dependent on padding bytes, it's that when you copy those padding bytes around you are leaking information that you probably never meant to copy.

This can be problematic if you are copying kernel space memory to a user space process, for example. Let's say there's a call into the kernel that returns a copy of this 4+1 struct with three more bytes of padding. Maybe what was on the stack before the space was assigned to those last padding bytes are some information the kernel definitely shouldn't leak to user space, like some bytes of a password, and now any user space process could potentially read them simply by calling some unrelated kernel function.

[robotresearcher]

It’s great that the compiler team could implement safer behaviour. But if the programmers’ intent was to zero all bits in the array they should express this explicitly in the code with a memset(). Otherwise a change in the compiler later could throw up this vulnerability again. The code should express the semantics as clearly as possible.

[adrianratnapala]

Then other other language lawyers will come around and tell you why you should use {0} instead of memset (e.g. because for some combinations of type and architecture the zero value isn't full of zero bytes).

This example also shows how "the semantics" is a fiddly concept. The reason the standard allows leaving bytes unzeroed is because they are not "semantically" important. But they actually do matter.

The problem with the mentality that it's always the programmer fault for not following "the rules" is they you eventually get to the point where the rules allow for no good solutions at all.

[robotresearcher]

I didn't say it was always the programmers' fault. But I believe the programmer should express their intent as clearly as possible. And memset( (void*)buf, 0, buflen ) says fill-a-contiguous-array-of-unsigned-chars-of-value-zero, which is semantically different from initializing an array of structs that may have padding, and better matches the programmers' intent. It doesn't matter if the zero value is all 0 bits or not - the important thing is that the whole contiguous memory region is zeroed.

I believe C99 says chars and unsigned chars have no padding.

https://stackoverflow.com/questions/13929462/can-the-unsigne...

[robotresearcher]

Another, better, response to your argument. Initializing an array of structs with = {0} does NOT tell the compiler that zeroing the entire contiguous chunk of memory matters - only that each struct should be zeroed. While memset( (void*)buf, 0, len) does tell the compiler that the entire contiguous memory chunk must be zeroed, which is what is intended.

Language lawyers need not apply.

[tlb]

It may be the right thing for the moment, but what happens to that code when new versions of the compiler come out? Someday it'll start leaking information again. Relying on non-standard behavior always ends in tears.

[EpicEng]

>Relying on non-standard behavior always ends in tears.

Who's relying on unspecified behavior? Do you mean some theoretical programmer? Well of course that's not a good thing, but I don't see what it has to do with the situation in the story and the MS engineers decideing to change how they implement that unspecified part of the spec.

[tlb]

The kernel programmer did, in this story.

An example of such an information leak is:

  struct foo {
    int a;
    char b;
  }

  void send_foo_01() {
    foo x {0, 1};
    write(fd, &x, sizeof(foo));
  }

which sends 3 bytes of the contents of the stack memory over the network, for almost every compiler except the one in the story. Running in a kernel, that could contain secrets.

[EpicEng]

Sorry, my mistake; yes, the kernel dev is now relying on the compiler to zero out those three bytes. I understand the decision to change this in the compiler, but I think the "fix" would have been to memset in the kernel code. I'd be surprised if they didn't do that, but maybe they can reasonably assume they'll never use another compiler to build the Windows kernel.

[legulere]

Signed integer overflow. It’s undefined because some platforms don’t use two’s complement.

Checking it after the fact is non-portable.

[rurban]

But not in sane compilers, they are better than the standard.

gcc, clang, icc all assume two's complement and happily overflow via -fwrapv. There's no such compiler on the Unisys anymore. They rather emulate two's complement. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm

[AlexMax]

Thankfully, not for very much longer...

http://wg21.link/P0907r2

C++, but apparently something similar is being considered for C as well.

https://twitter.com/jfbastien/status/989242576598327296?lang...

[JoeAltmaier]

Isn't that mostly theoretical? I can't seem to find any processor that doesn't use 2's complement.

[gh02t]

There are some old architectures that used one's complemeny and there are probably some still in service e.g.

https://en.m.wikipedia.org/wiki/UNIVAC_1100/2200_series

[davidgould]

I had the pleasure of programming on these. Having both positive and negative zeros is special and that they don't compare equal is extra special. And yet -- ones complement arithmetic is not even the oddest thing about Univac 1100s. I think the Univac corporate values statement must have included both "Dare to be different" and "Remember your past".

[JoeAltmaier]

Is anybody writing code for them? Probably not porting modern C anyway.

I assert this 'it might not be two's-complement' excuse is nonsense.

[8_hours_ago]

Presumably the compiler team wrote a test for that behavior so there won’t be a regression. I assume that Windows isn’t planning on switching compilers, so they can rely on that behavior as long as the Microsoft compiler passes all its tests.

[xvilka]

Well, MSVC team is the last one who can propose something since they didn't even support C11 completely today, and weren't able to add C99 support for almost two decades.

[nwellnhof]

They still don't support C99 fully.

[pjmlp]

They do, to the extent required by ANSI C++14, ANSI C++17 upgraded the requirements to C11.

They are pretty open the future of systems programming on Windows is C++ and .NET Native.

For C devs, they have helped clang to work on Windows and there is WSL for UNIX like software.

Just like they have improved Visual Studio integration with clang and gcc.

As for Visual C++, the name says it all.

[pjmlp]

Now that Microsoft is having again a new found love in C, either via Checked C research, the UDMF rewrite in C, WSL, or the upcoming Azure Sphere product, it would be interesting if they could contribute to improving the whole security story in C2X.

[slededit]

I don't think they are really that much in love with C aside from a few niche areas. The compiler itself still generates worse code when in C mode as it only exists for back-compat. When I worked there it was explicitly said that the new "universal" (or whatever they are called now) APIs were explicitly not targeting C. C++ on the other hand seemed to be undergoing a new renaissance. They have added some newer C features but only those required by the C++ standard.

[pjmlp]

Quite true, when speaking about Visual C++ and I have also stated the same thing multiple times here.

Their C love on the projects I mentioned, is via clang and gcc.

[minipci1321]

> Problem solved.

This "solution" also prevents code analysis tools from detecting reads from padding bytes as "non-initialized memory reads".

When a routine leaves sensitive data on the stack, the entire area used up by that sensitive data, must be wiped out before the routine returns (and still, wiping out padding bytes would not be required). How about the part of the stack space not covered by that array of 11 elements?

[BeeOnRope]

The kernel has a separate stack, inaccessible to user-space.

Otherwise, you'd be right: a shared stack would be a giant source of information leakage from kernel space to user-space unless it was very carefully managed (probably at a significant performance cost). Thus, separate stacks (it also has the advantage of not needing to make assumptions about how user-mode programs use their stack, e.g., if they are transiently using "unallocated" stack above rsp, etc).

Probably what happened here is that this structure was copied back to user space (e.g., as the result of a system call) exposing the kernel data.

[cryptonector]

At Sun Solaris and Studio were quite separated organizationally (and business-wise too). Mind you, the Solaris engineering group had early access to Studio and also, of course, access to the Studio team, so... not quite joined at the hip, but not too separated either.

[nwellnhof]

In C11, the spec was changed to make zeroing out padding mandatory.

[oblong]

> This UB was leading to disclosure of little bits of kernel memory back into user mode because Windows engineers assumed that = { 0 } was the same as leaving the variable uninitialized and then memsetting the whole thing to zero

But what on earth were they doing with the padding bits?

[Gibbon1]

They aren't doing anything with the padding bytes. But what probably happened was they copied the array into userland memory. Which potentially allows a malicious sandboxed program to read the padding bytes that contain bits of kernal memory.

[kabdib]

Total guess, but they could have been including them in a hash function or struct equality check (equality with memcmp, hash by just grabbing bytes, etc.).

That would not have gone well with uninitialized padding :-)

[buckminster]

It's not uncommon for a function to use a supplied output buffer as scratch space. So the padding could have contained pretty much anything.

[captain_perl]

I'm skeptical that it was implemented that way because of kernel memory leaking concerns.

More likely it was done to prevent Windows API calls from panicking when they accessed unset parameter structures.

As a former Windows programmer, that was one of the largest sources of errors back in the Win32 days.

[jclulow]

But you don't access the padding; it's generally implicitly added by the compiler to comply with the ABI rules for the platform. That argument might make sense if this was about zeroing _all_ stack objects not explicitly initialised, but it's explicitly talking about the padding.

[skookumchuck]

> This UB was leading to disclosure of little bits of kernel memory back into user mode

If you write inline assembler, you can access this stuff anyway. So I'm not seeing what the value is in zeroing it by the caller. The kernel callee should zero its stack frame before returning.

[BeeOnRope]

How so? The kernel presumably has a separate stack which is not accessible to user-space, but here information was disclosed because a structure copied back to user space was create on the stack, initialized to {0} and then member-wise assigned, with some padding bytes never being touched and thus containing whatever previous values happened to be on the kernel stack. So far this is all in kernel-space so nothing been exposed yet.

Then, however, if this structure is copied back to user-space, e.g,. as an output parameter of a kernel call, the padding bytes with the exposed data will be copied along with it (unless you get lucky and the copy routine happens to make the exact same decision with regard to padding handling).

If the kernel stack _itself_ was visible to user-space, you'd have a whole separate set of problems: you'd have to zero the whole stack (or at least the extent of the stack that could have been touched) on every kernel call.

[skookumchuck]

Yes, you're right.