Per standard wording if an evaluation contains an undefined operation then it is not a constant expression. It means that undefined behavior should result in compilation failure if it happens in a context where a constant expression is required, like in static_assert.
However the standard only requires this for language-level undefined behavior. For undefined behavior happening in the standard library it's unspecified whether the expression is a constant expression or not. So no, constexpr tests don't cover all possible UB.
Also even if in theory language-level undefined behavior should be caught in constant expressions, in practice compilers miss a number of undefined behaviors. They are generally good at catching out-of-range indexing, using objects outside of their lifetime, using uninitialized values, signed integer overflow and modifying const objects. However there are a number of subtle undefined behaviors that they don't catch, like unsequenced operations on the same object, invalid values for unscoped enums.
There might be some overlap with runtime tests with -fsanitize=undefined,address. For catching uninitialized values at runtime though you probably need msan, which is a pain to set up, but constexpr tests cover that. On the other hand the function you test might not be available at compile-time.
Anyway, constexpr tests are a valuable tool. It's not a silver bullet.
With C++20 almost anything can be used in constexpr context (vector, unique_ptr, virtual, function, etc.) and as long as it's in the scope it can be tested at compile time which guarantees memory safatey, no UB, etc. Additionally, since constexpr can be executed at run-time and code has been tested at compile-time already therefore 'static_assert' is (almost) all you need - https://godbolt.org/z/P4cqboGx6.
> A core constant expression is any expression whose evaluation would not evaluate any one of the following:
> 8. an expression whose evaluation leads to any form of core language undefined behavior (including signed integer overflow, division by zero, pointer arithmetic outside array bounds, etc).
This does not really "guarantee" anything. You can still have as much UB (incl leaks) as you want, as long as they are not evaluated at compile-time. i.e. it's at best equivalent to running valgrind.
The point is that the code is only checked for UB with the arguments it is given at compile time. There is no guarantee that it can't invoke UB with other arguments it might receive at runtime.
For example, here is a modification of the original program that does invoke UB, but compiles just fine:
That only works if you know the values at compile-time though:
constexpr int foo(int x) {
return 1024*1024*1024*x;
}
int main()
{
int y;
std::cin >> y;
static_assert(foo(1)); //all good
foo(y); //oops, UB if user enters 7
}
Yeah, the tests will only fail if the tests trigger UB. It's like all testing, it only detects issues if you trigger the issues in the tests. Using static_assert as your test system obviously doesn't obviate the need for writing good tests.
I hope there’s also some text in the standard prohibiting implementations from allowing any other expressions as a constant expression (which they otherwise could as a language extension), and thus requires compilation failure for such expressions?
Pragmatically, you can't stop extensions; if the fine print for `--std=cool++23` says that this mode is not actually C++23-compliant, nearly nobody will ever notice or care. Pragmatically, if a popular compiler makes `--std=cool++23` the default, and requires `--std=C++23 --iso-eic-jtc1-sc22-wg21 --pompous` to get standard-conforming behaviour, nearly nobody will do that; instead they will complain that other compilers lack the extensions.
Extensions can be standard-compliant, in the sense that they don’t violate any prescription by the standard, and thus a program cannot assume their absence. My question was whether the standard actually takes care to render the acceptance of constexprs-with-UB non-standard-compliant. That is, in addition to “must accept X”, does it also say “must only accept X”?
constexpr has to checked for leaks and UB so as long as there is coverage at compile-time (static_assert + constexpr) I would assume there shouldn't be neither leaks nor UB. But the context is limitted where that can be applied and actually compiles. For example, there is no way to do it with global variables but with limited scope that's possible.
I guess so, can't think of an example now but I'm pretty sure there are subtle corner cases (as always) and it depends on the testing, coverage and potential limitations of checking things at compile-time, though, IMHO, the technique is promissing and can help with a lot of use cases but defo not everything.
The standard explicitly disallows compiling of UB for constexpr.
Otherwise, what you write is provably correct - UB is not statically decidable in all cases (and depending on the type of UB, not even in a lot of cases).
If your compile time evaluations don't trigger signed integer overflow (or any other UB) does it follow that at runtime you couldn't pass a parameter that would trigger signed overflow?
I mean it's still useful because at least you know your test code is not artificially passing because of some UB makes it look like passing
From an overall performance point of view, I wonder how the timing works out for the compiler to run your unit tests like this vs to produce and invoke a binary.
I bet it's mostly a wash, and the ergonomics of conventional gtest macros look way better to my eye.
The idea to let the compiler run the tests only works if you constexpr everything, which means putting all code in headers.
This effectively means giving up on separate compilation. Worse, if you use some mixture (most code in headers but you still have >1 compilation unit), your compile times completely explode as essentially all code is compiled repeatedly for each unit.
There are other ways to do this. I made a proof of concept of using linker sections to allow you to sprinkle tests within the implementation inline once... https://github.com/cozzyd/examc (this is obviously not production-ready, just serves as a proof of concept).
Basically the idea is that the test code gets written to a different linker section that your test runner can iterate through, when tests are enabled.
This is easy on gcc because it generates automatic constants for the beginning and end of different linker sections. There may be away to do this with clang as well, but I never use clang.
Pragmatically you can write your code in such a way that you can get immediate feedback in your IDE as you write the code if a static assert fails as you are implementing your function.
You could of course set up your runtime tests in a similar way, having the ide run them back to back as you are writing code, but it is more complicated, especially if the code is in an intermediate state that it is not fully compilable.
So in the end it is not a huge breakthrough, but having compile time tests is still quite a nice feature.
Hmm, there are obvisouly trade offs (it depends on the compiler how many tests, how are they written, etc.) but for apples to apples comparision the gtest binary would have to be either compiled with sanitizers (that would be probably slower to compile than static_assert tests without sanitizers) or run with valgrind or similar (execution would be much slower, static_asserts tets don't have to be executed, compiles=green).
(Offtopic but) Kris, I have used your micro unit-test library in the past and it was a pleasure to look at your code. You're the kind of crazy guy (in a good way) that gives me the motivation to learn new stuff. Thanks.
This makes sense for straightforward tests, but static_assert is not all you need in general, because some things has be executed in runtime, after series of steps or after some timeframe. Good luck reproducing or testing these in compile-time.
Could you not put a series of steps or something that mocks time into constexpr? A comment mentioned above that in C++20, almost all features are available at compile time now.
With few limitations, yes. In C++20, you could for example test for constant evaluation[1], using that as a mechanism to fall back to a real clock during runtime.
The C++ code is a simple list type, and also a bunch of tests for that type to confirm that it works as intended. The crucial trick here is that because static asserts are used, the test values are computed during compilation, such computations are forbidden (by the standard) from having any leaks or Undefined Behaviour. Anything allocated must be freed by the time the tests complete, and no language Undefined Behaviour is permitted.
The latter is pretty normal for other languages but is a big deal in C++ where UB is a constant plague. However many languages have either forbid or have strict limits on compile time heap allocation - after all that heap isn't going to still exist at runtime. Requiring that you free everything allocated fixes that hole and means you get free leak detection.
The compiler is doing a bunch of complicated stuff at compile time. In fact it's both a C++ compiler and a C++ interpreter.
static_assert is a compile-time check.
[] { ... }(); is an immediately executed lambda function (IIFE in javascript parlance).
list<int> list{} is a linked list of integers (double-linked, forward and backward). push_back() allocates more memory. pop() / clean() deallocates memory.
It is creating a bunch of objects, modifying them, then asserting their value all at compile time. For example the first example creates a list and asserts its size is 0. List normally allocated on heap, so I am guessing they have made changes in thay area in c++20 by making it Constexpr, which is a fancy way to say an expression can be known at compile time.
Thanks! Most likely not yet applicable at Google's scale but smaller project can defo leverage the approach. Personally, I'm writing most of my tests this way and with TDD the red phase is always a compilation fail which is quicker than buiding and running in my experience. But that's for a medium size project. But as always it depends there are trade offs.
I’ve been using this technique as well, but I found that debugging static_asserts is quite hard. I often fall back to calling the failing test at runtime and stepping through. Any suggestions for a different workflow?
IMHO the best approach is to avoid the problem by applying TDD. Then there is very little need to debug anything. But otherwise, there is https://github.com/mikael-s-persson/templight for compile-time debugging which is pretty cool and having something like `expect(auto... args) static_asert(args...); assert(args...);` may help with being able to debug at run-time and get the coverage (though, the code has has to compile aka pass first).
However the standard only requires this for language-level undefined behavior. For undefined behavior happening in the standard library it's unspecified whether the expression is a constant expression or not. So no, constexpr tests don't cover all possible UB.
Also even if in theory language-level undefined behavior should be caught in constant expressions, in practice compilers miss a number of undefined behaviors. They are generally good at catching out-of-range indexing, using objects outside of their lifetime, using uninitialized values, signed integer overflow and modifying const objects. However there are a number of subtle undefined behaviors that they don't catch, like unsequenced operations on the same object, invalid values for unscoped enums.
There might be some overlap with runtime tests with -fsanitize=undefined,address. For catching uninitialized values at runtime though you probably need msan, which is a pain to set up, but constexpr tests cover that. On the other hand the function you test might not be available at compile-time.
Anyway, constexpr tests are a valuable tool. It's not a silver bullet.