[bhauer]

Obviously I am biased about the benchmarks project, but I disagree concerning the predictive power of the rounds to-date.

Thus far, the rank order of each round, as we add more tests, remains largely consistent. The most extensive test--Fortunes--exercises request routing, database connectivity and pooling, the ORM (if available), entity object instantiation, dynamic-sized collections, sorting, server-side templates, and XSS counter-measures. On the whole, where a framework has received an implementation of Fortunes, we see roughly the same order as in the other test types.

To clarify some points:

* The magnitude of the Worlds table is intentionally small enough to easily fit into the database server's in-memory cache. This is an exercise in measuring the frameworks' and platforms' database drivers, connection pooling, and ORM performance; not the performance of the database server. As an unintended side-effect--largely thanks to the contributions of readers--the scope of the project has broadened to include some Mongo and Postgres tests so it is to a very small degree a rough comparison of the request-processing capacity of three popular database platforms. But it is expressly not a benchmark of database servers.

* The response payload is intentionally extremely small because these tests are designed to exercise framework fundamentals such as request routing and header processing among others. Increasing the payload size directly increases the number of frameworks that will saturate gigabit Ethernet. As it is, even with a trivial payload, high-performance frameworks saturate gigabit Ethernet on the trivial JSON-encoding and plaintext tests.

* A larger payload JSON encoding test type is planned for the future [1], but I would caution that it is unlikely to shuffle the rank order seen in tests to-date in any notable fashion.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/13... (see #13)

[nickzoic]

> But it is expressly not a benchmark of database servers.

Oh, okay, cool. I'd misinterpreted it as being a kind of a "full stack" test. I didn't look at any of the "fortunes" examples, either, as it happens.

I can see where you're coming from, but I'd still be concerned that the concurrency and connection pooling behaviour could be quite different depending on the DB query behaviour.

What I should have said is "... predictive of how well your website will actually work under load". But nothing much is predictive of that except for trying it with synthetic data ...

[bhauer]

You're right that it's impossible to fully disentangle the performance of the database server from the frameworks' ORMs and their platforms' drivers and connection pools. We indicate which database server is being used in each test permutation and one can make very rough observations from the data--such as "MySQL appears very slightly faster in this type of use-case than Postgres"--but like I said, that's not really the purpose of the project. If one wants to compare database servers, there are many better resources for that insight.

You're also right that nothing can predict how your application will perform under load until you build it and test it.

By testing the fundamentals of web application frameworks, however, we hope to inform a preliminary selection process (along with self-selects such as comfort level with code type and community) to give you a rough idea of capacity before you build out the full application. I feel especially that the massive spread of the performance numbers--covering many orders of magnitude as it does--is illuminating to newbies and also valuable to seasoned pros.