Brian from TechEmpower here. I don't want to take away from these results, but I do want to provide an important piece of context. This link is to a rendering of data from a continuous run [1] that hasn't received the type of sanity checking we do for an official TFB round. You can tell from the title ("Test") and the UUID of the run visible in the gray box under the navigation. Based on this link's attention, we'll make a more prominent warning for renderings of continuous runs so that they are more obvious to readers not familiar with the project.<p>We execute runs like this continuously to allow maintainers of test implementations to observe the results of their contributions. Given the performance seen here, it is <i>very likely</i> that Lithium will be well ranked in the next official round. But we feel the ranking seen in a continuous run such as this should be taken with a grain of salt until that next official round is available.<p>[1] <a href="https://tfb-status.techempower.com/results/57b25c85-082a-4013-b572-b0939006eaff" rel="nofollow">https://tfb-status.techempower.com/results/57b25c85-082a-401...</a>
I had investigated what explained the huge gap between the few fastest frameworks vs the rest.
The answer is a deceptive one, they didn't achieve revolutionary optimizations.
The thing is: on many of those benchmarcks, the bottleneck obviously is the DB.
The ability to do DB queries asynchronously and with batching is the differentiating factor.
Only postrgresql can support such a feature but you need support on the postrgresql client too.
The official C postrgresql client used everywhere does not support said feature except with a patch from 2016.
Yes the secret of drogon (and probably of Lithium) is that they use a fork of libpq from 2016 because upstream can't agree on merging the patch and nobody is working on upstreaming it.
Actix web benefited from the feature because their client tokio-postgres is a reimplementation and does not use libpq.<p>The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty.
But when loom arrive every jdbc existing code will magically, automatically become truly asynchronous such spring should come on the top 4 place.<p>There is also a wrapper of the jdbc through kotlin coroutines and there are reactive jdbc implementations such as R2DBC. It is unclear as of today if such solution enable postrgresql async queries and batch processing. It seems that nobody has tried those on TechEmpowerUp which is sad.
Finally one could use libpq over JNI.
Edit: I have read that the next release of pgjdbc (43) will switch from std socket to the NIO non blocking socket.<p>What should be heuristically the fastest HTTP framework (H2O, in C) has refused to use the old libpq fork because the api is not stable and thus not production grade.
Is this a criteria that's important enough to warrant this much effort?<p>I can think of several metrics that would come first - even for sites with fairly high performance requirements.<p>How often is the web framework genuinely the bottleneck (at least in a way that's not easily fixable with caching or any other similar sledgehammer)?
Link to the lithium homepage: <a href="https://github.com/matt-42/lithium" rel="nofollow">https://github.com/matt-42/lithium</a>
Incredibly frustrating the benchmarks not linking to the actual frameworks - searching for "lithium web framework" gives me some PHP framework...<p>Edit: Comment lower down suggests: <a href="https://github.com/matt-42/lithium" rel="nofollow">https://github.com/matt-42/lithium</a> is the correct one.
I'm more interested in the fact that something called asp.net core is one of the five fastest, while asp.net not 'core' is the single slowest. What's the deal there? I dimly know that asp.net is a Microsoft thing, and remember it being used by a bunch of enterprise sites a decade or so ago...
I really do like these TechEmpower Fortunes benchmarks. It has quite an extensive list of frameworks, languages, and configurations with throughput, latency (w/ SD), and errors.<p>I always check any lesser known framework to see which ballpark it falls into. I'm always surprised to see that so many of the popular frameworks are ~10x worse than the best--although many of those 'best' don't do as much processing. I'm much more likely to pay attention to the error counts, max latency, or SD (σ).
I wonder how realistic/fair the code of some benchmarks is.<p>I just took some small peak at atix (because I happen to know it) and while first all looked fine it wasn't quite that realistic.<p>Mainly:<p>- It uses a fork of tokio-postgres specific for the test (which differs in that Client is no longer send and it has a Unsafely soundness hole by wrongly using Unsafe Cell, and no Issue tracking enabled on, through replacing that cell with RefCell probably yield very similar performance)<p>- Instead of using the default web::Json responder it uses simd-json (EDIT: explicitly encoding the data into a buffer instead of returning it wrapped in web::Json)<p>- Uses snmalloc instead of the default allocator (<a href="https://github.com/microsoft/snmalloc" rel="nofollow">https://github.com/microsoft/snmalloc</a>)<p>Just to be clear besides the first point all of this is not unrealistic for a context where you want to highly optimize your server. But not how actix is used most of the time.<p>---<p>Edit: Also just to be clear I didn't nit pick intentional on actix, it just happen to be the framework I'm more familiar with and especially using snmalloc for actix seems to be quite a reasonable idea.<p>EDIT2: The Unsefety unsoundness isn't triggered given the way this library is used, it's still there and would prevent this change from ever been merged upstream.
On lithium's JSON parser, they describe:<p>"Only cases where the structure of the object is known at compile time are covered."<p>Numbers are impressive, but there are some big trade-offs that come with.
Link to the project : <a href="https://github.com/matt-42/lithium" rel="nofollow">https://github.com/matt-42/lithium</a><p>not obvious from a google search
Offtopic: matt42 (or others); where do I learn modern optimization like this? I can optimize for cycles on embedded MCU's and older computers and I can optimize in .NET/JVM (and others) but are there any good sources for OS on modern metal optimizations? Besides reading the Lithium sourcecode ofcourse.
has there ever been a code created just specifically for this benchmark, in assembly language with everything hard-coded, just to see what the upper bound is ?
Lithium is the fastest web framework among the frameworks that nobody use<p>Seriously why is this reaching the front page, who really care about such benchmark ?