One way to add some trust is to make benchmarks open-source and reproducible:
<a href="https://github.com/ClickHouse/ClickBench/">https://github.com/ClickHouse/ClickBench/</a><p>(I work at ClickHouse)
Can you trust benchmarks Ever ? - No!<p>Or rather while some benchmarks which are relevant for your use case, are done correctly and unbiased may exist it is not easy to find and identify
Trust for what purpose? Are you a Marketing Droid for a product which looked good/bad in some particular benchmark? A fanboy yearning to proclaim yourself Right on the internet? A paid-per-click author cranking out web content? A PHB looking for validation of his half-assed decisions?<p>For the great majority of use cases, any looks-good benchmark will do the job.<p>(Vs. if you actually need to know about real-world performance - you'll have to do some serious work for that information.)
Full disclosure: I work at StarTree, which is powered by Apache Pinot.<p>ClickHouse's ClickBench is a good general tool. However, it's not the end-all, be-all of performance benchmarking and testing. Its results may or may not be applicable for guidance on the performance of your specific use case when you get to production.<p>It is definitely a stab at getting an objective suite of tools for the real-time analytics space. But just like you had YCSB as a good general performance test, eventually a subset of users wanted something specific for Cassandra and Cassandra-like databases (DSE, ScyllaDB, etc.), so you eventually saw cassandra-stress. We have to consider cases where certain databases may need to have testing suites that really capture their capabilities.<p>ClickHouse themselves publishes a list of Limitations that everyone should keep in mind as they run ClickBench:<p><a href="https://github.com/ClickHouse/ClickBench/#limitations">https://github.com/ClickHouse/ClickBench/#limitations</a><p>CelerData (based on StarRocks) also wrote up this:<p><a href="https://celerdata.com/blog/what-you-should-know-before-using-clickbench" rel="nofollow">https://celerdata.com/blog/what-you-should-know-before-using...</a><p>Plus, I want to direct people to the discussion generated when ClickBench was first posted to HN:<p><a href="https://news.ycombinator.com/item?id=32084571" rel="nofollow">https://news.ycombinator.com/item?id=32084571</a><p>As user AdamProut commented back at the time:<p>> It looks like the queries are all single table queries with group-bys and aggregates over a reasonably small data set (10s of GB)?<p>>I'm sure some real workloads look like this, but I don't think it's a very good test case to show the strengths/weaknesses of an analytical databases query processor or query optimizer (no joins, unions, window functions, complex query shapes ?).<p>> For example, if there were any queries with some complex joins Clickhouse would likely not do very well right now given its immature query optimizer (Clickhouse blogs always recommend denormalizing data into tables with many columns to avoid joins).<p>So, again, ClickBench is a good (great) beginning. As an industry we should not let it be seen as the end. I'd be interested in the community's opinions on what and how we should be doing better.