Comparison of Data Lake Table Formats (Iceberg, Hudi and Delta Lake)

A major problem with these table formats that will surface soon enough is that they use serial numerical ordering for versions.<p>It&#x27;s like inventing SVN for data. Soon enough git will have to be invented as well.

We&#x27;re currently looking into datalake implementations. Right now, we only have 1 or 2 data sources. Current thinking is reading them on the fly, combine them using pandas dataframe and query that. Anyone have experience with doing something similar?

How does the concept of a table here differ from that of a standard relational table (if at all)? Is it that the table is a logical abstraction over a distributed set of files?

Does anyone have experience running either of these three in production?

Does anyone have good real life stories of how data from a data lake made a real difference in a product or a business?

Wondering if this is basically what all data lakes will look like in the future. All data stored in these table formats…

The repo comparison was really cool. I guess that could be made into a product.

Thanks for bringing this comparison out!

Helpful to see a concise comparison!

Nicely summarised and visualised :+1

Great comparison!