Big data is dead

&quot;For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size.&quot;<p>The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being &quot;data driven&quot;. Most corporate decision making is highly political, the needs of&#x2F;whats best for the business is just one parameter in a complex equation.

I&#x27;ve made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been &quot;more and more more data&quot;. Sensor-the-heck out of your farm, and We&#x27;ll Tell You Things(tm).<p>Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or &quot;Aha&quot; moment where some correlation becomes apparent, but the thing about these is that once they&#x27;ve been observed and understood, the value of ongoing observation drops rapidly.<p>A great example of this is soil moisture sensors. Every farmer that puts these in goes geek-crazy for the first year or so. It&#x27;s so cool to see charts that illustrate the effect of their irrigation efforts. They may even learn a little and make some adjustments. But once those adjustments and knowledge have been applied, it&#x27;s not like they really need this ongoing telementry as much anymore. They&#x27;ll check periodically (maybe) to continue to validate their new assumptions, but 3 years later, the probes are often forgotten and left to rot, or reduced in count.

While I get that they&#x27;re sometimes useful to trigger debate, I don&#x27;t really subscribe to very bold statements.<p>We are drowning in data, it&#x27;s all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life.<p>It&#x27;s is an ocean and it&#x27;s all about the vessel you pick to navigate it. I don&#x27;t believe that the vessel should dictates the size of the ocean, it&#x27;s simply constrained by it&#x27;s capabilities. The trick is to pick the right vessel for the job, whether you want to go fast, go far or fish for insights (ok, I need to stop pushing on this metaphor )<p>This visionary paper from Michael Stonebreaker (2005) predicted it quite accurately and I think is still relevant: <a href="https:&#x2F;&#x2F;cs.brown.edu&#x2F;~ugur&#x2F;fits_all.pdf" rel="nofollow">https:&#x2F;&#x2F;cs.brown.edu&#x2F;~ugur&#x2F;fits_all.pdf</a><p>Databases come in various flavours and the &quot;trends&quot; are simply a reflection of what the current era needs<p>Disclaimer: I work at ClickHouse

This posting was great. Highly recommended reading through. It gets really good when the author hits &quot;Data is a Liability&quot;.<p><i>&gt; An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.”</i><p>This is exactly it. It&#x27;s way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can&#x27;t fathom throwing any data way, Just In Case.<p>Really appreciated the post overall. Very insightful.<p>As an anecdote to this article, when business folks have come up to me and asked about storing their data in a Big Data facility, I have never found the justification to recommend it. Like, if your data can fit into RAM, what exactly are we talking about Big Data for?

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated &#x27;chat&#x27; result.<p>Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn&#x27;t mean big data is dead. ChatGPT is literally changing how school will test their students, for a start.<p>Maybe what the author is trying to say is &#x27;small-scale big data is dead, but big data chugs on.&#x27;

MotherDuck has been making the rounds with a big funding announcement [1], and a lot of posts like this one. As a life-long data industry person, I agree with nearly all of what Jordan and Ryan are saying. It all tracks with my personal experience on both the customer and vendor side of &quot;Big Data&quot;.<p>That being said, what&#x27;s the product? The website says &quot;Commercializing DuckDB&quot;, but that doesn&#x27;t give much of an idea of what they&#x27;re offering. DuckDB is already super easy to use out of the box, so what&#x27;s their value-add? It&#x27;s still a super young company, so I&#x27;m sure all that is being figured out as we speak, but if any MotherDuckers are on here, I&#x27;d love to hear more about the actual thing that you&#x27;re building.<p>[1]: <a href="https:&#x2F;&#x2F;techcrunch.com&#x2F;2022&#x2F;11&#x2F;15&#x2F;motherduck-secures-investment-from-andreessen-horowitz-to-commercialize-duckdb&#x2F;" rel="nofollow">https:&#x2F;&#x2F;techcrunch.com&#x2F;2022&#x2F;11&#x2F;15&#x2F;motherduck-secures-investm...</a>

I agree with many of the points here.<p>My cheap no-name old laptop SSD writes with 170MB&#x2F;s.<p>A customer has a name, address, email and order. Let&#x27;s say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach.<p>My disk is 240GB, which means I can store data for 1.2 billion customers. It&#x27;ll take a while until I become that successful.

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database.<p>Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can query whatever you want, …<p>And if the database may get up to 1TB, still no problem with SQL. If exceed that, you may need a professional OPs team for your database and a few giant servers, but they should easily be able to go up to 10 TB, offload some queries to secondary servers, …

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry compared to some of the things others did on the system, particularly, hyrdrodynamics modellers. Moreover, I know I can probably process 100GB datasets on my own home PC, which isn&#x27;t too impressive, it would just take longer (say a day or so instead of a hour or a few minutes). And this is with idk, python scripts using MPI. Yes, MPI because I&#x27;m a computational scientist and that&#x27;s what HPC systems use, nothing fancy and likely the &quot;legacy systems&quot; he railed against in his pitches, but it worked.<p>I&#x27;m just awestruck, I could tell anyone that &quot;large data&quot; isn&#x27;t really a bottleneck, but making sense of it is the very difficult part. My mentors kept pushing me to mention the sheer size of the datasets I process in talks because it sounds impressive, and I do do so, but I always knew it didn&#x27;t matter because the interpretation and analysis is the hard part, not just the &quot;sheer size.&quot;<p>[0] not going to use my real name

Big Data was whatever someone couldn&#x27;t handle in a spreadsheet or on their laptop using R.<p>This paper is 8 years old and it was somewhat obvious then.<p>Scalability! But at what COST? <a href="https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;hotos15&#x2F;hotos15-paper-mcsherry.pdf" rel="nofollow">https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;hotos15&#x2F;hotos...</a><p>A big single machine can handle 98% of peoples data reduction needs. This has always been true. Just because your laptop only has 16GB doesn&#x27;t mean you need a Hadoop (or Spark, or Snowflake) cluster.<p>And it was always in the best interest of the BD vendors and Cloud vendors to say, &quot;collect it all&quot; and analyze on&#x2F;or using our platform.<p>The future of data analysis is doing it at the point of use and incorporating it into your system directly. Your actionable insights should be ON your grafana dashboard seconds after the event occurred.

big data isn&#x27;t big anymore.<p>1) 10 years ago, having access to 300tb of data that could sustain 10gigabytes&#x2F;s of throughput would require something like two racks of disks with some SSD cache and junk.<p>2) people thought hadoop was a good idea<p>3) People assumed that everything could be solved with map:reduce<p>3) machine learning was much less of a thing.<p>4) people realised that postgres does virtually everything that mongo claimed it could.<p>5) people realised that cassandra was a very expensive way to make a write only database.<p>I gave a talk about using big data, and basically at the time the best definition I could come up with was &quot;anything that&#x27;s too big to reasonably fit in one computer. so think 4, 60 disk direct attached SAS boxes&quot;.<p>Most of the time people were chasing the stuff for the CV, rather than actually stopping to think if it was a good idea. (think k8s two years ago, chatGPT now, chat bots in 2020). Most buisnesses just wanted metrics, and instead of building metrics into the app, they decided to boil the ocean by parsing unstructured logs.<p>Not surprisingly it turned to shit pretty quick. Nowadays people are much better at building metrics generation directly into apps, so its much easier to easily plot and correlate stuff.

Detailed web event telemetry is where I have seen the &quot;biggest&quot; data, not application-generated data. Orders, customers, products will always be within reasonable limits. Generating 100s of events (and their associated properties) for every single page&#x2F;app view to track impressions, clicks, scrolls, page-quality measurements can get you to billions of rows and TBs of data pretty quickly for a moderately popular site. Convincing technical leaders to delete old, unused data has been difficult; convincing product owners to instrument fewer events is even harder.

I love DuckDB and am cheering for MotherDuck, but I think bragging about how fast you can query small data is really no different than bragging about big data. In reality, big data&#x27;s success <i>is not</i> about data volume. It&#x27;s about enabling people to effectively collaborate on data and share a single source of truth.<p>I don&#x27;t know much about MotherDuck&#x27;s plans, but I hope they&#x27;re focused on making it as easy to collaborate on &quot;small data&quot; as Snowflake&#x2F;etc. have made it to collaborate on &quot;big data&quot;.

Some of mongo&#x27;s leveling off is the adoption of good jsonb columns in postgres.<p>mongo&#x27;s got sharding out of the box - which is nice - but you have to get your key right or it will suck.<p>Also no one should want to host a mongo db - unless that&#x27;s your business.

This reminds me of a great blog post by Frank McSherry (Materialize, timely dataflow, etc) talking about how using the right tools on a laptop could beat out a bunch of these JVM distributed querying tools because... data locality basically.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;frankmcsherry&#x2F;blog&#x2F;blob&#x2F;master&#x2F;posts&#x2F;2015-02-04.md">https:&#x2F;&#x2F;github.com&#x2F;frankmcsherry&#x2F;blog&#x2F;blob&#x2F;master&#x2F;posts&#x2F;2015...</a>

Big Data is dead? Seems well and alive to me. If you&#x27;re not a big company with big customers, it never affected you to begin with.

I am writing an essay series on this topic: last-mile analytics and how an abundance of data must be ultimately converted into (measurably correct) action.<p>If anyone wants to follow along, the series is here!<p><a href="https:&#x2F;&#x2F;alexpetralia.com&#x2F;2023&#x2F;01&#x2F;19&#x2F;working-with-data-from-start-to-finish&#x2F;" rel="nofollow">https:&#x2F;&#x2F;alexpetralia.com&#x2F;2023&#x2F;01&#x2F;19&#x2F;working-with-data-from-s...</a>

Not that big data is dead, more like real time data is coming to life, but you need the old stuff around to make a buck or two… Well, that my view. LLMs are transformer model technique are making data more relevant than ever. If you are a business, well you are in for a “now real” digital transformation.<p>Making data the centerpiece of your business business could mean that your effectiveness of business process could increase several order of magnitudes. Funny thing is, you will not use some else’s model, unless you are building a ChatBox to infer, but you will need to build your own model and be trained in your own business process to be successful.<p>Consider a bank, here is my prediction of expected outcomes:<p>Enhanced Customer Experience: The system can act as a virtual banking assistant, providing customers with instant access to their account information, real-time transactions, and balance updates. The system can also answer customer inquiries and provide relevant information, improving the overall customer experience. Improved Fraud Detection: The system can monitor the bank&#x27;s financial transactions in real-time and identify any potential fraud, helping the bank reduce its exposure to financial losses.<p>Automated Loan Processing: The system can analyze loan applications, credit scores, and other relevant data to approve or reject loan applications in real-time, reducing the time and effort required for manual loan processing. Personalized Marketing: The system can analyze customer behavior, transaction history, and demographic information to provide personalized marketing and cross-selling opportunities, increasing the bank&#x27;s revenue and customer loyalty.<p>Real-Time Insights: The system can provide real-time insights into the bank&#x27;s financial performance, customer behavior, and market trends, enabling the bank to make informed decisions and respond to market changes quickly.<p>What is interesting to me is, this is just the beginning of what could be…

&gt; <i>Are you in the big data one percent?</i><p>Exactly, and I&#x27;d go further.<p>Are you in the perf&#x2F;scale&#x2F;data one percent?<p>So many people worry about scaling when in reality 99% of web apps will never reach above 100reqs&#x2F;s.<p>I&#x27;ve been in web dev for 20+ years. Only once when working for a big international corporate client I had to worry about traffic spikes. And that was just for one of their multiple web apps.

Who has ever believed those claims? There&#x27;s a common saying &quot;garbage in, garbage out&quot; about what happens with all those fancy models if the data quality is not high. That&#x27;s really independent from dataset-size. There&#x27;s no magic insight you get because your dataset is bigger. You need a quality analyst to handle your data, irrelevant of its size.<p>Also, who thought their company would cease to function because surely they will hit google-scale dataset-sizes in the near future? Impossible for most except the biggest of the biggest

It is amusing that in 2005, &quot;VLDB&quot; (precursor term to &quot;big data&quot;) was defined in Wikipedia to be &quot;larger than 1TB&quot;.. after reading through the post and the author&#x27;s experience.. it would appear that this was not actually a completely terrible estimate, although there are larger and smaller: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;w&#x2F;index.php?title=Very_large_database&amp;oldid=20738417" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;w&#x2F;index.php?title=Very_large_databa...</a><p>The current version of that article states: &quot;There is no absolute amount of data that can be cited. For example, one cannot say that any database with more than 1 TB of data is considered a VLDB. This absolute amount of data has varied over time as computer processing, storage and backup methods have become better able to handle larger amounts of data.[5] That said, VLDB issues may start to appear when 1 TB is approached,[8][9] and are more than likely to have appeared as 30 TB or so is exceeded.[10]&quot; <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Very_large_database" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Very_large_database</a>

Not dead, just complying with the Gartner cycle for hypes.<p>There is probably a rational, well thought out classification of different types of data bigness, as in CERN-big, Google-big, MegaBank-big, down to wordpress-log big and on the basis of that one would probably find that different designs are indispensable, address different pain points and cannot really &quot;die&quot;. Hype has a more erratic lifecycle than real needs

This is an excellent summary, but it glosses over part of the problem (perhaps because the author has an obvious, and often quite good solution, namely DuckDB).<p>The implicit problem is that even if the dataset fits in memory, the software processing that data often uses more RAM than the machine has. And unlike using too much CPU, which just slows you down, using too much memory means your process is either dead or so slow it may as well be. It&#x27;s _really easy_ to use way too much memory with e.g. Pandas. And there&#x27;s three ways to approach this:<p>* As mentioned in the article, throw more money at the problem with cloud VMs. This gets expensive at scale, and can be a pain, and (unless you pursue the next two solutions) is in some sense a workaround.<p>* Better data processing tools: Use a smart enough tool that it can use efficient query planning and streaming algorithms to limit data usage. There&#x27;s DuckDB, obviously, and Polars; here&#x27;s a writeup I did showing how Polars uses much less memory than Pandas for the same query: <a href="https:&#x2F;&#x2F;pythonspeed.com&#x2F;articles&#x2F;polars-memory-pandas&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pythonspeed.com&#x2F;articles&#x2F;polars-memory-pandas&#x2F;</a><p>* Better visibility&#x2F;observability: Make it easier to actually see where memory usage is coming from, so that the problems can be fixed. It&#x27;s often very difficult to get good visibility here, partially because the tooling for performance and memory is often biased towards web apps, that have different requirements than data processing. In particular, the bottleneck is _peak_ memory, which requires a particular kind of memory profiling.<p>In the Python world, relevant memory profilers are pretty new. The most popular open source one at this point is Memray (<a href="https:&#x2F;&#x2F;bloomberg.github.io&#x2F;memray&#x2F;" rel="nofollow">https:&#x2F;&#x2F;bloomberg.github.io&#x2F;memray&#x2F;</a>), but I also maintain Fil (<a href="https:&#x2F;&#x2F;pythonspeed.com&#x2F;fil&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pythonspeed.com&#x2F;fil&#x2F;</a>). Both can give you visibility into sources of memory usage that was previous painfully difficult to get. On the commercial side, I&#x27;m working on <a href="https:&#x2F;&#x2F;sciagraph.com" rel="nofollow">https:&#x2F;&#x2F;sciagraph.com</a>, which does memory and also performance profiling for Python data processing applications, and is designed to support running in development but also in production.

The title might be hyperbole (intentionally), but the observations are more or less in line with what I experienced through a few the Big Data initiatives over the years under different enterprise environments (although I have reservation about the one 1%er comment). To me, Big Data was never about how &quot;big&quot; the data was, but more about the tools&#x2F;system&#x2F;practice needed to overcome the limitation of the previous generation. From that perspective, yes, the &quot;monolith&quot; may be having a &quot;coming back&quot; for now due to the improvement of underlying single node performance. But I do think Data size will keep growing, everything needed to make Big Data work would still be there when the pendulum swings back where a single node can&#x27;t handle it anymore.

I feel like big data has rarely lived in most organizations. My own experience working in large orgs largely supports the point that collected data is rarely queried. But this is rarely due to a lack of interest, it is mostly because a) nobody really has a great overview over what even is collected b) even if you know&#x2F;assume something is collected, you usually have no idea where c) if you find the data, there is a decent chance that it is in some sort of weird format that requires a ton of processing to be usable.<p>This has been - to varying extends - my own experience working in large organizations that don&#x27;t have tech as their core business.<p>Although there are some successful data analysis project, the potential of the collected data remains largely underutilized.

&gt; Most data is rarely queried<p>Right on point. In the past I have been obsessed with big data, looking for insights. Then I realized that a medium-sized specific data set is always better than a gargantuan general big data monster. There is so many applications in my field where only outliers matter anyways, and everything is very &quot;centralized&quot; to a few relevant observations. So the only thing about big data is that you maybe throw away 99.9% of the data right away and then you have some observations that you actually care about. There is soooo much data out there that is just noise, and so little that I actually care about. And that&#x27;s why I still end up hand collecting stuff every now and then.

I believe we are living in the &quot;emotional era&quot;, so data has being ignored and &#x27;feelings&#x27; come first when making decisions or creating processes. This is happening not only in companies but in our current society in general.

Customer pays data analytics vendor to tackle bunch of their [low quality, big size] data.<p>If you have no tangible capabilities to do above, asking customer &quot;ARE YOU IN THE BIG DATA ONE PERCENT?&quot; will be the quickest way out of the door.

I love DuckDB&#x27;s simplicity and think it will solve many problems. Still, transitioning from a local single file DB to concurrent updates and serving it online will be different. I&#x27;m curious about what MotherDuck will come up with to solve DuckDB at scale.<p>I love use cases like the Rill Data (<a href="https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=XvP2-dJ4nVM">https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=XvP2-dJ4nVM</a>), where you can suddenly run analytics with a single cmd line prompt and see your data just instantly visualized. Such use cases are only possible because of the &quot;small&quot; data approach that DuckDB tries.

Looks at 15 hr Spark job (running since this morning)<p>Sighs...

Confirmation bias exists almost everywhere. Confirmation bias especially among senior management is highly dangerous as decisions are based on not on data and facts, rather they are based on anecdotes, hunch&#x2F;feelings, with high probability of going wrong. This is precisely where data scientists play a significant role, by providing recommendations and presenting facts based on hard data and mathematical models, in order to ensure that senior management decisions are based on facts&#x2F;data, and not on anecdotes and hunches. Furthermore, a data driven organisation must have a supporting culture, where data driven decisions are given precedence, and data scientists (data messengers) must be empowered to present facts as is, no matter whether these facts are aligned or not with the basic assumptions and biases held by the senior management team. Creating such a supporting organization culture is extremely important but definite not easy. Culture is one of the factors that makes a difference between success or failure in a data driven organisation.

To the extent &quot;Big Data&quot; originally and is still often claimed to mean &quot;data beyond what fits on a single [process&#x2F;RAM&#x2F;disk&#x2F;etc]&quot;, it&#x27;s always been strange to me how much it&#x27;s identified with analytics pipelines doing largely trivial transformations producing ultra-expensive &quot;BI&quot; pablum.<p>Yes, thank goodness that part is dead. But meanwhile - we&#x27;ve still got more <i>actual data</i> than ever to store, and ever-tighter deadlines on finding and delivering it. If we can get back to that and let the PySpark bootcampers fade away, maybe things can get a little better for once.<p>In other words:<p><i>Even when querying giant tables, you rarely end up needing to process very much data. Modern analytical databases can do column projection to read only a subset of fields, and partition pruning to read only a narrow date range. They can often go even further with segment elimination to exploit locality in the data via clustering or automatic micro partitioning. Other tricks like computing over compressed data, projection, and predicate pushdown are ways that you can do less IO at query time. And less IO turns into less computation that needs to be done, which turns into lower costs and latency.</i><p>Big data is &quot;dead&quot; because data engineers (the programming ones, not the analysts-in-all-but-title) spent a ton of effort building DBs with new techniques that scale better than before, with other storage patterns than before. Someone still has to write and maintain those! And it would be even better if those tools and techniques could escape the half dozen major data cloud companies and be more directly accessible to the average small team.

&quot;90% of queries processed less than 100 MB of data. [in big query]&quot;<p>I think there is a problem when someone with such proclaimed knowledge of the sector gets to this, and similar, pieces of data, and does not attribute it to pricing. Could it be queries are short because bigquery pricing for analysis, as confusing as this models are, is based on amount of data?[0]<p>Because the other line of reasoning is that a big chunk of that 90% of professionals being paid to do their jobs, do NOT take into account pricing of the tool and are using it for small data, instead of thinking that people are using the best tool with the lowest price, because there&#x27;s plenty of options to process and analyse data right now in the cloud.<p>On the &quot;business have low amount of data&quot;, that matches my experience as well. At first I thought I was simply dealing with smaller sized companies, but it&#x27;s a trend of doing big data projects for data that&#x27;d fit a pendrive.<p>[0] <a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;bigquery&#x2F;pricing#analysis_pricing_models" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;bigquery&#x2F;pricing#analysis_pricing_m...</a>

Sampling has proven extremely useful. Pi can be approximated with it as were nuclear bombs designed using statistical methods. Flame graphs based on stack samples are used to optimize servers. Government does planning with it. Management does its thing by wandering around.<p>It usually does not take many data points for an actionable insight and most actions then will invalidate small details in old data anyhow. Better to start every round with fresh eyes.

This entire post reads like &quot;you probably don&#x27;t actually have big data&quot;.<p>What do these blockchains do that have to keep data around forever, with high throughput, and need to expose it quickly do? Are you saying they should delete parts of data in the chain?<p>Seriously, I&#x27;ve spent my career working on big data systems, and while the answer is sometimes &quot;yes you need to delete your data&quot;, I don&#x27;t think that&#x27;s going to always work.

Seems like just yesterday, every business magazine&#x27;s cover story was about &quot;big data.&quot; Wonder what the next batch of business buzz words will be?

Pretty funny to see this when every other headline on this site is about how large language models are about to revolutionize dentistry, beekeeping, etc.

Perhaps this is true for business data (though I&#x27;m skeptical of the claims), but, for example, for security data, this isn&#x27;t true at all. Collecting cloud, identity, SaaS, and network logs&#x2F;data can easily exceed hundreds of terabytes. A big reason why we&#x27;re building Matano as a data lake for security.<p>It seems an odd pitch in general to say, hey my product specifically performs poorly on large datasets.

To add to the “the real issue is…” pile:<p>Most orgs collect the data that is easy to collect, and they are extremely lucky if that happens to be the data that enables the insights they desire. When the data they really <i>need</i> looks too hard to get, the org tries to compensate by collecting more of the easy stuff, and hoping that if blood can’t be squeezed out of a stone, maybe it can be squeezed out of 100bn stones.

I remember the big data craze. People had very little data and low quality at that so they had a data problem before they had a big data one!

&gt; Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc<p>I’m no statistician, but I’m like 99% sure that’s an exponential, not a power law<p>There’s a world of difference. The point of an exponential is that you can ignore big things. The point of a power law is that you can’t.

With all the LLM craziness, this is just the beginning. How else are they going to train all those models? I&#x27;m not an expert, just imho.

We need to re-think how to make data _useful_. The fact that the value hasn&#x27;t materialized after decades of attempts, billions of dollars, and lots of tools and technology points to the fact that our core assumptions and patterns are wrong.<p>This post doesn&#x27;t go far enough. It challenges the assumption that everyone&#x27;s data is &quot;big data&quot; or that every company&#x27;s data will eventually grow to be big data. I agree that &quot;big data&quot; was the wrong model. We also need to challenge that all data should be stored in one place (warehouse, lake, lakehouse). We need to challenge that one tool can be used for every data need. We need to challenge how we build systems both from a technology and people standpoint. We need to embrace that the problems and needs of companies _are always changing_.<p>We are living with conceptual inertia. Many of our patterns are an evolution from the 70&#x27;s and 80&#x27;s and the first relational databases. It&#x27;s time to rethink how we &quot;do data&quot; from first principles.

I&#x27;m quite surprised with data sizes mentioned in the article, and wondering if I&#x27;m missing something...We are a very small 2yo company, handling route optimization and delivery management &#x2F; field service. Even with our very small number of customers, their relatively small sizes (e.g. number of &quot;tasks&quot; per day), being very early in development in terms of data that we collect - our database containing just customer data for 2 years is ~100GB. Which I previously considered small, and if we collected useful user metrics, had more elaborate analytics, location tracking history etc, I would expect it to be at least 3x.<p>We don&#x27;t use any &quot;BigData&quot; products yet, as there wasn&#x27;t any need for them, even when we provide full search and relatively nice and rich set of analytics over all the data. Yet, based on the article, we&#x27;re way above most of the companies relying heavily on such tools. Confusing.

Another problem with &quot;BigData&quot;: hiring and the tendency of the ecosystem to &quot;sustain&quot; itself (like any system). As a company hires traditional BigData Architects, Developers, Data Scientists, Engineers, etc. it will naturally have a tendency to choose the traditional BigData technology and solutions like BigQuery, Spark, storing everything in HDFS, etc.<p>A trick I saw is companies hiring experienced jack-of-all-trades back-end engineers into Data teams. A lot of things get migrated from Spark to Postgres, from Kafka to REST API calls, and keep working fine and become generally more responsive.<p>I&#x27;m on the same page as the author here: traditional BigData tech has its place and its uses, but before choosing it companies (CTOs, architects) should carefully consider if it is necessary, especially considering the cost of it and the risk of locking themselves down in a very specialized domain.

Tableau&#x27;s &quot;Medium Data&quot; April Fools Day ad from several years ago still rings amazingly true.

So what I&#x27;m hearing is it&#x27;s not the size of your data that matters, it&#x27;s how you use it?

From about 2008&#x2F;2009&#x2F;2010 or so on there was perhaps an over-emphasis on specialized tools for the mass acquisition of streams of data. Maybe in large part due to the explosion of $$ in ad-tech. Some people had legitimately insane click&#x2F;impression streams -- I worked at a couple companies like that. Development of DBs based on LSM trees or other write-specialized storage structures became important. Existing relational databases weren&#x27;t particularly well built for this stuff. This was part of, but not the whole story with the whole NoSQL thing. People were willing to go completely denormalized in order to gain some advantage or ability here. It helped that much of the data looked at was of perhaps little structural complexity.<p>In the meantime SSD storage took off, so the IOPS from a stock drive have skyrocketed, business domains for large data sets have broadened beyond click&#x2F;impression streams, and the challenge now is not &quot;can I store all this data&quot; it&#x27;s &quot;WTH do I do with it?&quot;<p>Regardless of quantity of data, structuring and analysis and querying of said data remains paramount. The challenge for anybody working with data is to represent and extract knowledge. I remain convinced that logic -- first order logic and its offshoot in the relational model -- remains the best tool for reasoning about knowledge. Codd&#x27;s prognostications on data from the 1970s are still profound.<p>I think we&#x27;re in a space now where we can turn our attention to knowledge management, not just accumulating streams of unstructured data. The challenge in a business is to discover and capture the rules and relationship in data. SQL is an existing but poor tool for this, based on some of the concepts in the relational model but tossing them together in a relatively uncomposable and awkward way (though it remains better than the dogs breakfast of &quot;NoSQL&quot; alternatives that were tossed together for a while there.)<p>My employer is working in this space, I think they have a really good product: <a href="https:&#x2F;&#x2F;relational.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;relational.ai&#x2F;</a>

My presentation from FOSDEM 2023 is very sympathetic to the &quot;Big data is dead&quot; statement: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=JlcI2Vfz_uk">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=JlcI2Vfz_uk</a><p>It is about using modern tools (ClickHouse) for data engineering without the fluff - when you can take whatever dataset or data stream and make what you need without the need for complex infrastructure.<p>Nevertheless, the statement &quot;big data is dead&quot; is short-sighted, and I don&#x27;t entirely follow this opinion.<p>For example, here is one of ClickHouse&#x27;s use-case:<p>&gt; Main cluster is 110PB nvme storage, 100k+ cpu cores, 800TB ram. The uncompressed data size on the main cluster is 1EB.<p>And when you have this sort of data for realtime processing, no other technology can help you.

we regularly run audits on over 12 years of customer order histories. This requires scanning of about 40TB of data and growing. They used to jump through hoops on the Oracle cluster just to get data out for one customer. We pushed all of the order history into s3 parquet using Spark and I can query this in about 20 seconds using Spark or Presto. It&#x27;s now streamed through kafka and Spark structured streaming so it&#x27;s up to date in about 3 minutes. The click-bait-y title notwithstanding, I get that not all data is &#x27;big&#x27; and duckdb (and datafusion, polars, etc) is probably great for certain use-cases but what I work on every day can&#x27;t be done on a single machine.

To be honest, I slightly disagree about data size. I think the big data is there to be had, the real story is that data science itself has not panned out to provide the business value that people asserted would come from it. Data volumes haven&#x27;t risen more because in the end, it turns out most of the things businesses need to know are easily ascertainable from much smaller data and their ability to action even these smaller very obvious things is already saturated.<p>It doesn&#x27;t help that we&#x27;ve shifted into a climate where hoarding data comes with a huge regulatory and compliance price tag, not to mention risk. But if the value was there we would do it, so this is not the primary driver.

It&#x27;s kinda weird to read this. The whole argument is &quot;we didn&#x27;t have databases that could handle the sizes and use cases emerging, we worked on the problem for 20 years and now it&#x27;s no biggie&quot;.<p>Mission accomplished more than big data is dead IMHO.

Long live Big Model, I guess? Instead of independent data warehouses, we are now moving towards a few centralized companies using supercomputer in physical data centers. The &quot;winner takes all&quot; effect will only increase as the trend goes on.

Congratulations on the birth of little data, to the proud dad Big Data are in order.<p>Well, obviously, the realization that management is largely emotion driven and little data driven, is a prelude for the CEO-AI yet in the makings.<p>Of course this still got a face to it. A CEO who speaks and talks, as the voice commands, but does not do the part that even humans who think they are good at it, are bad at, decision making. The ground truth is there (&quot;Cooperate history&quot;) going back to the merchants of sumeria. Lets learn that lesson, pack it into a decission tree, and wrap that bundle with Chat GPT smooth talking.

The database was the key technology in the 2001-2011 decade: it allowed companies to store massive amount of data in an organized way, so that they could provide basic functionality (search, monitoring) to users. Statistical learning is being the key &quot;technology&quot; of 2011-today: it allowed companies, which had stored massive amount of data, to feedback predictions to users. I think AR&#x2F;Computer Graphics will be the key technology of the next decade: it will allow users to interact directly and seamlessly with the insights produced by ML systems, and possibly feed-back information.

nosql is dead, client side SPAs are dead. Nice to see the complexity pendulum swinging back to the correct side again. Curious what the merchants of complexity will reach for next. Are applets going to be the new hot thing?

Great post and really resonates with my experience. Good to have some confirmation that most organizations aren&#x27;t using their large swaths of data.<p>Although I don&#x27;t think most organizations are blaming lack of actionable insights on the data size. It&#x27;s the lack of prioritizing data usage over data accessibility. We need to be teaching data people business levers and teaching business people data levers.<p>Data should be a byproduct of an actionable idea that you want to execute. It shouldn&#x27;t exist until you have that experiment in mind.

Big Data lives on in LLMs.

Agree. And thing I noticed is that tools like #apache spark have become the de-facto standard for any data engineer work even when data size does not require it. Result is that many jobs are much harder to mantain and often slower (due to all the shuffling) than running on a single node.

Somewhere along the line people were tricked into thinking that logging was data, and that to we needed to turn up every trace log to 11 on every production system.<p>Logs are where data goes to die.

My personal definition of Big Data has always been when you gather&#x2F;store data without having a planned use for it. Do we need this data? Don&#x27;t know, let&#x27;s just store it for now.<p>The article does allude to this definition when it states that &quot;Most data is rarely queried&quot;. We have become data hoarders. Technology has made it easy (and relatively cheap) to store data, but the ideas of what to do with this data have not scaled in comparison.

&quot;Among customers who were using the service heavily, the median data storage size was much less than 100 GB&quot;<p>Eye-opening. Especially when combined with a recent quote from Satya Nadella, &quot;First, as we saw customers accelerate their digital spend during the pandemic, we’re now seeing them optimize their digital spend to do more with less.&quot;<p>Conclusion: SaaS is easy to drop off in downturns. Just as easy as it is to buy initially.

Why would I use DuckDB instead of Clickhouse or similar? Is it just because I want to have the database embedded in my app and not connect to a server?

I agree with a lot of the sentiments of the MotherDuck people, but boy are they loud and proud for someone who never delivered anything more than blogposts and vague promise to somehow exploit the MIT licensed DuckDB.<p>Meanwhile for example boilingdata.com seems to have already done that - by using AWS Lambda + DuckDB as distributed compute engine which I can&#x27;t decide if its awesome, deranged or both.

Unlike quantum which cracks computationally complex algorithms, BigData was just about costs.<p>SSDs we’re limited in capacity and still expensive.<p>Parallelizing work with MapReduce allowed using cheap fault-prone commodity hardware and disks.<p>If you’re dealing with terabytes rather than petabytes of data, you probably don’t need BigData

&quot;Very often when a data warehousing customer moves from an environment where they didn’t have separation of storage and compute into one where they do have it, their storage usage grows tremendously...&quot;<p>Can someone explain why this is the case? Is it due to more replications or maintaining more indices?

Well then if businesses do not require data, then the &quot;AI world&quot; might need some. So changing practice to be a machine learning engineer might not seem too bad.

Well, we have less than 2 TB of data, and although we are running MySQL on a large instance with ~120 GB of RAM, it&#x27;s extremely slow when dealing with big tables (like a 25 GB table) and that&#x27;s why we need &quot;big data&quot; tools like BigQuery.

It&#x27;s always kind of amazed me how closely Big Data was followed by the KonMari method and it really seems like the nerds were not paying attention to that at all. Or just happy to take a paycheck from people who weren&#x27;t paying attention.<p>Hoarding is not a winning strategy.

I think server hardware solved the big data issue. The stuff we have now can blitz through data in the blink of an eye. For national governments like our own, mainframes still have a place. For me personally, I don&#x27;t even talk about big data anymore.

People don&#x27;t want to deal with having to rearchitect when their workload does not fit on a single instance. Yes, optimize for the small data case, but if you build a product that can handle only the small data case, you have a tough sell.

&gt; The most surprising thing that I learned was that most of the people using “Big Query” don’t really have Big Data.<p>wow, ya think? Must have been eye opening to see all those customers with a few million rows thinking they had &quot;Big data&quot; huh?

It&#x27;s not dead, it&#x27;s just entered the plateau of productivity. Where people use it for whatever it&#x27;s useful for and don&#x27;t try to solve every problem with it just because it&#x27;s the cool new thing.

On to the next hype theme(AI)!

Big data starts somewhere around a petabyte, maybe a bit lower than that. That&#x27;s when you need some serious, dedicated systems. But as always everyone wants to (pretend to) do what the big players do.

First they came for the sacred microservices, now they are after Bid Data. What. Is. Happening.<p>Don&#x27;t get me wrong, I love it. It&#x27;s about time people got off these stupid and shockingly expensive bandwagons.

Similar to the &quot;we must have microservices so that we can <i>scale</i>&quot; fad a lot of people thought they had big data even though their records easily fit on a single machine.

We&#x27;re going to reach a point where we might say the same thing about large language models. Fine tuned LMs (based off of their large parents) are going to be the bread and butter.

Agree with this post.<p>Big data was vendor generated hype that convinced many engineers to confuse the size of their dataset with their, ahem, shoe size.<p>They didn’t do their employers any favors.

Big data is dead because executives are rewarded when decisions are reactionary and politically savvy, data doesn&#x27;t enter the picture.

This is a bit ironic given that generative AI models like GPT-3 and Dall-E only work because they were trained on very large datasets.

Main goal of Big Data as i see is to profile performance and metrics. Number of user registration, number of converted users,...

I quite often say: If you need KPIs you&#x27;re too far removed from how the company actually conducts business.

So the argument is you can do everything with an OLAP Database because we shrunk &quot;Big Data&quot; back inside RAM?<p>K, good luck!

Medium data is where it&#x27;s at! diskframe.com and polars and arrows are good enough for most use cases!

The truth is that most &quot;big data&quot; problems aren&#x27;t big and can often be solved with awk and xargs.

What about IoT?

Big Data got replaced by Big Parameters.

On the contrary. Now that AI is here, big data is going to be more alive than ever.

I think the relevant phrase here is &#x27;Selling your book&#x27;

This is what most of the companies are doing.

Long live big data!

Long live Big Meaning

Listen to “Reason”

Interesting take from a Googler.<p>Big data hype never felt to me like anything more than a hype campaign to help big tech research ML&#x2F;AI.<p>Larry even rambled as much: <a href="https:&#x2F;&#x2F;arstechnica.com&#x2F;information-technology&#x2F;2013&#x2F;05&#x2F;larry-page-wants-you-to-stop-worrying-and-let-him-fix-the-world&#x2F;" rel="nofollow">https:&#x2F;&#x2F;arstechnica.com&#x2F;information-technology&#x2F;2013&#x2F;05&#x2F;larry...</a><p>It appears not all Googlers got the memo.<p>Everyone else is in the way of him solving big problems! Not like such work could not be distributed among technologists and researchers around the globe via the internet. Help Google do it!<p>I am leaning into “Deep Work” going forward; will slowly iterate on my own model creation and collaborate with like minded folks. I’m fucking done with intentionally empowering billionaire minority who convinced an ignorant political gerontocracy that minority is capable of magic.<p>Anyone prattling on with common tropes of “longtermism”; nation state nutters, religious, technocrats; are appealing to non-existent authority they see a a magical future for us! Give them your money to insure it arrives! They have zero ability to insure such outcomes and a lot of upside to making people believe such today.

Good riddance.