Hadoop “fails us”

This really strikes me as more of a marketing piece by Snowflake than a well-researched piece of reporting. The article mostly just quotes one person - Bob Muglia - who is, as they say on Wall Street, "talking his book" - i.e. giving an opinion that is not coincidentally in line with his own financial interests. Sure, Hadoop is getting old, and is quickly becoming replaced by spark. But loads of organizations have used, and continue to use hadoop /spark successfully. And the part about Kafka replacing Hadoop /Spark is just silly. They're completely different technologies, used for very different purposes, and many organizations use both side by side.

&gt; Hadoop is great if you’re a data scientist who knows how to code in MapReduce or Pig<p>I guess that kind of makes sense. What programmers like might not necessarily be a good basis for the long-term. I do have to say, though, that part of the failure of Hadoop is it generated a lot of hype and so better tools and alternatives developed to meet that need. So when you&#x27;re saying a better alternative is Spark or Kafka, I feel like it&#x27;s almost as if you&#x27;re saying &quot;Oracle is a failure that never materialized the promised benefits. Instead we should be using PostGres and MySQL.&quot;<p>But I also know that with a lot of big data hype, businesses were wanting to do Hadoop and NoSQL and all this stuff because it was the cool new thing, not that they actually needed it. I&#x27;ve heard data scientists make the joke that every business thinks they need these tools because they&#x27;re having difficulty running their business out of a spreadsheet.<p>I think it&#x27;s important to remember that for most businesses, Spark, Hive, whatever; those aren&#x27;t the right tools, either. SQL is still what most companies need. Businesses want machine learning, but usually what they need is boring old statistics. In an industry that always wants to be ahead of the curve, we tend to forget that it&#x27;s not always the right thing to have the newest toys. Sometime companies do well utilizing the latest and greatest, but sometimes they just use what already exists wisely. I suspect that for many of those companies that used Hadoop and felt it didn&#x27;t work them, the problem wasn&#x27;t that they needed Spark instead, it was they were trying to solve problems that didn&#x27;t exist. Man, I&#x27;m too young to sound this old. But, eh, yeah, we need to respect our elders&#x27; technology first, and consider the newest stuff only when we have a definable need for it.

My observation that traditional database technologies are transforming themselves into &#x27;hybrids&#x27; (as far document-oriented data types go).<p>Example is Postgres. Now has with JSONB as document-oriented field type. Now has Postgres XL, as horizontally scalable ACID database. Will have, by approximately august&#x27;, abiltity to maintain views in memory (aka lambda.architecture speed layer) via PipelineDB extension, for fast streaming analytics.<p>It seems that a combination of Postgres (with extensions) + Kafka +Redis -- is a strong stack for lambda architecture and, as initial data hub component of the overall puzzle<p>While Spark (or even Python+Dask) can be viewed as distributed data analytics platforms that replaces &#x27;non-ui centric&#x27; BI. I think ui-centric BI (eg adhoc reports&#x2F;visualizatiosn) are going to be still dominated within enterprises by Tableu&#x2F;QlickView type of solutions.<p>For traditional BI-oriented data marts (that organized downstream from the datahub) -- probably traditional column oriented databases, and the new open source ones make sense.<p>To me ,the promise of hadoop being a silver bullet for &#x27;all the big data needs&#x27; -- was always nothing more than unsubstantiated hype.<p>So it definitely failed the ones who believed, the hype -- but did not fail others who did not buy into it.

I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and then we migrated to other Big Data technologies. Just wanted to add my thoughts here.<p>Hadoop (and its general ecosystem, which includes HBase), is a fairly good idea. Its core ideas - map&#x2F;reduce on Hadoop, and a large distributed key&#x2F;value store for HBase - are actually pretty solid.<p>And for many years, there were simply no alternatives to Hadoop. Think of the years 2008 to 2012&#x2F;13. If you had to process terabytes or petabytes of data, what were your solutions? No wonder Yahoo and Facebook (and others) put in so much effort into their Hadoop solutions.<p>But, IMHO, there were several issues with Hadoop and their ilk.<p>1. The core infrastructure wasn&#x27;t stable enough. Hadoop &#x2F; HBase were supposed to be distributed systems, and they worked well, but small failures could bring down your entire cluster. Given that Hadoop and HBase were being used in mission-critical systems in the cloud, and given the amount of DevOps or sys-admin work that went into maintaining these, I&#x27;m not surprised people eventually migrated to distributed systems that were easier to maintain and run.<p>2. There are now plenty of &quot;hosted on the cloud&quot; solutions such as Amazon DynamoDB or similar cloud solutions. When your company depends on 99.99% or similar SLAs, you don&#x27;t want to have downtime on your database systems and spend time debugging complicated core dumps on your Hadoop or HBase clusters when you can just store it &quot;on the cloud&quot; and be done with it. Sure, there&#x27;s a higher price point, but those are the trade-offs you live with.<p>3. If you want to be in-house, there are plenty of alternatives out there as well today. Apache Spark for processing, Kafka for a messaging bus &#x2F; streaming data, ElasticSearch for large scale storage, with multiple indices. Many of them are much more robust than Apache Hadoop &#x2F; HBase, and I&#x27;m not surprised they&#x27;ve gotten more traction recently.<p>Ultimately, I think Hadoop &#x2F; HBase are just showing their age. They were fantastic for the first wave of Big Data technologies, and you had little alternative if you were building large-scale systems circa 2008 to 2013, but now, you just have a plethora of choices from various vendors.

The article is on point about how we need better data infrastructure to support data scientists and analysts. In the past I&#x27;ve worked to develop very scalable data infrastructure to support data science workloads on high variety sensor dat,a but it always felt like the only reason we were doing it was that nobody developed tools made for companies like ours.<p>We can build better data infrastructure for data scientists, but in practice it&#x27;s hard to sell &quot;10x easier to use&quot; into organizations with hadoop or half-broken bespoke infrastructure because the IT groups running the show don&#x27;t really care that they&#x27;re making their data scientists miserable.<p>If change is to come, it&#x27;s going to have to be from data scientists embedded within business units demanding better tools, because hadoop works just fine if you don&#x27;t care how hard it is for your users to access their data.

Hadoop didn&#x27;t fail us. We failed Hadoop.<p>The decline of Hadoop as a software category is Software Product Marketing 101: it did not identify pervasive killer use cases critical to running . Yes, it&#x27;s true that Hadoop was a revolutionary way to store and process massive datasets on commodity hardware, but what&#x27;s the use case for that? If you are Visa&#x2F;AMEX (fraud detection), Facebook&#x2F;Google (various ML-based data products) and a few other types of companies with obvious applications of massive data processing, yes, Hadoop has been great.<p>But here&#x27;s the thing: beyond a few such corner cases, it never found a use case that enterprise data warehouse couldn&#x27;t handle.<p>Then came Redshift, then BigQuery, and now Snowflake (as a BigQuery on AWS, really). While there are some key technical differences between Redshift and BigQuery&#x2F;Snowflake, they are all _much_ cheaper than the previous generation of data warehouses (Vertica, Netezza, Greenplum, etc.) The lower price meant greater access, and developers who previously couldn&#x27;t imagine using data warehouses could finally spin one up with a credit card swipe.<p>Hadoop, too, took a lot of collateral damage because many developers realized that they didn&#x27;t need much of Hadoop beyond SQL-on-Hadoop.<p>Redshift was a beautiful feat of product strategy and marketing: They just took what used to cost a lot and offered it for much less in an environment where developers already had a lot of data (AWS). This was much simpler to execute than what Hadoop had to do: introduce new technology, identify use cases, and finally compete with incumbent solutions.<p>We failed Hadoop (as you can see from www.cloudera.com, even Cloudera, the Hadoop company, hardly mentions Hadoop on its top page). Not the other way around.

Disclaimer : I work for a corporate IT consulting giant.<p>The trend I&#x27;m seeing the big Data sphere in my company is a by-and-large move away from technologies that implement MapReduce and and complicated batch processing with HDFS as a data store. More and more customers want insight as soon as they get&#x2F;produce their data so we&#x27;ve seen a particularly large increase in interest in technologies such as Kafka&#x2F; Samza and spark&#x2F; pySpark.<p>I see a trend in Kafka and but I think the community needs to jump behind it too, keep it as a pipeline tool and not a querying engine.<p>I dont see Hadoop-based solutions going away any time soon though.

What about competition like HPCC[1]?<p>[1]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;HPCC" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;HPCC</a>

Kafka relies on ZooKeeper of the Hadoop eco-system. ZooKeeper is not so great.