Data Wrangling at Slack

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30 random technologies and become an analytics company too.

Sometimes looking at people's stacks I wonder if we've made computing so complicated most of the time is spent dealing with stuff that is broken, and little time is left to do anything useful. Data science seems even more into this that programming in general; and sometimes you wonder if the result is actually worth all the pain.

Seems like a pretty typical set of problems. Dependency conflicts hard. Schema evolution hard. Upgrades hard.<p>The big data space still feels like an overengineered, fractured, buggy mess to me. I was hoping spark would simplify the user experience but it&#x27;s as much of a clusterf*ck as anything else.<p>How hard can fast, reliable distributed computation and storage for petabytes of data be? He said ironically.

We actually have pretty similar architecture and use Presto for ad-hoc analysis, Avro is used for hot data and ORC is used as columnar storage at <a href="https:&#x2F;&#x2F;rakam.io" rel="nofollow">https:&#x2F;&#x2F;rakam.io</a>. Similar to Slack, we have append-only schema (stored on Mysql instead of Hive), since Avro has field ordering the parser uses the latest schema and if it gets EOF in the middle of the buffer, fills the unread columns as null. We modified the Presto engine and built a real-time data warehouse, Avro is used when pushing data to Kafka, the consumers fetch the data in micro-batches, process and convert it to ORC format and save it to the both local SSD + AWS S3.

I had very similar experience with Parquet and cross system pains. Pretty much the whole big data space is a giant cluster fuck of poorly documented and ever so slightly incompatible technologies.. with hidden config flags you need to find to get it to work the way you want, classpath issues, tiny incompatibilities between data storage formats and SQL dialects and so on..<p>Hoping someone on this thread could answer a related question - how do you store data in Parquet when the schema is not known ahead of time? Currently we create an RDD and use Spark to save as Parquet (which I believe has an encoder&#x2F;decoder for Rows) but this is a problem because we can&#x27;t stream each record as it comes and use a lot of memory to buffer before writing to disk.

We&#x27;re actually having a debate now as we&#x27;re starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w&#x2F; Hive. I&#x27;m curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.

I&#x27;m curious about how much time is spent moving data back and forth from S3. It sounds like they don&#x27;t currently have an ETL per say.

We are implementing a very similar architecture, and have decided to use Avro for schema validation &#x2F; serialization, rather than Parquet.<p>Does anyone have experience with both that can talk to their strengths &#x2F; weaknesses?

Data Engineering is about developing technology for data management. Data management&#x2F;analysis is about using this technology to produce results.<p>So this is not about data engineering, but data management&#x2F;analysis.

We (adtech) use a very similar approach. We&#x27;re consuming a ton of data through Kafka and then using Secor to store it on S3 as Parquet files. We then use Spark for both aggregations as well as ad-hoc analyses.<p>One thing that sounds very interesting and worked surprisingly well when I played around with it was Amazon&#x27;s Athena (<a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;athena&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;athena&#x2F;</a>) which lets you query Parquet data directly without relying on Spark which can get expensive quickly. I wouldn&#x27;t trust production use cases just yet and it ties you more and more into the AWS ecosystem but might be worth exploring as a simple way to do basic queries on top of Parquet data. I suspect it&#x27;s simply a managed service on top of Apache Drill (<a href="https:&#x2F;&#x2F;drill.apache.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;drill.apache.org&#x2F;</a>).

With Qubole you can offload data engineering to their platform. Cluster management is super simple. Hand rolled solutions in my experience are a pain and elastic cloud features take up time to build. Qubole&#x27;s offering provides out of the box experience for most big data engines out there. Presto&#x2F; Spark&#x2F; Hive&#x2F; Pig - what have you - all work with your data living in S3 (or any other object storage). I believe they have offerings in other clouds too.<p>Some amount of S3 listing optimisation is done by Qubole&#x27;s engineering team for: <a href="https:&#x2F;&#x2F;www.qubole.com&#x2F;blog&#x2F;product&#x2F;optimizing-s3-bulk-listings-for-performant-hive-queries&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.qubole.com&#x2F;blog&#x2F;product&#x2F;optimizing-s3-bulk-listi...</a><p>They also have features that allow you to auto-provision for additional capacity in your compute clusters as your query processing times increase.

Apparently the concept on sampling has been lost in time.

Isn&#x27;t moving data back and forth from s3 rather expensive?

This is off-topic, but I can&#x27;t help myself:<p>Slack, Hive, Presto, Spark, Sqooper, Kafka, Secor, Thrift, Parquet.<p>I sometimes can&#x27;t tell the difference between real Silicon Valley product names and parodies. I&#x27;m starting to miss the days when it was all just letters and numbers.

Well for its worth my experience interviewing for the data team there was terrible. A long coding exercise that when submitted resulted in a 7 day wait and a 2 liner email. Wouldn&#x27;t recommend.