A point that a lot of nominal Big Data startups miss is that genuinely large-scale data management and analytics are not driven by visualizations at all nor fit in a web-driven SaaS-like environment. The purpose is to answer a complex question from unimaginably large volumes of data, not to draw charts and graphs. It is often too I/O intensive for virtualized clouds and the visualization component is almost superfluous to the purpose. Most of the problems that need to be solved in Big Data are low level, down at the computer science and infrastructure level. Many of the use cases are intrinsically poorly suited for web-based SaaS type offering.<p>To make matters worse, many high-value Big Data analytical problems are (literally) not meaningfully visualizable except for marketing purposes. It is rather tricky to visualize an analytic product when there are a hundred critical values that need to be rendered in some fashion for every pixel your monitor can display. A lot of high-value analytics have this characteristic but most of the nominal Big Data visualization tools ignore this case even though it is arguably the most important one.<p>Consequently, while labeling your startup "Big Data" is trendy and fashionable, there are very few genuine Big Data startups. Adding value in this market requires a combination of serious theoretical computer science chops plus very creative interface design. Few startups are actually addressing the needs of this market and are instead assuming the market wants the web app they have the skills to produce.
There's a lot of money to be made in the smaller verticals. I think for a startup, there's a lot of opportunity to allow people to just manipulate and sanitize data in simpler spreadsheets.<p>For example, one thing I'm being forced to implement myself has been a lot of string manipulation operations to sanitize different kinds of data I'm playing with in spreadsheets.<p>Even just having something misimported wastes a lot of time.<p>OpenRefine isn't bad, but can only get you so far. That being said, if I can come up with a complete solution myself, I wouldn't mind just adding it to the suite of tools I'm already offering :)<p>I'm also wondering about different kinds of tools already out there though.
Excel is good enough for most problems that small and medium sized firms have. Big Data tools and techniques are only worth the effort when the problems to solve are big. That's not small business.
I think there is definitely an opportunity to help companies understand their GA data, but our company Keen IO is proof that Big Data is not too fat for startups. We're finding a big opportunity in helping developers build custom analytics backends and white-labeled analytics. We've found the big bucks are made supporting customers with truly big data challenges. We can provide the scalability and reliability that would be arduous and expensive for them to build in-house.
I can certainly relate to the challenges of working on a big data platform intended to immediately satisfy varied customers...<p>But mainly I want to say I'm super impressed with the Google Analytics 'storification'. I can imagine the difficulties in bringing that level of quality to myriad data sources, but I'm excited to see you succeed!