I feel like developers who haven't worked with medium-sized data don't know where to draw the "big data" line. You see this with companies overspending for complicated "big data" solutions when the only have something on the order of billions of rows and nothing more. I deal with many billion row tables in the products, and SQL Server still works wonderfully there. Throw 1TB of RAM and a stack of Fusion IO drives on a pair of servers and I'm confident the pair (for mirroring obviously) and a DBA can easily handle the needs of any company that thinks they have "big data", save the current big data pioneers (e.g. Google, Facebook) who are only spending time on these problems because throwing more money (read: hardware) was hitting diminishing returns. In my specific case, we have billions of rows store in SQL tables whose schemas (including PK and indices) and queries were built by server engineers without any explicit DBA experience. If we can manage to store our data quite naively in SQL server by just putting it on a pair of superboxes, I find it hard to believe this solution wouldn't apply to all but a handful of tech companies.
Disagree with almost everything this guy wrote.<p>99% of startups and in house business apps will never need to scale at a level that an RDBMS can't support. We should all be so lucky to have the success that brings these horizontal scalability problems.<p>Also, I hope he lets us know what kind of incredible data he end up mining from his mouse movement logs. Surely every website and app on planet Earth has completely exhausted basic solutions like A/B testing and usability testing to make their apps better. The only solution now is to embark on a data mining expedition so that artificial intelligence will tell us how to make better software.
I agree with what you said. I worked with .net since the 1.0 days and after years of staunch support, I got a job last year working with python/javascript on a unix stack and kissed microsoft technologies goodbye.<p>You're absolutely right about Microsoft being behind the curve. There's a pattern of denial when it comes to new technologies, and a total ineptitude at spotting new trends and adapting to them. I watched with envy for years as all the rails kids played with their new toys until Microsoft got its shit together and came up with an MVC solution.<p>The whole world is using flash? Lets come up with a knock-off 5 years too late. Interactivity is a big deal? Let's try to convince them that webforms is MVC until its too late. Who needs javascript anyways? Distrubuted software you say? Let's not jump on the REST bandwagon, lets make WSDL defined web services and make a shitty API on top! Why have an ORM solution when putting all of your business logic in SQL is SO EASY with microsoft? See, I can drag and drop a database connection from Visual Studio!<p>I love C# as much as the next person. It's a great language. SQL Server is a pretty good relational database. But if you choose to devote all of your energy to Microsoft development and don't learn anything else, you have nobody to blame expect for yourself when you find your skill-set behind the curve.<p>.Net developers would never agree about the big-data part, because to be honest they haven't invested time into understanding what data science is. There are so many IT shops developing on the Microsoft stack that make less than say 50 million a year that have tons of data (billions of records they say!). Maybe they make CRM software, or school software, or an inventory system for a small grocery store. They have tons of test scores, purchase records, waste numbers, demographic info, yet if you ask them to develop something that gives you insight into what factors influence student performance, or to visualize perchasing trends, or to predict sales numbers so that the company doesn't over purchase perishables, you'll get a blank stare. The idea that data can be transformed into useful information, and that this isn't simply a matter of CRUD hasn't occurred to them. They probably don't even replicate their db for their half ass attempt at reporting, so they wouldn't begin to understand what something like hadoop even does.<p>You simply CANNOT explain to a .net zealot what big data is. They just don't get it. "It's stored in the database, and its a few terabytes...how big does it need to get? We aren't Google afterall!"
I'm in a similar boat except that I think I can keep the good (C# / .Net) and leave the bad (Microsoft Analysis Server).<p>We are looking at scaled DB backend using column stores, multiple compute instances feed through message queues running on Amazon or private cloud, and a mostly Javascript / HTML5 front-end. It is all tied together with C# and so far it doesn't look like it will be too bad.<p>Not surprisingly, the stickiest part of the puzzle is... Microsoft Excel, specifically Pivot Tables. Weird but true.
Sometimes when I talk to CS students at my university, I feel like we are living in parallel universes. I live in one where Python, Ruby and JavaScript are the most important languages, basic Unix skills is a must-have, etc. In their world, the whole programming landscape is divided in two sides: .NET and Java. Everything that doesn't fall in either of those categories is a toy.<p>I thought it was just the atmosphere of my university, or maybe realities of the job market in my country[1]. But reading this made me wonder id there are really two parallel universes in computer-land.<p>[1] Based on the name of OP, I am 90% sure we have the same nationality.
I agree there's an increasing demand for big data but I think the OP is overreacting. I think there's going to be a big demand for internal LOB apps, mobile apps etc for a long time. Not everything is big nor needs to scale. Can't hurt to learn unix or jvm languages though.
So his point is that databases are going to go away (odd, considering even Google went back to a nice ACIDy RDBMS for AdWords), and Microsoft is just going to ignore any sort of scaling technology and disappear. That's certainly an odd perspective.<p>MS has two problems with "big data" (and let's just pretend lots of people have big data problems - I've seen people deploy 100-node Hadoop clusters to deal with 30M rows/month of data because they think Hadoop is some magic sauce.)<p>First, they want to squeeze as much money out of enterprise customers as possible. Look at the new SQL Server pricing and limitations, where the SQL team is doing things they promised they wouldn't and previously mocked Oracle for doing (charging by core). Their in-memory solution (Hekaton) comes a bit late, and only for Enterprise edition.<p>Second, they generally want to deliver solutions that the majority of their customers can figure out. Again, look at Hekaton. From what I've read, it'll have extremely high compatibility with T-SQL. Shipping a limited release that only had a small subset of SQL, even though it'd be useful for certain apps, probably was never a serious consideration. Hell, look at C# and how their customers are begging them to please not innovate too much, since learning is hard.<p>There's also the Azure push. SQL Azure has federations, making it easy to shard a traditional SQL schema across many physical instances. They also promote Hadoop on Azure. (As I understand, Hadoop's a dead-end; even inside Google, MapReduce was toasted by Dremel, right?)<p>If Microsoft just added Hadoop-style functionality to one of their server products, it would not be friendly. People capable of writing an OK SQL query for a report can't necessarily format that same query in an efficient map-reduce style.<p>And really, since this need is far less than it's played out to be, MS is probably just fine pushing and profiting off their traditional solutions. Azure helps them secure a few leading-edge needs, and eventually they'll roll out an easy-to-use commercial solution.
I don't agree on big data part. But I certainly agree on not innovating and not have a solid plan. MS doesn't have any plan, their biggest problem is they don't like to innovate and instead just love to jump on the bandwagon of any trend. I mean what's the point of Silverlight? Adobe Flash is already dominated market. Instead of focusing on HTML5, they just built another thing like Flash.<p>MS needs innovation and solid plan; not jumping on the band wagon of next hot topic.
I'm a bit confused by this statement:<p>"Cannot say the same thing for middleware technologies, such as BizTalk or NServiceBus. Databases? Out of question."<p>I'm curious about his view that middleware is not here to stay. Surely the multiple vendor cloud trend that enterprises are following (Google + salesforce + Microsoft + oracle) - because no one vendor does everything - means middleware is going to be more prevalent in the future.<p>Thoughts?
It's like Microsoft can't catch a break: if they develop something that competes with other solutions - it's NIH syndrome; if they don't - they're behind the curve.<p>Fun fact is that DevDiv produces tons of things that allow .NET devs use existing OSS solutions for things that are not done by Microsoft themselves.