Humans vs. Microsoft Excel: The Quest For Smart Tools

The author said it himself, these errors were caused by humans, not excel. I don&#x27;t think making excel &quot;smart&quot; enough to prevent people from making dumb mistakes is a good idea. In fact I think that would transform excel into microsoft word, which is the only thing that could make life worse for your typical competent office worker.<p>Excel already has a lot of this crap. It already requires an act of god to prevent excel from changing your string of digits into a number or date. It already shows you little exclamation point icons when your formulas omit adjacent rows or are different from other formulas in the same row&#x2F;column.<p>A tool will never make it possible for dumb people to solve hard problems easily. It&#x27;s like trying to design a knife that makes it impossible to cut yourself. Nobody with any kind of a clue would want that knife.<p>A tool should be straightforward and intuitive, but it shouldn&#x27;t aim to be smarter than it&#x27;s user.

I work with an army of very smart people who need to build incredibly complex models to drive our business. Like most companies, we turn to Excel. It&#x27;s the only widely used application with a low enough barrier to entry that &quot;non-technical&quot; people can build legitimately complex (if not always robust) models.<p>But the problems with Excel and general spreadsheet modeling limitations offer such a great opportunity for improvement that would impact just about every business out there. I wish I saw more people tackling this, or I had a great idea to do it myself. The problems I see are around barrier to entry, in that it must be usable in just a couple minutes, simple enough for non-programmers to learn, and equally or more efficient to get a basic model functioning. Otherwise I just don&#x27;t see wide adoption despite possible maintenance, accuracy, and reliability benefits.<p>James Kwak has written some smart stuff about this (<a href="http:&#x2F;&#x2F;baselinescenario.com&#x2F;2013&#x2F;04&#x2F;18&#x2F;more-bad-excel&#x2F;" rel="nofollow">http:&#x2F;&#x2F;baselinescenario.com&#x2F;2013&#x2F;04&#x2F;18&#x2F;more-bad-excel&#x2F;</a>), and I&#x27;m interested in what Data Nitro (datanitro.com) is doing.

disagree completely. MS Excel is a masterpiece in software engineering, definitely the best piece of software MS ever created. Large pieces of the economy run on Excel, finance, controlling, etc. are unthinkable without it.<p>the flexibility, ease of use, speed is astounding. applying colors to cells and then being able to sort by color still blows my mind - not the feature per se, but the fact that they included it and solved a common problem (how many times did a client mark something by color in a big spreadsheet...).<p>excel is mobile, works offline. replacing it with some webapp or too much logic just shows the complete misunderstanding of its appeal.

This is a problem I&#x27;ve spent a lot of time pondering. The author makes several suggestions; one in particular (a GitHub for datasets) is an idea I felt strongly enough about that I founded a start-up called BuzzData to address this concern. Unfortunately, we did not find our critical mass and the product has since successfully pivoted into an entirely different product called LookBookHQ.<p>I do have another idea, which is to bring the concept of testing to the Excel world.<p>I think that the combination of:<p>- a framework for building up test cases that can be run against the data as part of the saving process (think rSpec)<p>- an English-like, indented grammar for defining user stories that can potentially generate code for tests (think Cucumber)<p>- a strong push for a culture of testing in the Excel community that would have to include Microsoft<p>I dunno, honestly. Could be crazy. The culture might never accept testing.<p>But then again, one of the basic ways to sell something is to give people insurance against humiliating themselves. It&#x27;s possible that this thinking would make a lot of sense to the twice shy, highly risk-averse board members of the major financial institutions and governments of the world.

The fundamental problem with big spreadsheets is that they&#x27;re complex pieces of software, and they&#x27;re being designed by people who haven&#x27;t studied software engineering.<p>Building a complex program is hard. It&#x27;s untenable if you don&#x27;t approach it properly. This is true whether you&#x27;re using Excel, Python, LISP, or anything else. Better tools can go a long way to help, but the ultimate answer is educating people on how to properly engineer systems.<p>On that last point - at DataNitro[1], we let people script Excel with Python, which is a better tool than VBA or than doing things by hand. The result is that for a given level of complexity, you see fewer errors and more robust spreadsheets; but when you give people more power, they naturally build more sophisticated spreadsheets. All else being equal, this might result in more, not fewer errors.<p>The great thing about introducing a new tool, though, is that there&#x27;s also a chance to introduce new ways of working. Suddenly, people start building spreadsheets that work with version control and run unit tests. This helps tremendously with robustness and reliability.<p>[1] <a href="https:&#x2F;&#x2F;www.datanitro.com" rel="nofollow">https:&#x2F;&#x2F;www.datanitro.com</a>

At a previous job, I tried to convince them (and succeeded in some cases) to build a domain-specific web app instead of sharing and e-mailing ever-changing Excel docs.<p>I used CakePHP, and it was really easy to whip up a complete CRUD web-app with domain-specific information and DB tables, allow people to collaborate through a web interface, and be happy users.<p>Some apps are tougher than others, I&#x27;ll admit.

I dare say that the problem is less about Excel being dumb than that if you&#x27;re interested in the underlying logic Excel hides all that from you the minute you click off a cell. There&#x27;s black box programming and then there&#x27;s just silliness.<p>As for data-entry, ideally, that&#x27;s not something that humans should be doing for anything but the most trivial concern. If you have huge columns of financial data or test results or what have you, then having someone type it into a computer is silliness of the highest order.

I think it&#x27;s impossible to replace Excel en masse, so lots of folks are starting to pick off bits of it to replace. And hopefully in doing so they&#x27;ll invest in making those bits a lot smarter than they are in Excel.<p>Examples from the S12 YC batch alone:<p>. VBA isn&#x27;t great, so DataNitro is letting you use Python.<p>. Grid is (mobiley) replacing the basic list-making use case of Excel.<p>. (Stretch example:) Financial modeling of big personal finance decisions is a nice use case of Excel, but too hard for most people, so SmartAsset bakes those kinds of models into an on-site calculator.<p>. Statwing is replacing pivot tables, which are clunky, devoid of statistics, do no error checking, etc.; in a few clicks Statwing will automatically choose, run, and interpret a statistical analysis, all the while looking for and alerting users to outliers or (some) other data issues. [I&#x27;m a cofounder. Also: <a href="https:&#x2F;&#x2F;www.statwing.com&#x2F;demo" rel="nofollow">https:&#x2F;&#x2F;www.statwing.com&#x2F;demo</a>]<p>I&#x27;d expect that soon someone will come along soon and pick off the financial modeling use case in a way that&#x27;s a bit more generalized than SmartAsset. Really tough UI challenge, but I think doable.

Having worked in nyc financial world, I can say excel is here to stay - at least for foreseeable future. Heck, the only reason I have windows running in parallel in my MBA is excel. We use to have what we called &quot;excel test&quot; for all quants and analyst positions. We use to QA our excel sheets just like any other software application. I am surprised that many companies don&#x27;t do that - it only makes sense.

You can turn Excel spreadsheets into web applications quite easily with Google Apps Script. We wrote a REST interface to a massive multi tab Excel file recently and it only took like an hour, as opposed to days of reverse engineering the calculations (the client did not know what the equations in the file were because the employee who created it left).<p>Basically I&#x27;m saying it will get much, much worse before it gets better.

There are entire organisations and conferences about the dangers of spreadsheets[1] and there are now programming standards for develop spreadsheets[2].<p>Basically, the normal rules apply. If you want safe spreadsheets, you have to plan and verify.<p>The reason this doesn&#x27;t happen is because firstly, spreadsheets are very accessible. Every working large system evolves from a working small system, as John Gall observed. Excel et al make that evolution relatively easy.<p>The second is that spreadsheets enable non-programmers to work with concrete concepts instead of the abstract ones that programming languages encourage. It&#x27;s easy to visualise locations on a sheet when you are looking at the sheet.<p>Finally, spreadsheets support calculation by propagation. They are an excellent declarative environment. Most programming languages do not support this model of programming (though we&#x27;re groping towards it with concepts of binding).<p>[1] <a href="http:&#x2F;&#x2F;www.eusprig.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.eusprig.org&#x2F;</a><p>[2] <a href="http:&#x2F;&#x2F;www.fast-standard.org&#x2F;document&#x2F;FastStandard_01b.pdf" rel="nofollow">http:&#x2F;&#x2F;www.fast-standard.org&#x2F;document&#x2F;FastStandard_01b.pdf</a>

If you are amused at what autocorrect does for writing, imagine the hilarity of economic decisions based on &quot;smart&quot; spreadsheets.

Excel will always be used BECAUSE its so free-form.<p><a href="http:&#x2F;&#x2F;readwrite.com&#x2F;2013&#x2F;06&#x2F;11&#x2F;excel-is-an-art-form-these-beautiful-images-are-proof#awesm=~o8w9EUvUTJDPlC" rel="nofollow">http:&#x2F;&#x2F;readwrite.com&#x2F;2013&#x2F;06&#x2F;11&#x2F;excel-is-an-art-form-these-b...</a>

Hmm... Excel-like front-end to Haskell?

Typed fields sound neat!

We need to think beyond the limitations of tools no matter how pervasive their use. It reminded me of : <a href="http:&#x2F;&#x2F;vimeo.com&#x2F;66085662" rel="nofollow">http:&#x2F;&#x2F;vimeo.com&#x2F;66085662</a> (you can skip to the demo)

I got theory. There is a limit what you can do with WYSIWYG editors. Once you reach the limit, the software starts getting in your way, hiding your errors and making you work harder.<p>It applies for many software. From HTML to 3D modeling.