This has always been my pet peeve with jobs that won't hire people without '5-7 years experience with X' - the vast majority of these jobs do not need an EXPERT on <i>language x</i>, or <i>framework y</i>. If they do, the rest of this post does not apply.<p>You can learn enough of a language to be dangerous in just a couple weeks. Enough to be productive in 2-3 months. However, there are 3 other things you have to learn:<p>1. How the code for the project you are hired for works. How it's laid out. All the weird shit about it.<p>2. How the framework/libraries used by the project works. Nothing to do with the project, but maybe you haven't used Angular, Swing, or Boost before.<p>3. How the data is laid out, and how it gets in/out of your system. What database tables are there, what's the workflow.<p>Those 3 things are the hard part and what take 6-12 months. Learning Javascript, Ruby, or whatever language is the least of your worries.
Glamorous Toolkit (<a href="https://gtoolkit.com" rel="nofollow">https://gtoolkit.com</a>) by feenk, which embodies the thesis of this blog post, is in my opinion the most exciting working development environment in decades. I think in 5-10 years people will be pointing to it as having a huge impact on the industry.<p>I implore anyone serious about changing how software development works to digest Andrei Chis' thesis Moldable Tools: <a href="http://scg.unibe.ch/archive/phd/chis-phd.pdf" rel="nofollow">http://scg.unibe.ch/archive/phd/chis-phd.pdf</a><p>Once you understand the consequences of reducing the cost of specialised tool development by orders of magnitude, it becomes obvious that a qualitative change in experience follows, that brings us much closer to what people like Engelbart were searching for.
Some of us (me, for example :) have to spend multiple weeks a year to draw the design architecture of certain parts of the product. During engineering meetings everybody raised the same question: i commit to new area in product, how can I understand what are the components and how to do certain things? You can ask your colleagues but they might know it as well.<p>So I decided to document main flows and designs in PlantUML diagrams. having these diagrams greatly improved onboarding process, cause you can quickly glance what component does what and what are the dependencies (the code base was in JS, so it is usually quite limited on refactoring/figuring out wtf is going on).<p>But the problem with such approach is: diagram quickly gets out of date. Someone makes the change and the diagram makes no sense at all now. With what I saw in Gtoolkit, you can always query the real source code and build custom dev tools that always produce current and real overview of the system. I would love to have a starter kit for JS projects that you can drag and drop and start building your own tooling for your product.
Couldn't agree more. I think we should redesign programming so that it is primarily a method of communicating between humans. If that becomes the focus, we can use the best methods we have of communicating words, graphics, and interactivity to explain complex systems.<p>One of my own projects in this area is a prototype where I made the best possible explanation of a JavaScript library I could: <a href="https://glench.github.io/fuzzyset.js/ui/" rel="nofollow">https://glench.github.io/fuzzyset.js/ui/</a>
The hard part is talking to hiring managers or potential clients about this without sounding like an idiot who doesn’t know how to put his or her shoes on the right feet. What do other people do in those situations to communicate this?
If anyone is interested, Tudor gave a wonderful presentation of Glamorous Toolkit at last month’s UK Smalltalk user group meeting:<p>Part 1 <a href="https://vimeo.com/496004749" rel="nofollow">https://vimeo.com/496004749</a>
Part 2 <a href="https://vimeo.com/498735070" rel="nofollow">https://vimeo.com/498735070</a>
This is my situation right now on a contract I started a few months ago. A consultancy was hired and spent over a year building a system. All the in-house employees don’t really know how it works, it’s essentially a black box to them. So I’m digging through the code to make sense of it and no one from the business can give me test data that reflects what the system was meant to handle. Only thing I can really do is best guess and slowly learn how the business operates while at the same time fix bugs and launch new features.
So, one aspect of this post is: a) a lot of prior work has assumed that "comprehension" and "reading" are the same and b) reading is a bad approach to understanding code.<p>For me, this also calls to mind an old blog post from Peter Seibel about a disconnect where many of us think that we should all be reading code for our own understanding, somewhat like literature, but very few of us do and it rarely yields much. And one of the reasons why it seems that style of reading is ineffective, is that coming to an understanding of code is more like a scientific investigation than just reading prose.<p><a href="http://www.gigamonkeys.com/code-reading/" rel="nofollow">http://www.gigamonkeys.com/code-reading/</a><p>I agree with that point. I also agree that we need to be able to create tools that deal with software systems more easily.<p>But I also think two other perspectives are important:<p>- One is the historical perspective. The code-base is rarely a coherent whole. Two different areas may accomplish similar things with different tactics. The design of a component may be ill-suited to the way it is now being used. What is an intentional choice, and what is an accident? Which past choices should my current project align with? We typically read the code as it exists currently, and look at specific parts of the history only as a supplement, because even visualizing history is complex. But understanding why choices were made, and in what context, can be critical to knowing which things can now be changed.<p>- The other is that reading code shows us a complex intensional definition, and reading tests gives us a partial view of an extensional definition (in case X we get behavior Y). But to "understand" programs enough to proficiently change them, we have to grokk something like the neighborhood around our current program: How would a given change in the code change the behavior? Being able to interactively change and re-run a program, and compare behavior before and after is in some sense like doing finite difference method differentiation.
This is so true, I Twitch streamed myself developing a game, for a total of 64 hours. And looking back, I found the parts where development slowed down is when I was confused--I almost want to yell at myself--hindsight really is 20/20.<p>What I took away is the faster I can recognize I'm in confusion, the faster I can get out of that state--it means I really need to focus on learning what I'm not understanding, and then come up with a plan to change the actual system into what I want it to do.<p>Another thing is, though I've really refined text editing skills, it's interesting to see yourself struggling to _simply edit text_. So while watching myself, I jotted down a few ideas of shortcuts that could help.<p>Final thing, that I don't know how to fix, is sometimes I'm not motivated, or not in the mood, or scatter brained. Honestly, looking at myself, I don't know what my deal is. If I can just force myself to turn on the stream, then my friends will pop in and encourage me, so that has really helped. Though I'll still hit work that I'm like <i>ugh</i>.
At the beginning of the presentation, a slide showed the similarities between IDEs. For whatever reason, it reminded me of the setup for every intro to Smalltalk presentation I've been through. Sure enough, Glamorous Toolkit was written in Pharo, and seems to really bring a lot of the smalltalk like tooling to other languages ... and wraps it up with a notebook style UI. It's the most interesting Smalltalk thing I've seen in a while.
Reading this leads me to wonder: Is a large part of what makes a "5% programmer" or whatever, that they are so much better at comprehension/figuring-it-out, and retaining what they've figured out for next time?
I've spent almost a year with Spark and I think I'm just scratching the surface now. There are so many knobs that just configuring out just the optimal cluster configuration for production jobs took weeks of testing (so much of that is dependent on the data size and specific use-case). The docs are pretty good, but really don't detail any of 'gotchas' that you'll find in production (you have google relentlessly for those), and the hacks you put in place to deal with those, well, you're on your own. Unless you have a staff that has worked with the toolset for years with it (which we don't.. I'm basically it), you will spend weeks in a try / hack loop. All that said, it's a great toolset for its intended purpose..scala is a great language, etc, etc.. but I've spent a long time 'figuring the system out'.
I've found Source Sourcetrail (<a href="https://www.sourcetrail.com/" rel="nofollow">https://www.sourcetrail.com/</a>) an invaluable tool in aiding program comprehension, especially in C++ projects.<p>There's another interesting program comprehension tool based on dynamic analysis (as opposed to Sourcetrail's static analysis), which I've yet to try: <a href="http://findtheflow.io/" rel="nofollow">http://findtheflow.io/</a>
The tooling approach is an interesting one, but I think the most important thing remains documentation. The "treat code like data" analogy breaks down because unlike (most) data, code is a thing that was intentionally crafted, one piece at a time, by a relatively small number of people, many of whom are probably still in the building. It isn't some foreign artifact, understood by no-one, that's been measured from impersonal processes. It was <i>made</i>. Almost by definition someone has <i>already had an understanding of it</i> at some point. Reverse-engineering a new understanding from scratch - even via powerful tooling - remains a wasteful path to take compared to simply reading a (written-down) understanding that already exists.
From my experience, when changing (or fixing) something on an existing system, I typically spend ~48% of the time figuring out where to do the change, then ~2% actually doing the change, and then another ~48% testing it and adapting unit tests broken by it.
I also remember a post on HN about Chris Granger's Lightable IDE?<p><a href="http://lighttable.com/" rel="nofollow">http://lighttable.com/</a><p><a href="https://www.chris-granger.com/lighttable/" rel="nofollow">https://www.chris-granger.com/lighttable/</a><p>Although as ground breaking, it does not seemt to have 'taken off' as much as it should?
The difference between operator and Engineer is, you need an Engineer when it's not obvious what to do next. That's the whole job - figuring the system out and finding a solution.
So let me get this straight: A programming environment to program programming environments in. What would be the startup cost of using this be? Hmm. Comprehending and navigating it, eh? So now, when I'm, say, doing embedded C/++ programming, I have to add yet another language to the stack, yet another toolkit, yet another environment _I have to build first_ with my preferences -- so that _another_ person getting into the project can not only use my environments, my toolkits, my code and comprehend those - but ALSO my hand built specialized for my needs programming environment? And this is supposed to _improve_ the situation?<p>OK, I only skimmed the site, but mostly because above question keeps nagging me and the mind boggles.<p>IOW, gtoolkit is today's sexy emacs (or, it claims to be what emacs claims/ed to be): Here's a toolkit for writing/extending/MOLDING your tool. Or, acknowledging it's smalltalk-heritaged, it's yet-another-smalltalk-IDE.<p>It's not like the "extra work" of making your product explorable disappears with glamorous toolkit, the silver bullet. gtoolkit just offers a streamlined set of APIs for you to (comprehend, and navigate, and then) use to make your product explorable, doesn't it. So we add another layer of abstraction, work, and potential for error? What was that with complexity and abstraction? Hmm.
I know that the programming language is only part of the story but I do wish more programming languages put a greater emphasis on readability and maintainability rather than writing. I find it particularly annoying when a language does not allow you call functions/methods with named parameters. E.g. it's possible in python, required in Swift and impossible in Golang.
Hence the value of documentation. NIH syndrome is perhaps often a rational "I need clarity and control, so I am not going to waste time grokking the existing implementations and just re-implement" decision.
Does someone have advice on how to figure out a system, especially a Java API? I'm supposed to own an API at work, but I never really understood how to learn an API. Do you type out the functionality in text? Do you draw a flow diagram? Do you keep things at a class level or dive down into functions and variables? The bright side is that this API isn't very big, so I do want to use this as an opportunity to really learn it and also learn how to approach studying API's in general.
This is why following conventions for whatever technologies you are using (not reinventing the wheel) has value. Same for documentation. Going the extra mile to document what you are doing and why, with some examples, or to update the documentation as things change, is not only valuable, but a clear sign of the maturity of a developer.
This will probably get worse as the value to engineers of fighting for architectural sanity diminishes in the modern semi-disposable project based job economy.<p>I have no stake in the the companies future? OK, add another layer of abstraction to 'fix' that nonissue. Whatever. I can avoid it for as long as I can see to care.
In other words, the secret sauce of a scalable, in terms of people, software project is the lack of any clever bs in it: such a project uses only boring predictable patterns in everything.
>We created Glamorous Toolkit to provide a concrete start for the "how not to read code" conversation. Glamorous Toolkit is a moldable development environment that makes it possible to create custom tools about software systems inexpensively.<p>That was the most blatant blogversising I have ever seen.