Over the last decade or so I’ve seen more and more people dismiss the idea of universities, and more generally, dismissing the idea of learning anything that isn’t explicitly needed for the career they’re studying for. This always felt like a huge mistake to me.<p>Universities have their problems, but getting students to see the value in subjects on the fringe, or completely outside, of their primary field of study is not one of them. These are the places new and novel solutions are born. Even if someone isn’t an expert, knowing enough to bring in an expert and facilitating the conversation can pay dividends.<p>I was once tasked with getting a new team up to speed quickly in a new site we were standing up. The manager at the time wanted to forgo training entirely to just let them figure it out, in the name of speed. I dug my heels in and still ran everyone through it. With some, it was extremely fast, and there was no way they were going to absorb it. However, I wanted them to at least hear it, so if something came up, they may not know what to do, but they will hopefully at least know enough to ask, so we can then dive deeper and show them the right way. The company had its own way of doing almost everyone, so someone doing what they thought was right based on previous experience often led to a mess.
The depth of knowledge required to work on something tends to scale with how much performance is needed, or how large the impact will be. I feel this should be obvious, but maybe the reason we're having to defend shallow technical knowledge here might be because we're getting pushed to understand things deeply regardless of the performance/impact of the actual jobs.<p>I also think it should be fairly obvious that some jobs can be accomplished by only having shallow or ad-hoc knowledge because they work on low performance/impact, but are still needed by someone, and thus require an employee.<p>What has not been obvious is why we can't differentiate roles that require deep vs. shallow knowledge officially, because there is still quite a lot of ambiguity in the actual work demands of "Software Engineer" (or "Software Developer") which makes this kind of defense in the OP necessary.
I think the title is a little, I don't know, trollish? Unnecessarily controversial?, but I think the insight is true: it's very good to have <i>some</i> knowledge about things outside of "your area," instead of being too hyperfocused.<p>However, I think that's a broader thing than just "technical" knowledge: you should know a little about what your customers do, what your manager does, what the role is of systems peripheral to yours is, etc.
I think it's critical to have both (1) actual experience working with something and (2) then reading about how it works. You gain natural intuition when you have done both.<p>When I'm listening to someone suggest an idea, you can tell<p>- if they're just working off something that they heard about (i.e. let's implement Redis, I heard it was fast),<p>- if they've done it before but don't really know how it works (i.e. MongoDB worked for me once, don't need to consider Postgres),<p>- or if they did both
I'd like to see an expert in the field of machine learning generate a cursed iceberg diagram specific just to machine learning. I suspect an LLM would do OK on generating the upper levels but very poorly on the lowest levels.<p><a href="https://news.ycombinator.com/item?id=26766722">https://news.ycombinator.com/item?id=26766722</a> (307 pts 60 comments)<p><a href="https://news.ycombinator.com/item?id=32612931">https://news.ycombinator.com/item?id=32612931</a> (341 pts 39 comments)
Giving yourselves breadcrumbs to knowledge is a fantastic life skill... but one that may (even more) fall out of favor with AI replacing the value. If I suddenly need to add a caching layer and have learned nothing about it I can ask AI to give me an overview and go ahead and recommend the best solution for my codebase and situation. Yes, you may miss some subtly and have to flail around to ask the right questions, but there is no doubt the value of "shallow technical knowledge" is less important.<p>What will remain important or grow in importance is general curiosity. Connecting completely disparate ideas or ways of thinking will lead you to new creative thoughts or solutions that AI would never produce because everyone else is working from the same standard ideas.<p>I was an English major in college and took classes in politics, philosophy, math, language, etc. based on personal interest. And I ended up as an engineer (with my trusty CS minor). TI've met several developers who have had a similar background and they tend to become the most well-rounded and business-aware ones on the team. I worry that this shift to higher cost/higher stakes/higher competition education is making this approach to learning feel untenable and my approach of 20 years ago comes across as totally irresponsible. But I would argue American education is leading to a factory approach at exactly the time when "structured thinking" is being fully replaced by AI. What is the value of crushing leetcode nowadays? Better to have a dev that has some intuition as to why people aren't clicking that new button.
This is something I am observing at work, teaching a few people and observing in some other people outside of work: It is good to understand how your stuff touches adjacent stuff, and how adjacent things touch your stuff. This eventually enables communication across layers and teams as well.<p>To pick up one one of his examples, a few people at work understand Postgres very, very well. But some of them have troubles to discuss topics with developers, because they have no knowledge how application servers interact with the Postgres (usually via some pooling, sometimes not), how different kinds of applications have different query patterns (think REST-based applications that are heavily indexed for low-quantity retrievals vs ETL based applications) and so on. I can't write a production ready app in Rails, Spring, Django right now, or a data analysis system in Flink, Spark or whatever, but I tend to have an idea what the dev needs to do so.<p>On the flipside, if you have a motivated customer or provider, I find it very valuable to spend some time to show and teach them how our systems want to be touched, one way or another. Some "idle" or "non productive" time with some senior-ish devs just sharing ideas and knowledge how our Postgres functions, some somewhat unintuitive thoughts like index selectiveness, and wants to work at a somewhat shallow level has paid off a lot at work.<p>Suddenly people ask good questions about the postgres ecosystem before starting their project and such so they don't have to spend time building a postgres extension in a worse way in their application. How silly is that.
> requires having reliable shallow intuitions about how things work<p>This is very insightful and applies to many fields, maybe specially within STEM<p>Like for example the water flow analogy of current. It’s a great analogy that works to a great degree to explain and model a lot of things about electricity, but eventually breaks down<p>For 99%+ of the people and cases, the approximate analogy is perfectly useful
I think the distinction here is between "I understand that X is a solution to Y", vs. "I understand that X is BASICALLY a mashup of A and B according to scheme C." A lot of times, people reach for X when they have a vaguely Y-ish problem, when that's actually inefficient or overkill, and/or you could easily validate if X would actually work by manually trying out A and B by hand.<p>In addition, there will be times when X cannot directly solve the version of Y you have, but there are simple ways to tweak A or B such that now you do have a solution to the problem. So you can become much, much more effective at solving Y-like problems by understand the the building blocks behind standard solutions.
The problem with shallow technical knowledge and, even worse, talking with confidence like the author does, is that it can propagate misinformation and loses a ton of nuance.<p>For example, the author talks very confidently about indexes, and makes a few conclusions, but they aren't as correct as his confidence suggests.<p>> That an index is only useful if it matches the actual query your application is making. If you index on “name plus email” and then start querying on “name plus address”, your index won’t be used and you’re back to the full table scan described in (1)<p>Not true. If you have single column indexes on both name and email, it could use the two indexes, though not as efficient as a single two-column index. If you query "name plus email" and the index is "name, email, age" then it could use the index.<p>> That indexes massively speed up reads, but must slow down database writes and updates, because each change must also be made again in the index “dictionary”<p>Must? No. The performance might be imperceptible and not material at all. If you have a ton of indexes, sure but not if you have a reasonable amount.<p>Shallow technical knowledge is fine but you should also have the humility to acknowledge that when you're dispensing said shallow knowledge. It can also lead to pretty bad engineering decisions if you think your shallow knowledge is enough.
Does the description of database indexes even count as _shallow_ knowledge? Sounds like a dictionary description with an example. Not even table stakes to warrant your inclusion in any conversation on the topic.<p>I know that some birds migrate depending on the season and they fly in certain formations for efficiency. I'd never,ever think I could have any serious conversation with a biologist or ornithologist.
With the pace of technology progress and the amount of new developments in computer science, it's quite easy today to remain on shallow knowledge, which could still be ok, accepted and further help progress itself.
At the end of the day, one of the key point of the progress is to make difficult things easier for a larger audience. This introduces abstraction layers, one on the other. Hence you may not know how the KV Cache works in Attention mechanism, but you might still be able to train a super useful AI model.<p>However I think those unique engineers with vertical and deep knowledge in a tech stack (e.g. C, Java, Maths under NN) are still very needed in the world, because they are capable of building and repairing the fundamentals of everything which gets built upon. So, if you are interested a such fundamental stack, hack it, crash it, it won't be wasted time and world will need you :)
I have absolutely no desire to become an expert in OpenVMS, but I have an account with DCL access that can run the Oracle Rdb sql utility.<p>There have been several times that my exposure has been wrapped into production applications.<p>I know what I need to know on that platform, and try to maintain little more.<p>Is this what everybody does?
I'm reminded of a Knuth quote[0]<p><pre><code> | If you find yourself doing only theory, do some practical work. It'll help with your theory.
| If you find yourself doing only practical, do some theory. It'll help with your practical work.
</code></pre>
The two complement one another. Theory informs you of what is possible while the hands on informs you on what is feasible. They are different kinds of intuitions. Theory gets you thinking in high abstractions, understanding the building blocks and how things are put together. The practice forces you to deal with everything that slips through the cracks, teaching you how to do things efficiently and effectively.<p>It's like making a building. You technically only need the construction skills to make a building. But the engineering designs make you able to build bigger and safer. The theory lets you find new novel designs and materials. You need all of this working together to build something great.<p>I've always found theory and practice to go hand in hand. Maybe because I came to CS from physics, but they have hard divisions there. I focused on experimental physics but I was known for strong math skills which allowed me to bridge these two sides. But the major reason I did it is because I couldn't see how they were different. So when I later heard Knuth, it really resonated with me.<p>But truthfully, I've struggled sometimes working with others, getting pulled in different directions. I left physics to become an engineer and all they saw me as was a theorist despite most of my work for them being practical (running simulations and then building physical devices and the necessary test infrastructure).<p>Now I'm about to defend my PhD in CS (ML) and the biggest struggle I've had is that it feels like any time I spend on theory is seen as wasted. I do so much less than before, but it's been the most helpful tool to my work. So I'm hoping others can help me understand this culture. We definitely see theory differently but I'm not sure why or how. Everything seems hyper focused on making products. To ship as fast as possible. No time to think about what we're making. For me, that slows me down! Just because lines of code aren't appearing on screen doesn't mean I'm not hard at work. A few hours or even a day of thought has saved me weeks worth of work. I'm not the quickest to first result[1] but the result is I can do more with less and quickly adapt to all the changes that happen as the project evolves. New experiments, measurements, features and such get integrated quickly because I expect this to happen. But still, considered slow, and I can't find out why.<p>So I'm wondering how others have dealt with this. I know I'm not alone here. I'm probably not going to change much because the end result works. But why is CS so focused on short term and how do I highlight longer term work when we're just measuring weekly updates. The strategy means more weeks with "not much to show" but other weeks that look like extreme productivity. But it's all the same. Because the truth is, the earlier work is just difficult or impossible to measure. I'm pretty sure this is why CS is the way it is, but I've got to say, I've got a lot of experience with measures and analysis and the fact is not everything is measurable. We only can use proxies and those easily become unaligned.<p>[0] I didn't copy paste it so it might be a little off but the message is right<p>[1] after proof of concept. Early on I try to be quick to fail. I'm talking about once that's going and we're doing more serious building.
> Database [anything]<p>Honestly, the author couldn't have chosen a worse example if they tried. RDBMS are so absurdly complex and have so many edge cases for everything, that they are my canonical answer to why it's a bad idea to have dev teams managing them, or designing schema. Disclaimer: I'm a DBRE.<p>Look at MySQL's ORDER BY [0], GROUP BY [1], and index merge [2] optimizations. Postgres' ORDER BY optimization [3] doesn't have nearly as many gotchas, but it more than makes up for it by having many more types of indices [4], as well as many types of operator classes within each of those. While you're at it, you should probably have a handle on collations [6], AKA "why did this ORDER BY do something odd?"<p>> You could learn about the data structures that represent the index<p>You mean a B+ tree (for standard types)? I would hope you covered that in DS&A already.<p>> about all the tricks that go into updating them efficiently<p>> But it will very rarely inform the database migration or query you’re writing.<p>I disagree. For example, if you didn't know that updating <i>any</i> indexed column in a Postgres table means the <i>entire</i> row is re-written (hope you haven't done anything silly, like storing medium-sized JSON blobs that aren't quite big enough to trigger TOAST), you would likely be much more cavalier with your indexing decisions.<p>I'm so very tired of the "I don't need to know how the things I use work" attitude. Do you need to be able to write your own database to effectively use one? No, of course not. Should you have read the official docs? Yes. You'll learn far more than from skimming random blogs and YouTube videos.<p>[0]: <a href="https://dev.mysql.com/doc/refman/8.0/en/order-by-optimization.html" rel="nofollow">https://dev.mysql.com/doc/refman/8.0/en/order-by-optimizatio...</a><p>[1]: <a href="https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html" rel="nofollow">https://dev.mysql.com/doc/refman/8.0/en/group-by-optimizatio...</a><p>[2]: <a href="https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimization.html" rel="nofollow">https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimiza...</a><p>[3]: <a href="https://www.postgresql.org/docs/current/indexes-ordering.html" rel="nofollow">https://www.postgresql.org/docs/current/indexes-ordering.htm...</a><p>[4]: <a href="https://www.postgresql.org/docs/current/indexes-types.html" rel="nofollow">https://www.postgresql.org/docs/current/indexes-types.html</a><p>[5]: <a href="https://www.postgresql.org/docs/current/indexes-opclass.html" rel="nofollow">https://www.postgresql.org/docs/current/indexes-opclass.html</a><p>[6]: <a href="https://www.postgresql.org/docs/current/collation.html" rel="nofollow">https://www.postgresql.org/docs/current/collation.html</a>
AI makes it completely defensible these days, as long as you have some "dev intuition".<p>Example: Someone just handed me a USB key with some old DOS software on it called SAGE made by a company which disappeared in 2011 (which apparently still powers a number of law offices... shocker), and they encountered a date handling issue (any case entered for 2026 borks some views and reports) and I'm facing either a decompilation or some other heavy analysis (none of which I'm familiar with) since the source code is obviously unobtainable (score another for open source? Seriously, when companies sink, they should be required to release their source code...). I'm not doing this for free, of course (I did tell them that failure was a distinct possibility), but I'm going to attack it with AI and rely on dev intuition for the rest and see what happens... (and use some kind of test harness, of course... to prevent regressions and prove the fixes)