The solution is to do more with less. Devs are taking advantage of a wealth of abstraction and being buried under it. But they actually don't need it.<p>You do not need some 3rd party library to write a command line interface for your python code. You do not need three different projects just to maintain your list of installed packages or virtual environment. And you don't need a package to wrap around loading environment variables.<p>Just learn how to do these things in a simpler way, without the extra deps. Will there be some more boilerplate code? Yes. But it will be <i>your</i> code, and it will be so simple that you will learn to write it very well. In addition: make your code loosely coupled and composeable, and overload existing code with additional functions when feasible, rather than starting a new project from scratch. Press your company to let you contribute back to OSS, as this lets somebody else do more of the maintenance, and has the side effect of reducing the total number of software projects. (It immediately saves you money and time. Kind of stupid that more of us don't do this already.)<p>Simplifying is hard. But when you get good at it, you find you're much more buoyant.
If you are really struggling, I'd suggest trying out the latest .NET before abandoning all hope.<p>I was recently able to build a 3d software rasterizer and streaming web service using .NET6 and I did not have to consume a single 3rd party dependency.<p>Even things like SIMD-optimized projection matrix calculations can be found in the box (System.Numerics).<p>There is obviously the Microsoft aspect to this equation, but I feel like you should at least try the porridge before throwing it away. The empire has a hell of a Death Star these days.
It seems there is one of those posts on HN every 6 weeks.<p>And yet there is no solutions besides yolo-live-with-it.<p>I thought it sucks too, I'm in the same boat, but I considered the opposite for a while: what if we didn't have all the code reuse and boatloads of libraries at our fingertips?<p>That would be a TON of code added to our applications and overall - shock full of bugs and lack of optimizations. These libraries, when popular are battle tested, cover much more grounds than we do today (but might need tomorrow) and have been looked at if slow.<p>This ecosystem relies mostly on everyone acting like adults; Libraries to respect semver and not making large, breaking API changes on minor versions; and an ecosystem that is "safe", where bad actors can't pretend to be someone else.<p>Ideally there would be an open "group" of people running penetration tests and security overview of most packages where the collective could donate against their work. Large donations would allow specifying which packages you'd want them to review, and fixes would flow money to the maintainers...<p>But I'm dreaming. We're all building on a gigantic pile of hacks written for someone's portfolio to get their next job, which won't be in the same language and the thing will become abandoned.
Read also this excellent article by Russ Cox (of Go fame) on the issue:<p><a href="https://research.swtch.com/deps" rel="nofollow">https://research.swtch.com/deps</a><p>This is at heart an economics problem: how to provide a public good (software security) in the presence of free riders. The way society handles this kind of risk is through insurance, but there are a number of things that need to happen to enable a sustainable economic basis to fund the necessary dependency-vetting to happen.<p>Edit: I wrote up my thoughts on the way forward here:<p><a href="https://blog.majid.info/supply-chain-vetting/" rel="nofollow">https://blog.majid.info/supply-chain-vetting/</a>
It's basically a batteries not included problem.<p>Say you create a new React app. The fact that you do is an indicator that the core tech (web tech in this case) does not satisfy your needs out of the box. So you grab a framework from "user land", one of many.<p>React though is not self-contained. To even get it to build anything, it depends on various other projects that each have deep dependencies. Webpack, linters, a choice of CSS framework, etc.<p>At this point, you app itself has zero functional dependencies. You haven't even produced a hello world, all of this stuff is needed just to get it to do anything at all.<p>As you'll then start to build actual functionality, you'll notice that React doesn't have facilities or an opinion on the most basic of stuff. Loading data, managing forms, standardized UI, the essentials are not there. So you go for robust/popular libraries for each of those gaps, which in turn have their own dependency trees.<p>This way even the simplest of apps require thousands of dependencies. It's not due to developers too lazy to write "leftpad". It's a tech stack issue where nothing is standardized or included out of the box.
We're not drowning. The boat was not made to be waterproof and yet we think it is.<p>Any software you build, like everything else needs maintenance. It's a fallacy to think that just because math is correct (boolean algebra) that your code will tick along fine indefinitely.
> You’ve probably seen run-of-the-mill web applications with hundreds of direct dependencies<p>This is one of the reasons I like working on legacy web apps. Other than maybe jquery, they typically don’t have any dependencies. It is so easy to follow the code logic because it is all right there and nothing is hidden 10 layers deep in dependency hell.
Who’s drowning? I feel better than ever. This is a great problem compared to the old way of stuff being out of date and it being a giant pain to use libraries and stuff.<p>Also (knock wood) I haven’t experienced any dependency related outages and have been able to run Java, ruby, Python, and javascript stuff for many years without problems. When I need reproducibility I pin to specific versions. And I vet what packages I use and don’t just randomly grab whatever is in the stackoverflow copy and paste.
I lightheartedly object to C# being mentioned in the same breath as the rest. I've been writing Windows Forms applications for 21 years. Most C# applications have fewer than 10 dependencies, and they tend not to have their own transitive dependencies. It's bliss. We just really don't have the same level of problems in this ecosystem.
Isn't this the consequence of a large, free, open, diverse system? I remember the constant complaints about their being "too many javascript" frameworks a few years ago, and how their are always new ones.<p>I'm less concerned with the moral or practical considerations here, but can this really be a bad thing? Or, perhaps more to the point, isn't it a sign of a healthy ecosystem?<p>It seems as the world gets more interconnected this problem of "too much"; it's an interesting one to solve... And I'm not saying we shouldn't, but I guess I'm curious about the philosophical question of complaining about complexity in a decentralized public ecosystem. Is there a solution to the ecosystem as a whole? Or will sort of curated subgroups of the ecosystem be birthed that attempt to create a walled garden. It doesn't seem possible to create a shared walled garden without stifling innovation, becoming more closed, all the things that I think most open source communities don't want to do.
In the "old" days ("old" being .. what, ten years ago?) packages were released as libraries, which bundled large amounts of functionality into a single atomic version. You could depend on OpenSSL for all your crypto code, GLib for argument parsing or atomics or UUIDs, libiconv for character encoding. There was some expectation (usually met) that the developers of these libraries would have some sort of basic testing strategy, and that they would try to maintain API compatibility over long periods of time.<p>A large program might have less than a dozen external dependencies, and they released every couples months at most, so keeping up to date was easy. And most of the time you didn't even need to update unless someone discovered a security vulnerability, which was not <i>that</i> common despite all the libraries being written in C or C++.<p>The approach of NPM (etc) is fundamentally different in that it's considered normal for a project to have hundreds or thousands of dependencies. To manage the update cadence, the users of these package repositories try to invent new schemes for version numbers -- or write quirky gif-laden blog posts about how they burn the monthly budget of a small country on testing in CI.<p>But the core problem is that they've locked themselves into a stupid and unsustainable paradigm. You can't do the sort of software development that involves <i>writing code</i> if you've decided that your job is to haphazardly glue together other people's code, especially when your're willing to depend on packages written by anything and anyone.<p>--<p>I honestly think a lot of the problem of over-dependency comes from the new package managers that make it too easy to depend on stuff recursively. If NPM required you to type out `npm add-project-dependency left-pad/1.0.0 --checksum sha256:a1b2c3...` then that would have been a strong forcing function against a huge number of tiny packages, because at some point <i>your fingers get tired</i>.<p>For example, in my personal projects I use Bazel as a build system, which makes me a bit more aware of dependency hell because each dependency needs to be registered.<p>There are projects like QEMU that are straightforward to build in Bazel because their dependency tree is bounded, but some of the stuff coming out of JavaScript land is just unavailable to me because I don't have the patience to chase down hundreds of packages to run a "hello world".<p>I've found I avoid certain parts of the Rust ecosystem because they're JS-ish, but there's other parts that have a more C/C++-ish style and those packages work fine for me. Similarly for Python. Go has largely avoided the issue, though I don't have an intuition as to why (different culture?).
The thing I hate the most is when some JavaScript library uses some obscure build system (or any build system really) but on GitHub they don't publish the built version. What could have been simple download now turns into several hours exercise in trying to understand and make work yet another build system, downloading intermediate bullshit from random websites. Just publish the damn library, thanks.
For simple, once used projects, I don’t concern much with these aspects. But for a big projects, choosing your framework and dependency is always a big concern and should do it carefully. Investigate all your dependency is a must, not just for the security aspect.
And when you do it, you will convince yourself less is more and your team will re-create many things just to reduce the dependency.
I'm going through this right now. At dayjob we have a lot of internal dependencies and libraries that are created sufficiently far from me that I treat them as third party and when things go wrong (which they do a lot) it's a nightmare to debug and work through. Easily half a day gone.<p>So for sideproject I'm going mostly bespoke and using very little 3rd party libraries. The result of that is there's sooooooo much boiler plate to write and it's very boring and it's hard to be motivated and productive to get through it. What's keeping me going is that once the boilerplate is done - it's mostly the foundational level of data queries + endpoints - then I shouldn't really have to touch that stuff again.<p>My conclusion is that there's no great answer.<p>Using lots of libraries is like riding a horse. It'll start moving right away but you don't have full control and you gotta find the right way to get it to do what you want, and may have to fight it if it really doesn't want to. Vs building most things yourself is like building a car. You don't go anywhere for a long time as you're building it, but once you've built it you can move very quickly and if anything goes wrong you can easily pop the hood and fix/change things up.<p>I think for long lived projects it's better to build the car, but also be disciplined in writing very good code + documentation. The initial build out sucks hard but that pain will get amortized over a very long future and ultimately be worth it.<p>And for absolutely required dependencies go with paid services. Specifically paid services where the company's primary focus _is_ providing that service. They are financially and existentially motivated to give good service, and are usually good about keeping backwards compatibility while usually staying on top of security/modernization upgrades without requiring work on your own end. A managed database is a good example of this kind of paid dependency.
Another downside of the deluge of dependencies that I don't see discussed often enough is bloat. If you import something that imports something else that imports something else ad nauseam, you're importing a <i>lot</i> of stuff you're not actually using and won't ever actually need. Tree shaking was supposed to help with that, but it barely moves the scale.<p>The author says that "Reusing external code is a huge advantage, and when everyone does it, it’s also a competitive necessity", but when I've taken it upon myself to "reinvent" something that there's already a moderately decent solution for, I've found that it really wasn't that much extra effort and the end result was <i>insanely</i> more flexible in the ways that I needed it to be than the supposedly off-the-shelf alternative.
Every library is code I have to maintain without even knowing what's inside.<p>I only use libraries for something I absolutely can't write myself, like PGP.<p>I also avoid languages and technologies which introduce breaking changes.<p>e.g. Python will never be a top-level language in my project, only an optional add-on for scripting, with failure checks.
How about building on solid ground for once? Datalisp.is the thing I keep posting here and it keeps getting overlooked but a solid foundation means longer lasting structures, just look at the 3-4-5 triangle; it still has a right angle.
This path was paved when NPM and Node.js decided that lots of small dependencies are a good thing. All of this damage and JavaScript's reputation as a maintenance nightmare is a result of that terrible, terrible decision.
I went through dependency hell today.<p>I wanted to start a new react project and someone who I was going to collaborate with said they wanted try storybook. Ok, follow the tutorial. It immediately spits out all these warnings<p>warning " > @testing-library/user-event@13.5.0" has unmet peer dependency "@testing-library/dom@>=7.21.4".
warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-syntax-flow@^7.14.5".
warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-transform-react-jsx@^7.14.9".
warning "@storybook/builder-webpack5 > fork-ts-checker-webpack-plugin@6.5.2" has unmet peer dependency "typescript@>= 2.7".
warning "@storybook/addon-essentials > @storybook/addon-docs > @babel/plugin-transform-react-jsx@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0".
warning "react-scripts > eslint-config-react-app > @typescript-eslint/eslint-plugin > tsutils@3.21.0" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
warning "@storybook/addon-actions > react-inspector@5.1.1" has incorrect peer dependency "react@^16.8.4 || ^17.0.0".
warning " > @storybook/addon-essentials@6.5.7" has unmet peer dependency "@babel/core@^7.9.6".
warning "@storybook/addon-essentials > @storybook/addon-docs > @mdx-js/react@1.6.22" has incorrect peer dependency "react@^16.13.1 || ^17.0.0".
warning "@storybook/addon-interactions > @devtools-ds/object-inspector > @devtools-ds/themes > @design-systems/utils@2.12.0" has unmet peer dependency "@types/react@<i>".
warning " > @storybook/preset-create-react-app@4.1.2" has unmet peer dependency "@babel/core@</i>".
warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin@1.0.2-canary.6.9d540b91e815f8fc2f8829189deb00553559ff63.0" has unmet peer dependency "typescript@>= 3.x".
warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin > react-docgen-typescript@2.2.2" has unmet peer dependency "typescript@>= 4.3.x".
warning " > @storybook/react@6.5.7" has unmet peer dependency "require-from-string@^2.0.2".
warning "@storybook/react > @babel/preset-flow@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0".
warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1".
warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react-dom@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1".<p>So that failed IMO. There should be zero warnings! A pile of warnings has no meaning because no will notice when it goes from 18 to 19 warnings.<p>I ended up aborting that and trying other things. It's been 10hrs now and I still have nothing working. Every path has lead to some kind of out dated article and/or dependency issue.
Funnily enough this becomes much less of a problem once you use a real type system instead of having to write manual tests. Both because whether your code compiles becomes a pretty good (not perfect, just as your test suite is not perfect) indication of whether this dependency upgrade is breaking for you or not, and because those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place.