How to write a silly post about JavaScript dependencies:<p>- Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems.<p>- Talk about devDependencies and dependencies without considering how they might be entirely different, say… one kind gets bundled, the other doesn't.<p>- Always use npm, so you can highlight duplicates. Don't use yarn.<p>- Adopt a narrow definition of 'dependency hell' so you can focus on the thing that JavaScript does imperfectly (minimizing number of dependencies) and avoid talking about the things that it does well (handling conflicting deps, solving diamond deps, resolving dependency chains)<p>- Don't worry about explaining how any of the issues you're discussing have real-world implications. Have companies been failing because they run out of disk space because of node_modules?<p>- Take automated security audits extremely seriously. A backtracking bug in the argument parser for the web framework's local CLI tool? Of course it's not exploitable, when… it's a web app, and the CLI tool is something you run locally. But add it to the count as if it is!
Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems.<p>"small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code."<p>Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open Source Software", The Linux Foundation & The Laboratory for Innovation Science at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L. Hoffman. <a href="https://www.coreinfrastructure.org/programs/census-program-ii/" rel="nofollow">https://www.coreinfrastructure.org/programs/census-program-i...</a><p>(This is a contrast of citations from other works; you can see the specific citations in the report, but I'm trying to do this from a phone. In any case, the issue is the massive difference between JavaScript and everything else, and I think this quotation shows it nicely.)
I think that we are in a situation where the number of dependency doesn't have any meaning anymore for people working on web frontend projects. Recently while revamping parts of the CI/CD at $WORK, I found out that running npm install for one of our frontend project downloads around 1 million dependencies. And npm audit reports a completely ridiculous number of security issues with them.<p>It's just so absurd and nonsensical, nobody can understand what that even mean.<p>Seems to me to be a crazy huge liability, but quite a lot of people seem to be fine with that.<p>I don't know what to do from that information, but trying to create some awareness around the issue feels like talking about the number of operations per second a CPU does with someone who has zero idea what a realistic range is. It's seen as a high number, but doesn't have any meaning attached to it.<p>Edit: I don't have access to the project anymore to check more details (such as installation time), but I checked some personal notes I took when I found this out, the redacted output from npm audit is the following<p><pre><code> $ npm audit
<REDACTED> vulnerabilities found - Packages audited: 927016
Severity: <REDACTED> Low | <REDACTED> Moderate
Done in 41.62s.
</code></pre>
So slightly below the 1 million mark.<p>Edit 2: Now I'm wondering ... maybe the "packages audited" from npm audit doesn't mean "you have that number of packages, and I audited them". But if that's the case, that's a terrible UX, and I have zero idea what that number mean, which kind of support my point.
I have a project with a total of eight dependencies. I walked away for six months as it was stable. I come back to add a few features and npm tells me I have over 38 000 vulnerabilities of different severity levels. So many that 'npm audit' just freezes up.<p>So there's that too.
I cannot understand how people work with this. I work with many different technologies and try to avoid JS, but some times I have to. The past weeks I have worked on a React Native project that was written by someone else; what a horror show. I mean it wasn't the worst code by the previous guy but even in a few months a lot is simply broken and not 'best practice' anymore. Compared to most other environments I work with, iteration is somewhat faster when you finally have everything working, but the dependencies and ecosystem is horrible imho.<p>A bigger issue with all of it is that in my line of work, for the backend of firmware I cannot just depend on 3rd party libs as they will get audited, so I need to audit them before. Ofcourse most people don't have that issue.
It's not really hell though, is it? Hell is infinite pain and agony. This is more like an occasional bother.<p>Yes, we get comically large numbers when we look through node_modules. But, real "dependency hell" is when you have situations that take unbounded manual effort to resolve.<p>How often do we end up with impossible circular dependencies? Or conflicting versions clobbering each other? Or non-deterministic builds introducing random behavior?<p>That all is commonplace even today with other platforms like python, and rarely an issue with node. I'd much rather occasionally sneer at the size of node_modules than any of that actual hell.
What’s worse is that the whole installing multiple versions of the same package for different sub dependencies doesn’t make any sense when when the language supports singletons and module level variables with has no way of specifying which version to import.<p>I was shocked this behavior exists when a site broke by upgrading a sub dependency. Turned out they both installed react so they used a different “creatContext” and now there were two context instances instead of one.
My concern with npm packages isn't the number and small size of the packages itself, but that most of them are maintained and owned by individuals who do work with little review by others and who could disappear any time. If an individual goes away or takes harmful action it takes time till this is noticed and till a surviving fork emerges.<p>If more libraries were maintained by groups in a shared/collaborative way, more of those risks would go away.
Sadly the alternative isn't much better: the old school model of not having packages often leads to libraries never getting updated at all. In C++ often adding a library is such a chore once you get through all the linking issues, as you battle the 4000 conflicting compiler switches, and compiler versions, that once it works you never want to change it again.<p>I think maybe the best solution we have so far are robust "batteries included" standard libraries like python, where even if they're huge theyre at least curated and maintained by a central group.
That page itself is an example what's wrong with using JavaScript on the web: It's used in places where it's not necessary. In this case the site doesn't apply a stylesheet with JavaScript disabled.<p>JavaScript is great when it's used for enabling interactive websites, but nowadays more and more websites use it to break features, which don't need JavaScript at all.
express (common web framework) seems to have only 51 deps in lockfile<p>jekyll has 13<p>I don't disagree that lots of deps = supply chain risk<p>But there have always been variations between projects & kinds of projects re how many deps they pull in. Try following someone's ipython data science tutorial, it's requirements.txt for days<p>I think lack of a standard lib early on plus hipster functional culture made the '10s JS leaders like small monofunctional libs.<p>Focus on transpilation plus lack of tree-shaking in early versions of webpack may have also created an incentive for small things.
If you want the 3D view of Gatsby dependencies without waiting for the initial 2D view to populate, you can go here:<p><a href="https://npm.anvaka.com/#/view/3d/gatsby" rel="nofollow">https://npm.anvaka.com/#/view/3d/gatsby</a>
<i>200 megabytes is not that bad. I’ve seen it rise above 700 MB easily.</i><p>A reasonable complaint! Instead of npm, try <i>p</i>npm [0]. It uses file links to avoid duplication.<p>[0] <a href="https://pnpm.js.org/" rel="nofollow">https://pnpm.js.org/</a>
This wouldn't be much of a problem if developers weren't as reliant on frameworks as JS developers are — who for some reason love to blame others for "reinventing the wheel". What do people even use lodash for? Do you really need a whole library because you can't write a function to sort an array? This all reminds me of the npm left-pad scandal. To me the problem was not that a single package broke thousands of projects, but that thousands of developers depended on a package for such a simple feature.
I treat Npm packages like a JavaScript and PHP. There was a need and it was done. In the short run it is good enough. But in the long run it may become complicated. For example what about essencial packages that wont be maintained anymore? What about depending on millions of lines of code from unknown sources? What about single dependency that is npm? It is a risk too.
So what exactly is the problem? I see a lot of talk about dependency hell and node_modules bloat but I don't really hear much about what the actual downside is.<p>Is it performance (I haven't experienced it)?
Is it taking up disk space (I have terabytes to spare)?
Does it add complexity (I would argue it reduces complexity)?<p>What exactly is the problem?
I'm not buying the narrative of "js dependency hell what are they thinking" anymore.<p>Try installing a package for Rust. Or Go. Or ... any language, really.
Sometimes I feel like I am becoming increasingly out-of-touch with reality with how far this NPM/JS ecosystem nonsense has been taken. Is anyone still using this crap for de novo business applications that have real-world consequences? I have no problem if you want to build fantastical, sprawling node implementations for fun and potentially personal profit, but when your decisions start to involve other professionals who are simply trying to get the features implemented so they can go home...<p>I feel like we need to really think about the consequences of making these kinds of technical choices, especially in 2020 given all the horror stories circulating. Someone is eventually going to have to come in after the fact and scrape all the gore off the walls and redo your webapp in .NET/Rust/Go/etc. Why not start there and be done with it? You could save so much frustration. Is it about subjective/artistic preferences? At what point should the business owner start getting involved in these technical decisions? If the business owner found out their webapp was trapped in hell because you had a personal aesthetic attraction to one language over another much more pragmatic language, how do you think they would respond?