Be <i>very</i> careful with the live-restore feature of docker. Running 'docker volume prune' just removed <i>all</i> my named volumes, which were used by running containers.<p>See <a href="https://github.com/moby/moby/issues/38883" rel="nofollow">https://github.com/moby/moby/issues/38883</a>
Automation of any sort will sometimes accidentally your data, whether due to periodic hiccups, system instabilities and bugs, operator misunderstandings or errors, or random cosmic ray strikes.<p>The exact reason it blows up isn't even necessarily all that important, other than in its effect on what you should be doing to reduce the probability of downtime. Well-engineered systems are routinely developed from less than completely reliable parts. Stuff fails, we design for it.<p>It's certainly not reason not to <i>use</i> it, if it's resulting in a net positive gain in your ability to get things done and maintain control and transparency over your deployed systems.<p>But it's certainly a good reason (among a long list of good reasons) to make sure you have a good backup routine in place, including regular testing of both their integrity and your ability to restore a working prod system from them quickly.
This definitely sounds like a bug.<p>docker volume prune says:<p>"Remove all unused local volumes. Unused local volumes are those which are not referenced by any containers"<p>If it removed a local volume that was being used by a container that is kinda bad.
1. Why are you running docker volume prune in production?<p>2. Why are you running docker on ad-hoc machines you need to prune?<p>3. Why do you even need root access on production machines to fiddle around with docker commands?<p>While this is obviously a bad bug (and there are many with Docker), it seems more of an operational procedures failure than anything else. You could be saying:<p>“Beware of rm -rf /, it just deleted 20gb of production data”<p>Ok. Sure. But why are you tools and procedures putting yourself in a position to make that mistake?
I really-really hope you are not relying on Docker only when protecting 19G of data. Docker volume operations are the equivalent of playing with sudo rm -rf, shit's going to happen once in a while.
In the Moby issue you mention that you are using live restore (<a href="https://docs.docker.com/config/containers/live-restore/" rel="nofollow">https://docs.docker.com/config/containers/live-restore/</a>) which is most likely where the problem is. Docker daemon restarts, existing containers are kept alive, but the restarted Docker daemon doesn’t know about those existing containers yet and thus thinks their volumes are unused.
This makes it sound like it's quite common to use docker containers operating in a heavily stateful fashion. Is that indeed common nowadays? (Though, the state in this case is only counted on to persist in the named volumes.)
My browser ate 16GB of ram while I've been reading this. The system crashed but the tabs were here there after a reboot. I'm not even mad anymore.
Computers are wonderful. They can do the same work that would require a thousand people to accomplish in the same amount of time.<p>Flip side . . .<p>Computers are terrible. They can screw things up so bad it would require a thousand people to accomplish in the same amount of time.
`docker volume prune` is specifically there to remove volumes, so backing up before using it seems to be mandatory, just in case. But yeah, if this is a bug, it's a nasty one.