Ask HN: How do you manage your ML experiments?

For work, I need to run many PyTorch&#x2F;TF&#x2F;MXNet experiments in parallel on a cloud instance with multiple GPUs. Currently, I use Tensorboard (and its variants) to log results and tmux to run experiments simultaneously on multiple GPUs.<p>However, I often run into these issues:<p>1. Some experiments fail due to run-time errors and tmux allows them to fail silently<p>2. Some experiments cause a GPU to run out of memory, and I have to dig through many tmux sessions to find and re-run that experiment<p>3. If many GPUs are close to full, I have to revert to running experiments in sequence, and have to wait until experiment_i is over before running experiment_i+1<p>4. When running different experiments, I have to manually estimate how much GPU memory a specific experiment will consume before I can deploy them onto multiple GPUs<p>5. When doing a particularly tedious task (eg. hyper-parameter search), there will often be on the order of a hundred experiments; this becomes extremely difficult to manually maintain using tmux<p>Ideally, a perfect solution for this workflow would be a tool that could 1) profile memory consumption for a set of experiments, 2) automatically deploy experiments onto a cluster of GPUs, 2) re-run, queue, or re-assign experiments to other GPUs if needed, and 4) send notifications&#x2F;keep track of all experiment progress.<p>I currently know of other tools like PyTorch Lightning (which only works with PyTorch and requires a significant code restructure) and Weights &amp; Biases (only has experiment progress&#x2F;logging ability), but I have yet to find something that is lightweight and flexible enough to handle all of these requirements.<p>What&#x27;s the best way to manage experiments like this?