Interesting. There's been decades of research on Derivative-Free Optimization (DFO) and stochastic/evolutionary algorithms (most of which are derivative-free). They're used in practical applications, but have been hard to reliably perf benchmark because solution paths are so dependent on initial guess and random chance.<p>This one focuses on maximizing sample efficiency. That's an interesting (and important) metric to benchmark, especially for functions that are computationally expensive to evaluate, like full-on simulations. Sounds like the algorithm would need to be able to efficiently come up with an accurate surrogate model for the expensive function -- which is hard to do in the general case, but if something is known about the underlying function, some specialization is possible.
NeurIPS, 2020: "We need more sample-efficient algorithms for finding better hyperparameters that specify how to train computationaly expensive deep learning models."<p>Rich Sutton, 2019: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." (<a href="https://news.ycombinator.com/item?id=23781400" rel="nofollow">https://news.ycombinator.com/item?id=23781400</a>)<p>I wonder if in the end simply throwing more and more computation at the problem of finding good hyperparameters will end up working better as computation continues to get cheaper and cheaper.
if anyone wants to do this. i have a threadripper build with 2 2080ti. would be cool to do a group project.
write pm if you want. i am located in amsterdam, europe