Hello HN!<p>We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: <a href="https://replicate.ai/" rel="nofollow">https://replicate.ai/</a><p>Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose.<p>We built <a href="https://www.arxiv-vanity.com/" rel="nofollow">https://www.arxiv-vanity.com/</a> together for fun, which led to us teaming up to build more tools for ML.<p>We spent a year talking to lots of people in the ML community and building all sorts of prototypes, but we kept on coming back to a foundational problem: not many people in machine learning use version control.<p>This causes all sorts of problems: people are manually keeping track of things in spreadsheets, model weights are scattered on S3, and results can’t be reproduced.<p>So why isn’t everyone using Git? Git doesn’t work well with machine learning. It can’t store trained machine learning models, it can’t handle key/value metadata, and it’s not designed to record information automatically from a training script. There are some solutions for these things, but they feel like band-aids.<p>We came to the conclusion that we need a native version control system for ML. It’s sufficiently different to normal software that we can’t just put band-aids on Git.<p>We believe the tool should be small, easy to use, and extensible. We found people struggling to migrate to “AI Platforms”. A tool should do one thing well and combine with other tools to produce the system you need.<p>Finally, we also believe it should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community.<p>Replicate is a first cut at something we think is useful: It is a Python library that uploads your files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get back to any point in time using the command-line interface, analyze your results inside a notebook using the Python API, and load your models in production systems.<p>We’d love to hear your feedback, and hear your stories about how you’ve done this before.<p>Also – building a version control system is rather complex, and to make this a reality we need your help. Join us in Discord if you want to be involved in the early design and help build it: <a href="https://discord.gg/QmzJApGjyE" rel="nofollow">https://discord.gg/QmzJApGjyE</a>
Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects:<p>* Kubeflow: <a href="https://github.com/kubeflow/kubeflow" rel="nofollow">https://github.com/kubeflow/kubeflow</a><p>* MLFlow: <a href="https://github.com/mlflow/mlflow" rel="nofollow">https://github.com/mlflow/mlflow</a><p>* Pachyderm: <a href="https://github.com/pachyderm/pachyderm" rel="nofollow">https://github.com/pachyderm/pachyderm</a><p>* DVC: <a href="https://github.com/iterative/dvc" rel="nofollow">https://github.com/iterative/dvc</a><p>* Polyaxon: <a href="https://github.com/polyaxon/polyaxon" rel="nofollow">https://github.com/polyaxon/polyaxon</a><p>* Sacred: <a href="https://github.com/IDSIA/sacred" rel="nofollow">https://github.com/IDSIA/sacred</a><p>* pytorch-lightning + grid:
<a href="https://github.com/PyTorchLightning/pytorch-lightning" rel="nofollow">https://github.com/PyTorchLightning/pytorch-lightning</a><p>* DeterminedAI: <a href="https://github.com/determined-ai/determined" rel="nofollow">https://github.com/determined-ai/determined</a><p>* Metaflow: <a href="https://github.com/Netflix/metaflow" rel="nofollow">https://github.com/Netflix/metaflow</a><p>* Aim: <a href="https://github.com/aimhubio/aim" rel="nofollow">https://github.com/aimhubio/aim</a><p>* And so many more...<p>In addition to this list, several other hosted platform offer experiments tracking and model management. How do you compare to all of these tools, and why do you think users should move from one of them to use replicate, thank you.
Congratulations with the launch.<p>We've used <a href="https://github.com/iterative/dvc" rel="nofollow">https://github.com/iterative/dvc</a> for a long time and quite happy. What's the main difference between replicate.ai and dvc?
Congrats on the launch! Very exciting tool. I'm one of the creators of <a href="https://DAGsHub.com" rel="nofollow">https://DAGsHub.com</a> which lets you host and collaborate on data science projects – think data science pull requests and data merging. We're integrated with DVC for data versioning and experiments, but what you're building is definitely super interesting.
I was looking for something similar today. I just adopted it. :)<p>Thank you for your amazing work!<p>Do you have the intention to integrate it with PT Lightning or as a PT Lightning Logger?<p>It would be nice to have it maintained there.<p>It’s used all over huggingface-Transformers examples.
Congrats on the launch.
Have you looked at <a href="https://comet.ml" rel="nofollow">https://comet.ml</a>
If so, how do you compare to them?
Congrats on the launch. To me this feels like DVC but with a slightly more convenient python API, and without the pipeline which might be really great. How are you organizing experiments though? Without Git it seems too easy to have experiments become unusable. And filtering according to accuracy is not really a solution..
This looks similar in philosophy and approach to <a href="https://guild.ai" rel="nofollow">https://guild.ai</a> although guild ai can parse stdout to extract metrics to track, and is integrated with tensorboard.
How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers.<p>However what I love about them is the amazing UI that allows me to compare experiments.
Congrats on the launch.<p>I have built an open source tool (called hyperML) for similar problem sometime back.
I think the problem is not just storing in version control but being able to quickly retrieve those in live /test systems or containers.<p>Mounting and loading datasets and models is painful. It kind of what makes local training a better option.<p>If only the weights were version controlled by libraries (tf, pytorch or scikit) this whole problem will be much easier to solve.
Great to see this! I've been looking at alternatives for experiment tracking, but most of what I've come across have looked way too heavy (and expensive) for a small team.<p>Is R support planned? Or might this just work already with reticulate?
Congrats on the launch! What's the business concept behind the tool? Can't find anything on the homepage, or in the post. I didn't know YC funded opensource tools like this, it's kind of refreshing.
nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).
What is the best way to do dataset versioning when using Replicate? I get that Replicate saves the dataset version but not the data itself. Is DVC a good fit?<p>Second question - can you tag or attach a comment to a group of experiments?