Hey HN, I wanted to share a new project we've been working on for the last couple of months called ART (<a href="https://github.com/OpenPipe/ART">https://github.com/OpenPipe/ART</a>).<p>ART is a new open-source framework for training agents using reinforcement learning (RL). RL allows you to train an agent to perform better at any task whose outcome can be measured and quantified.<p>There are many excellent projects focused on training LLMs with RL, such as GRPOTrainer (<a href="https://huggingface.co/docs/trl/main/en/grpo_trainer" rel="nofollow">https://huggingface.co/docs/trl/main/en/grpo_trainer</a>) and verl (<a href="https://github.com/volcengine/verl">https://github.com/volcengine/verl</a>). We've used these frameworks extensively for customer-facing projects at OpenPipe, but grew frustrated with some key limitations:<p>- Multi-turn workflows, where the agent calls a tool, gets a response, and calls another, are not well supported. This makes them a non-starter for any task that requires an agent to perform a sequence of actions.<p>- Other frameworks typically have low GPU efficiency. They may require multiple H100 GPUs just to train a small 7B parameter model, and aren't able to keep the GPUs busy consistently during both the "rollout" and "training" phases of the training loop.<p>- Existing frameworks are typically not a convenient shape for integrating with existing agentic codebases. Existing trainers expect you to call raw text completion endpoints, and don't automatically provide industry-standard chat completion APIs.<p>ART is designed to address these limitations and make it easy to train high-quality agents. We've also shared many details and practical lessons learned is in this post, which walks through a demo of training an email research agent that outperforms o3 (<a href="https://openpipe.ai/blog/art-e-mail-agent">https://openpipe.ai/blog/art-e-mail-agent</a>). You can also find out more about ART's architecture in our announcement post (<a href="https://openpipe.ai/blog/art-trainer-a-new-rl-trainer-for-agents">https://openpipe.ai/blog/art-trainer-a-new-rl-trainer-for-ag...</a>).<p>Happy to answer any questions you have!
Contributor here, we developed the Agent Reinforcement Trainer (ART) library to make it easy to train LLMs for anything.<p>No callbacks or straitjacket flows. Instead we serve an OpenAI API-compatible endpoint that you can use as a drop-in replacement for any proprietary APIs you may be hitting.<p>After collecting responses from the inference API, you can tune the model with your own custom rewards and repeat the process as long as you like, until performance converges. We believe this level of flexibility will make it easier for you to train state-of-the-art models for your own use cases, much like Kyle's new email agent[1].<p>Also happy to answer any questions you have about the framework.<p>[1] <a href="https://openpipe.ai/blog/art-e-mail-agent">https://openpipe.ai/blog/art-e-mail-agent</a>
Figured now was a good time to post this since we recently got surprisingly good results on training an email research agent. Link is above, but will put it here as well since I think it's a good example of RL's promise: <a href="https://openpipe.ai/blog/art-e-mail-agent">https://openpipe.ai/blog/art-e-mail-agent</a>
Thanks for sharing this! A couple of questions come to mind:<p>- How does training with RL differ from fine tuning?<p>- When would it make sense to fine tune instead of using RL?