lots of attempts at the llm eval game:<p>- <a href="https://github.com/BerriAI/bettertest">https://github.com/BerriAI/bettertest</a> <a href="https://twitter.com/ishaan_jaff/status/1665105582804832258" rel="nofollow noreferrer">https://twitter.com/ishaan_jaff/status/1665105582804832258</a><p>- <a href="https://github.com/AgentOps-AI/agentops">https://github.com/AgentOps-AI/agentops</a><p>- <a href="https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm-features-with-confidence">https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm...</a><p>- <a href="https://news.ycombinator.com/item?id=36958175">https://news.ycombinator.com/item?id=36958175</a><p>- <a href="https://github.com/promptfoo/promptfoo">https://github.com/promptfoo/promptfoo</a><p>can you articulate what insight on the market you have that will make you stand out over time?
Thanks for the enormous amount of interest and questions in this post. I wanted to make a follow-up comment to answer all the questions above about differentiation, where we see issues in the current solution space and clarify the problem we are trying to tackle.<p>Starting with the problem - the problem we are trying to tackle is to make iteration of LLMs and Agent applications as easy as possible for data teams - not for people to quickly edit and compare prompts (although that is 1 way of doing it). There are a number of solutions out there to help you test prompts etc. but many fail to fit within the data team's workflow (which consists of tooling like Pytest and CLI-first approaches) and do not make it easy to iterate and launch on things like agents.<p>From conducting 30+ interviews with ML Engineers and data scientists building in this space - all of them want to first build a LangChain agent/RAG pipeline and then build their own internal version of it (due to the fact that LangChain is quick to set up but lacks tooling). A lot of them are encountering issues with developing the right evaluation infrastructure (which DeepEval aims to solve through synthetic data creation and easy-to-use testing tools).<p>Our product roadmap is to not only build the initial unit testing for LLM but also to make it easy for other developers and other MLEs to quickly iterate off this. This means long-term, our plan is to ensure that our users and customers are able to build the best agents/LLM solutions possible.<p>And (to be frank) a lot of the existing solutions aren't the best looking, have limited visualisations (or are just dead)
Microsoft Guidance integration?<p><a href="https://github.com/guidance-ai/guidance">https://github.com/guidance-ai/guidance</a>