Hi HN!<p>I lead product for Vectara (<a href="https://vectara.com" rel="nofollow noreferrer">https://vectara.com</a>) and we recently worked with OpenSource connections to both evaluate our new home-grown embedding model (Boomerang) as well as to help users start more quantitatively evaluating these systems on their own data/with their own queries.<p>OSC maintains a fantastic open source tool, Quepid, and we worked with them to integrate Vectara (and to use it to quantitatively evaluate Boomerang). We're hoping this allows more vector/hybrid players to be more transparent about the quality of their systems and any models they use instead of everyone relying on and gaming a benchmark like BIER.<p>More details on OSC's eval can be found at <a href="https://opensourceconnections.com/blog/2023/10/11/learning-to-measure-ai-powered-search-with-vectara/" rel="nofollow noreferrer">https://opensourceconnections.com/blog/2023/10/11/learning-t...</a>