JSON Mode has been one of the biggest enablers for working with Large Language Models! JSON mode is even expanding into Multimodal Foundation models! But how exactly is JSON mode achieved?<p>There are generally 3 paths to JSON mode:<p>1. Constrained Generation (such as Outlines)<p>2. Begging the model for a JSON response in the prompt<p>3. A two stage process of generate-then-format (or generate-then-retry)<p>Although most of the field has converged on the first method, Let Me Speak Freely? is a new paper challenging the potential tradeoffs in achieving JSON mode with constrained generation.<p>I am BEYOND EXCITED to publish the 108th Weaviate Podcast with Zhi Rui Tam, the lead author of Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models!<p>As the title of the paper suggests, although constrained generation is awesome because of its reliability, we may be sacrificing the performance of the LLM by producing our JSON with this method.<p>The podcast dives into how these experiments identify this and all sorts of details about the potential and implementation details of Structured Outputs. I particularly love the conversation topic of incredibly Complex Structured Outputs, such as generating 10 values in a single inference or say HTML templates.<p>https://www.youtube.com/watch?v=UsVIX9NJ_a4