Structured Outputs with Ollama

If anyone needs a more powerful constrain outputs, llama.cpp support gbnf:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;blob&#x2F;master&#x2F;grammars&#x2F;README.md">https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;blob&#x2F;master&#x2F;grammars&#x2F;...</a>

This is wonderful news.<p>I was actually scratching my head on how to structure a regular prompt to produce csv data without extra nonsense like &quot;Here is your data&quot; and &quot;Please note blah blah&quot; at the beginning and end, so this is much welcome as I can define exactly what I want returned then just push structured output to csv.

Yay! It works. I used gemma2:2b and gave it below text<p><pre><code> You have spent 190 at Fresh Mart. Current balance: 5098 </code></pre> and it gave below output<p><pre><code> {\n\&quot;amount\&quot;: 190,\n\&quot;balance\&quot;: 5098 ,\&quot;category\&quot;: \&quot;Shopping\&quot;,\n\&quot;place\&quot;:\&quot;Fresh Mart\&quot;\n}</code></pre>

No way. This is amazing and one of the things I actually wanted. I love ollama be because it makes using an LLM feel like using any other UNIX program. It makes LLMs feel like they belong on UNIX.<p>Question though. Has anyone had luck running it on AMD GPUs? I&#x27;ve heard it&#x27;s harder but I really want to support the competition when I get cards next year.

Has anyone seen how these constraints affect the quality of the output out of the LLM?<p>In some instances, I&#x27;d rather parse Markdown or plain text if it means the quality of the output is higher.

So I can use this with any supported models? The reason I&#x27;m asking is because I can only run 1b-3b models reliably on my hardware.

PRs on this have been open for something like a year! I&#x27;m a bit sad about how quiet the maintainers have been on this.

I&#x27;m still running oobabooga because of its exlv2 support which does much more efficient inference on dual 3090s

What&#x27;s the value-add compared to `outlines`?<p><a href="https:&#x2F;&#x2F;www.souzatharsis.com&#x2F;tamingLLMs&#x2F;notebooks&#x2F;structured_output.html#outlines" rel="nofollow">https:&#x2F;&#x2F;www.souzatharsis.com&#x2F;tamingLLMs&#x2F;notebooks&#x2F;structured...</a>

Is there a best approach for providing structured input to LLMs? Example: feed in 100 sentences and get each one classified in different ways. It&#x27;s easy to get structured data out, but my approach of prefixing line numbers seems clumsy.

That&#x27;s very useful. To see why, try to get an LLM _reliably_ generate JSON output without this. Sometimes it will, but sometimes it&#x27;ll just YOLO and produce something you didn&#x27;t ask for, that can&#x27;t be parsed.

I must say it is nice to see the curl example first. As much as I like Pydantic, I still prefer to hand-code the schemas, since it makes it easier to move my prototypes to Go (or something else).

Could someone explain how this is implemented? I saw on Meta&#x27;s Llama page that the model has intrinsic support for structured output. My 30k ft mental model of LLM is as a text completer, so it&#x27;s not clear to me how this is accomplished.<p>Are llama.cpp and ollama leveraging llama&#x27;s intrinsic structured output capability, or is this something else bolted ex-post on the output? (And if the former, how is the capability guaranteed across other models?)

Wow neat! The first step to format ambivalence! Curious to see how well does this perform on the edge, our overhead is always so scarce!<p>Amazing work as always, looking forward to taking this for a spin!

This is a fantastic news! I spent hours on fine tuning my prompt to summarise text and output in JSON and still have some issues sometimes. Is this feature available also with Go?

Very annoying marketing and pretending to be anything other than just wrapper around llama.cpp.