Hi HN! Erik here from Banana (formerly the serverless GPU platform), excited to show you what we’ve been working on next:<p>Fructose<p>Fructose is a python package to call LLMs as strongly typed functions. It uses function type signatures to guide the generation and guarantee a correctly typed output, in whatever basic/complex python datatype requested.<p>By guaranteeing output structure, we believe this will enable more complex applications to be built, interweaving code with LLMs with code. For now, we’ve shipped Fructose as a client-only library simply calling gpt-4 (by default) with json mode, pretty simple and not unlike other packages such as marvin and instructor, but we’re also working on our own lightweight formatting model that we’ll host and/or distribute to the client, to help reduce token burn and increase accuracy.<p>We figure, no time like the present to show y’all what we’re working on! Questions, compliments, and roasts welcomed.
IMHO, in the future programming may look similar to this. Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code.<p>This strikes a happy medium, where machines are assisting programmers, making them much more productive. Yet the resulting code is understandable as a human has decomposed everything into functions, and also robust as it is formally verified.<p>I am working on a F# proof-of-concept system like this, there are other alternatives around implemented in Haskell and other languages with varying levels of automation. It is potentially an interesting niche for a startup.
This approach may be too high-level "magic" to the point of being difficult to work with and iterate upon.<p>Looking at the prompt templates (<a href="https://github.com/bananaml/fructose/tree/main/src/fructose/templates">https://github.com/bananaml/fructose/tree/main/src/fructose/...</a> ), they use LangChain-esque "just try to make the output to be valid JSON" when APIs such as GPT-4 Turbo which this model uses by default now support function calling/structured data natively and do a very good job of it (<a href="https://news.ycombinator.com/item?id=38782678">https://news.ycombinator.com/item?id=38782678</a>), and libraries such as outlines (<a href="https://github.com/outlines-dev/outlines">https://github.com/outlines-dev/outlines</a>) which is more complex but can better ensure a dictionary output for local LLMs.
Big proponent of guaranteed outputs for LLMs. I wrote a library awhile back (gpt-json) that did something similar by querying the OpenAI API. At the end of the day though while their responses are _highly likely_ to be valid JSON they're not guaranteed. There's only so much that can be done with remote calls to their model's black box.<p>The future here really lies in compiling down context free grammars. They let you model json, yml, csv, and other programming languages as finite state machines that can force LLM transitions. They end up being pretty magical: you can force value typing, enums, and syntax validation of multivariate payloads. For use in data pipelines they can't be beat.<p>I did some experiments a few weeks ago on training models to generate these formats explicitly with jsonformers/outlines. Finetuning in the right format is still important to maximize output. You can end up seeing a 7% lift if you finetune explicitly for your desired format. [^1] At inference time the CFGs will constrain your model to do what it's actually intended to.<p>[^1]: <a href="https://freeman.vc/notes/constraining-llm-outputs" rel="nofollow">https://freeman.vc/notes/constraining-llm-outputs</a>
> not unlike other packages such as marvin<p>This feels pretty much identical to Marvin? Like the entire API?<p>From a genuine place of curiosity: I get that your prompts are different, but like why in the name of open source would you just not contribute to these libraries instead of starting your own from scratch?
Does anyone else get bothered by how this seemingly results in code that won't compile?<p>Instead of this:<p>@ai()
def describe(animals: list[str]) -> str:
"""
Given a list of animals, use one word that'd describe them all.
"""<p>it would seem a lot more intuitive to do this:<p>def describe(animals: list[str]) -> str:
return ai("""Given a list of animals, use one word that'd describe them all.""", animals)
Good stuff. How does this compare to Instructor? I’ve been using this extensively<p><a href="https://jxnl.github.io/instructor/" rel="nofollow">https://jxnl.github.io/instructor/</a>
Since you are going down this route, I would recommend you guys to build some sort of unit test driven fine tuning framework, where you may provide input output examples expressed as simple function calls. You could then let the LLM generate examples and check them using the unit tests and keep the valid results to build up a valid data set. For bonus points, the unit tests themselves could also call the LLM to check if the output passes criteria expressed in natural language or not.
I love this emerging space at the intersection of programming and LLMs. It goes beyond having the LLMs generate code: that's an obvious and amazing use case, but it's far from the only one.<p>Another project I'm excited about in this area is GPTScript, which launched last week: <a href="http://github.com/gptscript-ai/gptscript">http://github.com/gptscript-ai/gptscript</a>.
TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling.<p>Langchain & Llamaindex plus Fructose really need to skip the structure adherence work & move to chunking/KG generation since that's the next pain point to tackle.
I find it grating that all of these types of things say "LLMs" when in fact they literally only work with OpenAI. There are hundreds of variations of LLM models. When it works with only gpt-4-turbo or gpt-3.5-turbo, it's inaccurate to say it's a tool for LLMs in general.
Maybe I’m not the audience for this but how is this a “product”. Coercing LLM outputs into a function call is built into OpenAI itself.<p>What is fructose doing extra here? It’s like productising copy&paste which every modern OS has, no?
Nice, I've just started building something similar in TypeScript. I wasn't a big fan of the Langchain model. I wanted to develop with normal functions in an imperative manner so the code is very easy to read. I'm also using decorators to add the required functionality to workflow steps so I can support retries, and build something like LangSmith on top too.<p>So far I've been able to make a little workflow that can complete real simple infrastructure requests in JIRA. Pick the right repository, make the changes, compile, and push up the merge request.
This is great, it might be really helpful for what I’ve been working on<p>Just put together a small project that uses GPT to find good job matches[1]<p>One of the most challenging things in making it useful for more users, is managing the prompt that include several pieces of user input, and need to return a specific format, with a structure that depends on what the user wants to include in the prompt<p>What’s the typical use case for this? Who needs it the most right now?<p>Thank you!<p>[1] <a href="https://news.ycombinator.com/item?id=39621373#39624542">https://news.ycombinator.com/item?id=39621373#39624542</a>
So what is this actually putting into the prompt to guide generation? I dislike libraries that come with a lot of pointless abstraction.<p>I'm about to write something that generates typescript code from pydantic models. If this just works out of box, it would make me very happy.<p>I'll take a look through the repo tomorrow, sorry if my response is a little lazy, I just got off work.
Definitely very excited to see this be a thing. Genuinely liked the approach to make function calls strongly typed and rely on functional programming principles.<p>During my senior year, I worked on a research project very similar to this and I’m glad to see this out there for everyone. I’d love to connect with the team if possible!
Thanks everyone for a great Show HN! This turned out much larger than expected, thanks for all the comments and github stars. We had a fun time with it.<p>Lots of takeaways, blogs to read, things to implement, issues to address. On it!
I've done a lot of work over the last year wrangling LLM outputs - both from the OpenAI API as well as local LLMs.<p>What are the benefits of using Fructose over LMQL, Guidance or OpenAI's function calling?
How does Fructose relate or compare to Instructor (<a href="https://github.com/jxnl/instructor">https://github.com/jxnl/instructor</a>)?
fyi: LM Studio can host a server that uses the OpenAI api to whatever model you are using locally<p>So as long as this library can be directed to localhost or configured, it can use any LLM
Yet another implementation of <a href="https://esolangs.org/wiki/English" rel="nofollow">https://esolangs.org/wiki/English</a>