I am an AI researcher. Most actual AI researchers and engineers use very few of these tools - the only one being model providers like OpenAI API and public clouds (AWS, Azure, GCP). The rest of these are infra-centric tools that a16z is highly incentivized to over-inflate the importance of.
> So, agents have the potential to become a central piece of the LLM app architecture (or even take over the whole stack, if you believe in recursive self-improvement). ... . There’s only one problem: agents don’t really work yet.<p>I really appreciate that they called out and separated some hype vs. practice, specifically with regards to Agents. This is something I keep hoping works better than it does, and in practice every attempt I've taken in this direction leads to disappointment.
This blog post is way more complex than it needs to be, a lot of what most people are doing with llms right now boils down to using vector databases to provide the "best" info/examples to your prompt. This is a slick marketing page but im not sure what they think they're providing beyond that.
Bit of self-promotion, but Milvus (<a href="https://milvus.io" rel="nofollow noreferrer">https://milvus.io</a>) is another open-source vector database option as well (I have a pretty good idea as to why it isn't listed in a16z's blog post). We also have milvus-lite, a pip-installable package that uses the same API, for folks who don't want to stand up a local service.<p><pre><code> pip install milvus
</code></pre>
Other than that, it's great to see the shout-out for Vespa.
interesting to see that the word "generative" does not appear in this blogpost (apart from the tags). 6 months ago Generative AI was all the rage: <a href="https://a16z.com/2023/01/19/who-owns-the-generative-ai-platform/" rel="nofollow noreferrer">https://a16z.com/2023/01/19/who-owns-the-generative-ai-platf...</a><p>I think this is a very well articulated breakdown of the "LLM Core Code Shell" (<a href="https://www.latent.space/p/function-agents#%C2%A7llm-core-code-shell" rel="nofollow noreferrer">https://www.latent.space/p/function-agents#%C2%A7llm-core-co...</a>) view of the world. but it is underselling the potential to leave the agents stuff to a three paragraph "what about agents?" piece at the end. the emerging architecture of "Code Core, LLM Shell" decentralizing and specializing the role of the LLM will hopefully get more airtime in the december a16z landscape chart!
this and other different end-to-end architectures are offered in deepset/haystack, one of the best and quite mature frameworks to work with LLMs (pre-GPT craze) and do augmented retrieval, etc.<p>I do feel the article presents old concepts as "emerging".<p>if you are curious about building something quickly, you can jump into one of the tutorials <a href="https://haystack.deepset.ai/tutorials" rel="nofollow noreferrer">https://haystack.deepset.ai/tutorials</a><p>Over a weekend I've used deepset/haystack to build a Q/A engine over open source communities slack and discord threads that can potentially have an answer - it was a joy and a breeze to implement. If you have question about Metaflow, K8s, Golang, Deepset, Deep Java Library and some other tech - try asking your quick question on <a href="https://www.kwq.ai" rel="nofollow noreferrer">https://www.kwq.ai</a> :-)
Microsoft guidance is legit and useful. It's a bunch of prompting features piled on top of handlebar syntax. ( And it has its own caching. Set temp to 0 and it caches. no need for LLM specific caching libs :) )<p><a href="https://github.com/microsoft/guidance">https://github.com/microsoft/guidance</a>
How prescient is the "Hidden Technical Debt" [1] paper from ~8 yrs ago compared to this? See the top of pg4 for a figure that I've personally found to be useful in explaining all the stuff necessary to put together a reasonable app using ML/DL stuff (up until today, anyway).<p>I see all the same bits called out:<p>- Data collection<p>- Machine /resource management<p>- Serving<p>- Monitoring<p>- Analysis<p>- Process mgt<p>- Data verification<p>There's some new concepts that aren't quite captured in the original paper like the "playground" though.<p>I've kind of been expecting a follow-up that shows an update to that original paper.<p>[1] <a href="https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf" rel="nofollow noreferrer">https://proceedings.neurips.cc/paper_files/paper/2015/file/8...</a>
> "For devs who see every database-shaped hole and try to insert Postgres"<p>I see sub second performance on >1m vectors with PgVector. Vector databases have a place but this statement seems disingenuous at best. Bringing on a vector database adds additional complexity and a giant chunk of use cases simply don't need it. Not to mention the additional latency you'd be adding.
>In-context learning solves this problem with a clever trick: instead of sending all the documents with each LLM prompt, it sends only a handful of the most relevant documents.<p>I don't even think this is a correct definition for "in-context learning". In-context learning is a type of few-shot learning in which examples of input/output pairs are provided as part of the prompt. The idea is that the model is able to "learn" the pattern of the task from the examples. Quoting from the GPT-3 paper:<p>>what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.<p>I really don't think it's standard to refer to the process of embedding-based retrieval as "in-context learning".
Is the emerging architecture made out to be more complicated than what most of the companies are currently building? Perhaps! But this is most likely the general direction where things will start trending towards as the auxiliary ecosystem matures.<p>Shameless plug: For fellow Ruby-ists we're building an orchestration layer for building LLM applications, inspired by the original, Langchain.rb: <a href="https://github.com/andreibondarev/langchainrb">https://github.com/andreibondarev/langchainrb</a>
Reading the comments, it seems like we need better human-agent interaction tools<p>Many are frustrated about not being able to better direct the agents<p>It’s like the agents have certain pre-learned things they can do, but they aren’t really learning how to apply those things to the environments their human operators want them to develop in<p>Or at least it is not easy/straightforward how to teach the model new tricks
I was looking at various forms of indexing solutions to solve search and clustering problems with TerminusDB for clients. When I compared solutions against embeddings from LLMs, they LLMs were just far easier to work with and got much better results. I believe traditional text indexing will die quickly, as will a lot of the Entity Resolution and traditional clustering methods to be replaced completely by LLM's. We found them so compelling we wrote our own open source vector database sidecar: <a href="https://github.com/terminusdb-labs/terminusdb-semantic-indexer">https://github.com/terminusdb-labs/terminusdb-semantic-index...</a>
It's a good article for people with no strong background in NLP or LLMs. Gives a comprehensive overview on how this could be applicable for startups.<p>Enterprise, not so quite, there are a lot of other stuff to consider like points missing, ethical application, filtering, security, points that are very important for enterprise customers.<p>Also, in-context learning is just one way to apply LLMs, there are way more applications like few short learning, fine tuning, depending on cost and application involved as I've highlighted here<p><a href="https://twitter.com/igorcosta/status/1671316499179667456" rel="nofollow noreferrer">https://twitter.com/igorcosta/status/1671316499179667456</a>
I agree on the excerpt on agents. Reliability and reproducibility of task completion is the biggest problem for agents to cross the chasm to real life use cases. When agents are given an objective, they think everything from first principles or scratch about next best action to complete the objective and agent trajectory ends up becoming more of a linguistic dance. But we are solving some of the agent specific problems at SuperAGI <a href="https://github.com/TransformerOptimus/SuperAGI">https://github.com/TransformerOptimus/SuperAGI</a> ( disclaimer : Im creator of it ) by doing agent trajectory fine tuning using recursive instructions. Think about objective as telling agent to go from A to B and instructions are akin to giving it directions about the route. And this instruction can be self created after every run and fed into subsequent runs to improve the trajectory.<p>Other problem with agent is : most independent agents are capable of doing very thin slice of use case, but for complex knowledge work tasks, more often than not, one agent is not enough. You need a team of agents. We introduced a concept of Agent Clusters - which operate in master slave architecture and coordinating among themselves to complete nuanced tasks and coordinating via shared memory and shared task list.<p>Another big bottleneck I think is lack of a notional concept of Knowledge for Agents. We have LTM and STM, but knowledge is specialized understanding of particular class of objectives ( ecommerce customer support, Account based marketing, medical diagnostics for particular condition etc ) plugged into the agent. Currently agents leverage on the knowledge available in the LLMs. LLMs are great for intelligence, but not necessarily knowledge required for an objective. So we added concept of knowledge - which is a embedding plugged into agent apart from LTM / STM<p>There lot of other challenges that need to solved like agent performance monitoring, agent specific models, agent to agent communication etc to truly solve for agents deployed in production. Not sure about point mentioned in the article that they might even take over the entire stack because autonomous agentic behaviour is good for certain use cases and not for all kinds of apps.
I think sidecar vector databases that work with existing dbs will emerge as more prevalent than the pure vector DB. I also think the vector & graph combo on highly interconnected data will have additional benefits for those building a wide range of LLM applications. A good example is the VectorLink architecture with TerminusDB [1] which is based on Hierarchical Navigable Small World graphs written in Rust.<p>[1] <a href="https://github.com/terminusdb-labs/terminusdb-semantic-indexer">https://github.com/terminusdb-labs/terminusdb-semantic-index...</a>
With everyone writing about LLMs, and not time to read them even 1% of it all, the reason to read A16z is for technical/analytical merit, or for an investment-pumping angle?
In future we will see ton of similar charts borrowing elements from graph and signal theories. There's no limit on amount of different LLM-multiagents.
I covered the subject during a python Atlanta talk last month. There isn’t much that’s new at the moment. Mostly because an LLM can be considered a software agent. That may change soon as things become more complex, though. Things like AWS’s Kendra show there’s some new patterns in the pipeline.<p>I’ll say this post is rather shallow to be considered technical or even fit the title
Ugh. Of course the Enterprise Architecture rears its ugly head here.<p>Just here to say that you can quickly build a robust feature with only OpenAI's APIs, redis, a text file for the prompt you parameterize (versioned), and a little bit of glue code (no LangChain). You can add instrumentation for observability around that like you would any other code.<p>I would wager that most enterprise use cases don't need most of the tools listed in this article, and using them is complete overkill.
Still relatively simple, the stack being LLM hopefully most of the actual “stack” work will be transferred inside the LLM. Example, if context size becomes unlimited, could do away with vector dbs.
Great starting point! These diagrams notably miss a LLM firewall layer, which is critical in practice to safe LLM adoption. Source: We work with thousands of users for logicloop.com/ai
are VCs doing architecture now ? huh... architecture astronauts much.<p>looks like they are playing catchup and trying to stay relevant.<p>What happened to their web3 vision ?
Something not obvious to me with these VC diagrams wrt the memory tier being just vector DBs vs also including knowledge graphs<p>Good: We're (of course) doing a lot of these architectures behind-the-scenes for louie.ai and client projects around that. Vector embeddings are an easy way to do direct recall for data that's bigger-than-context. As long as the user has a simple question that just needs recalling a text snippet that fairly directly overlaps with the question, vector embeddings are magical. Conversational memory for sharing DB queries across teammates, simple discussion of decades of PDF archives and internal wikis... amazing.<p>Not so good: What happens when the text data to answer your question isn't a directly semantic search match away? "Why does Team X have so many outages?" => What projects is Team X on" + "Outages for those projects" + "Analysis for outage" . AFAICT, this gets into:<p>A. Failure: Stick with query -> vector DB -> LLM summary and get the wrong answer over the wrong data<p>B. AutoGPT: Getting into an autoGPT langchain that iteratively queries the vector DB, and iteratively reasons over results, & iteratively plans, until it finds what it wants. But autoGPT seems to be more excitement than production use. Many questions like speed, cost, & quality...<p>C. Knowledge graphs: Getting into use the LLM to generate a higher-quality knowledge graph of the data that is more receptive to LLM querying. The above question now becomes a simpler multi-hop query over the KG, so both fast and cost-effective... If you've indexed correctly and taught your LLM to generate the right queries.<p>(Related: If you're into this kind of topic, we're hiring here to build out these systems + help use them on our customers in investigative areas like cyber, misinfo, & emergency response. See new openings up @ ttps://www.graphistry.com/careers !)