Hello everyone!<p>I am Jan, CTO and one of the creators of Pathway, the real-time data processing framework. I’m excited to share Pathway’s ready-to-use AI Pipelines, configurable with just YAML! These frameworks offer out-of-the-box solutions for AI search, RAG, and more—optimized for real-time indexing and in-memory processing.<p>What makes it simple? YAML templates!
The pipeline templates are fully customizable using YAMLs to fit your needs, from changing the data sources to the choice of the LLM model, all without touching Pathway’s Python code.
Thanks to the Pathway data processing engine, these pipelines are optimized for real-time, updating the pipeline by ingesting the data as it arrives.<p>Why it matters:
* YAML templates: that’s it, no Python code required.
* Full customization: Configure everything, from LLM models and indexes to tokenizers and retrieval strategies.
* Real-time data sync: Connect to data sources like Google Drive, SharePoint, and more, with pipelines automatically updating to reflect all additions, deletions, and changes in real-time.
* Powerful search capabilities: Internal in-memory indexing supports vector, hybrid, and full-text search. The indexing is automatically maintained as data sources update in real-time.
* Deploy in minutes: Quickly deploy on-premises or on cloud platforms like AWS, GCP, or Azure.
* It's open: don't hesitate to contribute!<p>Templates: <a href="https://pathway.com/developers/templates" rel="nofollow">https://pathway.com/developers/templates</a>
GH repo: <a href="https://github.com/pathwaycom/llm-app/">https://github.com/pathwaycom/llm-app/</a>
Docs: <a href="https://pathway.com/developers/ai-pipelines/welcome" rel="nofollow">https://pathway.com/developers/ai-pipelines/welcome</a><p>Deploy your personalized AI pipeline in minutes and let us know what you think!
I like the YAML abstraction. This should make it easier to programmatically try and evaluate multiple configurations for the whole AI pipeline (not just the LLM) against a dataset or real users through an API deployment.<p>Some feedback: It would be great to see in one place all the supported fields and values for the YAML config.