VectorFlow (open-sourced on GH) is a new tool for ingesting data into Vector Databases such as Weaviate! There is quite an interesting End-to-End stack emerging at the ingestion layer, from:<p>- Retrieving data from misc. sources such as Slack, Salesforce, GitHub, Google Drive, Notion, ...<p>- To then Chunking the Text (maybe with the use of Visual Document Layout parsers like what Unstructured is imagining)<p>- Extracting Metadata potentially (say the "age" of an NBA player as in the Evaporate-Code+ research)<p>- Then sending this data off to embedding model inference and unpacking that can of worms from inference acceleration to load balancing<p>- Finally -- importing the vectors themselves to Weaviate!<p>I learned so much from this conversation, I really hope you enjoy listening and please check out VectorFlow!<p>https://www.youtube.com/watch?v=efAvOsO8Gik