I am working on refactoring a system which I feel does some frequently encountered flows across organisations. Collects raw event streams, grabs unique URLs, crawls, extracts, creates vectors, uses these output in some downstream analysis and objects.<p>I have many different solutions which “work” but I am not sure which is optimal.<p>I find both the internet and LLMs to be a very poor resource for this type of work.<p>So where does one learn about how others have implemented such systems? Happy to reads books, blogs, repos!