Big problem, let's break it down...<p>1. Data issue identification
2. Solution and implementation<p>Most issues are discovered in the data warehouse. Entity matching customer data across different systems, some business process results in duplicate, or null data. I know there are existing, non-LLM, products that do this. I'm curious to compare those with new LLM first products.<p>On solution/implementation. Ideally you're able to fix this in the source system, either in your SaaS tool or in the way you write production data. You can also fix this in the data warehouse, munging/ETL'ing the data. Seems like LLMs could help to 1) identify and recommend a change in an external system, 2) submit a PR to solve this in the data warehouse.<p>Anyone know anyone working on these problems?
Obviously, the cost of using LLM to solve this problem is relatively high. At most, it can only treat LLM as one of the signals instead of processing all the information. Combining LLM with tree models or graph models and adding NER technology may achieve an approximate solution to 2. Can you really trust the answers provided by LLM to be used in a production environment?
1 is already a solved problem. My employer had originally put a ML based system to find DQ issues (already working) and is looking at pocs to add LLMs in the model mix. Hearsay is that our lakehouse vendor will have their own solution to this question via a acquisition.<p>2 is interesting, possible to do via LLM but I worry about data privacy and hallucinations making data more believable but not real.
We (<a href="https://dqc.ai" rel="nofollow">https://dqc.ai</a>) are doing something in the space, yes. Next to a mixture of ML, and heuristic based approaches, plus link & integrations into source systems. Happy to talk about it, feel free to reach out.