Specifically, how strong are the durability & backup guarantees?<p>The setup is:<p>-kafka is the underlying source-of-truth for the data.<p>-querying is done on caches (I.e. Redis, ktables) hydrated from kafka<p>-Kafka configured for durability (infinite topic retention, 3+ replication factor etc)<p>We are not allowed to lose data <i>in any circumstances</i><p>Is the proposed setup nuts?<p>In <i>theory</i> the replication should guarantee durability & resiliency. Confluent themselves encourage the above pattern.<p>The only flaws I can think of, are:<p>- cache blows up and needs to be rehydrated from scratch<p>-broker disk gets wiped/corrupted -> kafka rebalance, resulting in prolonged downtime if topics contain mountains of data<p>Has anyone run and battle tested this kind of setup in production? I.e. encountered disk corruption, brokers going down, but still retain data?<p>My gut feel is, this is a bad setup, because kafka wasn't originally designed for RMDBS levels of durability, but can't point to a concrete reason why this would be the case.
Can you talk more about the use case?<p>My question would be, why use that setup if you don't have to?<p>That setup is not something I would reach for unless I really really had to (and when you think you really really have to, check again, then check again, you probably don't really really have to).<p>How about a RDBMS like Postgres setup with streaming synchronous replication and durability-tweaked parameters?