Data-Oriented Architecture

We jump through so many hoops to avoid this terrible pattern. This puts more load and less scaling options on your DB. If you need to upgrade how something is stored, you need to touch every service. And they didn&#x27;t really solve their problem.<p>If you can&#x27;t keep track of what services call each other, what makes you think you&#x27;ll be able to keep track of who is writing out what data?<p>Reading how they did SOA (spaghetti of services with highly coupled interactions) I don&#x27;t think putting that at the data layer is useful in the long run. They even suggest many services operating on the same data as a way to decorate the data as it haphazardly makes it way through this mess of an app. Their original problem was a poorly planned separation of concerns leading to cross service coupling. In DOA that still have it. Its just coupled directly through the datastore with no hope of cleaning it up without a massive migration.

I liked the clarity of the article and the reasoning is solid. Still, I&#x27;m not too convinced:<p>- A key reason for splitting up a monolith into services is because collaboration becomes too costly with 100s to 1000s of developers working on the same code. Your build &amp; test system can&#x27;t handle the number of commits. Your code takes too long to build. Would the data-access layer not them become the development bottleneck in such scenarios, meaning that it doesn&#x27;t scale as well as SOA?<p>- What are the advantages and disadvantages of centralizing in the data-oriented layer over centralizing through a network-related tool like Envoy, Kubernetes, or Istio?<p>- The database layer itself often becomes a performance bottleneck, which requires us to run a sharded database. Some companies go the extra distance by running in-memory databases, document databases, and time-series databases. In such cases, wouldn&#x27;t the data access layer need to support federation, which is a hard problem according to database research?<p>- Is O(N^2) really that big of a problem? It seems like the problem can be reduced to something simpler: developers cannot easily understand which services communicate with one another. If that is the case, would a visualization tool be sufficient?

This was the predominant architectural design about 20 years ago.<p>There are many strong points for this the general challenges are...<p>Databases don&#x27;t&#x2F;didn&#x27;t have too much in the way of integration primitives...<p>Databases can become a performance bottleneck that can only be scaled vertically...<p>SQL language is not that great as an application programming language<p>EDIT: forgot the biggest one which is, it is completely up to discipline to produce any kind of separation between implementation details and public api since everything lives in the database. That&#x27;s really the biggest challenge.

It&#x27;s in a certain way a very roundabout way to describe a event based SOA system.<p>Many of the problems described are problems from SOA systems which are using a number of common anti-patterns for SOA systems.<p>Also the way this person describes DOA is prone to end up with a monolith in _data_ and a just the logic is not monolithic but might end up accidentally being quite tight coupled. (Through it does have some benefits).<p>Also even with DOA you can end up with internal state coupling between services if you do it wrong. It&#x27;s harder then in some bad designed SOA systems but IMHO roughly as likely as a SOA system communicating with events.<p>---<p>- So use events if you do SOA (for inter service communication)! - Never rely on the internal state of another Service. - Make sure you don&#x27;t send events to a specific service, instead &quot;just&quot; send events, then all services interested _in that event_ will receive it. (Make sure to subscribe for events independent of their source not services; E.g. use a appropriate event broker or service mech; Sometimes just broadcasting is fine; <i></i>Oh and naturally storing the event and making that trigger other services works, too. At which point we are back at DOA<i></i> )<p>----<p>DOA is not bad just IMHO misguiding. If you want to use it look at common problems with event based systems for vectors of potential problems wrt. accidental internal state coupling as many of this will apply to DOA, too (if your system becomes complex enough).<p>Note that I don&#x27;t mean all problems caused by combining eventual consistency + high horizontal scaling with event systems. Sadly this is just very often mixed up.

I like this architecture but I worry that in sufficiently complex systems it can lead to some significant complexity if there are entities within the database that have a lot of contention around them. For example, it may end up translating to deadlocks etc. when two services that are naive to each other start locking tables in interleaved sequences of interactions. It still seems wise even if you follow this pattern to segment areas of the database to management of particular services or consumers and then have those present APIs or message passing interfaces to each other. Which leads you half way back to SOA or microservices.

Hmm. Does this not turn the data access layer into a service bus?<p>This could get very messy at scale.

Yikes. You haven&#x27;t reduced inter-component communication. You&#x27;ve basically got an interface distributed across all your components, and defined as a data schema. Any change to this can now potentially break communication between any two or more processes.

using schemas rather than APIs to share information between components seems right<p>IMO one of the reasons CRUD is so hard now is that the schema is different at every layer of the product<p>Slight differences between layers are necessary for permissions &#x2F; privacy, but there are probably better ways to get that done than to reimplement the schema at every layer.

DOA isn&#x27;t a great acronym :)

Congrats, you just reinvented the monolithic database with a bunch of functions on top. Everything old is new again.

Would it be possible make interactive systems with this architecture? The requester needs to be notified about result availability, so some communication must be targeted to specific component (the requester) going against the foundation idea of this pattern.

This is nothing new and generally how SOA began in the 2000’s.<p>One of the critical improvements to SOA is DDD (domain-Driven Design) where context matters and boundaries should include the service and its data.<p>Data oriented architecture a bad idea. Period.

So in this scenario are we essentially reimplementing Apache Kafka? Producers who create events that are then read by consumers and persisted on a database layer. What am I missing?

A central message broker that intermediates between services also solves the problems described in the &quot;Problems of scale&quot; paragraph. It might be worth mentioning this.

If you put your data in CSV files on your file system and had different programs on the same server access those files.. would it be a data oriented architecture?

Another interesting point of this architecture, that you can plan your cache really well.

Cache rules everything around me.

So on a high level redux but for backend (Store-reducers-components) ?

Industrial-grade Data-Oriented Architecture:<p>[The Pure Function Pipeline Data Flow v3.0 with Warehouse &#x2F; Workshop Model](<a href="https:&#x2F;&#x2F;github.com&#x2F;linpengcheng&#x2F;PurefunctionPipelineDataflow" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;linpengcheng&#x2F;PurefunctionPipelineDataflow</a>)<p>1. Perfectly defeat other messy and complex software engineering methodologies in a simple and unified way.<p>2. Realize the unification of software and hardware on the logical model.<p>3. Achieve a leap in software production theory from the era of manual workshops to the era of standardized production in large industries.<p>4. The basics and the only way to `Software Design Automation (SDA)`, just like `Electronic Design Automation (EDA)`.