Thinking in Event Sourcing, and CQRS, and DDD, and Services, and …

I can’t remember how I came across CQRS and Event Sourcing exactly but the when was somewhere around 2010. I spent a year or so really digging in to it and getting familiar with concepts before I eventually had the opportunity (or balls) to try it on a real project. At that time, I along with probably every man and his dog was still using ORMs like NHibernate. Though I was acutely aware of the pain and cost it incurred I didn’t really know how to do it any differently. Discovery of the concepts in CQRS and Event Sourcing was the beginning of a journey that little by little, bit by bit began to change my mindset to the point where even if I had to work on a project that for all intents and purposes had no real relationship to such ideas, I would build things differently in order to at least make these projects somewhat maintainable, and a bit less painful to work on.

On this very blog, about four years ago I described in a series of articles the idea of messages flowing through code that had (and still does) have a massive impact on the structure of your typical CRUD application. Even now, somewhat refined, I still use those techniques because they add value both to me and anyone else having to read my code. The result is far fewer layers, more vertical slices (or end to end feature folders) and the ability to delete code without having to worry about the impact on the rest of the application. This one ability alone has the greatest pay off because it means I can rapidly rewrite features and the rest of the code won’t know or care.

Fast-forward to 2017. For me, these ideas and the way they materialised were a good way to move from the old painful CRUD, ORM mapping world but for the large majority of developers stuck in their boring old Enterprise jobs, they’re still churning out heavy layered CRUD apps, cookie cutter style. Talk to most about CQRS and/or Event Sourcing and their eyes will glaze over. They’ll look at you like your speaking a foreign language. After seven years or so, I feel like I have a pretty good grasp on what it’s all about but for those that don’t but want a quick high level overview here’s what I think you should know:

1. Event sourcing is not a technical alternative to mapping DTOs with an ORM

When you first talk to a developer about event sourcing they usually think about it in technical terms. Saving events vs saving DTOs but this is completely missing the point. Event sourcing is about capturing user intent/actions that are IMPORTANT to the business even if they don’t know HOW that data is important just yet. When they figure it out you’ll be ready to help them dig into it by building projections off that historical event data so they can answer questions right here, right now. This leads me on to the next point.

2. Data structure vs behaviour

When your ORM or your database is at the centre of your world it’s hard to think about nothing but structure and how you’re going to map the data back into objects and we all know about the object relational impedance mismatch right? Remember those books by Bruce Eckel – Thinking in Java, Thinking in C++, etc? What you actually need to do is start “Thinking in Behaviours” (Come on Bruce, a great book title right there!). Events are the results of actions being performed in the domain you’re working in. By working with domain experts you can use language that both can understand because events such as CustomerPlacedOrder are plain English and let you get to the heart of what a system should DO not what it should store. Thinking in behaviours leads to breakthroughs in understanding.

3. The view of the world changes over time

How often, working on a large codebase are you required to add in some extra functionality because a requirement has arisen that nobody foresaw? Now it means you’re probably going to have to jump through hoops to make it all work, you’ll struggle trying to introduce tests for it, and you’ll probably have just made the codebase that little bit dirtier. You’re fighting a losing battle with software entropy. Systems that prioritise structure and state are brittle. They do not cope easily with changes over time. However, with CQRS/ES this is fairly trivial. Working with the domain expert we probably arrive at some new behaviour that results in a bunch of new events, probably from a newly discovered aggregate. Combining these with some existing events we can write a new denormaliser to project those events into new reports, new screens, allowing the read side of the system to evolve independently.

4. DDD does not apply to monoliths

Remember when you first heard the term agile (no, not the marketing crap that it’s become)? I’m talking about how the idea that the waterfall approach that everyone knows is bad if applied over say six months, makes a whole load of sense if applied over a week, possibly two at most? Basically, we squeeze the tried and trusted steps of analysis -> design -> implementation -> test -> deployment down to a much smaller time frame. The same is true of Domain Driven Design. Developers learning DDD will try to apply the ideas at the macro level meaning everywhere across the entire system and before they know it they’re well on their way to another big mess that someone else will have to pick up and work with. Domain Driven Design should be applied at a smaller scale, in a bounded context (one or more but not everywhere). The determining factor being a context where the business derives great value. DDD is costly and even more so if applied indiscriminately. The great thing about DDD though is that it really does fit like a glove with the idea of Aggregates and Event Sourcing. Both encourage thinking about behaviour, and language. This leads me to point 5.

5. Get your granularity right

I like to think of the pieces of a properly designed system at different levels of granularity ranging from the micro to the macro. So for instance, at the micro level we have:

* Aggregates.

Aggregates represent a transactional boundary and are responsible for enforcing invariants and not allowing themselves to be bypassed. This is why in a properly designed Aggregate you will be unlikely to find state exposed via properties. Instead only behaviours which in turn emit our events. What does it mean to enforce invariants in a transactional boundary? As an example, imagine our aggregate (representing a customer, say) has behaviours that are only allowed if the customer is over 18. Let’s say we need to correct the customer’s date of birth. This has implications for those behaviours. If when we correct the birth date it turns out the customer is no longer allowed to perform those operations we need that state within the aggregate (e.g. IsAdult) to be consistent with the date of birth. What this means from a technical point of view is that the two should never get out of sync. We never allow the IsAdult state to be eventually consistent, it must always be immediately consistent.

* Services

Zooming out a bit from the view of aggregates we arrive at services which in DDD parlance represent a bounded context. So here we can apply all the goodness of Domain Driven Design within this small, narrowly focused service if, of course, it is rich enough to be of business value. In this bounded context the idea of the ubiquitous language will apply. If you find the language of a particular term changing meaning depending on who you speak to then that’s probably a sign that your service is too large and you should look to see how it can split into two different bounded contexts. Without getting into the whole service vs micro-service argument the point is that these contexts should not be so large that they can’t easily be torn down and rewritten/replaced within a very small period of time. Some of these services/contexts will be ideal candidates for event sourcing. Others will be a bit more mundane and can be implemented in a much more trivial manner. Don’t sweat the details here. They’re small enough that you can replace them, refine or rewrite them. They don’t need to be perfect. They just need to be stable with a consistent api.

* Top level architecture

Zooming out again we end up looking at how these services/bounded contexts communicate. This will likely be via messages in what is known as an EDA or Event-Driven Architecture. At this level of granularity CQRS and/or ES do not apply. At this point all communication is async and you are totally in the realms of eventually consistent views of the world. When you get here you’re in a position to take advantage of even bigger views of the world i.e. the cloud, containers (ie. docker, etc), and the ability to scale up and down as required.

Summary

As I get older, working on large codebases becomes an exercise in frustration because my brain seems less able to understand the big (massive) picture of the whole system. I can’t keep it all in my head! Breaking a system down and having knowledge of business events as opposed to technical mappings is a much stronger glue for system understanding in my opinion. Whilst there are likely many more parts to the whole, each one is manageable and digestible at least from a coding point of view (orchestration is another matter). Division of labour among teams is much easier without falling over each other in source control systems too.

One idea I really like and think of as a benefit of Event Sourcing is the occasionally connected client such as apps on phones and tablets. Even using ES here could reap real rewards. Capturing events in an app allows the user to continue working even when they lose signal. At some point later on when they reconnect, those events can be transmitted and the system brought up to date.

CQRS and Event Sourcing are not new ideas. They weren’t new when I first came across them but still even now among the majority of developers these ideas are not well known. I understand it can look complex from the outside but like anything you look at for the first time you realise there’s a learning curve but you just need to get stuck in and start playing with the ideas and it’ll start to click.

The one thing in common though with all these ideas is that it requires you to THINK and that’s probably the hardest thing of all to learn.

Advertisements
Thinking in Event Sourcing, and CQRS, and DDD, and Services, and …

MiniBus

I’ve been hacking away recently on a little MSMQ side project I call MiniBus that allows me to reliably integrate between small applications and services, etc. with a dose of NServiceBus-style goodness. This kind of begs the question though, why not just use NServiceBus? Well, for a start, getting a business to pay for it can be a difficult conversation, and if you’re going to use it at all then really it should be used as part of an overall system architecture that can justify it such as the new mobile eCommerce platform we’ve recently been building. Whilst it is clearly capable of handling simple application integration scenarios too, that isn’t what it was really designed for and can be overkill. It’s also quite complex and requires a good understanding of distributed systems architecture, something not all developers are willing to learn about. As much as I love NServiceBus (and hey, it must be good ’cause its managed to convince some people that I have a clue about architecture, but don’t tell my boss!) sometimes the project at hand calls for a simpler approach.

I often find myself working on small integration projects when we bring new clients on board. Some of the older apps that we have that already do similar work are written in such a way that there exists a direct temporal coupling between us and our clients, or even to various other parts of our own system. It makes for a rather brittle architecture. My desire to improve this situation had me looking around for something that, in NServiceBus terms, would essentially give me a bus.Send but not bus.Publish. All I needed was the ability to break that coupling and let me integrate apps reliably by offloading work, when appropriate, to other services but…I also wanted some of the nice things that NServiceBus has like automatic retries, JSON serialization, transaction support, and error queues for failed messages. Oh, and I also wanted something small and lean and I think I’ve achieved that goal so I’m happy with the end result.

MiniBus is not an ESB and it doesn’t do pub-sub, but it does have some bus-like properties. In a nutshell, it offers the following features:

  • Send to one or more queues (local and/or remote)
  • Load balancing via round-robin dispatching
  • Read messages synchronously or asynchronously
  • Choice of XML or JSON serialization
  • Enlist in ambient transactions
  • Configurable automatic retries
  • Move to error queue on failure
  • Automatically create local queues
  • Install MSMQ if not detected
  • Simple logging support
  • Ability to return error messages back to the read queue

Whilst the source code and full documentation can be found on GitHub here’s a quick overview of using MiniBus. Essentially you create a bus instance via the BusBuilder class and then use it to either send messages:


// create a bus for sending messages
IBus bus = new BusBuilder()
    .WithLogging(new FileLogger())
    .InstallMsmqIfNeeded()
    .DefineErrorQueue("MiniBus.errors")
    .DefineWriteQueue("MiniBus.messages1")
    .DefineWriteQueue("MiniBus.messages2@remotepc")
    .DefineWriteQueue("MiniBus.messages3")
    .CreateLocalQueuesAutomatically()
    .EnlistInAmbientTransactions()
    .JsonSerialization()
    .CreateBus();

// create your message type
var bob = new Person { p.Name = "Bob", Age = 22 };

// send it
bus.Send(bob);

or receive them:


// create a bus for reading messages
IBus bus = new BusBuilder()
    .WithLogging(new FileLogger())
    .DefineErrorQueue("MiniBus.errors")
    .DefineReadQueue("MiniBus.messages1")
    .JsonSerialization()
    .NumberOfRetries(3)
    .CreateBus();

// register a message handler
bus.RegisterHandler<Person>(new PersonHandler());

// process messages on the read queue
bus.Receive<Person>();

Receiving messages can be done synchronously as shown in the example or asynchronously. If, for example, you want to process messages only at specific times then the Receive method is the better choice. If you want to process messages as and when they arrive then call the ReceiveAsync method as soon as you’ve created the bus and configured your handler. Either way, the handler specific code runs in the context of a transaction and should an exception occur MiniBus will retry if you’ve configured the bus to do so. If after the maximum number of retries the message still can’t be successfully processed, MiniBus will move it to an error queue. When you’re ready to try again, the ReturnAllErrorMessages method will move failed messages off the error queue and back to the read queue. On the subject of handlers, you simply create one or more classes that implement the IHandleMessage<T> interface and then register them with the bus.

MiniBus also has a rudimentary “load balancing” option via the AutoDistributeOnSend method. Ordinarily, having defined multiple write queues, sending a message would mean that a copy is sent to all of those queues which is fine if you need different handlers for the same message type. On the other hand, if you want to spread the load by having two or more different endpoints deal with messages in the same way setting the AutoDistributeOnSend option will cause MiniBus to adopt a round-robin style balancing approach whereby the first call to Send will send the message to the first queue, the second call to Send will send the message to the second queue, and so on before going back to the first. This means that no single endpoint is tasked with having to do all the work. In other words, you gain some parallelism.

In order to keep it small and simple, MiniBus has no dependencies on any other library. Logging is up to the consuming application. An interface is provided (ILogMessages) that you can use to wrap your logging library of choice. If you don’t like specifying queue names in code, it’s up to you to read them out of your config file and set the various bus properties. Finally, If you need a hosting process I recommend TopShelf for Windows Services. Right now, MiniBus leaves all of this up to you though that might change in future iterations.

There’s probably a lot more that could be done with MiniBus but you have to draw the line somewhere. There’s no point in attempting to make another ESB when there’s already plenty of great options out there like NServiceBus, and Rebus, another cool library. With MiniBus I had a simple use case in mind and I’ve more or less built what I needed but I think it could be useful to others too and for that reason I think it’s worth sharing so I’ve decided to put it “out there” by making it available on both GitHub and Nuget. It’s small, simple to use and reliable too so feel free to give it a go, and if you find a use for it then great! If you do happen to find any issues though, be sure to let me know or better yet, submit a patch! :)

Updates

I’ll keep this page updated with changes as and when they happen

24/11/2013

MiniBus 0.1.3 released

  • ReturnErrorMessages renamed to ReturnAllErrorMessages
  • Added ReturnErrorMessage method to move specific message from error queue
  • Added Fail fast option
  • The fail fast option is useful for scenarios where it’s important that messages are processed in order. As soon as an error is detected no more messages are processed and the failing message stays on the read queue.

    MiniBus