The Messaging Lightbulb

Over the last eighteen months or so I’ve developed a couple of applications that make heavy use of messaging and queues and found myself having to think about the implications of asynchronous, out of process code, something that ordinarily most of us don’t even consider. We’re naturally inclined to think about everything as request and response, and whenever a remote call is required we immediately reach inside our web services toolkit and bring out WCF or even asmx. With messaging you have to, to quote a fruit-based computer company, “think differently”. The good thing for me was that these projects brought certain ideas more clearly into focus giving me a whole new perspective.

During that time (well, for a lot longer actually) I’ve absorbed whatever I could find from Greg Young and Udi Dahan, be it blog posts or videos (hurry up with the book, Greg!). These two people are the guys that have probably influenced me the most, especially when it comes to CQRS and architecture. It’s their knowledge (and Greg’s focus on “messages” in particular) that has continuously kept me challenging my own assumptions and together with the experience I’ve gained from those messaging projects I feel like another piece of the puzzle has slowly but surely dropped into place. In fact it feels like that moment when I felt the object oriented lightbulb switch on after years of procedural programming, only, thanks to this new step forward, I now realize that the OOP moment wasn’t quite the epiphany I thought it was.

The thing about messaging and the patterns around it is that they are not just for distributed systems. Messaging concepts have a place within ordinary applications and I’d go as far as to say that, actually, the idea of the message as an explicit concept in your code is a key component of good object oriented software, one that was originally pushed by the pioneers but has somehow been diluted over the years. I’m pretty sure that this is exactly the idea the that Greg Young in particular has been trying to get across to developers for some time now. Go back and watch some of the videos on Skillsmatter such as Assert.That(We.Understand) where he talks about incredibly rich tests that you could give to a business person and they would see a specification written entirely in English, generated out of a command (message) that ultimately became an event (another message) that he could then use to build that business-friendly output.

He also talks about this concept of passing messages around in a video from Oredev a few years ago called 19 1/2 Things To Make You A Better Object Oriented Programmer. You can find it here, go watch it, it’s another excellent talk.

Once you have an explicit concept defined in your system as a message you can actually do some interesting stuff in your code. As an example, one of the patterns well known in the distributed messaging world is the pipe and filters pattern. This pattern allows for the transformation of a message through a series of filters (message handlers) as it passes from queue to queue over the transport mechanism (pipe) until it reaches it’s final destination at which point the message may well look very different from how it started out. This same technique is easy to do in an application by creating a set of classes that act as the filters for the particular message type. These are chained together so that when the first filter has finished transforming the message in some way its output becomes the input of the next filter and so on. Using this technique I was able to refactor an application that had some quite horrible to read code for basically doing nothing more than importing an excel spreadsheet into a database as a batch job. There were a number of complicated steps along the way that taken together made for one ugly path through the code because there didn’t seem to be a nice way to structure it. However, breaking these steps out into classes, each responsible for doing their bit on the message (the Excel dataset) and then passing it on to the next made a dramatic difference to the readability of the code. Not only that, new steps could also be plugged in at any point in the chain without affecting the other code.

This should feel like nothing new, after all, we all know the benefits of OOP and the principles surrounding it such as SRP, Open/Closed, etc but for all that we still end up working on codebases that seem to completely ignore them. And now I think I know why – messages.

Messages are the concepts that really allow us to unlock the benefits of object oriented languages. We need to cut down on passing around the primitive types and instead think at a higher level of abstraction, i.e. in terms of messaging.

As another example, you can pass a command message to your application and push it through an in-memory bus so that the correct handler is invoked, again because you’ve defined an explcit message. Just imagine how much cleaner your MVC controllers would look by just doing Bus.Send(msg) and having it routed to a handler that acts on it. That same handler could be wrapped with other handlers that “intercept” the message to do cross cutting concerns such as logging, transactions, security, etc. Your controller would neither know nor care. Once again, possible because you made a concept explicit. This is also evident in CQRS in that messages come in as commands, are handled in a way that invoke behaviour on domain objects which then end up generating events that eventually make their way out to event handlers.

This focus on explicit messages has affected the way I write software. I tend to focus much more on behaviour and my code feels cleaner and easier to read because of it. Extension points tend to occur naturally because messages tease out a more object oriented solution. They also make keeping the bigger picture of the software in your head a little bit easier too because instead of thinking about methods and passing primitives around, you can now visualize the flow of a message as it passes through the various parts of your system.

I love those all too rare moments when you see things from a new perspective. I’d heard Greg Young talk about these things long before I grasped the real meaning, always thinking, yeah I’m doing that, etc but never really knowing if I’d understood correctly but at last the light is shining. I think it took the actual experience of working on a distributed message queing system for me to see things in that new light. All of a sudden things clicked and made me want to revisit old code to see if what I had learned could be applied, and of course, I found it could. Messages are such an important concept, distributed or not, that I now wonder why more hasn’t been made of the idea by various authors, practitioners, etc. over the years. Maybe they have and I just didn’t grok it or maybe they haven’t had their lightbulb moment yet? Whatever the reason I think messaging should be given proper consideration as a central component of any moderately complex project. The benefits are many with, as far as I have experienced, few drawbacks, the main one being the required shift in perspective but the effort is worth it.

The Messaging Lightbulb