Put a service on it: How web services power Getty Images
I am constantly amazed with two aspects of our Editorial Tools platform: the size of our system and how well structured it remains after five years of coding.
We all know good coding practices are important. Decades of design patterns, SOLID principles and TDD teach us how to write better code. But what is often overlooked is how this all comes together to build complex, multi-faceted, interconnected systems that stand the test of time.
I had a naive understanding of Enterprise Systems — even after reading Fowler – until I realized we had built one and I saw the core principles applied in practice. These patterns are weaved into a system we still love with thousands upon thousands of business rules powered by a “why bother counting” amount of code. It is ever-changing, sometimes breaking but always running; a global system that does–and no matter how much we add, will always do — almost everything.
Don’t get me wrong — five years of coding on the same suite of systems does not leave one without skeletons in the closest. Despite dozens of independent sub-systems, thousands of classes, refactoring efforts, performance optimizations, and scalability challenges, I see the same overall structure, same effort to define service boundaries and consistent approach to tiered applications throughout the system. The patterns existed from the start, evolved with time, and saved us time and time again. The service-oriented approach, pioneered by our core architecture team, is to layer services on top of each other following a simple set of rules. Each tier follows a pattern of aggregation and facades to the layer below. The approach creates both well defined boundaries between systems and creates predictable layers of functionality across all our systems.
A Service-Oriented Architecture
Adopting an architecture focused on interconnected services has paid itself off with dividends. I remember writing our first full-stack feature and wondering why the hell we were doing SOA. Our simple mandate was “Build a service.” There were so many layers to go through and a lot of repeating properties — adding a field to show on a UI was not a one line change when traversing service boundaries. However, what eventually emerged is that we have two distinct levels of service: Basic and Process.
A Basic service is a self-contained entity with no dependencies outside of itself. It is used for data storage and enforcement of validation rules for core entities. Basic entities follow the simple mantra:
“The Widget entity is managed by the Widget Service. You can do anything you want to do with a Widget, but the source of record is the Widget Service. Widgets are always stored and ultimately referenced from the Widget Service.”
The Widget service only cares about itself. It enforces its own validation, but it does not care how it is used since that would involve knowing how it interacts with other entities, violating a Basic service rule.
A Process service is where we enforce business rules, control flow, and aggregation of various Basic services. This allows us to prevent tight coupling of disparate entities that would prohibit agility. By abstracting functionality into Process services built on Basic services we can create a suite of systems each focusing on a specific purpose.
“Users can buy Widgets from the Widget Ordering Service. The Widget Ordering Service interacts with the following Basic Services: the Widget Service, the User Service and the Shopping Cart Service.”
Basic services may be referenced in dozens of Process services- but those Process services work on a specific problem domain and can be tuned for that domain with little overhead or tradeoff. In the above example we may also want to sell Gizmos in addition to Widgets. The rules for selling Gizmos may be different from Widgets, but they still involve Users and a Shopping Cart. A Gizmo Ordering service allows us to fine tune the rules for selling Gizmos without involving Widgets. Basic services do not map to individual entities; they represent something similar to an Aggregate Root but are not as strict. A Basic service is a set of entities that are so closely related boundaries cannot exist between them.
The Basic plus Process approach creates visible system boundaries which allow us to refactor and scale dependencies without worrying about unnecessary consequences. As long as our service contracts remain the same, we can do whatever we want under the hood. By taking a defensive first approach to service dependency, a dependent service can be down without affecting the whole. This provides great flexibility in deploying code and controlling cascading effects with unplanned outages.
Another important benefit of the service-first approach is that it allows the service to be consumed by anyone. We’re seeing this in the industry today where one API powers a company’s a website, mobile apps and partner integration. At first a website site may be the only consumer of the service. Five years later, one is working with teams in their organization that did not exist when the project started. Additonally, one is engaging with customers in ways which did not seem possible years ago. Having a service-first mentality allows teams to engage with minimal friction.
Defining service boundaries are always challenging. Your natural tendency is to push everything down into a single service. You struggle with aggregating domain objects, when to cache and performance optimization. You constantly want to reuse code when working with the same service in multiple projects. These problems get easier with time, experience, good framework tools and automation.
Repository patterns, inheritance techniques and tools like AutoMapper allow you to structure layers and transform data objects between them. A top-tier layer may be an aggregate of a view Basic service, or may need to represent itself in a couple of different of ways to layers below. We use AutoMapper to map a complex object into a simple one for serialization and leverage the repository pattern for the implement it. It requires more code, but we find it’s always better to have a service. Where we have gotten into the most trouble is when we did not expose functionality as a service.
One of the interesting consequences of the site + service approach is the plethora of layers it creates from top-to-bottom. Most people consider N-tier a simple three-tier solution. However, the site + service approach creates lots of layers. This is a good. A well built system can function without one — or any — of its base layers. In a similar way to how ViewModels have evolved to aggregate various Models for Views, tiers can aggregate functionality in rich domain objects which would be too complex to exist in a single holistic system. This allows consumers to cater functionality to match specific needs without worrying about other systems in the ecosystem. The layered approach also allows consumers to dip into any part of the stack they want too; from Basic services to the richer Process services built on top.
SOA is the Object Oriented Programming for Enterprise Systems
In the past five years of Editorial Tools (which is one of just many systems adopting SOA at Getty), SOA has helped us do things we never planned for nor could have done easily when needed. Combining entities from disparate places forces loose coupling and allows fine-grained control on system behavior. At scale this is invaluable, allowing us to deploy functionality into production as if we were hot swapping classes. You can subdivide and reuse functionality without worrying about DLL hell. Layers allow you to isolate unit tests for better code coverage and provide defensive programming strategies to deal with failure. Mapping teams to services allow for better organization for large teams. After five years of coding the same set of systems the mantra is simple: “Put a service on it!”
Editor’s note: The author of this post, Michael Hamrah, is the Director of Engineering for Editorial Technology in New York. You can follow him on twitter via @mhamrah.