The way we think about data and databases must adapt to fit with dynamic cloud infrastructure and Continuous Delivery. The need for rapid deployments and feedback from software changes combined with an increase in complexity of modern distributed systems and powerful new tooling are together driving significant changes to the way we design, build, and operate software systems. These changes require new ways of writing code, new team structures, and new ownership models for software systems, all of which in turn have implications for data and databases.
I wrote an article recently for the Simple Talk website called Common database deployment blockers and Continuous Delivery headaches, where I outline some of the common problems preventing databases from being deployable – a major blocker to Continuous Delivery.
Deployability is now a first-class concern for databases, and there are several technical choices (conscious and accidental) which band together to block the deployability of databases. Can we improve database deployability and enable true Continuous Delivery for our software systems? Of course we can, but first we have to see the problems.
The recommendations include:
- Minimize changes in Production
- Reduce accidental complexity
- Archive, distinguish, and split data
- Name things transparently
- Source Business Intelligence from a data warehouse
- Value more highly the need for change
- Avoid Production-only tooling and config where possible [I mention this in my talk How to choose tools for DevOps and Continuous Delivery]
To address [these things] individually perhaps doesn’t seem too challenging, but to tackle deployability requires close, effective collaboration between developers, DBAs, and operations teams to achieve the right balance between rapid deployment and access to data.
I recently posted a review of Patterns for Performance and Operability by Ford et al on the SoftwareOperability website. I think that this book is exceptionally useful in its treatment of both performance and operability, and anyone who cares about how well software works in Production should buy and read a copy (there are paper and eBook editions).
Two other reviews might be useful too: my colleague Anant East (Head of Architecture and Infrastructure, thetrainline.com) wrote up a detailed review of Patterns for Performance and Operability on the tech blog at thetrainline.com, and I posted a short review on Amazon.
I attended QConLondon 2013 last week; what I took from the first four sessions in the Building for Clouds track was: cloud API and infrastructure automation tools have now solved most of the ‘easy’ cloud problems, but harder challenges (such as automating clusters) remain. The sessions were from Tim Savage (@timjsavage) and Zenon Hannick (@zenonhannick) on Comic Relief’s unique challenges with performance testing, Gareth Rushgrove (@garethr) on how to avoid PaaS lock-in, Stephen Nelson-Smith (@LordCope) on how to use Chef to give you ‘optionality’ with different cloud vendors, and Andrew Crump (@acrmp) and Chris Hedley (@ChristHedley) on the CloudFoundry cloud platform.
When raising log events in code it can be difficult to choose a severity level (such as Error, Warning, etc.) which will be appropriate for Production; moreover, the severity of an event type may need to be changed after the application has been deployed based on experience of running the application. Different environments (Development (Dev), User Acceptance Testing (UAT), Non-Functional Testing (NFT), Production, etc.) may also require different severity levels for testing purposes. We do not want to recompile an application just to change log severity levels; therefore, the severity level of all events should be configurable for each application or component, and be decoupled from event-raising code, allowing us to tune the severity without recompiling the code.
A simple way to achieve this power and flexibility is to define a set of known event IDs by using a sparse enumeration (enum in C#, Java, and C++), combined with event-ID-to-severity mappings contained in application configuration, allowing the event to be logged with the appropriate configured severity, and for the severity to be changed easily after deployment.
I recently went to a Devs in the ‘Ditch meetup at 7digital to hear Chris O’Dell (@ChrisAnnODell) explain 7digital’s journey to Continuous Delivery and Steve Freeman (@sf105) speak on GOOS and system testing. We had some useful discussions on dependency injection and how to use logging well, and Steve’s perspectives on ‘code shapes’ and the purpose of tests were revealing.
Learning from ‘Working Effectively with Legacy Code’ by Michael C. Feathers
Working Effectively with Legacy Code by Michael C. Feathers is a classic software development text. Not only is it still completely relevant when working with code, seven years after it was published, but also the approaches it advocates at the source code level are becoming essential to apply at the component and system integration levels too as our software systems become more complicated and polyglot, and we learn to fully automate our deployments and treat our infrastructure as code.
For an overview of the book itself, read my review of ‘Legacy Code’ on Amazon.co.uk. I describe the different sections and how each is useful in a particular context. The rest of this article will focus on other aspects of Legacy Code, particularly the ‘why’ of changing code, and the implications for Continuous Delivery, infrastructure-as-code, and deployment automation.