Migrating ‘Mature’ Code to Continuous Delivery
Originally posted on TechBeacon
When starting up a greenfield project, it's easy to take advantage of the most modern development practices. An engineering team can create a new project in Azure or AWS and with a few clicks have it auto-deploying onto their platform of choice without a lot of fuss.
But what about the rest of us, who are working on codebases greater than five minutes old? How do you take code that's four years and hundreds of thousands of SLOC, and turn that into a lean, mean, continuous-deploying machine? How do you build a roadmap to go from a product that may take weeks or months to deploy out to customers into one that may deploy several times an hour? How do you ensure that your product that’s deploying several times an hour is doing so safely and sanely, and isn’t shoveling bad patch after bad patch onto your production system?
There’s more to the process than simply ‘go slow and iterate’. It starts with identifying clearly and honestly where your code base and team are with regard to deployments and knowing where you want to go. Not every project will be appropriate to continuously deploy to production, and that’s okay! Any step taken down the road to automate the build and deploy pipeline will pay dividends in overall developer productivity and code quality.
But what are those dividends? Why does continuous delivery matter? Nicole Forsgren, Jez Humble, and Gene Kim recently published a book entitled ‘Accelerate’, which enumerates on a number of studies they have carried out over the last several year, to track performance metrics - ranging from company success to employee retention - across organizations with varying levels of continuous delivery practices. Amongst the tangible benefits that were statistically significant, they found specifically that high performing organizations:
Had a 50% advantage in market capitalization growth over three years as compared to low-performing organizations
Spent 50% less time remediating security issues
Were 2x more likely to be recommended as ‘a great place to work’ by their employees
What does it mean to be a high performing organization? And how does an organization transition from low performing to high?
Pillars of Continuous Capabilities
In continuous delivery, there are four pillars of capabilities necessary in order to achieve full continuous delivery, each with distinct stages.
The four pillars are:
Source Control – does the entire team use source control? Do they have practices that are conducive to continuous delivery, such as GitHub flow or trunkline development?
Software Quality – are there automated tests? Do they pass reliably? Do they extend beyond simply unit tests and into the more specialized testing areas? Can they be run at will?
Deployment – are production builds created and deployed entirely by automation? Is there infrastructure as code to allow predictability and maintainability in the deployment architecture? Can individual services be deployed individually, or does everything deploy as a monolith?
Monitoring – do we collect logs? Do we collect metrics? Are they set up to alert for failures? Do we have traceability for failures across the entire stack?
It’s important to know not just what stage your project currently is in, but to know what it means to be in a stage. At each point, significantly different types of work are needed in order to advance to the next stage in the process, and each stage is correlated with a reasonable delivery strategy along the spectrum from fully-manual to fully-automated.
Similarly, it doesn’t make sense to try and tackle the work from a far-future stage if you have not yet cleared all the requirements for the most immediate next stage. There’s not a lot of benefit to tracking your UI code coverage metrics across builds if you are not even reliably running your unit tests during those builds.
This is very much a new-and-evolving field of software engineering. Processes that were previously imaginable at only the very largest and most engineering-wealthy companies such as Google and Twitter are now achievable for those of us at small and medium-sized organizations as well.
Each practice has been broken up across four stages: a score of 1 indicates that this practice is relatively immature, while a 4 indicates an extremely mature practice based on currently available technology.
Source Control Practices
A source control system might exist. Maybe. Probably.
A source control system definitely exists but has inefficient use of branches – either too many with deeply nested levels, or else too few. Merges might be a complex and fraught practice.
Developers generally use a current source control system and follow a defined practice, such as GitHub Flow. Branches probably live a week on average, and merges are medium-to-large.
All developers practice trunk-based development with frequent, easily digestible commits to master – perhaps even daily.
Quality Practices
Very few automated tests, and definitely no automated test runs. Not all tests pass. The most common refrain is, “It works on my machine!”
Automated unit tests exist and are run with every merge to master and are generally reliable. Integration/service tests may exist but are probably manual or else small in number.
Test metrics such as code quality are routinely collected and tracked and used as gates to merging commits. Integration/service tests are automated and reliable. Specialty tests exist but are probably run manually.
Specialty tests such as performance, security, etc. are run automatically. All test passes are automatic and can run on-demand for any change at any state in development.
Deployment Practices
Builds are sometimes created and deployed from a developer’s machine and pushed into production.
Builds are created from an automated build system and are deployed from within the automated system. There may be some manual steps, but they are of extremely limited scope and count.
Everything is created automatically - the only manual steps are no further than ‘push deploy button’ or ‘push rollback button’. Builds are likely monolithic.
All components are built automatically and are able to be deployed independently from the whole. There are no manual steps in a deployment beyond pushing the button. Builds are fast enough to run multiple times a day.
Monitoring Practices
Logs exist on individual servers but aren’t aggregated in any way. Tracing issues is a problem.
Logs are semi-structured at least, and are centrally aggregated for searching across machines and environments
Metrics / counters exist to answer aggregate questions about system health and to supplement server logs providing deep dives on individual issues.
Full observability achieved; logs are well-structured and appropriately leveled. Counters are standardized. It is possible to readily trace a request across many different parts of a system.
Putting It Together
In order to build a reasonable roadmap, an organization must first and foremost clearly determine at what stage they are currently operating at. This involves taking an honest survey of the displayed capabilities at each of the four pillars. Ideally, a team will be evenly spread across all capabilities. Practically speaking, however, most orgs will find that they have an uneven spread of abilities. Perhaps they are heavily invested in their quality technology – a 3 – but have put very little effort into their monitoring story – a 1.
The team must then decide where they would like to go. It is neither appropriate nor necessary for every single software organization to achieve full, continuous, automated deployment with every single change releasing immediately to production. Many organizations will be satisfied with a level 3 functionality, where it is straightforward and routine for someone to ‘push the button’ to ship a change or set of changes live.
Once a team knows where they are, and where they would like to go, it is a straightforward thing to determine what needs to be accomplished in order to arrive at their destination. Consider our team from the previous example. They have determined that a push-button deployment is an appropriate goal given their stakeholder concerns and investment they are willing to make (delivery stage 3). Assuming they have self-assessed to the following scores:
Quality: 3
Monitoring: 1
Source Control: 3
Deployment: 2
With their current scores, they can appropriately deliver software at stage 2 (early automation). If they are not currently delivering that way, then with no additional improvements upon the pillars, it would be safe for them to implement up to this level. If they wish to move on to push-button deployments, however, their course of action becomes clear. The most ‘bang for the buck’ tends to come from moving from a level 1 to a level 2 capability. Thus, they should begin by improving their monitoring practices. They would get relatively little additional utility for choosing instead to, say, improve their source control practices (from level 3 to 4).
Wrap Up
For an organization that is early in their continuous delivery journey, deciding where to begin can seem a daunting task. However, if they take a step back and can evaluate not only their current capabilities, but also how much investment they are willing to make, it can be straightforward to build a roadmap that will get them, step by step, to whatever destination they desire.
