Nowadays developers actively talk about DevOps; build pipelines are a reality in most teams and the majority of companies have a well-defined code review and continuous integration processes.
However, the process of releasing software is more complex than just a matter of tools usage. There are scenarios where teams are working in parallel to roll out new features or new business decisions are made from user insights. In order to accomplish those new requirements, releases must happen independently, at any time. The need to release software frequently comes with new challenges, which often requires a massive culture change. This means that all people involved in the release process are impacted. The important thing is that the Software has to work. There are a few practices that could help you to get over with it such as Continuous Delivery.
Continuous Delivery, as Jez Humble and David Farley described in their book, is the ability to get changes of all types into production in a sustainable way.
We can’t deny that every change in a working software involves risk. This might justify the fact that many companies define their own layers of protection, such as deploy windows. Ideally, working software shouldn’t change, or it should be stable. However, we can’t avoid change and we must find alternatives to mitigate this pain. The key to achieve predictable and sustainable releases are doing small and frequently. The bigger are the changes, bigger are the risks. That’s why we encourage to roll out small chunks of code as possible, which takes less work to fix whatever might goes wrong or give us a buffer to try out new experiments.
A common mistake that some people make is deal with CD as an achievable thing but it’s not. Continuous Delivery is a journey that you’re improving over time, learning from mistakes and adapting to what make sense for your business, embracing and challenging the restrictions.
The cultural shift to start practicing CD is hard to tackle without some ground rules in place. Ensure you put some thoughts and build a plan to achieve the requirements below:
Continuous deployable state
For this post, let’s focus on how to ensure our code is always in a deployable state. One of the techniques that enable it is called feature toggle.
Feature toggling is a technique that allows us to turn one or more features on or off. It gives us the flexibility to be in control about what’s being launched out of the door. A feature can be disabled in case of unexpected behaviour or it can be used to explore different scenarios (e.g. A/B testing).
There are different categories of feature toggles:
- Experiment Toggles: a proof of concept or just an A/B test at the user level.
- Operational Toggles: useful when rolling out a new feature which has unclear performance implications. Also applicable if you want to test a specific scenario in production that uses a new piece of infrastructure.
- Seasonal Toggles: to handle seasonal events, promotions, Olympics, world cup, Black Friday. It can be done at the user level and at the application level.
- Release Toggles: allows in-progress features to be checked into production at any time while new releases happen as normal.
From those categories, we’ll put some highlight on Release Toggles category, which is more related to our goals within Continuous Delivery Journey.
In order to make every commit production ready, feature toggle is one of the techniques that allow in-progress features to be checked in while still allows that codebase to be deployed to production at any time. It also means that we’re separating release from deploy.
It’s common to associate Release Toggles with the trunk base development but it’s not a mandatory. For the following examples, we’ll use feature branches instead. There’s no one better than another, it depends on what your team needs, what are your restrictions, what matters is to have an effective way of communication at some sort and deliver working software.
Consider we have a team of 2 developers and each of them is working in parallel with their own feature branches on different pieces of the software. Let’s say we have a release planned soon, so we intend to ship it once they’re done their jobs.
What if we find a bug after a merge with the second branch? What should we do?
Revert the merge?
Revert a specific commit?
Revert all the things?
Dig in to find the bug…
I’ve been in a situation where an entire week of work turns out to trash. Beyond the changes that we’ve made, a database migration implemented by another person didn’t work as expected and we all were working on the same codebase. We had to postpone the deploy and we also were unsure if we could push more changes until the fix is made.
Changing the scenario, we’ll consider to adding a toggle for every change. Let’s remember that the release process is everyone responsibility.
Toggle OFF, start to commit/push your changes.
Release it, test it (e.g. different environments, limit the range of users).
Then you can toggle ON.
If you notice a critical bug or some unexpected behaviour, instead of rollback or revert all the things, consider roll forward, turn the feature off and take your time to fix it.
The importance of toggle recycling
A common problem for teams starting with feature toggles is that the toggle code is never removed.
Keeping them will transform your software in to a complex system that is hard to maintain. As the time goes on and new members join, no one will know who or what depends on that code anymore. The impacts of a possible change will be unknown and adding new feature toggles can become a risk.
When is the right time to remove them? After the release, test it and see how it goes on production. At some point you need to make sure you removed all released toggles, so consider this: try to change your mindset of "Done" concept and start thinking on “Hypothesis validated” instead. Once you collected enough data to validate your hypothesis, go back to your code, remove the feature toggle relating to that release and keep your codebase clean.
A code example on Github
You can follow a code example on github which shows the example of a heart monitor. It is a quite simple way to simulate how would we apply feature toggling (and recycling) at the application level. It’s organised by branches, so the master branch will contain the final state and you can travel in time through the branches feature#1, feature#2 and so on.
You can also keep track of the feature toggling journey through the commits.
Some developers hate the idea of the feature toggles because it's really tempting to have a chain of toggles elsewhere. Make the best out of feature toggles by considering to
have a good separation of concerns, keep your feature isolated as possible,
aim for a design that can be modular where you can replace it anytime.
use different techniques like Branch by Abstraction that enables you gradually make changes in a total decoupled.
Beyond adding conditionals to the code, the same concept can be applied having different Microservices in general. Have you imagined how easy would be to flip from one Serverless function to another? You might also consider moving towards an Event Driven Architecture and have all the flexibility that feature toggles can provide by having some sort of event subscription handling.
Get in touch!
Learning and sharing is important to us at Pragmateam. We regularly visit other organisations, facilitating learning lunches to chat through our observations and common challenges. We hope you have enjoyed this blog and welcome the opportunity to visit you. Get in touch today - firstname.lastname@example.org.