Velocity in agile development measures the quantity of work a team can accomplish in a sprint. It can be measured in story points, hours or days. The higher the velocity of a team, the more features it delivers, the more value it brings to customers. Sprint velocity is a good measure in sprint project management to evaluate and estimate team productivity.

The measure of the velocity is based on multiple factors: the continuous integration (CI) process, the time to qualify the code changes, to test the regression, the security, the delivery, etc…

In today’s world, product managers want features as soon as possible in production so the processes and the tools have to follow the velocity of the development. If the velocity of the development team is faster than the qualification of the changes, the result will probably be a delay in the deployment of the new features and potentially a decrease in the competitiveness of the company in a market that moves rapidly.

Fortunately, to avoid that, multiple strategies exist today like the feature flag. They are an important part of a broader strategy to have an application that is adaptable to both technical and market needs.

What is Feature Flagging?

The feature flagging concept (also called feature toggling or feature switching) is widely credited to Martin Fowler, Jez Humble, and Facebook’s Engineering team. Within the context of continuous delivery, feature flags provided the foundation for a framework that would allow developers to release software faster with less risk and more control.

From a coding point of view, feature flagging can be seen as simple as an "if" statement which wraps around a new piece of code. When the "if" statement evaluates to true, the feature flag is on, then the new code is applied.

In a real software delivery example, the 'if' statement described above would be evaluated differently depending on the environment in which the software is running. For example, if the application is running on a QA server, the feature flag will return true and the new functionality will be seen by the QA team to qualify the functionality. But, if the code is executed on a production server, the feature flag will return false and the functionality will be hidden.

Feature flags play a key part in CI schemes where features are constantly being deployed but not necessarily "released" into production.

Since 2018, a lot of organizations use this concept in production. Their goal is to get the best from the product management team and the development team by improving the link between product management and release management. The idea is to defer the decision on questions related to the runtime as part of development strategy, DevOps strategy, and product strategy. Those questions can be:

Who should see the new admin screen that we're rolling out, and when?
When should we move to the new database?
Should the new button appear to every user? Should it be dedicated to a specific customer?

Feature flags allow you to enable features remotely on an application without needing to re-build/re-deploy the application. It allows teams to deploy the code to production but not release the feature until it has been qualified and defined as ready for production. Feature flags can target specific users to easily reduce the customer impact and run tests in production on a subset of beta testers. One nice side effect of it is speeding up the development team velocity.

They're bits of code that [allow] teams to modify system behavior without changing code. -- Martin Fowler

With that being said, feature flags are not the right solution in all cases and does not resolve all the issues in a delivery pipeline. Be careful in the implementation of it, or you may end up with a delivery pipeline more complicated than it needs to be.

FEATURE-FLAGS (1).png

Why is Feature Flagging important?

Feature flags are important in many ways. The top one is probably to decouple software deployments from feature releases. It brings the ability to push feature’s commits to production while choosing who amongst the customers gets to see that feature. Rather than managing features with branches in a source-control management tool (SCM), features can be turned on or off on the fly in a specific environment, allowing teams to release code with features that shouldn't be enabled yet.

Embracing this simple idea lays the foundation for many best practices, including:

Continuous deployment/delivery: multiple code pushes to production in a day.
Trunk/Mainline development: feature branches should be created only for pull requests, not for long lived feature development.
No more release trains to bog things down.
QA/Perf testing in production: real QA and performance testing is on production infrastructure with production traffic. Don’t waste time building extensive performance labs and staging environments.
Experimentation: know how a new feature moves the needle on KPIs.
Avoiding hotfixes or rollbacks when problems happen: both hotfixes and code rollbacks are stressful, take a long time, and lead to more problems than necessary. Instead, turn the feature off.

One goal is probably to defer important decision logic in the code until runtime where we used to have more information. A second one is to remove the dependency between the marketing, the product and the engineering team. Continuously deploying code with the ability to turn on and off features is better than developing a process to manually control the features that are ready for production.

Many people are against the idea that testing in production is good and they were right, but this new approach eliminates the risk introduced by code testing in production and potentially avoids the management of a dedicated environment for QA.

Before using feature flags, the all or nothing concept was applied, meaning either all customers got the feature or none of them did. So the risk of testing code in production was high. Feature flags allow the user to roll out a new feature to a small percentage of the users at a time. This allowed them to collect valuable feedback and data about a new feature without risking any potential issues to the entire customer base.

One other interesting aspect of the feature flags is the ability to create kill switches during heavy workload to improve the performance of the application. Kill switches are a safety mechanism used to shut off a feature during an emergency, when it cannot be shut down in the usual manner.

So, why is feature flagging so important ? Because it allows us to roll out new features gradually, introducing new features to power users first, getting feedback from them and improving it before rolling it out to all users without multiplying the environments.

How to implement Feature Flags?

Since the introduction of the feature flags by Martin Fowler, the concept has been improved in different ways.

Today, feature flags can be added to the code in multiple ways:

Create your own library.
Use a third-party feature flag library, consider maybe an open-source project.
Use a cloud-based feature flag management service.
Use a config file included in the deployment package, keep in mind that it should not mean that the application has to restart to apply the config.
Use a database and load the data at run-time.

Adding feature flags to an application or developing your own library means adding complexity and code that has to be maintained. The worst result of that would be to increase the technical debt and it is obviously not the objective.

So, one goal is to have a comprehensive feature flag management strategy, which includes a few components:

Toggle points used to switch behavior for new features.
Multiple toggle points come together to form a toggle project that determines the state of a project.
Toggle context provides the necessary contextual information (environment, specific user, etc).
A naming strategy to easily identify and understand the goal of a flag
One or multiple activation strategies to enable a feature for specific users, for a small subset of users or IPs.
A toggle report to quickly identify the potential technical debt introduced by the feature flag management
An history of each changes to ensure a quick rollback if needed

The feature flag implementation has to be controlled and monitored to be efficient. A process has to be defined to enable a feature in production without impacting customers. An audit has to be done regularly to avoid stale features and keep the code clean.

Unleash your Feature Flags

Today, many organizations use it in production and some of them have developed their own implementation and sell it as SaaS like Floodgate, LaunchDarkly, Split, etc… But there are also some interesting open-source projects to easily implement the concept like Unleash.

Unleash is an open-source feature management platform based on a client/server architecture. The feature toggles are evaluated in client SDKs which run as part of the application and support all the popular languages (Java, NodeJS, Go, Python, etc). The server side is composed of an API and a UI, the API holds all feature toggles and their configurations, the UI is composed of a dashboard used to manage feature toggles, define new strategies, look at metrics, etc.

The architecture can be easily deployed and scaled on every container orchestration platform (like Kubernetes ), on bare metal hosts and even in the cloud thanks to the Enterprise version.

Unleash (the free plan) covers most of the main features required to easily manage the feature flags like:

Boolean feature toggles to easily enable and disable a feature via the dashboard.
User management with role based access control to open access to the owners of the difficult task to enable a feature.
Canary and targeted release to improve the continuous deployment
Custom activation strategies to enable a feature only to certain customers
Experimentation (A/B testing)
Kill switches to automatically disable feature in case of heavy load
Audit logs, because observability is important
Add-ons integrations with other popular tools (Slack, Teams, Datadog, etc.)
Dashboard to manage technical debt

This is just a subset of features that Unleash provides in the free plan that can be deployed almost everywhere.

Unleash is definitely a good start to introduce feature flags to an existing application. It makes it easy to implement and to manage via the onboarded dashboard. Give it a try !

Next?

As mentioned earlier, the feature flag is not the answer to everything, it introduces multiple benefits to improve the continuous delivery and deployment pipelines by decoupling the velocity of development from the product lifecycle.

The concept is powerful but it requires to be controlled. It requires for example a control and a process to enable a new feature (who can enable a new feature? How is it tested? How is it monitored?). Feature flags require a specific attention from the development team to not abuse it and avoid increasing the technical debt. Relying on existing tools like Unleash is also a good beginning to reduce the potential technical debt that feature flags sometimes introduce. Keep it clean and don't abuse it!

For more information on feature flag principles:

About the authors

Hicham Bouissoumer - Site Reliability Engineer (SRE) - DevOps

Nicolas Giron - Site Reliability Engineer (SRE) - DevOps

KumoMind's Blog