As we all know and firmly believe, applications and infrastructures need to be secured, but the shipping processes of this whole ecosystem also needs to be.

In a previous article, we introduced GitOps as a methodology to improve the velocity of the development and the management of an entire infrastructure. But there are many other benefits from GitOps, and one of them is the potential improvement of the security.

Improve the security with GitOps

Security is the concern of everyone, everywhere. It's probably the only aspect that impacts everyone in a company at a different level. It never stops to be improved and that's why today, it is important to consider it as soon as possible in every new project to limit the risks when the project goes live.

As with many other aspects, DevOps has had an impact on the security management. It introduced the DevSecOps mindset and the security as code concept. DevSecOps wants to take full advantage of the agility and responsiveness of a DevOps approach by integrating security into the application lifecycle. If an efficient DevOps approach ensures rapid and frequent deployments, outdated security practices can negate the benefits of the most effective DevOps projects.

DevSecOps asks everyone to be responsible for security with the goal of safely distributing security decisions at speed and scale to those who hold the highest level of context. The idea is to think of security as a shared responsibility, integrated from the beginning to the end of the project, from the development to the infrastructure needed to host the applications.

GitOps obviously improves the security in many different ways during the lifecycle of an application:

• Single source of truth, as it allows to centralize all the configurations in a declarative way.
• Configuration as code, so no manual actions are made on any ecosystem.
• Auditable changes through leveraging the VCS features.
• Improved observability thanks to the collected metrics through different processes as we talked about in one of our previous article.

GitOps finally introduces the Continuous Security concept needed to run on every iteration, to catch errors and vulnerabilities at an early stage and eliminate them, long before any code is deployed to production.

Obviously, GitOps doesn't solve everything. As every new concept, it requires a specific attention on the management of the permissions and the sensitive data.

Permission management

GitOps is a suite of processes that enables the current state to converge towards the desired state. Having everything in Git allows you to leverage the features of the VCS providers regarding permission management.

Depending on how well organized your ecosystem is, you will be able to assign ownership, quality gates and give different types of access.

First of all, since everything is stored in Git, pull request reviews are going to be the first step to permission management. Without approval(s), no modification to the desired state will be allowed. Regarding ownership and accountability, we can define code owners and make sure a particular change would be audited by a certain team; or even a different team depending on the environment.

Multiple profiles could be considered like an operation or platform team with admin access, development teams with write-access to only certain files/folders or even a management team with read-only permissions handled through your Git tool. Sensitive data management

Keeping unencrypted confidential data in a version control system (VCS) like Git is widely recognized as a bad practice for many reasons. Once source code enters a git repository, it can be spread into multiple locations, and that potentially means losing control over sensitive data.

The management of this kind of data is always a challenge and subject to multiple talks and refactoring. Can we rely on a basic encoding like the Kubernetes secrets? What kind of encryption is required for security compliance? Should the confidential data be managed by git or an external tool? What is the purpose of encrypting data with a token if I cannot secure the access to this token?

When it comes to keeping sensitive information confidential inside a GitOps approach, choices can be complicated. The sensitive data has to be controlled in each step of the GitOps workflow to ensure that the data is not compromised. Multiple GitOps-friendly alternatives that preserve confidentiality have emerged to facilitate the management and the consumption of any kind of sensitive data.

But why exactly storing encrypted confidential data in a Git repository?

• To have a revision history of changes.
• To ensure that your secrets are backed up and will remain intact even if the environment is compromised.
• To secure the access to the secret.
• To uniform the way sensitive data and source code are managed.

Choosing the right solution for the real use case is not easy and depends on multiple factors like:

• The complexity of the installation and the management of the solution.
• The maturity of the product (even for a cloud based service).
• The features required by the workflow.
• The security aspect to fill the compliance requirements.

Hashicorp Vault remains a secure and widely adopted solution. HashiCorp Vault is a secrets management tool specifically designed to control access to sensitive credentials in a low-trust environment. It can be used to store sensitive values and at the same time dynamically generate access for specific services/applications on lease. Combined with a Vault secret injector, this solution is really powerful. In the context of GitOps however, it would distort the git as a single source of truth principle.

SOPS (Secret OPerationS) is a widely used, well-supported open-source tool (developed by Mozilla) that can be used to store secrets safely in a public or private Git repository. Basically, SOPS is a command line tool that can encrypt any value of a file leaving the key readable to keep an understanding of how this sensitive data is used. SOPS is currently well managed by multiple GitOps tools, like Fluxv2 or ArgoCD, to easily manage the confidential data in the GitOps workflow.

Many other tools exist like Kamus or Bitnami Sealed Secret, each one deserves a chance when it's time to update or change the sensitive data management. One recommendation is to keep things as simple as possible. Unfortunately, people used to not be involved in the life cycle of sensitive data when the process is complicated or requires too many steps.

Audit the GitOps flow

Auditability can generally be defined as the examination of certain records in order to verify their accuracy. In security, auditability has a slightly different meaning, and can be defined as the systematic evaluation of a process by measuring how well it conforms to a set of established criteria.

GitOps empowers teams and companies to automate the whole continuous deployment flow. But as we all know, the implementation of an infrastructure or applications does not just end when everything is deployed to production. Being able to audit a system is one of the most important things, especially when it comes to the process that enables you to deploy your whole infrastructure and applications. Knowing that is important, but what kind of data are necessary to be able to audit a GitOps flow? How to store all these data? How to monitor the changes and how to graphically represent them? To be able to do so, one needs, as mentioned above, a set of information to examine, analyse and compare; monitoring, traceability and logging, as specified in our previous article regarding Observability and GitOps, will be the principal elements that will facilitate auditing the GitOps flow.

The first tool that comes to mind regarding auditability for GitOps is obviously Git. Indeed, Git stores all the logs related to any modification that has been made to the source of truth; this way, you are able to identify the changes brought to the desired state. GitOps auditing comes in many forms and that is why Git logs can be enriched with custom metadata and stored in the cloud.

Also, depending on the nature of the tool you use, you can gather information based on your process. For instance, if you use Flux created by WeaveWorks, based on a pulling methodology, it automatically comes with a dashboard based on Prometheus and Grafana that allows you to understand metrics such as reconciliation time and status for each application that needs to be deployed. In that sense, you would be able to know the status of each application through your Git logs and also through your Flux dashboard.

Again, if you are using a push-based solution like Spinnaker, where you can implement a certain flow, you can measure and store some metrics for analysis throughout the entire lifecycle of your pipelines.

Next ?

Security is obviously everyone's business. It greatly depends on the context, constraints and goals that each one sets.

The purpose of this article was to provide an overview of the security in the GitOps methodology and present some tools that can improve the management of confidential data. Our next article will put security in context. Stay connected!

For more information on the security in GitOps methodology:

• Securing GitOps pipelines
• Tops 20 Dockerfile best practices
• Inject secrets into pods

About the authors

Hicham Bouissoumer - Site Reliability Engineer (SRE) - DevOps

Nicolas Giron - Site Reliability Engineer (SRE) - DevOps