This article is in line with our latest article «YAKD: Yet Another Kubernetes Dashboard» in which we listed several dashboard projects dedicated to managing Kubernetes clusters. In this article, we focused on development tools and we tried to highlight considerations that everyone should have in mind to correctly identify which tool is the best in which context.
Every SRE, DevOps engineer, developer and Kubernetes operator had to answer this question at least once in their life: what tool would be the best to manage my Kubernetes resources?
There is obviously not a single answer to this question because it depends on multiple factors as developers and operators have a different way of consuming Kubernetes resources. One wants to focus on speed, while the other potentially focuses on security and automation aspects.
Important Considerations
Several considerations should be taken into account when choosing the tools used to operate and consume Kubernetes resources. Operators and developers have, by their trade, a different use of this platform and therefore, potentially require a different client to handle their resources.
Unlike developers, operators have constraints to respect or impose to ensure the sustainability of the cluster and compliance with mandatory security policies while allowing easy access to resource management.
On the development side, the constraints are higher. The aim is to minimize infrastructure constraints and friction so that it accelerates the development of new features.
Identifying and understanding each person’s needs is necessary in order to work harmoniously as a team.
Several considerations must therefore be taken into account when choosing such tools:
- The level of maturity of the teams is obviously an important factor in the choice of the tool. Ease of access to Kubernetes management is paramount to ensure the adoption and proper use of resources.
- User Interface (UI) versus Command Line (CLI), this aspect is generally unique to each one but can be an important factor stemming from the maturity of the teams regarding the management of Kubernetes.
- The frequency of use is very important to take into account because it is a good indicator of the adoption of the platform and the velocity of its users. Two important points to measure to quickly identify improvement points (such as the choice of a CLI tool rather than a UI).
- Following from the previous point, the type of use is also to be taken into account. Developers are potentially looking for a tool that allows them to automatically update remote resources without having to run a single command (live coding) while allowing them quick access to logs to quickly debug any errors.
- As a result of the type of use, the target environment (development, staging, production) is another important parameter to take into account. Access to these environments generally involves different constraints that are important to respect. Roles and access permissions, for example, are generally specific to each environment, thus limiting access to resources and the relevance of certain Kubernetes customers.
- Team collaboration is also a factor in the choice of development tools. Working as a team is important to increase code quality and velocity.
- The components of the integration and continuous deployment environment. Although these components have a limited impact in the choice of such tools, it is important to minimize impacts with continuous chains.
- The complexity of management is an important point. It is essential that the tool can be managed as simply as possible to ensure its adoption once again. Automation and documentation are highly recommended to quickly onboard any new user.
- The security aspect is unfortunately often overlooked but remains essential, especially in access management to limit the management of remote resources and thus minimize the possible impact on existing resources.
- Auditability is obviously an important aspect in the choice of the tool, it is essential for a company to use tools and processes that comply with the security rules imposed by compliant certifications.
In view of these considerations, the objective is therefore not to find a dashboard tool that will give quick access to the status of remote resources but rather to meet specific user needs in order to improve their productivity, their speed, their learning and then the adoption of the Kubernetes platform while respecting the company’s security constraints.
Why Is Kubectl Not The Best Option?
Kubectl is the preferred tool for managing a Kubernetes cluster. Its ease of installation and use makes it possible to manage absolutely all resources on any Kubernetes cluster (local, on-premise or cloud), but it nevertheless has limitations in our daily work.
Developer Point Of View
Today, companies are looking for tools and processes that will make the life of developers as simple as possible so that they can focus on the constant improvement of the application and thus, the competitiveness of the developed product rather than losing time on aspects not critical to the business. The aim is therefore to increase the productivity and speed of the development teams in order to improve the Speed-To-Market.
To achieve this, the tools used must make it possible to facilitate the daily tasks of developers such as the deployment and configuration of a development environment, debugging, accessibility of logs, metrics, live coding, etc.
One of the big advantages of Kubernetes is its ability to generate and logically isolate environments in namespaces. Each developer can therefore have their own development environment without impacting the work of their colleagues.
This isolation requires two types of automation. The first being the creation of the environment itself and the second being the update of deployed resources. The first can be managed by the operationalization teams, the second coming directly from the application development. This is called live coding where each change made locally by a developer is automatically reflected on the remote Kubernetes cluster.
Unfortunately, Kubectl is not able to manage this important aspect of the development cycle today. Kubectl is a very good tool to manage any resource on a Kubernetes cluster but in its rather static way (editing YAML files followed by the execution of command lines) to manage resources potentially results in a significant loss of productivity in a market requiring ever faster production.
From this point of view, Kubectl is potentially not the best option to manage the daily tasks of the development teams.
Operator Point Of View
From the operator’s point of view, Kubectl is probably the preferred tool for managing a Kubernetes cluster. However, when it comes to managing the access of an entire engineering team, the task becomes more complex and can even become a nightmare.
Kubectl requires the use of a KUBECONFIG file that defines the context (cluster, API URL, users, roles, etc.) for all executed commands. This file is usually generated by the cluster security team to limit everyone’s access to what they need to do. This file is then transmitted to the recipient who becomes the owner and can therefore do what they want with it.
From an operational point of view, several considerations must be taken into account:
- Who is in charge of generating this configuration file?
- How to automate the process to facilitate the onboarding of new users?
- How to manage access rights securely?
- How to manage the rotation of these accesses in a secure manner?
- How to ensure that the file is not shared by users?
- How to share this file securely with the end user?
- How to revoke accesses to properly offboard a user?
This process is probably the simplest and riskiest way to give access to a Kubernetes cluster. Once the KUBECONFIG file is sent to the recipient, the security team unfortunately no longer has any control over it and must then actively supervise the accesses to quickly identify deviant behaviors.
What Are Your Options?
Fortunately, today, there are a multitude of tools for operators as well as developers to easily manage their interactions with Kubernetes and above all, optimize the time and actions needed to manage resources.
Again, the choice depends on the context of use and the considerations cited above. The list starts voluntarily with development tools that are very simple to use and ends with more complex concepts of operationalization. The objective is to give a global vision to the development teams and the operations teams of the possibilities and constraints of each.
Dashboards / IDE
A dashboard with IDE functionality (like Lens) is probably the first thing everybody uses to work with Kubernetes and run some tests.
From a developer’s point of view, an IDE can be seen as a perfect tool to manage its applications on a daily basis while quickly accessing logs, metrics and a terminal needed to debug its applications. Unfortunately, an IDE can have a static side in resource management that can lead to lower productivity in development. Live coding and team collaboration, for example, are two missing features of this type of tool.
From an operational point of view, the maintainability of these tools, access management via config files (such as Kubeconfig), extension and rights management, etc., can become complicated. The development of a playbook to configure the end-user workstation finally becomes an interesting option to minimize the necessary actions and ensure a fast onboarding, but this obviously requires proper maintenance and documentation.
Despite this, the dashboards remain very powerful tools, used by everyone for their ease of use. To find a list of available dashboards, refer to our previous post.
Development Toolkit
Unlike dashboards, there are now a multitude of tools that allow developers and operators to focus on what matters most, writing code.
From the developer’s point of view, this means having an environment available (potentially close to production) in order to test in the best possible conditions the features developed and to measure, in real time, the impacts to debug the code accordingly.
From an operational point of view, this means making available to developers a simple and automated way of consuming a resource pool (local or remote) in order to quickly set up an environment isolated from the rest, to minimize impact on the work of their colleagues.
What is expected of this type of tool is its ease of use, to facilitate its adoption. Here is a list of expected features:
- Live coding
- Team collaboration
- Access to real-time logs
- Debugging
- Connection to local and remote clusters
- Decentralization of the development environment (from laptop to remote cluster)
- No need to compile the application locally
- Automated integration in the real dev environment, no need to mock resources
- Scalability, availability and performance
- Easy onboarding as the dev environment can be built automatically on a Kubernetes cluster instead of configuring a laptop, which reduces the tasks to onboard.
Today, some projects try to meet these expectations:
These tools should be used primarily in the development and debug phases. From an operations point of view, they lack features to properly secure and audit their usage.
Identity Based Access
Kubernetes is a platform based on authentication and authorization of REST API requests. This means that each client is responsible for providing the necessary components (service account, role, role binding) to the cluster to allow the request to be executed. Even if a normal user cannot be added via an API call, any user who presents a valid certificate signed by the cluster certification authority is considered authenticated. This type of management involves a lack of control in access management, especially when it comes to auditing and securing the permissions of each user.
From an operations point of view, it is essential to centralize access management in order to easily onboard and offboard any user, manage and audit permissions, and thus ensure a minimum level of security required by all applications used in production.
Identity-based access management is not native to Kubernetes. This requires an agent deployed in the cluster and connected to a remote LDAP dictionary in which users, groups and permissions will be managed.
This type of centralized management is obviously completely different from the previous points. It requires developers and operators to authenticate their client before sending any request but they nevertheless have several interesting aspects to move Kubernetes in production:
- User access controlled by a centralized / external platform,
- Improved access control,
- Auditability of the accesses, required for security compliance,
- Easy onboarding / offboarding of any user,
- No config file to manage, user can connect and get a token to authenticate themselves on Kubernetes,
- Can be used by any local Kubernetes clients, installed locally, that support OIDC.
Today, some projects try to meet these expectations:
This is a subset of open source tools that can be used to manage identity based access of an on-premise Kubernetes cluster. On the Cloud, relying on service managed is also a good option to not increase the technical debt and the maintenance required to keep this tools up to date.
Next?
As you will have understood, there is no tool that meets all the requirements of developers and operators. The use cases differ so much from one company to another that it is not possible to define a single, simple and secure way to operate all the resources deployed on a Kubernetes cluster.
It is up to everyone to find the right compromise and the right tools with limited impacts on the productivity of the teams.
For more information on tools mentioned in this article:
About the authors
Hicham Bouissoumer — Site Reliability Engineer (SRE) — DevOps
Nicolas Giron — Site Reliability Engineer (SRE) — DevOps