The Hidden Costs of Homegrown K8s Management

Alexander Fridman

August 17, 2023

Minute Read

In recent years, Kubernetes has become the standard in container orchestration.

Originally designed by Google, it outran its competition. It's much more popular than OpenShift Container Platform and Apache Mesos. In addition, it's significantly more popular than Docker Swarm, a product developed by Docker, the company that developed the most popular containerization technology that has the same name.

There are several options to deploy a Kubernetes cluster, including public cloud, private cloud, and a homegrown solution either on-premises or in the cloud.

This post will highlight the hidden costs of the homegrown management of a Kubernetes cluster.

A Homegrown Cluster

A lot of companies tend to distance themselves from managed solutions for Kubernetes. The rationale is usually that a homegrown solution will provide them with better control over functionality and costs.

Let's explore the benefits of a homegrown solution a bit further.

Benefits of a Homegrown Kubernetes Deployment

Tailored Solution

With a homegrown Kubernetes deployment, you have complete control over the configuration, deployment, and management of your Kubernetes cluster.

This allows you to customize the setup to meet the specific needs and requirements of your applications and infrastructure. You can implement custom features, workflows, and integrations that may not be available in managed Kubernetes services or vendor distributions.

Cost Control

By building and managing your own Kubernetes cluster, you have the ability to optimize resource allocation, scale based on your specific needs, and potentially reduce costs associated with managed Kubernetes services or vendor distributions.

Basically, the idea is that they get a tailored solution for their needs at the minimal possible ongoing cost.

However, there are some caveats to those assumptions, as presented below.

Hidden Costs

Development Costs

Building a homegrown Kubernetes platform requires a significant investment of time and resources, including the development of custom tooling and integrations.

You need experienced DevOps engineers to set up the control plane, configure networking, and so forth. Once you integrate the cluster, you need to make sure it plays nicely with the other systems in your LAN or network.

Moreover, those costs are just for the initial setup; we haven't yet talked about the ongoing costs.

Maintenance Costs

A homegrown Kubernetes platform requires ongoing maintenance and updates to ensure that it remains secure and stable.

You need to patch the deployment for security and version updates. In addition, there are usually changes to be made to the networking via the service mesh. Likewise, node load can require changes in the auto-scaling configuration.

Security and Stability

In general, it'll be hard for a homegrown K8s solution to achieve the same level of availability and security that managed solutions boast. This makes sense, as a managed solution is essentially a product designed by a company specializing in it. It's their bread and butter to make the cluster available 99.99% of the time and to mitigate any security issues as soon as they arise.

Usually, other organizations with a core offering that's not a managed Kubernetes cluster can't dedicate the required resources to achieve anything even remotely near those numbers.

Furthermore, in terms of security, they don't have a SOC team that is strictly dedicated to managing Kubernetes. This, in turn, makes sense too, as the security and operations teams have multiple responsibilities. The core one is supporting the main product the company develops.

Common Security Risks

Misconfiguration—Misconfiguration of Kubernetes resources, such as pods, services, and ingress rules, can introduce vulnerabilities that can be exploited by attackers. For example, exposing sensitive information through misconfigured environment variables, using insecure network configurations, or allowing overly permissive access controls can result in data breaches.
Lack of proper authentication and authorization—Inadequate authentication and authorization mechanisms can lead to unauthorized access to the Kubernetes cluster. This may include weak or default credentials or improper use of privileged accounts. These vulnerabilities can allow attackers to gain unauthorized access to the cluster and potentially gain control over critical resources.
Lack of up-to-date expertise—Kubernetes is a rapidly evolving technology with frequent updates. As stated above, those are security patches and best practices. A homegrown Kubernetes deployment may lack the expertise or resources to consistently keep up with the latest changes.

There have been several incidents in the past where homegrown Kubernetes deployments were compromised.

For example, the Tesla Kubernetes cluster was exploited in 2018 by attackers who exploited a misconfigured Kubernetes pod to mine cryptocurrency.

Another example is the Verkada security camera breach in 2021, where attackers gained unauthorized access to a Kubernetes cluster, leading to the exposure of sensitive data from thousands of cameras.

Likewise, in 2018, SoundCloud faced a major outage due to a misconfiguration in their homegrown Kubernetes deployment, resulting in downtime and loss of service for several hours. This incident highlighted the importance of robust operational processes and thorough testing in homegrown Kubernetes deployments.

Standardization

Standardization is also an important consideration when it comes to building a homegrown Kubernetes platform. Without a standardized approach to configuration, deployment, and management, it can be difficult to ensure consistency across the entire platform, especially as it grows and becomes more complex.

This can result in issues such as

inconsistencies in naming conventions and configurations;

difficulty in scaling and managing resources across the platform;

increased risk of human error due to the lack of standardized processes and workflows; and

challenges in maintaining and updating the platform, as changes may affect different components in different ways if they are not standardized.

These standardization issues can ultimately lead to increased costs, decreased efficiency, and a higher risk of errors and downtime.

Lack of Vendor Support

Homegrown Kubernetes deployments lack formal vendor support, resulting in potential challenges when troubleshooting issues or seeking assistance.

This can lead to increased troubleshooting time, delays in issue resolution, and potential impacts on application availability. In addition, it can result in slower innovation and feature adoption.

The Kubernetes ecosystem continuously evolves, with new features, tools, and best practices being introduced frequently. Managed Kubernetes services or vendor distributions typically have faster innovation cycles and quicker adoption of new features compared to a homegrown Kubernetes deployment.

Opportunity Costs

Nothing comes free in this world. You can't eat your pie and have it too.

Every organization has limited resources. Building and managing a homegrown Kubernetes deployment may divert resources and attention from other strategic initiatives or core business activities, resulting in opportunity costs.

These costs may include missed business opportunities, delayed product development, or reduced focus on core business objectives.

Conclusion

It's important to carefully consider these hidden costs when evaluating the feasibility of a homegrown Kubernetes deployment.

While it may provide flexibility and customization options, it also may come with significant investments in terms of time, expertise, and ongoing maintenance.

Alternatively, leveraging managed Kubernetes services provided by cloud providers or Kubernetes distributions from reputable vendors may help mitigate some of these hidden costs and provide a more streamlined and supported Kubernetes deployment experience.

This post was written by Alexander Fridman. Alexander is a veteran in the software industry with over 11 years of experience. He worked his way up the corporate ladder and has held the positions of Senior Software Developer, Team Leader, Software Architect, and CTO. Alexander is experienced in frontend development and DevOps, but he specializes in backend development.

Kubernetes Insights

Cost Optimization

The Hidden Costs of Homegrown K8s Management

Table of Contents

A Homegrown Cluster

Benefits of a Homegrown Kubernetes Deployment

Tailored Solution

Cost Control

Hidden Costs

Development Costs

Maintenance Costs

Security and Stability

Common Security Risks

Standardization

Lack of Vendor Support

Opportunity Costs

Conclusion

Related blog posts

vCluster vs. HyperShift: Choosing the Right Path for Kubernetes Multi-Tenancy

Native Ambient Mesh Support with vCluster v0.25

Kubernetes Multi-tenancy: Are You Doing It the Hard Way?

Table of Contents

A Homegrown Cluster

Benefits of a Homegrown Kubernetes Deployment

Tailored Solution

Cost Control

Hidden Costs

Development Costs

Maintenance Costs

Security and Stability

Common Security Risks

Standardization

Lack of Vendor Support

Opportunity Costs

Conclusion

Related blog posts

vCluster vs. HyperShift: Choosing the Right Path for Kubernetes Multi-Tenancy

Native Ambient Mesh Support with vCluster v0.25

Kubernetes Multi-tenancy: Are You Doing It the Hard Way?

Sign up for our newsletter