Kubernetes, often stylized as “K8s,” emerged victorious in the realm of container orchestration tools years ago. Despite its dominance, implementing Kubernetes and integrating it with diverse infrastructures still presents a multifaceted landscape with a wide array of tools, some more robustly supported than others. A particularly noteworthy trend in this domain is the emergence of managed Kubernetes offerings from leading cloud providers, namely:

• Microsoft Azure’s Azure Kubernetes Service (AKS)
• Amazon Web Services’ Amazon Elastic Kubernetes Service (EKS)
• Google Cloud’s Google Kubernetes Engine (GKE)

From a DevOps standpoint, a critical assessment of these platforms is essential. Do they deliver on their touted capabilities? How do their creation times and other performance indicators measure up? How seamlessly do they mesh with their respective platforms, particularly their command-line interface (CLI) tools? What are the nuances of managing and working with these offerings? The following analysis delves into these questions and more.

Note: For those seeking a primer on Kubernetes clusters, Dmitriy Kononov provides an insightful introduction.

AKS vs. EKS vs. GKE: A Comparative Look at Advertised Features

To facilitate a structured comparison, the features of each managed Kubernetes version are categorized as follows:

• Global Overview
• Networking
• Scalability and Performance
• Security and Monitoring
• Ecosystem
• Pricing

Note: The information provided is subject to change as cloud providers frequently update their products.

Global Overview

As the originator of the Kubernetes project, it’s only natural that Google was the first to introduce a hosted version back in 2014.

Among the three contenders, Azure followed suit with AKS, which has undergone significant refinement. Those familiar with its predecessor, acs-engine, will appreciate the substantial improvements Microsoft has made with its successor, aks-engine.

AWS was the last to launch its offering, EKS, which occasionally lags behind in terms of features, although it is actively catching up.

Pricing, as always, is in flux. Google aligned with AWS’s price point of $0.10 per hour, effective June 2020. Azure stands out by offering the AKS service free of charge, but the sustainability of this model remains uncertain.

Another key distinction lies in the cluster upgrade mechanism. GKE boasts the most automated upgrades, enabled by default. In contrast, AKS and EKS require manual intervention to upgrade master or worker nodes.

Networking

In the realm of networking, the three cloud providers are relatively evenly matched. All three empower customers to implement network policies using tools like Calico. When it comes to load balancing, they seamlessly integrate with their respective load balancer resources, granting engineers flexibility in their choices.

A notable difference emerges in the realm of service mesh integration. AKS lacks out-of-the-box support for service mesh, although manual installation of Istio is feasible. AWS has developed its proprietary service mesh, App Mesh. Meanwhile, Google has released its integration with Istio (currently in beta), which customers can readily incorporate during cluster creation.

Top Choice: GKE

Scalability and Performance

When evaluating performance and scalability, GKE appears to have an edge. It supports the highest node count (5,000) and provides comprehensive documentation on optimal cluster scaling practices. Features ensuring high availability are readily available and easily customizable. Furthermore, GKE recently introduced Anthos, an initiative aimed at fostering an ecosystem around GKE and its functionalities, enabling on-premises GKE deployments.

However, AWS holds a distinct advantage: it is the sole provider allowing bare-metal nodes to power Kubernetes clusters.

As of June 2020, AKS lacks high availability for the master node, a crucial factor to consider. However, this could change in the future.

Top Choice: GKE

Security and Monitoring

From a compliance perspective, all three cloud providers are on par. However, EKS and GKE provide an additional layer of security through their integrated key management services.

In terms of monitoring, Azure and Google Cloud offer their proprietary monitoring ecosystems tailored for Kubernetes. Notably, Google’s offering has been recently revamped to leverage Kubernetes Engine Monitoring, specifically designed for Kubernetes environments.

Azure provides its container monitoring system, initially conceived for a basic, non-Kubernetes container landscape. Support for Kubernetes-specific metrics and resources (cluster health, deployments) has been introduced in preview mode as of June 2020.

AWS offers lightweight monitoring for the control plane directly within CloudWatch. For worker node monitoring, Kubernetes Container Insights Metrics are available via a dedicated CloudWatch agent installable within the cluster.

Top Choice: GKE

Ecosystem

Examining their ecosystems reveals distinct strengths for each provider. AKS boasts comprehensive documentation and ranks second in terms of Stack Overflow posts. EKS, despite having the fewest Stack Overflow posts, benefits from the richness of the AWS Marketplace. As the most mature platform, GKE leads in Stack Overflow posts and offers a substantial number of marketplace apps alongside the most extensive documentation.

Top Choices: GKE and EKS

Pricing

Despite adopting the $0.10 per hour pricing for all clusters, GKE remains the most cost-effective option due to Google’s sustained use discounts, applied when monthly usage of on-demand resources surpasses a certain threshold.

It’s crucial to acknowledge that the price examples provided exclude potential charges for traffic to the Kubernetes cluster.

AWS’s free tier is not applicable for testing EKS clusters because EKS necessitates larger machines than the tX.micro tier, and EKS hourly pricing falls outside the free tier scope.

Nevertheless, testing these managed Kubernetes offerings with a reasonable load can still be economical using spot/preemptible nodes from each provider, potentially yielding savings of 80-90%. However, deploying stateful production workloads on such ephemeral instances is not advisable.

Advertised Features and Google’s Advantage: A Recap

Based on the advertised features, a clear correlation exists between a managed Kubernetes offering’s market maturity and its feature set. Google’s pioneering role in the Kubernetes project translates into a significant advantage, resulting in tighter and more robust integration with its cloud platform.

However, as AKS and EKS mature, they should not be underestimated. Both possess unique strengths, such as AWS’s support for bare-metal nodes and its extensive marketplace.

With a clear understanding of the advertised features, let’s delve into a more practical comparison through hands-on testing.

Kubernetes: A Practical Comparison of AWS, GCP, and Azure

Marketing materials only tell part of the story. How do these platforms fare when tasked with handling real-world production loads? As a cloud engineer, I understand the importance of cluster provisioning and de-provisioning times, especially when adhering to infrastructure-as-code principles. Additionally, I was keen on exploring the capabilities of each CLI and assessing the ease of cluster creation for each cloud provider.

Cluster Creation: A User Experience Perspective

AKS

Creating a cluster on AKS mirrors the process of instance creation on AWS. Navigate to the AKS menu and proceed through a series of intuitive menus. Once the configuration is validated, cluster creation, a two-step process, commences. The straightforward nature allows engineers to swiftly launch clusters with default settings.

EKS

Cluster creation on EKS is demonstrably more intricate than on AKS. By default, AWS mandates a detour to IAM to define a new role for the Kubernetes control plane and assign it to the engineer. It’s important to note that cluster creation does not encompass node creation. The measured 11-minute average only accounts for master node creation. Node group creation is a separate step, again requiring a dedicated worker role with three essential policies configurable through the IAM control panel.

GKE

GKE provides the most streamlined manual cluster creation experience. After locating Kubernetes Engine within the Google Cloud Console, a simple click initiates cluster creation. Various setting categories are neatly organized in a left-hand menu. Google prepopulates the new cluster with an easily modifiable default node pool. Notably, GKE boasts the fastest cluster provisioning time, as highlighted in the subsequent table.

Cluster Spawning Time: A Benchmark

To ensure consistency and minimize regional variations, these tests were conducted in the same geographic regions (Frankfurt and West Europe for AKS). Node sizes were standardized as much as possible: three nodes, each equipped with two vCPUs and seven to eight GB of memory, representing a typical configuration for running light workloads and initial experimentation. Three cluster creations were performed for each provider to calculate an average.

GKE consistently outperformed its counterparts, exhibiting provisioning times consistently below three minutes.

Kubernetes CLIs: An Overview

While CLIs are not created equal, in this instance, all three contenders are essentially modules within larger CLI frameworks. Let’s explore the process of getting started with each cloud provider’s CLI toolchain.

AKS CLI (via az)

After installing the az tooling followed by the AKS module (az aks install-cli), engineers must authorize the CLI to interact with the Azure account associated with the project. This involves retrieving credentials to update the local kubeconfig file using the command az aks get-credentials --resource-group myResourceGroup --name myAKSCluster.

Similarly, cluster creation is invoked using: az aks create --resource-group myResourceGroup --name myAKSCluster

EKS CLI (via aws or eksctl)

AWS adopts a different approach, offering two official CLI tools for EKS cluster management. As expected, the ubiquitous aws CLI can connect to AWS resources, including clusters. Populating the local kubeconfig with credentials is accomplished via: aws eks update-kubeconfig --name cluster-test.

Additionally, engineers can leverage eksctl, a Go-based tool developed by Weaveworks, to simplify EKS cluster creation and management. A significant advantage of eksctl is its compatibility with YAML configuration files, facilitating infrastructure-as-code (IaC) by integrating with CloudFormation. This proves invaluable when incorporating EKS clusters into broader AWS infrastructure deployments.

Creating a cluster using eksctl is remarkably simple: eksctl create cluster, requiring no additional parameters.

GKE CLI (via gcloud)

GKE follows a similar pattern: install gcloud, then authenticate using gcloud init. From there, engineers can create, delete, describe, retrieve credentials for, resize, update, upgrade, or list clusters.

The syntax for cluster creation with gcloud is straightforward: gcloud container clusters create myGCloudCluster --num-nodes=1

AKS vs. EKS vs. GKE: Test Drive Results

In practical terms, GKE emerges as the clear frontrunner in terms of both console simplicity and cluster provisioning speed. Its UX, featuring a convenient connect button adjacent to each cluster, makes connecting to clusters remarkably easy.

Regarding CLI tooling, all three providers offer comparable functionalities. However, Weaveworks’ eksctl tool for EKS deserves special recognition. eksctl proves to be an ideal choice for implementing infrastructure-as-code on top of existing AWS infrastructure, seamlessly integrating EKS with other services.

Managed Kubernetes: AWS, GCP, and Azure Take the Lead

For those embarking on their Kubernetes journey, GKE stands out as the most approachable option. Its straightforward setup, intuitive and responsive provisioning experience, and tight integration with the Google Cloud Platform ecosystem make it a compelling choice.

Despite being a late entrant, AWS boasts undeniable strengths, including support for bare-metal nodes and its association with the most widely adopted cloud provider.

AKS has made remarkable strides since its inception. Feature parity and tooling maturity are likely on the horizon, leaving ample room for innovation. As with any managed Kubernetes offering, existing users of the parent platform will find its seamless integration particularly appealing.

Once a team commits to a Kubernetes cloud provider, it’s beneficial to learn from the experiences of others, especially their setbacks. These post-mortems serve as valuable resources, reflecting real-world scenarios and providing a solid foundation for developing robust best practices. I encourage you to share your insights and feedback below.