Managing Clusters with Cluster Registry

The Cluster Registry is part of the kubernetes foundation but is not included in the distribution by default. It’s added through a simple kubectl apply command. The cluster registry provides a simple yet powerful feature in that it enables a new resource type of Cluster to be added to your api calls. This means you can make a request for kubectl get clusters to retrieve a list of cluster matching specific queries. Since these are presented as standard Kubernetes API requests and resources, you can combine them with other commands to more dynamically manage your ecosystem.

In this example we’ll install the registry, load some metadata then use the registry to act on multiple clusters.

While you can accomplish this on your laptop, I’ll be demonstrating through Google Cloud Shell, which already includes many of the tools we’ll be using, as well as a standard environment we can all work from.

If you’re new to GCP, Google offers a free $300 credit as well as a generous Free Tier to get you started. So create your project and lets get started

Set Variables

To get started let’s create some variables and grab kubectx to manage our contexts easier

export PROJECT=$(gcloud config get-value project)
gcloud config set project $PROJECT

export PROJECT_DIR=$HOME/clsRegistry
mkdir -p $PROJECT_DIR/bin
PATH=$PROJECT_DIR/bin/:$PATH

Get tooling

Kubectx is a fantastic tool that simplifies managing and switching cluster contexts.

curl -LO https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx && \
chmod +x kubectx && \
mv kubectx $PROJECT_DIR/bin/.

Create your control cluster

For this example we’ll use a centralized control cluster to act as our administrative system. Since we’re just doing a small example we’ll also set the number of nodes to 1

gcloud container clusters create central --zone us-central1-a --num-nodes=1
kubectx central=gke_${PROJECT}_us-central1-a_central

Install the cluster registry CRD

Once the control cluster is up and running, it’s time to create the Cluster Registry resource definition.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/cluster-registry/master/cluster-registry-crd.yaml

Generate Yaml Files describing your clusters

In general it’s good practice to store your desired state in a source repository. We’ll create yaml files describing our clusters. While we’re not doing it here, you could actually use these definitions as the specs for creating the clusters themselves.

Create 3 files, one for each cluster. In these examples I’ve included labels for lifecycle as well as the zone in a custom data structure. You can add any data needed for your use case.

cat <<EOF >>$PROJECT_DIR/central.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: central
  namespace: default
  labels:
    lifecycle: "control"
spec:
  kubernetesApiEndpoints:
    serverEndpoints:
      - clientCIDR: "0.0.0.0/0"
        serverAddress: "100.0.0.0"
  myCustomData:
    zone: "us-central1-a"
status: {}
EOF

cat <<EOF >>$PROJECT_DIR/east.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: east
  namespace: default
  labels:
    lifecycle: "prod"
spec:
  kubernetesApiEndpoints:
    serverEndpoints:
      - clientCIDR: "0.0.0.0/0"
        serverAddress: "100.0.0.0"
  myCustomData:
    zone: "us-east1-b"
status: {}
EOF


cat <<EOF >>$PROJECT_DIR/west.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: west
  namespace: default
  labels:
    lifecycle: "prod"
spec:
  kubernetesApiEndpoints:
    serverEndpoints:
      - clientCIDR: "0.0.0.0/0"
        serverAddress: "100.0.0.0"
  myCustomData:
    zone: "us-west1-b"
status: {}
EOF

kubectl apply -f $PROJECT_DIR/central.yaml --context central

Create your remote cluster

OK lets spin up our clusters and load the registry. I’m also renaming the default context name here for easy use later on.

gcloud container clusters create west --zone us-west1-b --num-nodes=1
kubectx west=gke_${PROJECT}_us-west1-b_west
kubectl apply -f $PROJECT_DIR/west.yaml --context central

gcloud container clusters create east --zone us-east1-b --num-nodes=1
kubectx east=gke_${PROJECT}_us-east1-b_east
kubectl apply -f $PROJECT_DIR/east.yaml --context central

Deploy to all Prod clusters

In this section we’ll use the data in the registry along with a simple for loop to deploy an nginx image to all production clusters. We’re going to loop over the result of kubectl get clusters -l lifecycle=prod which selects resources labeled lifecycle=prod. Additionally we’re requesting the output to be a specific value in the json path located at .items[*].metadata.name

kubectx central

for cluster in $( kubectl get clusters -l lifecycle=prod -o jsonpath='{.items[*].metadata.name}' ); do

    kubectl run --image nginx myapp --context ${cluster}

done

List all deployments on all clusters

Now let’s run a simpler command to list all the deployments from all the clusters. You’ll note myapp deployed to east and west with nothing on central

for cluster in $( kubectl get clusters -o jsonpath='{.items[*].metadata.name}' ); do
    kubectx $cluster
    kubectl get deploy 
    echo 
done

Recap

So that about wraps it up. There’s really not much to setting up cluster registry but it provides a solid base for dynamically managing your fleet of kubernetes clusters.

Cleanup

Be sure to tear down your cluster to avoid additional costs

delete_cluster() {
 
        CLUSTER_NAME=$1
        CLUSTER_ZONE=$2

        # Delete the clusters 
        gcloud container clusters delete ${CLUSTER_NAME} \
        --zone=${CLUSTER_ZONE} \
        -q --async

        # Delete 
        kubectx -d ${CLUSTER_NAME}
}


delete_cluster central us-central1-a
delete_cluster west us-west1-b
delete_cluster east us-east1-b