Launch a Kubernetes cluster on AWS with kube-up and let it autoscale

In the following we will walk through how to launch a Kubernetes (k8s) cluster on AWS using kube-up, push a Docker image to ECR, start a server in k8s based on that image, and autoscale the cluster. The kube-up scripts needs a proper GNU/Linux system to operate. Therefore, if you usually use Windows you either need a VM running GNU/Linux or Windows 10 Anniversary update with Windows Subsystem for Linux (WSL).

You need a Docker host up and running. Depending on whether you are using pure GNU/Linux, GNU/Linux in a VM, or WSL installing and using Docker differs. In the first two cases you may see here on how to install Docker. If you are using WSL then you need to install Docker in Windows (either Docker Toolbox, or – preferably – Docker CE). If you can install Docker CE then there may be a way to have a Docker host within WSL. If you can only use Docker Toolbox in Windows then, at this juncture, there is no way to have a Docker host within WSL.

In my case I use WSL and I am bound to Docker Toolbox. Thus, I need to switch between WSL and Windows at some point in order to build the docker image. If you can use a proper GNU/Linux, or have GNU/Linux running within a VM you can stay there and can ignore the [In WSL] / [In Windows] switches later on.

[In WSL & Windows]

First, we need kubectl on both, Windows and WSL. To install kubectl see

• https://kubernetes.io/docs/user-guide/prereqs/
• https://kubernetes.io/docs/tasks/kubectl/install/

The awscli is also needed on both Windows and WSL. To install awscli see

• https://docs.aws.amazon.com/cli/latest/userguide/installing.html

[In WSL]

Now that kubectl and awscli are installed we can get the kube-up script. We will to some extent follow the instructions given here. However, since we need to make some adaptions to the script and its guts we will first download the script suite, but do not start the cluster immediately. Therefore

    $ mkdir ~/scripts
    $ cd
    $ export KUBERNETES_RELEASE=v1.4.0 #later versions do not work
    $ export KUBERNETES_PROVIDER=aws
    $ export KUBERNETES_SKIP_CREATE_CLUSTER=true
    $ wget -q -O - https://get.k8s.io >~/k8s_setup.sh && chmod +x ~/k8s_setup.sh && ~/k8s_setup.sh

This will download a lot of scripts to a folder ~/kubernetes. To use AWS as provider we need to adapt some variables. To save work later we write a helper in ~/scripts/prepare_k8s_aws.sh and put the following stuff and make sure to change the value of AWS_S3_BUCKET to something unique):

    # edit to your liking
    # and particularly CHANGE the value of AWS_S3_BUCKET
    export KUBERNETES_PROVIDER=aws
    echo "setting KUBERNETES_PROVIDER=$KUBERNETES_PROVIDER"
    export KUBE_AWS_ZONE=eu-central-1a
    echo "setting KUBE_AWS_ZONE=$KUBE_AWS_ZONE"
    export NUM_NODES=2
    echo "setting NUM_NODES=$NUM_NODES"
    export NUM_NODES_MIN=2   # needed for autoscaling
    echo "setting NUM_NODES_MIN=$NUM_NODES_MIN"
    export NUM_NODES_MAX=5   # needed for autoscaling
    echo "setting NUM_NODES_MIN=$NUM_NODES_MAX"
    export MASTER_SIZE=m3.medium
    echo "setting MASTER_SIZE=$MASTER_SIZE"
    export NODE_SIZE=m3.medium
    echo "setting NODE_SIZE=$NODE_SIZE"
    export AWS_S3_REGION=eu-central-1
    echo "setting AWS_S3_REGION=$AWS_S3_REGION"
    export AWS_S3_BUCKET=yourProjectUniqueName-kubernetes-artifacts
    echo "setting AWS_S3_BUCKET=$AWS_S3_BUCKET"
    export KUBE_AWS_INSTANCE_PREFIX=k8s
    echo "setting KUBE_AWS_INSTANCE_PREFIX=$KUBE_AWS_INSTANCE_PREFIX"

To make kube-up actually using these variables we do

    $ sed -i '2i. ~/scripts/prepare_k8s_aws.sh' ~/k8s_setup.sh
    $ sed -i '2i. ~/scripts/prepare_k8s_aws.sh' ~/kubernetes/cluster/kube-down.sh
    $ sed '/--vpc-zone-identifier/i\      --min-size ${NUM_NODES_MIN} \\' -i \
      ~/kubernetes/cluster/aws/util.sh
    $ sed '/--vpc-zone-identifier/i\      --max-size ${NUM_NODES_MAX} \\' -i \
      ~/kubernetes/cluster/aws/util.sh

To enable autoscaling later, we need the IAM role that is assigned to the minions during cluster startup to have some more permissons. Edit a file e.g. ~/as_policy.json and insert the following content (be sure to have no empty lines in the file):

    {
        "Effect": "Allow",
        "Action": [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:TerminateInstanceInAutoScalingGroup"
        ],
        "Resource": "*"
    }

Then add the content to the policy template:

    $ cd ~ # or change to the directory where as_policy.json resides
    $ sed -i '43s|}|},|' ~/kubernetes/cluster/aws/templates/iam/kubernetes-minion-policy.json
    $ sed -i -e '43r as_policy.json' ~/kubernetes/cluster/aws/templates/iam/kubernetes-minion-policy.json

Now we are ready to start the cluster:

    $ unset KUBERNETES_SKIP_CREATE_CLUSTER
    $ ~/k8s_setup.sh

Answer Y to the question, i.e. keep the ~/kubernetes directory, since otherwise all your previous changes will be lost. If anything fails during the process and you need to start over you need to run

$ ~/kubernetes/cluster/kube-down.sh

first.

The cluster setup will take up to ten minutes. In the meantime, we will implement a Hello-World service that we will use later on to hog the pod’s CPU capacity and trigger a scale-out. Take e.g. the Getting-Started Actuator Service as described here. Clone the project from here and change into gs-actuator-service/complete. In src/main/java/hello/Greeting.java add the following methods

    private int ackermann(int n, int m) {
        if (n == 0) return m + 1;
        if (m == 0) return ackermann(n - 1, 1);
        return ackermann(n - 1, ackermann(n, m - 1));
    }
    private void hogCPU() {
        ExecutorService executor = Executors.newWorkStealingPool();
        List<Callable<Integer>> callables = Collections.nCopies(35, () -> ackermann(3, 11));
        try {
            executor.invokeAll(callables)
                    .stream()
                    .map(future -> {
                        try {
                            return future.get();
                        }
                        catch (Exception e) {
                            throw new IllegalStateException(e);
                        }
                    })
                    .forEach(System.out::println);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

and call hogCPU() in the last line of the constructor. You may reduce (or increase) the number of parallel processes depending on the instance type you chose in NODE_SIZE above. In our case, NODE_SIZE is m3.medium which has one virtual CPU. You also need to import the following packages:

    import java.util.Collections;
    import java.util.List;
    import java.util.concurrent.Callable;
    import java.util.concurrent.ExecutorService;
    import java.util.concurrent.Executors;

To build Docker images for this project one may use the fabric8 maven plugin. See here on how to install it. (At the time of writing the latest version is 3.2.7 rather than 3.2.28.). Since I cannot have a Docker host within WSL I need to switch to Windows for building the image. This has one more implication: Since I start the cluster within WSL all configuration that is generated by kube-up will be written to ~/.kube/config within WSL. However, I need that configuration in Windows as well in order to use kubectl properly. So, after the cluster has been brought up successfully, I copy the generated kubectl config from WSL to Windows:

$ cp ~/.kube/config /mnt/c/Users/<yourUserName>/.kube/config

Also, after the cluster has been brought up a few URIs will be printed to the terminal. Find the URI for Kubernetes-dashboard and open it in a browser. You will get an SSL warning. After you accepted the warning you will be presented a login form. To login change back to the terminal and type

    $ CTX=$(kubectl config view | grep current-context | sed 's|current-context: ||')
    $ kubectl config view | grep -A2 $CTX-basic-auth | grep password | \
      sed 's| *password: ||'

This will print the password. The username is always admin. After logging in to the dashboard navigate to Workloads->Deployments and then change back to the terminal.

[In Windows]

Remember that I only need to switch to Windows since I am bound to Docker Toolbox for technical reasons. I start Docker Toolbox and use the provided shell.

Change to the Hello-World project and build it using fabric8:

    $ cd ~/path/to/gs-actuator-service/complete
    $ mvn clean package fabric8:build

You can check the image that has been created using

$ docker images

In our case there should be an output similar to

    REPOSITORY                              TAG                IMAGE ID            CREATED             SIZE
    springframework/gs-actuator-service     0.1.0              e3b475c4e906        17 hours ago        112.3 MB

Since we want to use Amazon’s Elastic Container Registry (ECR) to store our image we need to tag the image accordingly. First, you should create a repository for the project:

    $ ECR_ID=$(aws ecr create-repository --repository-name hellorepo --query repository.registryId --output text)

Then we tag the image:

    $ docker tag springframework/gs-actuator-service:0.1.0 \
      ${ECR_ID}.dkr.ecr.eu-central-1.amazonaws.com/hellorepo:0.1.0

To actually upload the image to ECR we need Docker to authenticate against ECR. As described here this _shall_ be (it’s NOT) as simple as

$ $(aws ecr get-login)

However, the login command that is returned by the awscli is, at this junction, indeed wrong: The repository URI contains an https:// which will later lead to a message that there is “no such repository” if you try to push the image with Docker. (This was a thing that drove me nuts for an entire day until I figured out what was actually going wrong). See on stackoverflow for the solution. Be sure to have GnuWin32 installed in Windows at this point. The “correct” way to authenticate Docker against ECR is then

    $ $(aws ecr get-login | sed 's|https://||' | sed 's|-e none ||')

Now push the image to ECR:

$ docker push ${ECR_ID}.dkr.ecr.eu-central-1.amazonaws.com/hellorepo:0.1.0

At this point we could make a deployment and expose it as a service. But we want to demonstrate autoscaling. In order to make that work we need to understand two things: (1) Deployments (and thus pods) can have explicit resource requirements. If there are no nodes available in the k8s cluster that could fulfil these requirements, a new node (i.e. a minion, i.e. an EC2 instance) would need to be instantiated. (2) This horizontal autoscaling for nodes is not yet built-in to kubernetes for AWS. But there is a cluster autoscaler available as deployment.

So we edit a file ~/autoscaler.yml with the following content:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: cluster-autoscaler
      labels:
        app: cluster-autoscaler
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: cluster-autoscaler
      template:
        metadata:
          labels:
            app: cluster-autoscaler
        spec:
          containers:
            - image: gcr.io/google_containers/cluster-autoscaler:v0.4.0
              name: cluster-autoscaler
              resources:
                limits:
                  cpu: 100m
                  memory: 300Mi
                requests:
                  cpu: 100m
                  memory: 300Mi
              command:
                - ./cluster-autoscaler
                - --v=4
                - --cloud-provider=aws
                - --skip-nodes-with-local-storage=false
                - --scale-down-delay=10m
                - --nodes=2:5:k8s-minion-group-eu-central-1a
              env:
                - name: AWS_REGION
                  value: eu-central-1
              volumeMounts:
                - name: ssl-certs
                  mountPath: /etc/ssl/certs/ca-certificates.crt
                  readOnly: true
              imagePullPolicy: "Always"
          volumes:
            - name: ssl-certs
              hostPath:
                path: "/etc/ssl/certs/ca-certificates.crt"

This configuration will scale the minions in the autoscaling group k8s-minion-group-eu-central-1a between 2 and 5 nodes. And it will take a node out of the cluster after it has been underutlized for 10 minutes.  Be sure to edit the value of AWS_REGION (in the example above eu-central-1), and the autoscaling group name (in the above example k8s-minion-group-eu-central-1a) to fit your config. The name of your autoscaling group name can be determined using

    $ aws autoscaling describe-auto-scaling-groups \
      --query AutoScalingGroups[].AutoScalingGroupName --output text

Deploy the cluster autoscaler:

$ kubectl create -f ~/autoscaler.yml

You can check the successful deployment in the Kubernetes-dashboard: refresh the Deployments page; the autoscaling deployment should be visible.

Now we can deploy the image to the k8s cluster:

    $ kubectl run gs-actuator \
      --image ${ECR_ID}.dkr.ecr.eu-central-1.amazonaws.com/hellorepo:0.1.0 \
      --requests='cpu=700m,memory=150Mi' \
      --limits='cpu=900m,memory=200Mi'

The requests and limits parameters are essential to trigger the autoscaling: We state that the deployment needs a pod with (at least) 70% CPU time and 150 MiB memory available. If you refresh the Deployments page you will see an error message that no node has enough capacity to take the pod. In the background it will launch a new EC2 instance and after ca. 6 minutes the new node will be available and the pod gets deployed on that node. Wait until the deployment is successful. It should not take longer than two more minutes, but depending on how fast AWS initializes the instance it can take up to 10 minutes in total.

Now let’s start a service of the gs-actuator deployment and make port 9000 available to the public (check that this meets the server.port in the application.properties of gs-actuator):

$ kubectl expose deployment gs-actuator --type=LoadBalancer --port=9000

This will start an Elastic Load Balancer (ELB) and you get the public DNS name with

    $ ELB_DNS_NAME=$(aws elb describe-load-balancers \
      --query LoadBalancerDescriptions[0].DNSName --output text) && \
      echo $ELB_DNS_NAME

It may take a while until the DNS name can be resolved.

The last step for full autoscaling is to enable Horizontal Pod Autoscaling (HPA) within the cluster:

$ kubectl autoscale deployment gs-actuator --min=1 --max=3 --cpu-percent=80

This will run at least one pod and at most three pods where a new pod gets launched as soon the (average) CPU utilization of the already running pods exceeds 80%.

Now generate a request for the service:

$ curl http://${ELB_DNS_NAME}:9000/hello-world

This will utilize the pod’s CPU fully for more than a minute which should then trigger the HPA. (If the service responds earlier in your case you may need to fire more requests in parallel or consider to adapt Greetings.hogCPU() to start more parallel processes.) This will start a new pod which – due to the specified resource requirements – will trigger a scale out at the nodes. Watch the kubernetes Dashboard Workloads -> Pods and Admin -> Nodes. A new pod will be created that does not fit in any node. Eventually (after 2 minutes or so), a new node will appear.

Let the service rest. HPA will eventually remove the pod. 10 minutes later the cluster autoscaler will remove the additional node.

To clean up run

$ ~/kubernetes/cluster/kube-down.sh

Remove the registry in ECR:

$ aws ecr delete-repository --repository-name hellorepo --force

Optionally, you may remove the S3 bucket. Do NOT use this if you have other buckets than the one created using the kube-up script): Be CAREFUL and double check. To be safe use the values you have set for AWS_S3_REGION and AWS_S3_BUCKET in ~/scripts/prepare_k8s_aws.sh. (You cannot plainly use $AWS_S3_REGION and $AWS_S3_BUCKET since these variables were set in a subprocess and are not known to the parent process: i.e. the child Bash that actually executes k8s_setup.sh cannot set variables in the parent Bash which called k8s_setup.sh)

    $ S3_BUCKET=$(aws --region eu-central-1 s3api list-buckets --query Buckets[].Name --output text)
    $ aws s3 rm s3://$S3_BUCKET --recursive # only deletes content but not bucket itself
    $ aws s3api delete-bucket --bucket $S3_BUCKET

This concludes our walkthrough. You

• started a kubernetes cluster on AWS
• set up kubernetes for autoscaling
• built a Docker image and pushed it to ECR
• let kubernetes use the image to start a deployment
• exposed the deployment as a service
• generated requests to the service in order to trigger autoscaling
• let the cluster scale-out and scale-in
• teared down the cluster
• cleaned up after you

I am eager to get your comments how this was helpful to you or receive your questions.

This post was written by: