Generate Kustomize Manifests from Helm Charts

Generate Kustomize Manifests from Helm Charts

Aug 30, 2021

Related to and how I used helm templatewith a values.yaml file to generate manifests

  • What is Kustomize
  • Issues with Kustomize
  • Kustomize vs Helm
  • cluster-autoscaler example? (https://github.com/Worldremit/devops-tf-aws-kubetools/tree/master/tools/cluster-autoscaler#update-instructions)
    • Upgrade Instructions from README

      Requirements: helm

      Source configuration for cluster-autoscaler has been gathered from: https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler

      Add the autoscaler repository

      helm repo add autoscaler https://kubernetes.github.io/autoscaler

      Check the latest chart version

      helm search repo autoscaler/cluster-autoscaler
      NAME                               	CHART VERSION	APP VERSION	DESCRIPTION
      autoscaler/cluster-autoscaler      	9.9.2        	1.20.0     	Scales Kubernetes worker nodes within autoscali...
      autoscaler/cluster-autoscaler-chart	2.0.0        	1.18.1     	Scales Kubernetes worker nodes within autoscali...

      Next, we need to download the latest version

      helm fetch autoscaler/cluster-autoscaler --untar --untardir $PWD/tmp --version 9.9.2

      Once that is done, we will utilise helm tempalate to generate manifests that can be used with kubectl in our pipeline

      values.yaml has been provided in an effort to ease the upgrade process which contain the changes in our environment compared to the default values that the halm chart gives us. You can find the full value list at https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/values.yaml (make sure to change to the correct helm chart/tag version)

      helm template $PWD/tmp/cluster-autoscaler --values values.yaml --namespace cluster-autoscaler --output-dir $PWD/tmp --debug

      Note that we are hardcoding the namespace above, make sure to change it if required.

      Once we generate the manifests, we still need to make a few tweaks:

      1. Merge all rbac related manifests into one
      2. TMP_LOC=$PWD/tmp/cluster-autoscaler/templates && cd $TMP_LOC && cat clusterrole.yaml clusterrolebinding.yaml role.yaml rolebinding.yaml serviceaccount.yaml > rbac.yaml && cd -
      3. Move the following into the base directory from tmp/cluster-autoscaler/templates
        • deployment.yaml
        • pdb.yaml¬†if there are any changes
        • service.yaml¬†if there are any changes
        • newly generated¬†rbac.yaml
      4. There are some values that we can't replace via values.yaml as they are built into Helm so we have to remove them
      5. sed -i '' '/app.kubernetes.io\/instance/d' base/*.yaml
        sed -i '' 's/RELEASE-NAME-cluster-autoscaler/cluster-autoscaler/g' base/*.yaml
      6. Remove duplicate AWS_REGION that contains the REMOVE-ME value from deployment.yaml

      Now all that is left is to generate a manifest and test locally. Follow instructions in the Generate Kubernetes manifest for verification section of this documentation.

      Testing the autoscaling functionality

      The autoscaler will scale up the cluster size when there are pods that aren't schedulable. An easy way to do this is to just spin up a large amount of containers on a small cluster.

      For example the following deployment can help, just change the replica count depending on the size of your cluster:

      • -- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app.kubernetes.io/name: nginx-deployment replicas: 20 template: metadata: labels: app.kubernetes.io/name: nginx-deployment spec: containers: - name: nginx image: nginx ports: - containerPort: 80 resources: limits: cpu: 300m memory: 500Mi requests: cpu: 300m memory: 500Mi

      Don't forget to remove the deployment once to confirm scaledown also works.

    • the project doesn't have kustomize specific files (link a project that does), but it does have helm
    • helm fetch
    • values.yaml
    • helm tempalate
    • replace helm build-in values & cleanup
    • kubectl kustomize env

Cover Photo by NAME on Unsplash

Blog Posts


Linkedin Github Twitter Curriculum Vitae

Damir Dulic | Powered by Notion & Super