Below are some notes on the steps you need to take before deploying Garden projects to a remote Kubernetes cluster, and an overview of the things you'll want to configure.
Many of the steps are not specific to Garden as such, so you may have already performed some of these steps and/or may need to follow the provided links in each section for details on how to perform the steps you have not yet completed.
Our cloud provider setup guide includes instructions for getting started with a few prominent hosted Kubernetes providers, and for configuring your Garden project to connect with them.
Start by making sure you have a kubectl context set up on your development machine to access your cluster. How you set this up will vary by how and where you have deployed your cluster.
Then configure the project and provider in your project
garden.yml, along with the kubectl context you use to connect to your cluster.
kind: Projectname: my-projectenvironments:- name: devproviders:- name: kubernetescontext: my-dev-context # the name of the kubectl context for the cluster...defaultEnvironment: dev
Note that you need to have permissions to create namespaces and to create deployments, daemonsets, services and ingresses within the namespaces created.
The plugin will create two or more namespaces per user and project, one to run services, another to manage metadata and configuration (this is so that your environment can be reset without clearing your configuration variables), and potentially more to support specific plugins/providers.
When you're connecting to a new cluster, or after you have updated your provider configuration or Garden itself, you need to install/update cluster-wide services that Garden needs to operate. When using
local-kubernetes this happens automatically when you're deploying or testing, but for remote clusters this requires a manual step. This is so that different users don't end up "competing" with different configurations or versions.
To initialize or update your cluster-wide services, run:
garden --env=<environment-name> plugins kubernetes cluster-init
Garden supports multiple methods for building images and making them available to the cluster. Below we detail how to configure for the standard out-of-cluster build flow, but do make sure to look at the in-cluster building for details on how to build images directly inside the cluster.
This is the default build mode. It is the least efficient one for remote clusters, but requires no additional services to be deployed to the cluster. For remote clusters, you do however need to explicitly configure a deployment registry, and to have Docker running locally. For development clusters, you may in fact get set up quicker if you use in-cluster building.
When you deploy to your environment (via
garden deploy or
garden dev) using the local Docker mode, images are first built locally and then pushed to the configured deployment registry, where the K8s cluster will then pull the built images when deploying. This should generally be a private container registry, or at least a private project in a public registry.
Similarly to the below TLS configuration, you may also need to set up auth for the registry using K8s Secrets, in this case via the
kubectl create secret docker-registry helper.
Note that you do not need to configure the authentication and imagePullSecrets when using GKE along with GCR, as long as your deployment registry is in the same project as the GKE cluster.
Once you've created the auth secret in the cluster, you can configure the registry and the secrets in your
garden.yml project config like this:
kind: Projectname: my-projectenvironments:- name: devproviders:- name: kubernetescontext: my-dev-context...deploymentRegistry:# The hostname of the registry, e.g. gcr.io for GCR (Google Container Registry)hostname: my.registry.io# Namespace (aka project ID) to use in the registry for this project.# For GKE/GCR, use the project ID where your cluster is.namespace: my-project-idimagePullSecrets:# The name of the secret you stored using `kubectl create secret docker-registry`- name: my-registry-secret# Change this if you store the secret in another namespace.namespace: defaultdefaultEnvironment: dev
You also need to login to the
docker CLI, so that images can be pushed to the registry. Please refer to your registry's documentation on how to do that (for Docker Hub you simply run
By default, Garden will not install an ingress controller for remote environments. This can be toggled by setting the
setupIngressController flag to
nginx. Alternatively, you can set up your own ingress controller, e.g. using Traefik, Ambassador or Istio. You can find examples for using Garden with Ambassador and with Istio in our examples directory.
You'll also need to point one or more DNS entries to your cluster, and configure a TLS certificate for the hostnames you will expose for ingress. How you configure DNS and prepare the certificates will depend on how you manage DNS and certificates in general, so we won't cover that in detail here.
Once you have the certificates in hand (the
.key files), create a Secret for each cert in the cluster so they can be referenced when deploying services:
kubectl create secret tls mydomain-tls-secret --key <path-to-key-file> --cert <path-to-crt-file>
Then configure each certificate/secret in your
garden.yml provider configuration:
kind: Projectname: my-projectdefaultEnvironment: devenvironments:- name: devproviders:- name: kubernetescontext: my-dev-contextdefaultHostname: mydomain.comtlsCertificates:- name: main# Optionally set particular hostnames to use this certificate for# (useful if you have multiple certs for the same hostname).hostnames: [mydomain.com]secretRef:# Change to whatever name you chose for the secret above.name: my-tls-secret# Change this if you store the secret in another namespace.namespace: default- name: wildcardsecretRef:name: wildcard-tls-secretnamespace: default...