One of Garden's most powerful features is the ability to build images in your Kubernetes development cluster, thus avoiding the need for local Kubernetes clusters. This guide covers the requirements for in-cluster building and how to set it up.
This guide assumes you've already read through the Remote Kubernetes guide.
First off, you should only use in-cluster building in development clusters! Production clusters should not run the builder services for multiple reasons, both to do with resource and security concerns.
You should also avoid using in-cluster building in clusters where you don't control/trust all the code being deployed, i.e. multi-tenant setups (where tenants are external, or otherwise not fully trusted).
In-cluster building works with most Kubernetes clusters, provided they have enough resources allocated. We have tested on GKE, AKS, EKS, DigitalOcean, and some custom installations.
Specifically, the clusters need the following:
hostPort, and for reaching
hostPorts from the node/Kubelet. This should work out-of-the-box in most standard setups, but clusters using Cilium for networking may need to configure this specifically, for example.
At least 2GB of RAM on top of your own service requirements. More RAM is strongly recommended if you have many concurrent developers or CI builds.
PersistentVolumeClaims and enough disk space for layer caches and the in-cluster image registry.
You can—and should—adjust the allocated resources and storage in the provider configuration, under resources and storage. See the individual modes below as well for more information on how to allocate resources appropriately.
Garden supports multiple methods for building images and making them available to the cluster:
The Cluster Docker and Kaniko modes build container images inside your development cluster, so you don't need to run Docker on your machine, and avoid having to build locally and push build artifacts over the wire to the cluster for every change to your code.
The Local Docker mode is the default. You should definitely use that when using Docker for Desktop, Minikube and most other local development clusters, and also if you're using Garden to deploy to staging/production clusters (more on security considerations above).
Let's look at how each mode works, and how you configure them:
The Cluster Docker mode installs a standalone Docker daemon into your cluster, that is then used for builds across all users of the clusters, along with a handful of other supporting services. Enable this mode by setting
buildMode: cluster-docker in your
kubernetes provider configuration.
In this mode, builds are executed as follows:
Your code (build context) is synchronized to a sync service in the cluster, making it available to the Docker daemon.
A build is triggered in the Docker daemon.
The built image is pushed to an in-cluster registry (which is automatically installed), which makes it available to the cluster.
After enabling this mode (we currently still default to the
local-docker mode), you will need to run
garden plugins kubernetes cluster-init --env=<env-name> for each applicable environment, in order to install the required cluster-wide services. Those services include the Docker daemon itself, as well as an image registry, a sync service for receiving build contexts, two persistent volumes, an NFS volume provisioner for one of those volumes, and a couple of small utility services.
Optionally, you can also enable BuildKit. In most cases, this should work well and be more performant, but remains optional for now. If you have
cluster-docker set as your
buildMode you can enable BuildKit for an environment as follows:
Make sure your cluster has enough resources and storage to support the required services, and keep in mind that these services are shared across all users of the cluster. Please look at the resources and storage sections in the provider reference for details.
This mode works mostly the same way as Cluster Docker, but replaces the Docker daemon with Kaniko. Enable this by setting
buildMode: kaniko in your
kubernetes provider configuration, and running
garden plugins kubernetes cluster-init --env=<env-name> to install required cluster-wide service.
You can provide extra arguments to Kaniko via the
extraFlags field. Users with projects with a large number of files should take a look at the
--use-new-run options as these can provide significant performance improvements. Please refer to the official docs for the full list of available flags.
The Kaniko project is still improving, but it provides a compelling alternative to the standard Docker daemon because it can run without special privileges on the cluster, and is thus more secure. It may also scale better because it doesn't rely on a single daemon shared across users, so builds are executed in individual Pods and don't share the same resources of a single Pod. This also removes the need to provision another persistent volume, which the Docker daemon needs for its layer cache.
The trade-off is generally in performance, at least for the moment, partly because it relies on the Docker registry to cache layers. There are also some known issues and incompatibilities, so your mileage may vary.
Note the difference in how resources for the builder are allocated. See the builder resources reference for details.
This is the default mode. It is the least efficient one for remote clusters, but requires no additional services to be deployed to the cluster. For remote clusters, you do however need to explicitly configure a deployment registry, and to have Docker running locally.
See the Local Docker builds section in the Remote Clusters guide for details.
You can publish images that have been built in your cluster, using the
garden publish command.
The only caveat is that you currently need to have Docker running locally, and you need to have authenticated with the target registry. When publishing, we pull the image from the in-cluster registry to the local Docker daemon, and then go on to push it from there. We do this to avoid having to (re-)implement all the various authentication methods (and by extension key management) involved in pushing directly from the cluster.
As usual, you need to specify the
image field on the
container module in question. For example:
kind: Modulename: my-moduleimage: my-repo/my-image:v1.2.3 # <- omit the tag here if you'd like to use the Garden-generated version tag...
In order to avoid disk-space issues in the cluster, the
kubernetes provider exposes a utility command:
garden --env=<your-environment> plugins kubernetes cleanup-cluster-registry
The command does the following:
Looks through all Pods in the cluster to see which images/tags are in use, and flags all other images as deleted in the in-cluster registry.
Restarts the registry in read-only mode.
Runs the registry garbage collection.
Restarts the registry again without the read-only mode.
When using the
cluster-docker build mode, we additionally untag in the Docker daemon all images that are no longer in the registry, and then clean up the dangling image layers by running
docker image prune.
There are plans to do this automatically when disk-space runs low, but for now you can run this manually or set up your own cron jobs.
The in-cluster builder may need to be able to pull base images from a private registry, e.g. if your Dockerfile starts with something like this:
my-private-registry.com requires authorization.
For this to work, you need to create a registry secret in your cluster (see this guide for how to create the secret) and then configure the imagePullSecrets field in your
kubernetes provider configuration:
kind: Projectname: my-project...providers:- name: kubernetes...imagePullSecrets:# The name of the registry auth secret you created.- name: my-registry-secret# Change this if you store the secret in another namespace.namespace: default
This registry auth secret will then be copied and passed to the in-cluster builder. You can specify as many as you like, and they will be merged together.
Note: Any time you add or modify imagePullSecrets after first initializing your cluster, you need to run
garden plugins kubernetes cluster-initagain for them to work when pulling base images!
You can also use your private registry to store images after building and for deployment. If you've completed the steps above for configuring your
imagePullSecrets, you can also configure a
deploymentRegistry in your provider configuration:
kind: Projectname: my-project...providers:- name: kubernetes...imagePullSecrets:- name: my-registry-secretnamespace: defaultdeploymentRegistry:hostname: my-private-registry.comnamespace: my-project # <--- make sure your configured imagePullSecrets can write to repos in this namespace
This is often more scalable than using the default in-cluster registry, and may fit better with existing deployment pipelines. Just make sure the configured
imagePullSecrets have the privileges to push to repos in the configured namespace.