One of Garden's most powerful features is the ability to build images in your Kubernetes development cluster, thus avoiding the need for local Kubernetes clusters. This guide covers the requirements for in-cluster building and how to set it up.

This guide assumes you've already read through the Remote Kubernetes guide.

Security considerations

First off, you should only use in-cluster building in development and testing clusters! Production clusters should not run the builder services for multiple reasons, both to do with resource and security concerns.

You should also avoid using in-cluster building in clusters where you don't control/trust all the code being deployed, i.e. multi-tenant setups (where tenants are external, or otherwise not fully trusted).

General requirements

In-cluster building works with most Kubernetes clusters, provided they have enough resources allocated and meet some basic requirements. We have tested it on GKE, AKS, EKS, DigitalOcean, and various other custom installations.

The specific requirements vary by the build mode used, and whether you're using the optional in-cluster registry or not.

In all cases you'll need at least 2GB of RAM on top of your own service requirements. More RAM is strongly recommended if you have many concurrent developers or CI builds.

For the cluster-docker and kaniko modes, and the (optional) in-cluster image registry, support for PersistentVolumeClaims is required, with enough disk space for layer caches and built images. The in-cluster registry also requires support for hostPort, and for reaching hostPorts from the node/Kubelet. This should work out-of-the-box in most standard setups, but clusters using Cilium for networking may need to configure this specifically, for example.

You can—and should—adjust the allocated resources and storage in the provider configuration, under resources and storage. See the individual modes below as well for more information on how to allocate resources appropriately.

We also strongly recommend a separate image registry to use for built images. Garden can also—and does by default—deploy an in-cluster registry. The latter is convenient to test things out and may be fine for individual users or small teams. However, we generally recommend using managed container registries (such as ECR, GCR etc.) since they tend to perform better, they scale more easily, and don't need to be operated by your team. See the Configuring a deployment registry section for more details.

Build modes

Garden supports multiple methods for building images and making them available to the cluster:

1. ​kaniko — Individual Kaniko pods created for each build in the garden-system namespace.

2. ​cluster-buildkit (experimental)— A BuildKit deployment created for each project namespace.

3. ​cluster-docker — A single Docker daemon installed in the garden-system namespace and shared between users/deployments.

4. local-docker — Build using the local Docker daemon on the developer/CI machine before pushing to the cluster/registry.

The local-docker mode is set by default. You should definitely use that when using Docker for Desktop, Minikube and most other local development clusters.

The other modes—which are why you're reading this guide—all build your images inside your development/testing cluster, so you don't need to run Docker on your machine, and avoid having to build locally and push build artifacts over the wire to the cluster for every change to your code.

The remote building options each have some pros and cons. You'll find more details below but here are our general recommendations at the moment:

• ​kaniko is a solid choice for most cases and is currently our first recommendation. It is battle-tested among Garden's most demanding users (including the Garden team itself). It also scales horizontally, since individual Pods are created for each build.

• ​cluster-buildkit is a new addition and is for now considered experimental, but we are hoping to make that the default in the future. Unlike the other options, which deploy cluster-wide services in the garden-system namespace, a BuildKit Deployment is dynamically created in each project namespace and requires no other cluster-wide services. This mode also offers a rootless option, which runs without any elevated privileges, in clusters that support it.

• ​cluster-docker was the first implementation included with Garden. It's pretty quick and efficient for small team setups, but relies on a single Docker daemon for all users of a cluster, and also requires supporting services in garden-system and some operations to keep it from filling its data volume. It is no longer recommended and we may deprecate it in future releases.

Let's look at how each mode works in more detail, and how you configure them:

kaniko

This mode uses an individual Kaniko Pod for each image build.

The Kaniko project provides a compelling alternative to the standard Docker daemon because it can run without special privileges on the cluster, and is thus more secure. It may also scale better because it doesn't rely on a single daemon shared across users, so builds are executed in individual Pods and don't share the same resources of a single Pod. This also removes the need to provision another persistent volume, which the Docker daemon needs for its layer cache.

In this mode, builds are executed as follows:

1. Your code (build context) is synchronized to a sync service in the cluster, making it available to Kaniko pods.

2. A Kaniko pod is created for the build in the garden-system namespace.

3. Kaniko pulls caches from the deployment registry, builds the image, and then pushes the built image back to the registry, which makes it available to the cluster.

Comparison

The trade-off compared to the cluster-docker is generally in performance, partly because it relies only on the Docker registry to cache layers, and has no local cache. There are also some occasional issues and incompatibilities, so your mileage may vary.

Compared to cluster-buildkit, Kaniko may be a bit slower because it has no local cache. It also requires cluster-wide services to be installed and operated, and for each user to have access to those services in the garden-system namespace, which can be a problem in some environments. It is however currently considered more "battle-tested", since the cluster-buildkit mode is a recent addition.

Configuration and requirements

Enable this by setting buildMode: kaniko in your kubernetes provider configuration, and running garden plugins kubernetes cluster-init --env=<env-name> to install required cluster-wide service.

By default, Garden will install an NFS volume provisioner into garden-system in order to be able to efficiently synchronize build sources to the cluster and then attaching those to the Kaniko pods. You can also specify a storageClass to provide another ReadWriteMany capable storage class to use instead of NFS. This may be advisable if your cloud provider provides a good alternative, or if you already have such a provisioner installed.

Note the difference in how resources for the builder are allocated between Kaniko and the other modes. For this mode, the resource configuration applies to each Kaniko pod. See the builder resources reference for details.

You can provide extra arguments to Kaniko via the extraFlags field. Users with projects with a large number of files should take a look at the --snapshoteMode=redo and --use-new-run options as these can provide significant performance improvements. Please refer to the official docs for the full list of available flags.

cluster-buildkit

With this mode, a BuildKit Deployment is dynamically created in each project namespace to perform in-cluster builds.

In this mode, builds are executed as follows:

1. BuildKit is automatically deployed to the project namespace, if it hasn't already been deployed there.

2. Your code (build context) is synchronized directly to the BuildKit deployment.

3. BuildKit imports caches from the deployment registry, builds the image, and then pushes the built image and caches back to the registry.

Comparison

This mode is a recent addition and is still considered experimental. However, the general plan is for this to become the recommended approach, because it has several benefits compared to the alternatives.

• It requires no cluster-wide services or permissions to be managed, and thus no permissions outside of a single namespace for each user/project.

• By extension, operators/users don't need to run a cluster initialization command ahead of building and deploying projects. The BuildKit deployment is automatically installed and updated ahead of builds, as needed.

• It does not rely on persistent volumes. Other modes need to either install an NFS provisioner, or for a ReadWriteMany storage class to be provided and configured by the user.

• BuildKit offers a rootless mode (see below for how to enable it and some caveats). If it's supported on your cluster, this coupled with the per-namespace isolation, makes cluster-buildkit by far the most secure option.

• BuildKit is a very efficient builder, and uses a combination of local and registry-based caching, so it should perform better than kaniko in most cases, and for long-running namespaces as good as cluster-docker.

Beyond being less tested in the wild (for the moment), there are a couple of drawbacks to consider:

• It doesn't scale quite as horizontally as Kaniko, since there is a single deployment per each project namespace, instead of a pod for every single build.

• The local cache is ephemeral, and local to each project namespace. This means users only share a cache at the registry level, much like with Kaniko. The cluster-docker daemon has a persistent local cache that is shared across a cluster (but in turn needs to be maintained and cleaned up). The effect of this is most pronounced for short-lived namespaces, e.g. ones created in CI runs, where the local cache won't exist ahead of the builds.

Configuration and requirements

Enable this mode by setting buildMode: cluster-buildkit in your kubernetes provider configuration. Unlike other remote building modes, no further cluster-wide installation or initialization is required.

In order to enable rootless mode, add the following to your kubernetes provider configuration:

clusterBuildkit:
  rootless: false

Note that not all clusters can currently support rootless operation, and that you may need to configure your cluster with this in mind. Please see the BuildKits docs for details.

You should also set the builder resource requests/limits. For this mode, the resource configuration applies to each BuildKit deployment, i.e. for each project namespace. See the builder resources reference for details.

The BuildKit deployments will always have the following toleration set:

key: "garden-build",
operator: "Equal",
value: "true",
effect: "NoSchedule"

This allows you to set corresponding Taints on cluster nodes to control which nodes builder deployments are deployed to.

cluster-docker

The cluster-docker mode installs a standalone Docker daemon into your cluster, that is then used for builds across all users of the clusters, along with a handful of other supporting services.

In this mode, builds are executed as follows:

1. Your code (build context) is synchronized to a sync service in the cluster, making it available to the Docker daemon.

2. A build is triggered in the Docker daemon.

3. The built image is pushed to the deployment registry, which makes it available to the cluster.

Comparison

The Docker daemon is of course tried and tested, and is an efficient builder. However, it's not designed with multi-tenancy and is a slightly awkward fit for the context of building images in a shared cluster. It also requires a fair bit of operation and several supporting services deployed along-side it in the garden-system namespace.

As of now, we only recommend this option for certain scenarios, e.g. clusters serving individuals, small teams or other low-load setups.

Configuration and requirements

Enable this mode by setting buildMode: cluster-docker in your kubernetes provider configuration.

After enabling this mode, you will need to run garden plugins kubernetes cluster-init --env=<env-name> for each applicable environment, in order to install the required cluster-wide services. Those services include the Docker daemon itself, as well as an image registry, a sync service for receiving build contexts, two persistent volumes, an NFS volume provisioner for one of those volumes, and a couple of small utility services.

By default, Garden will install an NFS volume provisioner into garden-system in order to be able to efficiently synchronize build sources to the cluster and then attaching those to the Kaniko pods. You can also specify a storageClass to provide another ReadWriteMany capable storage class to use instead of NFS. This may be advisable if your cloud provider provides a good alternative, or if you already have such a provisioner installed.

Optionally, you can also enable BuildKit to be used by the Docker daemon. This is not to be confused with the cluster-buildkit build mode, which doesn't use Docker at all. In most cases, this should work well and offer a bit of added performance, but it remains optional for now. If you have cluster-docker set as your buildMode you can enable BuildKit for an environment by adding the following to your kubernetes provider configuration:

clusterDocker:
  enableBuildKit: false

Make sure your cluster has enough resources and storage to support the required services, and keep in mind that these services are shared across all users of the cluster. Please look at the resources and storage sections in the provider reference for details.

Local Docker

This is the default mode. It is the least efficient one for remote clusters, but requires no additional configuration or services to be deployed to the cluster. For remote clusters, you do however need to explicitly configure a deployment registry, and obviously you'll need to have Docker running locally.

See the Local Docker builds section in the Remote Clusters guide for details.

Configuring a deployment registry

To deploy a built image to a remote Kubernetes cluster, the image first needs to be pushed to a container registry that is accessible to the cluster. We refer to this as a deployment registry. Garden offers two options to handle this process:

1. An in-cluster registry.

2. An external registry, e.g. a cloud provider managed registry like ECR or GCR. (recommended)

The in-cluster registry is a simple way to get started with Garden that requires no configuration. To set it up, leave the deploymentRegistry field on the kubernetes provider config undefined, and run garden plugins kubernetes cluster-init --env=<env-name> to install the registry. This is nice and convenient, but is not a particularly good approach for clusters with many users or lots of builds. When using the in-cluster registry you need to take care of cleaning it up routinely, and it may become a performance and redundancy bottleneck with many users and frequent (or heavy) builds.

So, for any scenario with a non-trivial amount of users and builds, we strongly suggest configuring a separate registry outside of your cluster. If your cloud provider offers a managed option, that's usually a good choice.

To configure a deployment registry, you need to specify at least the deploymentRegistry field on your kubernetes provider, and in many cases you also need to provide a Secret in order to authenticate with the registry via the imagePullSecrets field:

kind: Project
name: my-project
...
providers:
  - name: kubernetes
    ...
    deploymentRegistry:
      hostname: my-private-registry.com      # <--- the hostname of your registry
      namespace: my-project                  # <--- the namespace to use within your registry
    imagePullSecrets:
      - name: my-deployment-registry-secret  # <--- the name and namespace of a valid Kubernetes imagePullSecret
        namespace: default

Now say, if you specify hostname: my-registry.com and namespace: my-project-id for the deploymentRegistry field, and you have a container module named some-module in your project, it will be tagged and pushed to my-registry.com/my-project-id/some-module:v:<module-version> after building. That image ID will be then used in Kubernetes manifests when running containers.

For this to work, you in most cases also need to provide the authentication necessary for both the cluster to read the image and for the builder to push to the registry. We use the same format and mechanisms as Kubernetes imagePullSecrets for this. See this guide for how to create the secret, but keep in mind that for this context, the authentication provided must have write privileges to the configured registry and namespace.

See below for specific instruction for working with ECR.

Using in-cluster building with ECR

For AWS ECR (Elastic Container Registry), you need to enable the ECR credential helper once for the repository by adding an imagePullSecret for you ECR repository.

First create a config.json somewhere with the following contents (<aws_account_id> and <region> are placeholders that you need to replace for your repo):

{
  "credHelpers": {
    "<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login"
  }
}

Next create the imagePullSecret in your cluster (feel free to replace the default namespace, just make sure it's correctly referenced in the config below):

kubectl --namespace default create secret generic ecr-config \
  --from-file=.dockerconfigjson=./config.json \
  --type=kubernetes.io/dockerconfigjson

Finally, add the secret reference to your kubernetes provider configuration:

kind: Project
name: my-project
...
providers:
  - name: kubernetes
    ...
    imagePullSecrets:
      - name: ecr-config
        namespace: default

Using in-cluster building with GCR

To use in-cluster building with GCR (Google Container Registry) you need to set up authentication, with the following steps:

1. Create a Google Service Account (GSA).

2. Give the GSA the appropriate permissions.

3. Create a JSON key for the account.

4. Create an imagePullSecret for using the JSON key.

5. Add a reference to the imagePullSecret in your Garden project configuration.

First, create a Google Service Account:

# You can replace the gcr-access name of course, but make sure you also replace it in the commands below
gcloud iam service-accounts create gcr-access --project ${PROJECT_ID}

Then, to grant the Google Service account the right permission to push to GCR, run the following GCR commands:

# Create a role with the required permissions
gcloud iam roles create gcrAccess \
  --project ${PROJECT_ID} \
  --permissions=storage.objects.get,storage.objects.create,storage.objects.list,storage.objects.update,storage.objects.delete,storage.buckets.create,storage.buckets.get
​
# Attach the role to the newly create Google Service Account
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member=serviceAccount:[email protected]${PROJECT_ID}.iam.gserviceaccount.com \
  --role==projects/${PROJECT_ID}/roles/gcrAccess

Next create a JSON key file for the GSA:

gcloud iam service-accounts keys create keyfile.json --iam-account [email protected]${PROJECT_ID}.iam.gserviceaccount.com

Then prepare the imagePullSecret in your Kubernetes cluster. Run the following command, if appropriate replacing gcr.io with the correct registry hostname (e.g. eu.gcr.io or asia.gcr.io):

kubectl --namespace default create secret docker-registry gcr-config \
  --docker-server=gcr.io \
  --docker-username=_json_key \
  --docker-password="$(cat keyfile.json)"

Finally, add the created imagePullSecret to your kubernetes provider configuration:

kind: Project
name: my-project
...
providers:
  - name: kubernetes
    ...
    imagePullSecrets:
      - name: gcr-config
        namespace: default

Publishing images

You can publish images that have been built in your cluster, using the garden publish command.

The only caveat is that you currently need to have Docker running locally, and you need to have authenticated with the target registry. When publishing, we pull the image from the remote registry to the local Docker daemon, and then go on to push it from there. We do this to avoid having to (re-)implement all the various authentication methods (and by extension key management) involved in pushing directly from the cluster, and because it's often not desired to give clusters access to directly push to production registries.

Unless you're publishing to your configured deployment registry, you need to specify the image field on the container module in question, to indicate where the image should be published. For example:

kind: Module
name: my-module
image: my-repo/my-image:v1.2.3   # <- if you omit the tag here, the Garden module version will be used by default
...

By default we use the tag specified in the container module image field, if any. If none is set there, we default to the Garden module version.

You can also set the --tag option on the garden publish command to override the tag used for images. You can both set a specific tag or you can use template strings for the tag. For example, you can

• Set a specific tag on all published modules: garden publish --tag "v1.2.3"

• Set a custom prefix on tags but include the Garden version hash: garden publish --tag 'v0.1-${module.hash}'

• Set a custom prefix on tags with the current git branch: garden publish --tag 'v0.1-${git.branch}'

Generally, you can use any template strings available for module configs for the tags, with the addition of the following:

• ${module.name} — the name of module being tagged

• ${module.version} — the full Garden version of module being tagged, e.g. v-abcdef1234

• ${module.hash} — the Garden version hash of module being tagged, e.g. abcdef1234 (i.e. without the v- prefix)

Cleaning up cached images

In order to avoid disk-space issues in the cluster when using the in-cluster registry and/or either of the kaniko or cluster-docker build modes, the kubernetes provider exposes a utility command:

garden --env=<your-environment> plugins kubernetes cleanup-cluster-registry

The command does the following:

1. Looks through all Pods in the cluster to see which images/tags are in use, and flags all other images as deleted in the in-cluster registry and.

2. Restarts the registry in read-only mode.

3. Runs the registry garbage collection.

4. Restarts the registry again without the read-only mode.

5. When using the cluster-docker build mode, we additionally untag in the Docker daemon all images that are no longer in the registry, and then clean up the dangling image layers by running docker image prune.

There are plans to do this automatically when disk-space runs low, but for now you can run this manually or set up your own cron jobs.

You can avoid this entirely by using a remote deployment registry and the cluster-buildkit build mode.

Pulling base images from private registries

The in-cluster builder may need to be able to pull base images from a private registry, e.g. if your Dockerfile starts with something like this:

FROM my-private-registry.com/my-image:tag

where my-private-registry.com requires authorization.

For this to work, you need to create a registry secret in your cluster (see this guide for how to create the secret) and then configure the imagePullSecrets field in your kubernetes provider configuration:

kind: Project
name: my-project
...
providers:
  - name: kubernetes
    ...
    imagePullSecrets:
      # The name of the registry auth secret you created.
    - name: my-registry-secret
      # Change this if you store the secret in another namespace.
      namespace: default

This registry auth secret will then be copied and passed to the in-cluster builder. You can specify as many as you like, and they will be merged together.