kubernetes

Description

The kubernetes provider allows you to deploy container actions to Kubernetes clusters, and adds the helm and kubernetes action types.

For usage information, please refer to the guides section. A good place to start is the Remote Kubernetes guide guide if you're connecting to remote clusters. The Quickstart guide guide is also helpful as an introduction.

Note that if you're using a local Kubernetes cluster (e.g. minikube or Docker Desktop), the local-kubernetes provider simplifies (and automates) the configuration and setup quite a bit.

Please note that Garden is committed to supporting the latest officially supported versions. The information on the Kubernetes support and EOL timelines can be found here.

Below is the full schema reference for the provider configuration. For an introduction to configuring a Garden project with providers, please look at our configuration guide.

The reference is divided into two sections. The first section contains the complete YAML schema, and the second section describes each schema key.

Complete YAML Schema

The values in the schema below are the default values.

providers:
  - # List other providers that should be resolved before this one.
    dependencies: []

    # If specified, this provider will only be used in the listed environments. Note that an empty array effectively
    # disables the provider. To use a provider in all environments, omit this field.
    environments:

    # The container registry domain that should be used for pulling Garden utility images (such as the
    # image used in the Kubernetes sync utility Pod).
    #
    # If you have your own Docker Hub registry mirror, you can set the domain here and the utility images
    # will be pulled from there. This can be useful to e.g. avoid Docker Hub rate limiting.
    #
    # Otherwise the utility images are pulled directly from Docker Hub by default.
    utilImageRegistryDomain: docker.io

    # Choose the mechanism for building container images before deploying. By default your local Docker daemon is
    # used, but you can set it to `cluster-buildkit` or `kaniko` to sync files to the cluster, and build container
    # images there. This removes the need to run Docker locally, and allows you to share layer and image caches
    # between multiple developers, as well as between your development and CI workflows.
    #
    # For more details on all the different options and what makes sense to use for your setup, please check out the
    # [in-cluster building guide](https://docs.garden.io/kubernetes-plugins/guides/in-cluster-building).
    buildMode: local-docker

    # Configuration options for the `cluster-buildkit` build mode.
    clusterBuildkit: {}
      # Use the `cache` configuration to customize the default cluster-buildkit cache behaviour.
      #
      # The default value is:
      # clusterBuildkit:
      #   cache:
      #     - type: registry
      #       mode: auto
      #
      # For every build, this will
      # - import cached layers from a docker image tag named `_buildcache`
      # - when the build is finished, upload cache information to `_buildcache`
      #
      # For registries that support it, `mode: auto` (the default) will enable the buildkit `mode=max`
      # option.
      #
      # See the following table for details on our detection mechanism:
      #
      # | Registry Name                   | Registry Domain                    | Assumed `mode=max` support |
      # |---------------------------------|------------------------------------|------------------------------|
      # | AWS Elastic Container Registry  | `dkr.ecr.<region>.amazonaws.com` | Yes (with `image-manifest=true`) |
      # | Google Cloud Artifact Registry  | `pkg.dev`                        | Yes                          |
      # | Azure Container Registry        | `azurecr.io`                     | Yes                          |
      # | GitHub Container Registry       | `ghcr.io`                        | Yes                          |
      # | DockerHub                       | `hub.docker.com`                 | Yes                          |
      # | Any other registry              |                                    | No                           |
      #
      # In case you need to override the defaults for your registry, you can do it like so:
      #
      # clusterBuildkit:
      #   cache:
      #     - type: registry
      #       mode: max
      #
      # When you add multiple caches, we will make sure to pass the `--import-cache` options to buildkit in the same
      # order as provided in the cache configuration. This is because buildkit will not actually use all imported
      # caches
      # for every build, but it will stick with the first cache that yields a cache hit for all the following layers.
      #
      # An example for this is the following:
      #
      # clusterBuildkit:
      #   cache:
      #     - type: registry
      #       tag: _buildcache-${slice(kebabCase(git.branch), "0", "30")}
      #     - type: registry
      #       tag: _buildcache-main
      #       export: false
      #
      # Using this cache configuration, every build will first look for a cache specific to your feature branch.
      # If it does not exist yet, it will import caches from the main branch builds (`_buildcache-main`).
      # When the build is finished, it will only export caches to your feature branch, and avoid polluting the `main`
      # branch caches.
      # A configuration like that may improve your cache hit rate and thus save time.
      #
      # If you need to disable caches completely you can achieve that with the following configuration:
      #
      # clusterBuildkit:
      #   cache: []
      cache:
        - # Use the Docker registry configured at `deploymentRegistry` to retrieve and store buildkit cache
          # information.
          #
          # See also the [buildkit registry cache
          # documentation](https://github.com/moby/buildkit#registry-push-image-and-cache-separately)
          type:

          # The registry from which the cache should be imported from, or which it should be exported to.
          #
          # If not specified, use the configured `deploymentRegistry` in your kubernetes provider config.
          #
          # Important: You must make sure `imagePullSecrets` includes authentication with the specified cache
          # registry, that has the appropriate write privileges (usually full write access to the configured
          # `namespace`).
          registry:
            # The hostname (and optionally port, if not the default port) of the registry.
            hostname:

            # The port where the registry listens on, if not the default.
            port:

            # The registry namespace. Will be placed between hostname and image name, like so:
            # <hostname>/<namespace>/<image name>
            namespace:

            # Set to true to allow insecure connections to the registry (without SSL).
            insecure: false

          # This is the buildkit cache mode to be used.
          #
          # The value `inline` ensures that garden is using the buildkit option `--export-cache inline`. Cache
          # information will be inlined and co-located with the Docker image itself.
          #
          # The values `min` and `max` ensure that garden passes the `mode=max` or `mode=min` modifiers to the
          # buildkit `--export-cache` option. Cache manifests will only be
          # stored stored in the configured `tag`.
          #
          # `auto` is the same as `max` for some registries that are known to support it. Garden will fall back to
          # `inline` for all other registries.
          #  See the [clusterBuildkit cache option](#providersclusterbuildkitcache) for a description of the detection
          # mechanism.
          #
          # See also the [buildkit export cache documentation](https://github.com/moby/buildkit#export-cache)
          mode: auto

          # This is the Docker registry tag name buildkit should use for the registry build cache. Default is
          # `_buildcache`
          #
          # **NOTE**: `tag` can only be used together with the `registry` cache type
          tag: _buildcache

          # If this is false, only pass the `--import-cache` option to buildkit, and not the `--export-cache` option.
          # Defaults to true.
          export: true

      # Enable rootless mode for the cluster-buildkit daemon, which runs the daemon with decreased privileges.
      # Please see [the buildkit docs](https://github.com/moby/buildkit/blob/master/docs/rootless.md) for caveats when
      # using this mode.
      rootless: false

      # Exposes the `nodeSelector` field on the PodSpec of the BuildKit deployment. This allows you to constrain the
      # BuildKit daemon to only run on particular nodes.
      #
      # [See here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) for the official Kubernetes
      # guide to assigning Pods to nodes.
      nodeSelector: {}

      # Specify tolerations to apply to cluster-buildkit daemon. Useful to control which nodes in a cluster can run
      # builds.
      tolerations:
        - # "Effect" indicates the taint effect to match. Empty means match all taint effects. When specified,
          # allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
          effect:

          # "Key" is the taint key that the toleration applies to. Empty means match all taint keys.
          # If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
          key:

          # "Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal".
          # Defaults to
          # "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a
          # particular category.
          operator: Equal

          # "TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute",
          # otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate
          # the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately)
          # by the system.
          tolerationSeconds:

          # "Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be
          # empty,
          # otherwise just a regular string.
          value:

      # Specify annotations to apply to both the Pod and Deployment resources associated with cluster-buildkit.
      # Annotations may have an effect on the behaviour of certain components, for example autoscalers.
      annotations:

      # Specify annotations to apply to the Kubernetes service account used by cluster-buildkit. This can be useful to
      # set up IRSA with in-cluster building.
      serviceAccountAnnotations:

    # Setting related to Jib image builds.
    jib:
      # In some cases you may need to push images built with Jib to the remote registry via Kubernetes cluster, e.g.
      # if you don't have connectivity or access from where Garden is being run. In that case, set this flag to true,
      # but do note that the build will take considerably take longer to complete! Only applies when using in-cluster
      # building.
      pushViaCluster: false

    # Configuration options for the `kaniko` build mode.
    kaniko:
      # Specify extra flags to use when building the container image with kaniko. Flags set on `container` Builds take
      # precedence over these.
      extraFlags:

      # Change the kaniko image (repository/image:tag) to use when building in kaniko mode.
      image: >-
  gcr.io/kaniko-project/executor:v1.11.0-debug@sha256:32ba2214921892c2fa7b5f9c4ae6f8f026538ce6b2105a93a36a8b5ee50fe517

      # Choose the namespace where the Kaniko pods will be run. Defaults to the project namespace.
      namespace:

      # Exposes the `nodeSelector` field on the PodSpec of the Kaniko pods. This allows you to constrain the Kaniko
      # pods to only run on particular nodes. The same nodeSelector will be used for each util pod unless they are
      # specifically set under `util.nodeSelector`.
      #
      # [See here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) for the official Kubernetes
      # guide to assigning pods to nodes.
      nodeSelector:

      # Specify tolerations to apply to each Kaniko builder pod. Useful to control which nodes in a cluster can run
      # builds. The same tolerations will be used for each util pod unless they are specifically set under
      # `util.tolerations`
      tolerations:
        - # "Effect" indicates the taint effect to match. Empty means match all taint effects. When specified,
          # allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
          effect:

          # "Key" is the taint key that the toleration applies to. Empty means match all taint keys.
          # If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
          key:

          # "Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal".
          # Defaults to
          # "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a
          # particular category.
          operator: Equal

          # "TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute",
          # otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate
          # the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately)
          # by the system.
          tolerationSeconds:

          # "Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be
          # empty,
          # otherwise just a regular string.
          value:

      # Specify annotations to apply to each Kaniko builder pod. Annotations may have an effect on the behaviour of
      # certain components, for example autoscalers. The same annotations will be used for each util pod unless they
      # are specifically set under `util.annotations`
      annotations:

      # Specify annotations to apply to the Kubernetes service account used by kaniko. This can be useful to set up
      # IRSA with in-cluster building.
      serviceAccountAnnotations:

      util:
        # Specify tolerations to apply to each garden-util pod.
        tolerations:
          - # "Effect" indicates the taint effect to match. Empty means match all taint effects. When specified,
            # allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
            effect:

            # "Key" is the taint key that the toleration applies to. Empty means match all taint keys.
            # If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
            key:

            # "Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal".
            # Defaults to
            # "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a
            # particular category.
            operator: Equal

            # "TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute",
            # otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate
            # the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately)
            # by the system.
            tolerationSeconds:

            # "Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be
            # empty,
            # otherwise just a regular string.
            value:

        # Specify annotations to apply to each garden-util pod and deployments.
        annotations:

        # Specify the nodeSelector constraints for each garden-util pod.
        nodeSelector:

    # A default hostname to use when no hostname is explicitly configured for a service.
    defaultHostname:

    # Configuration options for code synchronization.
    sync:
      # Specifies default settings for syncs (e.g. for `container`, `kubernetes` and `helm` services).
      #
      # These are overridden/extended by the settings of any individual sync specs.
      #
      # Sync is enabled e.g by setting the `--sync` flag on the `garden deploy` command.
      #
      # See the [Code Synchronization guide](https://docs.garden.io/guides/code-synchronization) for more information.
      defaults:
        # Specify a list of POSIX-style paths or glob patterns that should be excluded from the sync.
        #
        # Any exclusion patterns defined in individual sync specs will be applied in addition to these patterns.
        #
        # `.git` directories and `.garden` directories are always ignored.
        exclude:

        # The default permission bits, specified as an octal, to set on files at the sync target. Defaults to 0o644
        # (user can read/write, everyone else can read). See the [Mutagen
        # docs](https://mutagen.io/documentation/synchronization/permissions#permissions) for more information.
        fileMode: 420

        # The default permission bits, specified as an octal, to set on directories at the sync target. Defaults to
        # 0o755 (user can read/write, everyone else can read). See the [Mutagen
        # docs](https://mutagen.io/documentation/synchronization/permissions#permissions) for more information.
        directoryMode: 493

        # Set the default owner of files and directories at the target. Specify either an integer ID or a string name.
        # See the [Mutagen docs](https://mutagen.io/documentation/synchronization/permissions#owners-and-groups) for
        # more information.
        owner:

        # Set the default group on files and directories at the target. Specify either an integer ID or a string name.
        # See the [Mutagen docs](https://mutagen.io/documentation/synchronization/permissions#owners-and-groups) for
        # more information.
        group:

    # Require SSL on all `container` Deploys. If set to true, an error is raised when no certificate is available for
    # a configured hostname on a `container`Deploy.
    forceSsl: false

    # References to `docker-registry` secrets to use for authenticating with remote registries when pulling
    # images. This is necessary if you reference private images in your action configuration, and is required
    # when configuring a remote Kubernetes environment with buildMode=local.
    imagePullSecrets:
      - # The name of the Kubernetes secret.
        name:

        # The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate
        # namespace before use.
        namespace: default

    # References to secrets you need to have copied into all namespaces deployed to. These secrets will be
    # ensured to exist in the namespace before deploying any service.
    copySecrets:
      - # The name of the Kubernetes secret.
        name:

        # The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate
        # namespace before use.
        namespace: default

    # Resource requests and limits for the in-cluster builder..
    resources:
      # Resource requests and limits for the in-cluster builder. It's important to consider which build mode you're
      # using when configuring this.
      #
      # When `buildMode` is `kaniko`, this refers to _each Kaniko pod_, i.e. each individual build, so you'll want to
      # consider the requirements for your individual image builds, with your most expensive/heavy images in mind.
      #
      # When `buildMode` is `cluster-buildkit`, this applies to the BuildKit deployment created in _each project
      # namespace_. So think of this as the resource spec for each individual user or project namespace.
      builder:
        limits:
          # CPU limit in millicpu.
          cpu: 4000

          # Memory limit in megabytes.
          memory: 8192

          # Ephemeral storage limit in megabytes.
          ephemeralStorage:

        requests:
          # CPU request in millicpu.
          cpu: 100

          # Memory request in megabytes.
          memory: 512

          # Ephemeral storage request in megabytes.
          ephemeralStorage:

      # Resource requests and limits for the util pod for in-cluster builders.
      # This pod is used to get, start, stop and inquire the status of the builds.
      #
      # This pod is created in each garden namespace.
      util:
        limits:
          # CPU limit in millicpu.
          cpu: 256

          # Memory limit in megabytes.
          memory: 512

          # Ephemeral storage limit in megabytes.
          ephemeralStorage:

        requests:
          # CPU request in millicpu.
          cpu: 256

          # Memory request in megabytes.
          memory: 512

          # Ephemeral storage request in megabytes.
          ephemeralStorage:

    # One or more certificates to use for ingress.
    tlsCertificates:
      - # A unique identifier for this certificate.
        name:

        # A list of hostnames that this certificate should be used for. If you don't specify these, they will be
        # automatically read from the certificate.
        hostnames:

        # A reference to the Kubernetes secret that contains the TLS certificate and key for the domain.
        secretRef:
          # The name of the Kubernetes secret.
          name:

          # The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate
          # namespace before use.
          namespace: default

    # Exposes the `nodeSelector` field on the PodSpec of system services. This allows you to constrain the system
    # services to only run on particular nodes.
    #
    # [See here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) for the official Kubernetes guide
    # to assigning Pods to nodes.
    systemNodeSelector: {}

    # The name of the provider plugin to use.
    name: kubernetes

    # The kubectl context to use to connect to the Kubernetes cluster.
    context:

    # The registry where built containers should be pushed to, and then pulled to the cluster when deploying services.
    #
    # Important: If you specify this in combination with in-cluster building, you must make sure `imagePullSecrets`
    # includes authentication with the specified deployment registry, that has the appropriate write privileges
    # (usually full write access to the configured `deploymentRegistry.namespace`).
    deploymentRegistry:
      # The hostname (and optionally port, if not the default port) of the registry.
      hostname:

      # The port where the registry listens on, if not the default.
      port:

      # The registry namespace. Will be placed between hostname and image name, like so: <hostname>/<namespace>/<image
      # name>
      namespace:

      # Set to true to allow insecure connections to the registry (without SSL).
      insecure: false

    # The ingress class or ingressClassName to use on configured Ingresses (via the `kubernetes.io/ingress.class`
    # annotation or `spec.ingressClassName` field depending on the kubernetes version)
    # when deploying `container` services. Use this if you have multiple ingress controllers in your cluster.
    ingressClass:

    # The external HTTP port of the cluster's ingress controller.
    ingressHttpPort: 80

    # The external HTTPS port of the cluster's ingress controller.
    ingressHttpsPort: 443

    # Path to kubeconfig file to use instead of the system default.
    kubeconfig:

    # Set a specific path to a kubectl binary, instead of having Garden download it automatically as required.
    #
    # It may be useful in some scenarios to allow individual users to set this, e.g. with an environment variable. You
    # could configure that with something like `kubectlPath: ${local.env.GARDEN_KUBECTL_PATH}?`.
    #
    # **Warning**: Garden may make some assumptions with respect to the kubectl version, so it is suggested to only
    # use this when necessary.
    kubectlPath:

    # Specify which namespace to deploy services to, and optionally annotations/labels to apply to the namespace.
    #
    # You can specify a string as a shorthand for `name: <name>`. Defaults to `<project name>-<environment
    # namespace>`.
    #
    # Note that the framework may generate other namespaces as well with this name as a prefix. Also note that if the
    # namespace previously exists, Garden will attempt to add the specified labels and annotations. If the user does
    # not have permissions to do so, a warning is shown.
    namespace:
      # A valid Kubernetes namespace name. Must be a valid RFC1035/RFC1123 (DNS) label (may contain lowercase letters,
      # numbers and dashes, must start with a letter, and cannot end with a dash) and must not be longer than 63
      # characters.
      name:

      # Map of annotations to apply to the namespace when creating it.
      annotations:

      # Map of labels to apply to the namespace when creating it.
      labels:

    # Set this to `nginx` to install/enable the NGINX ingress controller.
    setupIngressController: false

Configuration Keys

providers[]

providers[].dependencies[]

providers > dependencies

List other providers that should be resolved before this one.

Example:

providers:
  - dependencies:
      - exec

providers[].environments[]

providers > environments

If specified, this provider will only be used in the listed environments. Note that an empty array effectively disables the provider. To use a provider in all environments, omit this field.

Example:

providers:
  - environments:
      - dev
      - stage

providers[].utilImageRegistryDomain

providers > utilImageRegistryDomain

The container registry domain that should be used for pulling Garden utility images (such as the image used in the Kubernetes sync utility Pod).

If you have your own Docker Hub registry mirror, you can set the domain here and the utility images will be pulled from there. This can be useful to e.g. avoid Docker Hub rate limiting.

Otherwise the utility images are pulled directly from Docker Hub by default.

providers[].buildMode

providers > buildMode

Choose the mechanism for building container images before deploying. By default your local Docker daemon is used, but you can set it to cluster-buildkit or kaniko to sync files to the cluster, and build container images there. This removes the need to run Docker locally, and allows you to share layer and image caches between multiple developers, as well as between your development and CI workflows.

For more details on all the different options and what makes sense to use for your setup, please check out the in-cluster building guide.

providers[].clusterBuildkit

providers > clusterBuildkit

Configuration options for the cluster-buildkit build mode.

providers[].clusterBuildkit.cache[]

providers > clusterBuildkit > cache

Use the cache configuration to customize the default cluster-buildkit cache behaviour.

The default value is:

clusterBuildkit:
  cache:
    - type: registry
      mode: auto

For every build, this will

  • import cached layers from a docker image tag named _buildcache

  • when the build is finished, upload cache information to _buildcache

For registries that support it, mode: auto (the default) will enable the buildkit mode=max option.

See the following table for details on our detection mechanism:

In case you need to override the defaults for your registry, you can do it like so:

clusterBuildkit:
  cache:
    - type: registry
      mode: max

When you add multiple caches, we will make sure to pass the --import-cache options to buildkit in the same order as provided in the cache configuration. This is because buildkit will not actually use all imported caches for every build, but it will stick with the first cache that yields a cache hit for all the following layers.

An example for this is the following:

clusterBuildkit:
  cache:
    - type: registry
      tag: _buildcache-${slice(kebabCase(git.branch), "0", "30")}
    - type: registry
      tag: _buildcache-main
      export: false

Using this cache configuration, every build will first look for a cache specific to your feature branch. If it does not exist yet, it will import caches from the main branch builds (_buildcache-main). When the build is finished, it will only export caches to your feature branch, and avoid polluting the main branch caches. A configuration like that may improve your cache hit rate and thus save time.

If you need to disable caches completely you can achieve that with the following configuration:

clusterBuildkit:
  cache: []

providers[].clusterBuildkit.cache[].type

providers > clusterBuildkit > cache > type

Use the Docker registry configured at deploymentRegistry to retrieve and store buildkit cache information.

See also the buildkit registry cache documentation

providers[].clusterBuildkit.cache[].registry

providers > clusterBuildkit > cache > registry

The registry from which the cache should be imported from, or which it should be exported to.

If not specified, use the configured deploymentRegistry in your kubernetes provider config.

Important: You must make sure imagePullSecrets includes authentication with the specified cache registry, that has the appropriate write privileges (usually full write access to the configured namespace).

providers[].clusterBuildkit.cache[].registry.hostname

providers > clusterBuildkit > cache > registry > hostname

The hostname (and optionally port, if not the default port) of the registry.

Example:

providers:
  - clusterBuildkit:
      ...
      cache:
        - registry:
            ...
            hostname: "gcr.io"

providers[].clusterBuildkit.cache[].registry.port

providers > clusterBuildkit > cache > registry > port

The port where the registry listens on, if not the default.

providers[].clusterBuildkit.cache[].registry.namespace

providers > clusterBuildkit > cache > registry > namespace

The registry namespace. Will be placed between hostname and image name, like so: //

Example:

providers:
  - clusterBuildkit:
      ...
      cache:
        - registry:
            ...
            namespace: "my-project"

providers[].clusterBuildkit.cache[].registry.insecure

providers > clusterBuildkit > cache > registry > insecure

Set to true to allow insecure connections to the registry (without SSL).

providers[].clusterBuildkit.cache[].mode

providers > clusterBuildkit > cache > mode

This is the buildkit cache mode to be used.

The value inline ensures that garden is using the buildkit option --export-cache inline. Cache information will be inlined and co-located with the Docker image itself.

The values min and max ensure that garden passes the mode=max or mode=min modifiers to the buildkit --export-cache option. Cache manifests will only be stored stored in the configured tag.

auto is the same as max for some registries that are known to support it. Garden will fall back to inline for all other registries. See the clusterBuildkit cache option for a description of the detection mechanism.

See also the buildkit export cache documentation

providers[].clusterBuildkit.cache[].tag

providers > clusterBuildkit > cache > tag

This is the Docker registry tag name buildkit should use for the registry build cache. Default is _buildcache

NOTE: tag can only be used together with the registry cache type

providers[].clusterBuildkit.cache[].export

providers > clusterBuildkit > cache > export

If this is false, only pass the --import-cache option to buildkit, and not the --export-cache option. Defaults to true.

providers[].clusterBuildkit.rootless

providers > clusterBuildkit > rootless

Enable rootless mode for the cluster-buildkit daemon, which runs the daemon with decreased privileges. Please see the buildkit docs for caveats when using this mode.

providers[].clusterBuildkit.nodeSelector

providers > clusterBuildkit > nodeSelector

Exposes the nodeSelector field on the PodSpec of the BuildKit deployment. This allows you to constrain the BuildKit daemon to only run on particular nodes.

See here for the official Kubernetes guide to assigning Pods to nodes.

Example:

providers:
  - clusterBuildkit:
      ...
      nodeSelector:
          disktype: ssd

providers[].clusterBuildkit.tolerations[]

providers > clusterBuildkit > tolerations

Specify tolerations to apply to cluster-buildkit daemon. Useful to control which nodes in a cluster can run builds.

providers[].clusterBuildkit.tolerations[].effect

providers > clusterBuildkit > tolerations > effect

"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".

providers[].clusterBuildkit.tolerations[].key

providers > clusterBuildkit > tolerations > key

"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.

providers[].clusterBuildkit.tolerations[].operator

providers > clusterBuildkit > tolerations > operator

"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.

providers[].clusterBuildkit.tolerations[].tolerationSeconds

providers > clusterBuildkit > tolerations > tolerationSeconds

"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.

providers[].clusterBuildkit.tolerations[].value

providers > clusterBuildkit > tolerations > value

"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.

providers[].clusterBuildkit.annotations

providers > clusterBuildkit > annotations

Specify annotations to apply to both the Pod and Deployment resources associated with cluster-buildkit. Annotations may have an effect on the behaviour of certain components, for example autoscalers.

Example:

providers:
  - clusterBuildkit:
      ...
      annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'

providers[].clusterBuildkit.serviceAccountAnnotations

providers > clusterBuildkit > serviceAccountAnnotations

Specify annotations to apply to the Kubernetes service account used by cluster-buildkit. This can be useful to set up IRSA with in-cluster building.

Example:

providers:
  - clusterBuildkit:
      ...
      serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role

providers[].jib

providers > jib

Setting related to Jib image builds.

providers[].jib.pushViaCluster

providers > jib > pushViaCluster

In some cases you may need to push images built with Jib to the remote registry via Kubernetes cluster, e.g. if you don't have connectivity or access from where Garden is being run. In that case, set this flag to true, but do note that the build will take considerably take longer to complete! Only applies when using in-cluster building.

providers[].kaniko

providers > kaniko

Configuration options for the kaniko build mode.

providers[].kaniko.extraFlags[]

providers > kaniko > extraFlags

Specify extra flags to use when building the container image with kaniko. Flags set on container Builds take precedence over these.

providers[].kaniko.image

providers > kaniko > image

Change the kaniko image (repository/image:tag) to use when building in kaniko mode.

providers[].kaniko.namespace

providers > kaniko > namespace

Choose the namespace where the Kaniko pods will be run. Defaults to the project namespace.

providers[].kaniko.nodeSelector

providers > kaniko > nodeSelector

Exposes the nodeSelector field on the PodSpec of the Kaniko pods. This allows you to constrain the Kaniko pods to only run on particular nodes. The same nodeSelector will be used for each util pod unless they are specifically set under util.nodeSelector.

See here for the official Kubernetes guide to assigning pods to nodes.

providers[].kaniko.tolerations[]

providers > kaniko > tolerations

Specify tolerations to apply to each Kaniko builder pod. Useful to control which nodes in a cluster can run builds. The same tolerations will be used for each util pod unless they are specifically set under util.tolerations

providers[].kaniko.tolerations[].effect

providers > kaniko > tolerations > effect

"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".

providers[].kaniko.tolerations[].key

providers > kaniko > tolerations > key

"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.

providers[].kaniko.tolerations[].operator

providers > kaniko > tolerations > operator

"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.

providers[].kaniko.tolerations[].tolerationSeconds

providers > kaniko > tolerations > tolerationSeconds

"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.

providers[].kaniko.tolerations[].value

providers > kaniko > tolerations > value

"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.

providers[].kaniko.annotations

providers > kaniko > annotations

Specify annotations to apply to each Kaniko builder pod. Annotations may have an effect on the behaviour of certain components, for example autoscalers. The same annotations will be used for each util pod unless they are specifically set under util.annotations

Example:

providers:
  - kaniko:
      ...
      annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'

providers[].kaniko.serviceAccountAnnotations

providers > kaniko > serviceAccountAnnotations

Specify annotations to apply to the Kubernetes service account used by kaniko. This can be useful to set up IRSA with in-cluster building.

Example:

providers:
  - kaniko:
      ...
      serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role

providers[].kaniko.util

providers > kaniko > util

providers[].kaniko.util.tolerations[]

providers > kaniko > util > tolerations

Specify tolerations to apply to each garden-util pod.

providers[].kaniko.util.tolerations[].effect

providers > kaniko > util > tolerations > effect

"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".

providers[].kaniko.util.tolerations[].key

providers > kaniko > util > tolerations > key

"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.

providers[].kaniko.util.tolerations[].operator

providers > kaniko > util > tolerations > operator

"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.

providers[].kaniko.util.tolerations[].tolerationSeconds

providers > kaniko > util > tolerations > tolerationSeconds

"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.

providers[].kaniko.util.tolerations[].value

providers > kaniko > util > tolerations > value

"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.

providers[].kaniko.util.annotations

providers > kaniko > util > annotations

Specify annotations to apply to each garden-util pod and deployments.

Example:

providers:
  - kaniko:
      ...
      util:
        ...
        annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'

providers[].kaniko.util.nodeSelector

providers > kaniko > util > nodeSelector

Specify the nodeSelector constraints for each garden-util pod.

providers[].defaultHostname

providers > defaultHostname

A default hostname to use when no hostname is explicitly configured for a service.

Example:

providers:
  - defaultHostname: "api.mydomain.com"

providers[].deploymentStrategy

providers > deploymentStrategy

Experimental: this is an experimental feature and the API might change in the future.

Deprecated: This field will be removed in a future release.

Sets the deployment strategy for container deploy actions.

Note that this field has been deprecated since 0.13, and has no effect. The "rolling" will be applied in all cases. The experimental support for blue/green deployments (via the "blue-green" strategy) has been removed.

Note that this setting only applies to container deploy actions (and not, for example, kubernetes or helm deploy actions).

providers[].sync

providers > sync

Configuration options for code synchronization.

providers[].sync.defaults

providers > sync > defaults

Specifies default settings for syncs (e.g. for container, kubernetes and helm services).

These are overridden/extended by the settings of any individual sync specs.

Sync is enabled e.g by setting the --sync flag on the garden deploy command.

See the Code Synchronization guide for more information.

providers[].sync.defaults.exclude[]

providers > sync > defaults > exclude

Specify a list of POSIX-style paths or glob patterns that should be excluded from the sync.

Any exclusion patterns defined in individual sync specs will be applied in addition to these patterns.

.git directories and .garden directories are always ignored.

Example:

providers:
  - sync:
      ...
      defaults:
        ...
        exclude:
          - dist/**/*
          - '*.log'

providers[].sync.defaults.fileMode

providers > sync > defaults > fileMode

The default permission bits, specified as an octal, to set on files at the sync target. Defaults to 0o644 (user can read/write, everyone else can read). See the Mutagen docs for more information.

providers[].sync.defaults.directoryMode

providers > sync > defaults > directoryMode

The default permission bits, specified as an octal, to set on directories at the sync target. Defaults to 0o755 (user can read/write, everyone else can read). See the Mutagen docs for more information.

providers[].sync.defaults.owner

providers > sync > defaults > owner

Set the default owner of files and directories at the target. Specify either an integer ID or a string name. See the Mutagen docs for more information.

providers[].sync.defaults.group

providers > sync > defaults > group

Set the default group on files and directories at the target. Specify either an integer ID or a string name. See the Mutagen docs for more information.

providers[].forceSsl

providers > forceSsl

Require SSL on all container Deploys. If set to true, an error is raised when no certificate is available for a configured hostname on a containerDeploy.

providers[].imagePullSecrets[]

providers > imagePullSecrets

References to docker-registry secrets to use for authenticating with remote registries when pulling images. This is necessary if you reference private images in your action configuration, and is required when configuring a remote Kubernetes environment with buildMode=local.

providers[].imagePullSecrets[].name

providers > imagePullSecrets > name

The name of the Kubernetes secret.

Example:

providers:
  - imagePullSecrets:
      - name: "my-secret"

providers[].imagePullSecrets[].namespace

providers > imagePullSecrets > namespace

The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate namespace before use.

providers[].copySecrets[]

providers > copySecrets

References to secrets you need to have copied into all namespaces deployed to. These secrets will be ensured to exist in the namespace before deploying any service.

providers[].copySecrets[].name

providers > copySecrets > name

The name of the Kubernetes secret.

Example:

providers:
  - copySecrets:
      - name: "my-secret"

providers[].copySecrets[].namespace

providers > copySecrets > namespace

The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate namespace before use.

providers[].resources

providers > resources

Resource requests and limits for the in-cluster builder..

providers[].resources.builder

providers > resources > builder

Resource requests and limits for the in-cluster builder. It's important to consider which build mode you're using when configuring this.

When buildMode is kaniko, this refers to each Kaniko pod, i.e. each individual build, so you'll want to consider the requirements for your individual image builds, with your most expensive/heavy images in mind.

When buildMode is cluster-buildkit, this applies to the BuildKit deployment created in each project namespace. So think of this as the resource spec for each individual user or project namespace.

providers[].resources.builder.limits

providers > resources > builder > limits

providers[].resources.builder.limits.cpu

providers > resources > builder > limits > cpu

CPU limit in millicpu.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        limits:
          ...
          cpu: 4000

providers[].resources.builder.limits.memory

providers > resources > builder > limits > memory

Memory limit in megabytes.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        limits:
          ...
          memory: 8192

providers[].resources.builder.limits.ephemeralStorage

providers > resources > builder > limits > ephemeralStorage

Ephemeral storage limit in megabytes.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        limits:
          ...
          ephemeralStorage: 8192

providers[].resources.builder.requests

providers > resources > builder > requests

providers[].resources.builder.requests.cpu

providers > resources > builder > requests > cpu

CPU request in millicpu.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        requests:
          ...
          cpu: 100

providers[].resources.builder.requests.memory

providers > resources > builder > requests > memory

Memory request in megabytes.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        requests:
          ...
          memory: 512

providers[].resources.builder.requests.ephemeralStorage

providers > resources > builder > requests > ephemeralStorage

Ephemeral storage request in megabytes.

Example:

providers:
  - resources:
      ...
      builder:
        ...
        requests:
          ...
          ephemeralStorage: 8192

providers[].resources.util

providers > resources > util

Resource requests and limits for the util pod for in-cluster builders. This pod is used to get, start, stop and inquire the status of the builds.

This pod is created in each garden namespace.

providers[].resources.util.limits

providers > resources > util > limits

providers[].resources.util.limits.cpu

providers > resources > util > limits > cpu

CPU limit in millicpu.

Example:

providers:
  - resources:
      ...
      util:
        ...
        limits:
          ...
          cpu: 256

providers[].resources.util.limits.memory

providers > resources > util > limits > memory

Memory limit in megabytes.

Example:

providers:
  - resources:
      ...
      util:
        ...
        limits:
          ...
          memory: 512

providers[].resources.util.limits.ephemeralStorage

providers > resources > util > limits > ephemeralStorage

Ephemeral storage limit in megabytes.

Example:

providers:
  - resources:
      ...
      util:
        ...
        limits:
          ...
          ephemeralStorage: 8192

providers[].resources.util.requests

providers > resources > util > requests

providers[].resources.util.requests.cpu

providers > resources > util > requests > cpu

CPU request in millicpu.

Example:

providers:
  - resources:
      ...
      util:
        ...
        requests:
          ...
          cpu: 256

providers[].resources.util.requests.memory

providers > resources > util > requests > memory

Memory request in megabytes.

Example:

providers:
  - resources:
      ...
      util:
        ...
        requests:
          ...
          memory: 512

providers[].resources.util.requests.ephemeralStorage

providers > resources > util > requests > ephemeralStorage

Ephemeral storage request in megabytes.

Example:

providers:
  - resources:
      ...
      util:
        ...
        requests:
          ...
          ephemeralStorage: 8192

providers[].resources.sync

providers > resources > sync

Deprecated: This field will be removed in a future release.

Resource requests and limits for the code sync service, which we use to sync build contexts to the cluster ahead of building images. This generally is not resource intensive, but you might want to adjust the defaults if you have many concurrent users.

providers[].resources.sync.limits

providers > resources > sync > limits

Deprecated: This field will be removed in a future release.

providers[].resources.sync.limits.cpu

providers > resources > sync > limits > cpu

Deprecated: This field will be removed in a future release.

CPU limit in millicpu.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        limits:
          ...
          cpu: 500

providers[].resources.sync.limits.memory

providers > resources > sync > limits > memory

Deprecated: This field will be removed in a future release.

Memory limit in megabytes.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        limits:
          ...
          memory: 512

providers[].resources.sync.limits.ephemeralStorage

providers > resources > sync > limits > ephemeralStorage

Deprecated: This field will be removed in a future release.

Ephemeral storage limit in megabytes.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        limits:
          ...
          ephemeralStorage: 8192

providers[].resources.sync.requests

providers > resources > sync > requests

Deprecated: This field will be removed in a future release.

providers[].resources.sync.requests.cpu

providers > resources > sync > requests > cpu

Deprecated: This field will be removed in a future release.

CPU request in millicpu.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        requests:
          ...
          cpu: 100

providers[].resources.sync.requests.memory

providers > resources > sync > requests > memory

Deprecated: This field will be removed in a future release.

Memory request in megabytes.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        requests:
          ...
          memory: 90

providers[].resources.sync.requests.ephemeralStorage

providers > resources > sync > requests > ephemeralStorage

Deprecated: This field will be removed in a future release.

Ephemeral storage request in megabytes.

Example:

providers:
  - resources:
      ...
      sync:
        ...
        requests:
          ...
          ephemeralStorage: 8192

providers[].tlsCertificates[]

providers > tlsCertificates

One or more certificates to use for ingress.

providers[].tlsCertificates[].name

providers > tlsCertificates > name

A unique identifier for this certificate.

Example:

providers:
  - tlsCertificates:
      - name: "www"

providers[].tlsCertificates[].hostnames[]

providers > tlsCertificates > hostnames

A list of hostnames that this certificate should be used for. If you don't specify these, they will be automatically read from the certificate.

Example:

providers:
  - tlsCertificates:
      - hostnames:
          - www.mydomain.com

providers[].tlsCertificates[].secretRef

providers > tlsCertificates > secretRef

A reference to the Kubernetes secret that contains the TLS certificate and key for the domain.

Example:

providers:
  - tlsCertificates:
      - secretRef:
            name: my-tls-secret
            namespace: default

providers[].tlsCertificates[].secretRef.name

providers > tlsCertificates > secretRef > name

The name of the Kubernetes secret.

Example:

providers:
  - tlsCertificates:
      - secretRef:
            name: my-tls-secret
            namespace: default
          ...
          name: "my-secret"

providers[].tlsCertificates[].secretRef.namespace

providers > tlsCertificates > secretRef > namespace

The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate namespace before use.

providers[].systemNodeSelector

providers > systemNodeSelector

Exposes the nodeSelector field on the PodSpec of system services. This allows you to constrain the system services to only run on particular nodes.

See here for the official Kubernetes guide to assigning Pods to nodes.

Example:

providers:
  - systemNodeSelector:
        disktype: ssd

providers[].name

providers > name

The name of the provider plugin to use.

Example:

providers:
  - name: "kubernetes"

providers[].context

providers > context

The kubectl context to use to connect to the Kubernetes cluster.

Example:

providers:
  - context: "my-dev-context"

providers[].deploymentRegistry

providers > deploymentRegistry

The registry where built containers should be pushed to, and then pulled to the cluster when deploying services.

Important: If you specify this in combination with in-cluster building, you must make sure imagePullSecrets includes authentication with the specified deployment registry, that has the appropriate write privileges (usually full write access to the configured deploymentRegistry.namespace).

providers[].deploymentRegistry.hostname

providers > deploymentRegistry > hostname

The hostname (and optionally port, if not the default port) of the registry.

Example:

providers:
  - deploymentRegistry:
      ...
      hostname: "gcr.io"

providers[].deploymentRegistry.port

providers > deploymentRegistry > port

The port where the registry listens on, if not the default.

providers[].deploymentRegistry.namespace

providers > deploymentRegistry > namespace

The registry namespace. Will be placed between hostname and image name, like so: //

Example:

providers:
  - deploymentRegistry:
      ...
      namespace: "my-project"

providers[].deploymentRegistry.insecure

providers > deploymentRegistry > insecure

Set to true to allow insecure connections to the registry (without SSL).

providers[].ingressClass

providers > ingressClass

The ingress class or ingressClassName to use on configured Ingresses (via the kubernetes.io/ingress.class annotation or spec.ingressClassName field depending on the kubernetes version) when deploying container services. Use this if you have multiple ingress controllers in your cluster.

providers[].ingressHttpPort

providers > ingressHttpPort

The external HTTP port of the cluster's ingress controller.

providers[].ingressHttpsPort

providers > ingressHttpsPort

The external HTTPS port of the cluster's ingress controller.

providers[].kubeconfig

providers > kubeconfig

Path to kubeconfig file to use instead of the system default.

providers[].kubectlPath

providers > kubectlPath

Set a specific path to a kubectl binary, instead of having Garden download it automatically as required.

It may be useful in some scenarios to allow individual users to set this, e.g. with an environment variable. You could configure that with something like kubectlPath: ${local.env.GARDEN_KUBECTL_PATH}?.

Warning: Garden may make some assumptions with respect to the kubectl version, so it is suggested to only use this when necessary.

providers[].namespace

providers > namespace

Specify which namespace to deploy services to, and optionally annotations/labels to apply to the namespace.

You can specify a string as a shorthand for name: <name>. Defaults to <project name>-<environment namespace>.

Note that the framework may generate other namespaces as well with this name as a prefix. Also note that if the namespace previously exists, Garden will attempt to add the specified labels and annotations. If the user does not have permissions to do so, a warning is shown.

providers[].namespace.name

providers > namespace > name

A valid Kubernetes namespace name. Must be a valid RFC1035/RFC1123 (DNS) label (may contain lowercase letters, numbers and dashes, must start with a letter, and cannot end with a dash) and must not be longer than 63 characters.

providers[].namespace.annotations

providers > namespace > annotations

Map of annotations to apply to the namespace when creating it.

Example:

providers:
  - namespace: ''
      ...
      annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'

providers[].namespace.labels

providers > namespace > labels

Map of labels to apply to the namespace when creating it.

providers[].setupIngressController

providers > setupIngressController

Set this to nginx to install/enable the NGINX ingress controller.

Outputs

The following keys are available via the ${providers.<provider-name>} template string key for kubernetes providers.

${providers.<provider-name>.outputs.app-namespace}

The primary namespace used for resource deployments.

${providers.<provider-name>.outputs.default-hostname}

The default hostname configured on the provider.

Last updated