kubernetes
Description
The kubernetes
provider allows you to deploy container
actions to Kubernetes clusters, and adds the helm
and kubernetes
action types.
For usage information, please refer to the guides section. A good place to start is the Remote Kubernetes guide guide if you're connecting to remote clusters. The Quickstart guide guide is also helpful as an introduction.
Note that if you're using a local Kubernetes cluster (e.g. minikube or Docker Desktop), the local-kubernetes provider simplifies (and automates) the configuration and setup quite a bit.
Please note that Garden is committed to supporting the latest officially supported versions. The information on the Kubernetes support and EOL timelines can be found here.
Below is the full schema reference for the provider configuration. For an introduction to configuring a Garden project with providers, please look at our configuration guide.
The reference is divided into two sections. The first section contains the complete YAML schema, and the second section describes each schema key.
Complete YAML Schema
The values in the schema below are the default values.
Configuration Keys
providers[]
providers[]
providers[].dependencies[]
providers[].dependencies[]
providers > dependencies
List other providers that should be resolved before this one.
Example:
providers[].environments[]
providers[].environments[]
providers > environments
If specified, this provider will only be used in the listed environments. Note that an empty array effectively disables the provider. To use a provider in all environments, omit this field.
Example:
providers[].utilImageRegistryDomain
providers[].utilImageRegistryDomain
providers > utilImageRegistryDomain
The container registry domain that should be used for pulling Garden utility images (such as the image used in the Kubernetes sync utility Pod).
If you have your own Docker Hub registry mirror, you can set the domain here and the utility images will be pulled from there. This can be useful to e.g. avoid Docker Hub rate limiting.
Otherwise the utility images are pulled directly from Docker Hub by default.
providers[].buildMode
providers[].buildMode
providers > buildMode
Choose the mechanism for building container images before deploying. By default your local Docker daemon is used, but you can set it to cluster-buildkit
or kaniko
to sync files to the cluster, and build container images there. This removes the need to run Docker locally, and allows you to share layer and image caches between multiple developers, as well as between your development and CI workflows.
For more details on all the different options and what makes sense to use for your setup, please check out the in-cluster building guide.
providers[].clusterBuildkit
providers[].clusterBuildkit
providers > clusterBuildkit
Configuration options for the cluster-buildkit
build mode.
providers[].clusterBuildkit.cache[]
providers[].clusterBuildkit.cache[]
providers > clusterBuildkit > cache
Use the cache
configuration to customize the default cluster-buildkit cache behaviour.
The default value is:
For every build, this will
import cached layers from a docker image tag named
_buildcache
when the build is finished, upload cache information to
_buildcache
For registries that support it, mode: auto
(the default) will enable the buildkit mode=max
option.
See the following table for details on our detection mechanism:
In case you need to override the defaults for your registry, you can do it like so:
When you add multiple caches, we will make sure to pass the --import-cache
options to buildkit in the same order as provided in the cache configuration. This is because buildkit will not actually use all imported caches for every build, but it will stick with the first cache that yields a cache hit for all the following layers.
An example for this is the following:
Using this cache configuration, every build will first look for a cache specific to your feature branch. If it does not exist yet, it will import caches from the main branch builds (_buildcache-main
). When the build is finished, it will only export caches to your feature branch, and avoid polluting the main
branch caches. A configuration like that may improve your cache hit rate and thus save time.
If you need to disable caches completely you can achieve that with the following configuration:
providers[].clusterBuildkit.cache[].type
providers[].clusterBuildkit.cache[].type
providers > clusterBuildkit > cache > type
Use the Docker registry configured at deploymentRegistry
to retrieve and store buildkit cache information.
See also the buildkit registry cache documentation
providers[].clusterBuildkit.cache[].registry
providers[].clusterBuildkit.cache[].registry
providers > clusterBuildkit > cache > registry
The registry from which the cache should be imported from, or which it should be exported to.
If not specified, use the configured deploymentRegistry
in your kubernetes provider config.
Important: You must make sure imagePullSecrets
includes authentication with the specified cache registry, that has the appropriate write privileges (usually full write access to the configured namespace
).
providers[].clusterBuildkit.cache[].registry.hostname
providers[].clusterBuildkit.cache[].registry.hostname
providers > clusterBuildkit > cache > registry > hostname
The hostname (and optionally port, if not the default port) of the registry.
Example:
providers[].clusterBuildkit.cache[].registry.port
providers[].clusterBuildkit.cache[].registry.port
providers > clusterBuildkit > cache > registry > port
The port where the registry listens on, if not the default.
providers[].clusterBuildkit.cache[].registry.namespace
providers[].clusterBuildkit.cache[].registry.namespace
providers > clusterBuildkit > cache > registry > namespace
The registry namespace. Will be placed between hostname and image name, like so: //
Example:
providers[].clusterBuildkit.cache[].registry.insecure
providers[].clusterBuildkit.cache[].registry.insecure
providers > clusterBuildkit > cache > registry > insecure
Set to true to allow insecure connections to the registry (without SSL).
providers[].clusterBuildkit.cache[].mode
providers[].clusterBuildkit.cache[].mode
providers > clusterBuildkit > cache > mode
This is the buildkit cache mode to be used.
The value inline
ensures that garden is using the buildkit option --export-cache inline
. Cache information will be inlined and co-located with the Docker image itself.
The values min
and max
ensure that garden passes the mode=max
or mode=min
modifiers to the buildkit --export-cache
option. Cache manifests will only be stored stored in the configured tag
.
auto
is the same as max
for some registries that are known to support it. Garden will fall back to inline
for all other registries. See the clusterBuildkit cache option for a description of the detection mechanism.
See also the buildkit export cache documentation
providers[].clusterBuildkit.cache[].tag
providers[].clusterBuildkit.cache[].tag
providers > clusterBuildkit > cache > tag
This is the Docker registry tag name buildkit should use for the registry build cache. Default is _buildcache
NOTE: tag
can only be used together with the registry
cache type
providers[].clusterBuildkit.cache[].export
providers[].clusterBuildkit.cache[].export
providers > clusterBuildkit > cache > export
If this is false, only pass the --import-cache
option to buildkit, and not the --export-cache
option. Defaults to true.
providers[].clusterBuildkit.rootless
providers[].clusterBuildkit.rootless
providers > clusterBuildkit > rootless
Enable rootless mode for the cluster-buildkit daemon, which runs the daemon with decreased privileges. Please see the buildkit docs for caveats when using this mode.
providers[].clusterBuildkit.nodeSelector
providers[].clusterBuildkit.nodeSelector
providers > clusterBuildkit > nodeSelector
Exposes the nodeSelector
field on the PodSpec of the BuildKit deployment. This allows you to constrain the BuildKit daemon to only run on particular nodes.
See here for the official Kubernetes guide to assigning Pods to nodes.
Example:
providers[].clusterBuildkit.tolerations[]
providers[].clusterBuildkit.tolerations[]
providers > clusterBuildkit > tolerations
Specify tolerations to apply to cluster-buildkit daemon. Useful to control which nodes in a cluster can run builds.
providers[].clusterBuildkit.tolerations[].effect
providers[].clusterBuildkit.tolerations[].effect
providers > clusterBuildkit > tolerations > effect
"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
providers[].clusterBuildkit.tolerations[].key
providers[].clusterBuildkit.tolerations[].key
providers > clusterBuildkit > tolerations > key
"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
providers[].clusterBuildkit.tolerations[].operator
providers[].clusterBuildkit.tolerations[].operator
providers > clusterBuildkit > tolerations > operator
"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
providers[].clusterBuildkit.tolerations[].tolerationSeconds
providers[].clusterBuildkit.tolerations[].tolerationSeconds
providers > clusterBuildkit > tolerations > tolerationSeconds
"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
providers[].clusterBuildkit.tolerations[].value
providers[].clusterBuildkit.tolerations[].value
providers > clusterBuildkit > tolerations > value
"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.
providers[].clusterBuildkit.annotations
providers[].clusterBuildkit.annotations
providers > clusterBuildkit > annotations
Specify annotations to apply to both the Pod and Deployment resources associated with cluster-buildkit. Annotations may have an effect on the behaviour of certain components, for example autoscalers.
Example:
providers[].clusterBuildkit.serviceAccountAnnotations
providers[].clusterBuildkit.serviceAccountAnnotations
providers > clusterBuildkit > serviceAccountAnnotations
Specify annotations to apply to the Kubernetes service account used by cluster-buildkit. This can be useful to set up IRSA with in-cluster building.
Example:
providers[].jib
providers[].jib
providers > jib
Setting related to Jib image builds.
providers[].jib.pushViaCluster
providers[].jib.pushViaCluster
providers > jib > pushViaCluster
In some cases you may need to push images built with Jib to the remote registry via Kubernetes cluster, e.g. if you don't have connectivity or access from where Garden is being run. In that case, set this flag to true, but do note that the build will take considerably take longer to complete! Only applies when using in-cluster building.
providers[].kaniko
providers[].kaniko
providers > kaniko
Configuration options for the kaniko
build mode.
providers[].kaniko.extraFlags[]
providers[].kaniko.extraFlags[]
providers > kaniko > extraFlags
Specify extra flags to use when building the container image with kaniko. Flags set on container
Builds take precedence over these.
providers[].kaniko.image
providers[].kaniko.image
Change the kaniko image (repository/image:tag) to use when building in kaniko mode.
providers[].kaniko.namespace
providers[].kaniko.namespace
providers > kaniko > namespace
Choose the namespace where the Kaniko pods will be run. Defaults to the project namespace.
providers[].kaniko.nodeSelector
providers[].kaniko.nodeSelector
providers > kaniko > nodeSelector
Exposes the nodeSelector
field on the PodSpec of the Kaniko pods. This allows you to constrain the Kaniko pods to only run on particular nodes. The same nodeSelector will be used for each util pod unless they are specifically set under util.nodeSelector
.
See here for the official Kubernetes guide to assigning pods to nodes.
providers[].kaniko.tolerations[]
providers[].kaniko.tolerations[]
providers > kaniko > tolerations
Specify tolerations to apply to each Kaniko builder pod. Useful to control which nodes in a cluster can run builds. The same tolerations will be used for each util pod unless they are specifically set under util.tolerations
providers[].kaniko.tolerations[].effect
providers[].kaniko.tolerations[].effect
providers > kaniko > tolerations > effect
"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
providers[].kaniko.tolerations[].key
providers[].kaniko.tolerations[].key
providers > kaniko > tolerations > key
"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
providers[].kaniko.tolerations[].operator
providers[].kaniko.tolerations[].operator
providers > kaniko > tolerations > operator
"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
providers[].kaniko.tolerations[].tolerationSeconds
providers[].kaniko.tolerations[].tolerationSeconds
providers > kaniko > tolerations > tolerationSeconds
"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
providers[].kaniko.tolerations[].value
providers[].kaniko.tolerations[].value
providers > kaniko > tolerations > value
"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.
providers[].kaniko.annotations
providers[].kaniko.annotations
providers > kaniko > annotations
Specify annotations to apply to each Kaniko builder pod. Annotations may have an effect on the behaviour of certain components, for example autoscalers. The same annotations will be used for each util pod unless they are specifically set under util.annotations
Example:
providers[].kaniko.serviceAccountAnnotations
providers[].kaniko.serviceAccountAnnotations
providers > kaniko > serviceAccountAnnotations
Specify annotations to apply to the Kubernetes service account used by kaniko. This can be useful to set up IRSA with in-cluster building.
Example:
providers[].kaniko.util
providers[].kaniko.util
providers[].kaniko.util.tolerations[]
providers[].kaniko.util.tolerations[]
providers > kaniko > util > tolerations
Specify tolerations to apply to each garden-util pod.
providers[].kaniko.util.tolerations[].effect
providers[].kaniko.util.tolerations[].effect
providers > kaniko > util > tolerations > effect
"Effect" indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are "NoSchedule", "PreferNoSchedule" and "NoExecute".
providers[].kaniko.util.tolerations[].key
providers[].kaniko.util.tolerations[].key
providers > kaniko > util > tolerations > key
"Key" is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be "Exists"; this combination means to match all values and all keys.
providers[].kaniko.util.tolerations[].operator
providers[].kaniko.util.tolerations[].operator
providers > kaniko > util > tolerations > operator
"Operator" represents a key's relationship to the value. Valid operators are "Exists" and "Equal". Defaults to "Equal". "Exists" is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
providers[].kaniko.util.tolerations[].tolerationSeconds
providers[].kaniko.util.tolerations[].tolerationSeconds
providers > kaniko > util > tolerations > tolerationSeconds
"TolerationSeconds" represents the period of time the toleration (which must be of effect "NoExecute", otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
providers[].kaniko.util.tolerations[].value
providers[].kaniko.util.tolerations[].value
providers > kaniko > util > tolerations > value
"Value" is the taint value the toleration matches to. If the operator is "Exists", the value should be empty, otherwise just a regular string.
providers[].kaniko.util.annotations
providers[].kaniko.util.annotations
providers > kaniko > util > annotations
Specify annotations to apply to each garden-util pod and deployments.
Example:
providers[].kaniko.util.nodeSelector
providers[].kaniko.util.nodeSelector
providers > kaniko > util > nodeSelector
Specify the nodeSelector constraints for each garden-util pod.
providers[].defaultHostname
providers[].defaultHostname
providers > defaultHostname
A default hostname to use when no hostname is explicitly configured for a service.
Example:
providers[].deploymentStrategy
providers[].deploymentStrategy
providers > deploymentStrategy
Experimental: this is an experimental feature and the API might change in the future.
Deprecated: This field will be removed in a future release.
Sets the deployment strategy for container
deploy actions.
Note that this field has been deprecated since 0.13, and has no effect. The "rolling"
will be applied in all cases. The experimental support for blue/green deployments (via the "blue-green"
strategy) has been removed.
Note that this setting only applies to container
deploy actions (and not, for example, kubernetes
or helm
deploy actions).
providers[].sync
providers[].sync
providers > sync
Configuration options for code synchronization.
providers[].sync.defaults
providers[].sync.defaults
Specifies default settings for syncs (e.g. for container
, kubernetes
and helm
services).
These are overridden/extended by the settings of any individual sync specs.
Sync is enabled e.g by setting the --sync
flag on the garden deploy
command.
See the Code Synchronization guide for more information.
providers[].sync.defaults.exclude[]
providers[].sync.defaults.exclude[]
providers > sync > defaults > exclude
Specify a list of POSIX-style paths or glob patterns that should be excluded from the sync.
Any exclusion patterns defined in individual sync specs will be applied in addition to these patterns.
.git
directories and .garden
directories are always ignored.
Example:
providers[].sync.defaults.fileMode
providers[].sync.defaults.fileMode
providers > sync > defaults > fileMode
The default permission bits, specified as an octal, to set on files at the sync target. Defaults to 0o644 (user can read/write, everyone else can read). See the Mutagen docs for more information.
providers[].sync.defaults.directoryMode
providers[].sync.defaults.directoryMode
providers > sync > defaults > directoryMode
The default permission bits, specified as an octal, to set on directories at the sync target. Defaults to 0o755 (user can read/write, everyone else can read). See the Mutagen docs for more information.
providers[].sync.defaults.owner
providers[].sync.defaults.owner
providers > sync > defaults > owner
Set the default owner of files and directories at the target. Specify either an integer ID or a string name. See the Mutagen docs for more information.
providers[].sync.defaults.group
providers[].sync.defaults.group
providers > sync > defaults > group
Set the default group on files and directories at the target. Specify either an integer ID or a string name. See the Mutagen docs for more information.
providers[].forceSsl
providers[].forceSsl
providers > forceSsl
Require SSL on all container
Deploys. If set to true, an error is raised when no certificate is available for a configured hostname on a container
Deploy.
providers[].imagePullSecrets[]
providers[].imagePullSecrets[]
providers > imagePullSecrets
References to docker-registry
secrets to use for authenticating with remote registries when pulling images. This is necessary if you reference private images in your action configuration, and is required when configuring a remote Kubernetes environment with buildMode=local.
providers[].imagePullSecrets[].name
providers[].imagePullSecrets[].name
providers > imagePullSecrets > name
The name of the Kubernetes secret.
Example:
providers[].imagePullSecrets[].namespace
providers[].imagePullSecrets[].namespace
providers > imagePullSecrets > namespace
The namespace where the secret is stored. If necessary, the secret may be copied to the appropriate namespace before use.