Configuration
Supported k8s versions
As of R6, our tests cover 1.25.latest ... 1.28.latest. All of them pass the sonobuoy CNCF conformance tests.
Overview over the parameters in clusterctl.yaml and environment-XXX.tfvars
The provenance capo means that this setting comes from the templates used by the cluster-api-provider-openstack, while SCS denotes that this setting has been added by the SCS project.
Parameters CAPI management server
Parameters controlling the Cluster-API management server (capi management server) creation:
| environment | clusterctl.yaml | provenance | default | meaning |
|---|---|---|---|---|
prefix | SCS | capi | Prefix used for OpenStack resources for the capi mgmt node | |
kind_flavor | SCS | SCS-2V-4 | Flavor to be used for the k8s capi mgmt server | |
image | SCS | Ubuntu 22.04 | Image for the capi mgmt server. Use Ubuntu 22.04 or Debian 12. Check also the ssh_username parameter | |
ssh_username | SCS | ubuntu | Name of the default user for the image | |
clusterapi_version | SCS | 1.6.3 | Version of the cluster-API incl. clusterctl | |
capi_openstack_version | SCS | 0.9.0 | Version of the cluster-api-provider-openstack (needs to fit the CAPI version) | |
cilium_binaries | SCS | v0.15.23;v0.13.0 | Versions of the cilium and hubble CLI in the vA.B.C;vX.Y.Z format | |
restrict_mgmt_server | SCS | ["0.0.0.0/0"] | Allows restricting access to the management server by the given list of CIDRs. Empty value (default) means public. | |
mgmt_cidr | SCS | 10.0.0.0/24 | IPv4 address range (CIDR notation) for management cluster | |
mgmt_ip_range | SCS | {start:"10.0.0.11", end:"10.0.0.254"} | IP range from defined mgmt_cidr variable for management cluster. It is recommended to reserve the first 10 IPs. |
Common parameters management server and clusters
Parameters controlling both management server creation and cluster creation:
| environment | clusterctl.yaml | provenance | default | meaning |
|---|---|---|---|---|
cloud_provider | OPENSTACK_CLOUD | capo | OS_CLOUD name in clouds.yaml | |
external/external_id | OPENSTACK_EXTERNAL_NETWORK_ID | capo | "" | Name/ID of the external (public) OpenStack network, default "" uses the detected external network (for clouds with one external network). Required for clouds with more than one external network |
dns_nameservers | OPENSTACK_DNS_NAMESERVERS | capo | [ "5.1.66.255", "185.150.99.255" ] | Array of nameservers for capi mgmt server and for cluster nodes, replace the FF MUC defaults with local servers if available |
availability_zone | OPENSTACK_FAILURE_DOMAIN | capo | Availability Zone(s) for the mgmt node / workload clusters | |
kind_mtu | MTU_VALUE | SCS | 0 | MTU for the mgmt server; Calico is set 50 bytes smaller; 0 means autodetection |
http_proxy | SCS | Global setting for HTTP Proxy is set on the management host including all cluster-api components running in the bootstrap-cluster. Specify with protocol: e.g http://10.10.10.10:3128 | ||
no_proxy | SCS | Global setting for HTTP Proxy exception list. If http_proxy is not set this setting has no effect. If http_proxy is set, the default value for the NO_PROXY environment variable on all affected components is set to .svc,.svc.cluster,.svc.cluster.local,127.0.0.0/8,169.254.169.254/32,fd00:ec2::254/128,${var.node_cidr},${var.pod_cidr},${var.service_cidr}. The content of no_proxy is appended to this list. This setting has no effect on apt and snap commands, the way http_proxy is set for apt and snap does not allow the configuration of proxy exceptions. |
Parameters clusters
Parameters controlling the cluster creation:
| environment | clusterctl.yaml | provenance | default | meaning |
|---|---|---|---|---|
node_cidr | NODE_CIDR | SCS | 10.8.0.0/20 | IPv4 address range (CIDR notation) for workload nodes |
pod_cidr | POD_CIDR | SCS | 192.168.0.0/16 | IPv4 address range (CIDR notation) for pods |
service_cidr | SERVICE_CIDR | SCS | 10.96.0.0/12 | IPv4 address range (CIDR notation) for services |
use_cilium | USE_CILIUM | SCS | true | Use cilium as CNI instead of calico, it can be set to vX.Y.Z and true results in v1.15.1, also see cilium_binaries |
calico_version | CALICO_VERSION | SCS | v3.27.3 | Version of the Calico CNI provider (ignored if use_cilium is set) |
kubernetes_version | KUBERNETES_VERSION | capo | v1.28.x | Kubernetes version deployed into workload cluster (.x means latest patch release) |
| `` | OPENSTACK_IMAGE_NAME | capo | ubuntu-capi-image-${KUBERNETES_VERION} | Image name for k8s controller and worker nodes. Ubuntu 22.04 image is used for k8s versions >= 1.27.3, 1.26.6, 1.25.11, Ubuntu 20.04 otherwise. |
kube_image_raw | OPENSTACK_IMAGE_RAW | SCS | true | Register images in raw format (instead of qcow2), good for ceph COW |
image_registration_extra_flags | OPENSTACK_IMAGE_REGISTATION_EXTRA_FLAGS | SCS | "" | Extra flags passed during image registration |
| `` | OPENSTACK_SSH_KEY_NAME | capo | ${prefix}-keypair | SSH key name generated and used to connect to workload cluster nodes |
controller_flavor | OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR | capo | SCS-2V-4-20s | Flavor to be used for control plane nodes |
worker_flavor | OPENSTACK_NODE_MACHINE_FLAVOR | capo | SCS-2V-4-20s | Flavor to be used for worker nodes |
controller_count | CONTROL_PLANE_MACHINE_COUNT | capo | 1 | Number of control plane nodes in testcluster (0 skips testcluster creation) |
| `` | CONTROL_PLANE_MACHINE_GEN | SCS | genc01 | Suffix for control plane node resources, to be changed for rolling upgrades |
worker_count | WORKER_MACHINE_COUNT | capo | 3 | Number of worker nodes in testcluster |
| `` | WORKER_MACHINE_GEN | SCS | genw01 | Suffix for worker node resources, to be changed for rolling upgrades |
| `` | CONTROL_PLANE_ROOT_DISKSIZE | SCS | 20 | If diskless flavors are used for control plane nodes, this is the allocated root volume disk size (in GB) |
| `` | WORKER_ROOT_DISKSIZE | SCS | 20 | If diskless flavors are used for worker nodes, this is the allocated root volume disk size (in GB) |
anti_affinity | OPENSTACK_ANTI_AFFINITY | SCS | true | Use anti-affinity server groups to prevent k8s nodes on same host (soft for workers, hard for controllers) |
soft_anti_affinity_controller | OPENSTACK_SOFT_ANTI_AFFINITY_CONTROLLER | SCS | false | Allow the use of soft-anti-affinity for the controllers (if anti_affinity is true) |
| `` | OPENSTACK_SRVGRP_CONTROLLER | SCS | nonono | Autogenerated if anti_affinity is true, eliminated otherwise |
| `` | OPENSTACK_SRVGRP_WORKER | SCS | nonono | Autogenerated if anti_affinity is true, eliminated otherwise |
deploy_occm | DEPLOY_OCCM | SCS | true | Deploy the given version of OCCM into the cluster. true (default) chooses the latest version matching the k8s version. You can specify master to chose the upstream master branch. Don't disable this. |
deploy_cindercsi | DEPLOY_CINDERCSI | SCS | true | Deploy the given (or latest matching for the default true value) of cinder CSI. |
etcd_unsafe_fs | ETCD_UNSAFE_FS | SCS | false | Use barrier=0 for filesystem on control nodes to avoid storage latency. Use for multi-controller clusters on slow/networked storage, otherwise not recommended. |
testcluster_name | (cmd line) | SCS | testcluster | Allows setting the default cluster name, created at bootstrap (if controller_count is larger than 0) |
restrict_kubeapi | RESTRICT_KUBEAPI | SCS | [ ] | Allows restricting access to kubernetes API by list of CIDRs. Empty list (default) means public, [ "none" ] means internal access only. |
controller_metadata | OPENSTACK_CONTROL_PLANE_MACHINE_METADATA | SCS | { } | Adds additional metadata for instances running the k8s management nodes |
worker_metadata | OPENSTACK_NODE_MACHINE_METADATA | SCS | { } | Adds additional metadata for instances running the k8s worker nodes |
| `` | OPENSTACK_CLUSTER_GEN | SCS | geno01 | Generation counter for the OpenStackClusterTemplate resource. Increase, when changing restrict_kubeapi or other OC settings |
capo_instance_create_timeout | CLUSTER_API_OPENSTACK_INSTANCE_CREATE_TIMEOUT | capo | 5 | Time to wait for an OpenStack machine to be created (in minutes) |
containerd_registry_files | SCS | {"hosts":["./files/containerd/docker.io"], "certs":[]} | Containerd registry hosts config files, see related docs for details. |
Optional services deployed to cluster:
| environment | clusterctl.yaml | provenance | default | script | meaning |
|---|---|---|---|---|---|
deploy_metrics | DEPLOY_METRICS | SCS | true | apply_metrics.sh | Deploy metrics service to nodes to make kubectl top work |
deploy_nginx_ingress | DEPLOY_NGINX_INGRESS | SCS | true | apply_nginx_ingress.sh | Deploy NGINX ingress controller (this spawns an OpenStack Loadbalancer), pass version to explicitly choose the version, true results in v1.9.6 (supported k8s >= 1.25) |
| `` | NGINX_INGRESS_PROXY | SCS | true | (dito) | Configure LB and nginx to get real IP via PROXY protocol; trouble for pod to LB connections has been resolved by setting hostname |
use_ovn_lb_provider | USE_OVN_LB_PROVIDER | SCS | false | apply_nginx_ingress.sh | Clouds using OVN networking can deploy the OVN provider that has low overhead (L3) and makes real client IPs visible without proxy protocol hacks. Set to auto to enable. |
deploy_gateway_api | DEPLOY_GATEWAY_API | SCS | false | create_cluster.sh | Deploy Gateway APIs CRDs and enable ciliums Gateway API implementation. This only works in conjunction with USE_CILIUM=true. Also this will break at least one CNCF conformance test. This feature is considered a tech-review. Keep in mind that Gateway API itself is under development and not GA. Also note that ciliums implementation of Gateway API is considered "beta". |
deploy_cert_manager | DEPLOY_CERT_MANAGER | SCS | false | apply_cert_manager.sh | Deploy cert-manager, pass version (e.g. v1.14.2) to explicitly choose a version |
deploy_flux | DEPLOY_FLUX | SCS | false | create_cluster.sh | Deploy flux2 into the cluster |
deploy_harbor | SCS | false | deploy_harbor.sh | Deploy harbor into the cluster. When enabled, it will overwrite the settings above. Harbor forces deployment of flux and based on config, it can force deployment of other dependencies (cert-manager, ingress-nginx and Cinder CSI) | |
harbor_config | SCS | {domain_name:"", issuer_email:"", persistence:false, database_size:"1Gi", redis_size:"1Gi", trivy_size:"5Gi"} | (dito) | Harbor container registry configuration options, see related docs for details. |