Introduction:
As you know, spinning up a Kubernetes cluster on OpenStack with Magnum is super easy. The speed and scaling features it offers are pretty great:
- Master scaling
- Worker scaling / Worker auto-scaling (ability to scaling up to a number you specify)
- Self-healing (autohealing)
When you create a cluster in this environment, the storage your pods need are requested from Cinder component of the underlying OpenStack deployment, thanks to the cinder-csi-plugin (driver) inside Kubernetes. In other words, your pods storage claim's appears as a volume on the OpenStack.
Let's take a look at our shiny new, healthy cluster consisting of 1 Master and 1 Worker node for Kubernetes version 1.36.2.
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin coe cluster list
+--------------------------------------+-------+---------+------------+--------------+-----------------+---------------+
| uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+-------+---------+------------+--------------+-----------------+---------------+
| 63bf139b-4269-43ab-987a-cb3f09d46586 | k1362 | erdem | 1 | 1 | CREATE_COMPLETE | HEALTHY |
+--------------------------------------+-------+---------+------------+--------------+-----------------+---------------+
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin coe cluster show 63bf139b-4269-43ab-987a-cb3f09d46586
+----------------------+--------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------------------------------------------------+
| status | CREATE_COMPLETE |
| health_status | HEALTHY |
| cluster_template_id | 41ab5ef1-fc1c-4dca-9743-81b782c65296 |
| node_addresses | [] |
| uuid | 63bf139b-4269-43ab-987a-cb3f09d46586 |
| stack_id | kube-w2dkt |
| status_reason | None |
| created_at | 2026-07-20T20:49:59+00:00 |
| updated_at | 2026-07-20T20:55:33+00:00 |
| coe_version | v1.36.2 |
| labels | {'kube_tag': 'v1.36.2', 'availability_zone': 'nova', 'auto_scaling_enabled': 'False', |
| | 'auto_healing_enabled': 'False', 'master_lb_floating_ip_enabled': 'True'} |
| labels_overridden | {} |
| labels_skipped | {} |
| labels_added | {'availability_zone': 'nova', 'auto_scaling_enabled': 'False', 'auto_healing_enabled': 'False', |
| | 'master_lb_floating_ip_enabled': 'True'} |
| fixed_network | cd821f9f-fd5c-463c-bc4a-e6cda74d58e0 |
| fixed_subnet | 77eb74cd-b017-415c-9f1d-cbe2a59b4423 |
| floating_ip_enabled | True |
| faults | |
| keypair | erdem |
| api_address | https://192.168.1.81:6443 |
| master_addresses | [] |
| master_lb_enabled | True |
| create_timeout | 60 |
| node_count | 1 |
| discovery_url | |
| docker_volume_size | None |
| master_count | 1 |
| container_version | None |
| name | k1362 |
| master_flavor_id | 4cpu8ram |
| flavor_id | 4cpu8ram |
| health_status_reason | {'kube-w2dkt-5crpz-kbw2m.Ready': 'True', 'kube-w2dkt-default-worker-j2dpl-dltqw-w6v98.Ready': |
| | 'True', 'api': 'ok'} |
| project_id | 5efd985b9a2042cf98d60b0ca6fa55a3 |
+----------------------+--------------------------------------------------------------------------------------------------+The kube-w2dkt-5crpz-kbw2m is master node and kube-w2dkt-default-worker-j2dpl-dltqw-w6v98 is worker node, they are both ready. Let's create a storage class as shown below and try to create a 5GB storage for a pod with busybox container. Lets name the file test.yaml.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cinder-default
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cinder-pvc-demo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
name: cinder-pod-demo
spec:
containers:
- name: busybox
image: busybox:latest
command: [ "sleep", "3600" ]
volumeMounts:
- mountPath: /data
name: my-cinder-volume
volumes:
- name: my-cinder-volume
persistentVolumeClaim:
claimName: cinder-pvc-demokubectl apply -f test.yamlNow let's check its status on the OpenStack.
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin volume list
+----------------------------------+----------------------------------+--------+------+-----------------------------------+
| ID | Name | Status | Size | Attached to |
+----------------------------------+----------------------------------+--------+------+-----------------------------------+
| 390c3236-9f59-4696-9f41- | pvc-c48ab519-1f8a-4a64-9baf- | in-use | 5 | Attached to kube-w2dkt-default- |
| f1d6ba2cf517 | 40e8afecec14 | | | worker-j2dpl-dltqw-w6v98 on |
...
+----------------------------------+----------------------------------+--------+------+-----------------------------------+Everything is as expected.
Problem:
So if "Everything is as expected", what’s the problem? Actually, you just saw the scenario where it works without any issues. The problem basically looks like this:
erdem@EWUL13:~/.kube/test$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6b4b6457d5-6dc98 1/1 Running 0 6m59s
kube-system calico-node-858qg 1/1 Running 0 6m4s
kube-system calico-node-h6l94 1/1 Running 0 6m59s
kube-system coredns-7d764666f9-m9htf 1/1 Running 0 7m48s
kube-system coredns-7d764666f9-v4kgm 1/1 Running 0 7m48s
kube-system etcd-kube-d9dxm-g4zsz-jzdtt 1/1 Running 0 8m48s
kube-system k8s-keystone-auth-2hk9q 1/1 Running 0 6m22s
kube-system kube-apiserver-kube-d9dxm-g4zsz-jzdtt 1/1 Running 0 8m7s
kube-system kube-controller-manager-kube-d9dxm-g4zsz-jzdtt 1/1 Running 1 (8m46s ago) 8m48s
kube-system kube-proxy-79mds 1/1 Running 0 6m4s
kube-system kube-proxy-vf7jz 1/1 Running 0 7m48s
kube-system kube-scheduler-kube-d9dxm-g4zsz-jzdtt 1/1 Running 1 (8m35s ago) 8m48s
kube-system openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh 1/6 CrashLoopBackOff 26 (22s ago) 7m
kube-system openstack-cinder-csi-nodeplugin-6hxpk 3/3 Running 0 6m4s
kube-system openstack-cinder-csi-nodeplugin-cd2kj 3/3 Running 0 7m
kube-system openstack-cloud-controller-manager-qt8rn 1/1 Running 0 6m35sAs can be observed above, openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh inside kube-system somehow fails to run.
So, what are we going to see in the events when we run a describe on this pod.
erdem@EWUL13:~/.kube/test$ kubectl -n kube-system describe pod openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh
...
Normal Started 7m2s (x2 over 7m5s) kubelet spec.containers{cinder-csi-plugin}: Container started
Warning BackOff 6m18s (x2 over 6m23s) kubelet spec.containers{csi-attacher}: Back-off restarting failed container csi-attacher in pod openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh_kube-system(1e2a5234-543c-45d2-ac28-287b24d782f0)
Warning BackOff 6m12s (x3 over 6m18s) kubelet spec.containers{csi-provisioner}: Back-off restarting failed container csi-provisioner in pod openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh_kube-system(1e2a5234-543c-45d2-ac28-287b24d782f0)
Warning BackOff 6m8s (x3 over 6m13s) kubelet spec.containers{csi-snapshotter}: Back-off restarting failed container csi-snapshotter in pod openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh_kube-system(1e2a5234-543c-45d2-ac28-287b24d782f0)
Normal Pulled 2m10s (x5 over 6m54s) kubelet spec.containers{csi-attacher}: Container image "registry.k8s.io/sig-storage/csi-attacher:v4.7.0" already present on machine and can be accessed by the pod
Warning BackOff 114s (x56 over 7m1s) kubelet spec.containers{cinder-csi-plugin}: Back-off restarting failed container cinder-csi-plugin in pod openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh_kube-system(1e2a5234-543c-45d2-ac28-287b24d782f0)Nothing clear here. I wonder what's in the log.
erdem@EWUL13:~/.kube/test$ kubectl -n kube-system logs openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh -c cinder-csi-plugin
2026/07/18 22:33:21 Running command:
Command env: (log-file=, also-stdout=false, redirect-stderr=true)
Run from directory:
Executable path: /bin/cinder-csi-plugin
Args (comma-delimited): /bin/cinder-csi-plugin,-v=2,--endpoint=unix://csi/csi.sock,--cloud-config=/etc/kubernetes/cloud.conf,--cluster=e323bad7-a6ba-4e98-93ec-431197562841,--provide-node-service=false
2026/07/18 22:33:21 Now listening for interrupts
I0718 22:33:21.917078 12 driver.go:101] Driver: cinder.csi.openstack.org
I0718 22:33:21.917121 12 driver.go:102] Driver version: [email protected]
I0718 22:33:21.917127 12 driver.go:103] CSI Spec version: 1.8.0
I0718 22:33:21.917132 12 driver.go:104] Topology awareness: bool
...
I0718 22:33:21.919214 12 driver.go:165] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0718 22:33:21.919219 12 driver.go:165] Enabling node service capability: EXPAND_VOLUME
I0718 22:33:21.919223 12 driver.go:165] Enabling node service capability: GET_VOLUME_STATS
I0718 22:33:21.919333 12 openstack.go:168] InitOpenStackProvider configFiles: [/etc/kubernetes/cloud.conf]
I0718 22:33:21.919731 12 openstack.go:103] Global: ""
I0718 22:33:21.919760 12 openstack.go:106] Block storage opts: {0 false false false}
W0718 22:33:22.202321 12 main.go:127] Failed to GetOpenStackProvider : No suitable endpoint could be found in the service catalog.Failed to GetOpenStackProvider : No suitable endpoint could be found in the service catalog. We found the culprit. It clearly says "I can't find the necessary endpoint in the catalog to get storage space." Well, Cinder is installed and running fine. Our VMs can get the storage space they need. What on earth is happening to the Kubernetes cluster that it can't get storage for pods? Let's take a look at the endpoints.
(client) erdem@virtbase:~/vms/isos$ openstack --os-cloud=kolla-admin endpoint list
+------------------------+--------+--------------+-----------------+---------+-----------+-------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+------------------------+--------+--------------+-----------------+---------+-----------+-------------------------+
...
| 42d78b8108db49c08f4b6a | Test1 | cinder | block-storage | True | public | http://192.168.1.25:877 |
| 3fa87a4949 | | | | | | 6/v3 |
| 4a6b57cdf4ec4b5f8f19a6 | Test1 | cinder | block-storage | True | internal | http://10.10.22.25:8776 |
...
+------------------------+--------+--------------+-----------------+---------+-----------+-------------------------+Cinder is present, no problem there, but why is the service-type block-storage? Why isn't it registered as volume or volumev3? Did the type turn out differently during deployment, because we're using NFS instead of Ceph for storage, or did the types change in the OpenStack Gazpacho release? Still, it shouldn't be an issue, let's do a quick Google search... cinder csi plugin... Code is Here and under the hood it uses a library called "github.com/gophercloud/gophercloud/v2/openstack/blockstorage/v3/volumes"...
So, a bit more googling... gophercloud issues... There's an old issue, it got merged and closed... A newer one, on line 34 we can see that with a commit dated March 2025, the current types are as follows.
- block-store
- volume
- volumev2
- volumev3
Our type is block-storage, not block-store. So, could these have changed over to this time? Let's check... gophercloud v2... Yep here, changed indeed . If we look between lines 405 and 420 here, there are 3 supported types.
- v1 volume
- v2 block-storage
- v3 block-storage
So our type fits this condition as well. Then why on earth can't we find the right endpoint? Wait, what was our cinder-csi-plugin version again?
erdem@EWUL13:~/.kube/test/pods$ kubectl describe pods -n kube-system openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh | grep -i cinder-csi-plugin
cinder-csi-plugin:
Image: registry.k8s.io/provider-os/cinder-csi-plugin:v1.32.0
Image ID: registry.k8s.io/provider-os/cinder-csi-plugin@sha256:8dbc1d5c42c143d26ba7fea5ca36a38ab512983b7df16329b486421615381f96
/bin/cinder-csi-pluginA lot of time has passed since cinder-csi-plugin v1.32.0; the current version is 1.36.0. In that case, which version of gophercloud was our cinder-csi-plugin compiled against...
The cinder-csi-plugin container inside the openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh pod doesn't ship with bash, sh, or anything like that, so we jump in using busybox in debug mode.
erdem@EWUL13:~/.kube/test/pods$ kubectl -n kube-system debug -it openstack-cinder-csi-controllerplugin-57466c4589-6xkcs --image=busybox --target=cinder-csi-plugin
Targeting container "cinder-csi-plugin". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-qzk85.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
/ #We're inside busybox right now. The container we're looking for is almost certainly under /proc/1 path. In fact, its filesystem should be under /proc/1/root. Once we navigate there, we'll be in the root (/) directory of the cinder-csi-plugin container. let's take a quick look to the folder "bin".
/ # cd /proc/1/root/
/proc/1/root # ls bin/
btrfs btrfs-find-root btrfs-map-logical btrfsck cinder-csi-plugin dmesg mount umount
btrfs-convert btrfs-image btrfs-select-super btrfstune csi-deps-check.sh findmnt udevadmThe cinder-csi-plugin executable is right here. Now, since go commands aren't working for some reason I can't quite wrap my head around (there's no go installed in the container, just the go-runner), let's use the strings command to grep for things related to gophercloud and openstack inside the cinder-csi-plugin executable.
/proc/1/root # cd bin/
/proc/1/root/bin # strings cinder-csi-plugin | grep -i "gophercloud" | grep -i openstack
...
github.com/gophercloud/gophercloud/v2/openstack/compute/v2/volumeattach.VolumeAttachmentResult.Extract
...
github.com/gophercloud/gophercloud/[email protected]/openstack/blockstorage/v3/volumes/urls.go
github.com/gophercloud/gophercloud/[email protected]/openstack/blockstorage/v3/volumes/results.go
github.com/gophercloud/gophercloud/[email protected]/openstack/identity/v2/tokens/requests.go
...Out of dozens of lines, there's a section like the one above, and now we realize that our cinder-csi-plugin version 1.32.0 was compiled against gophercloud 2.5.0. So what are the supported types in gophercloud version 2.5.0? Let's check GitHub real quick. Right here, between lines 398 and 410, we see that the only supported types are the following:
- volume
- volumev2
- volumev3
We've made it this far. Now we know exactly what's missing. The main question now is: "What happens if we create this type of endpoint?" Let's test it out without breaking or messing up our existing service. We'll create a service named cinderv3 with the type volumev3, as well as the corresponding volumev3 endpoints.
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin service create --name cinderv3 --description "Cinder Volume Service V3" volumev3
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin endpoint create --region Test1 volumev3 public http://192.168.1.25:8776/v3
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin endpoint create --region Test1 volumev3 internal http://10.10.22.25:8776/v3
(client) erdem@virtbase:~$ openstack --os-cloud=kolla-admin endpoint list
+----------------------------------+--------+--------------+-----------------+---------+-----------+-------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+--------+--------------+-----------------+---------+-----------+-------------------------------------------+
...
| 42d78b8108db49c08f4b6a3fa87a4949 | Test1 | cinder | block-storage | True | public | http://192.168.1.25:8776/v3 |
| 4a6b57cdf4ec4b5f8f19a6ad85efe750 | Test1 | cinder | block-storage | True | internal | http://10.10.22.25:8776/v3 |
| 90d6c1de3b8b4bbfa4a2f440226e17f1 | Test1 | cinderv3 | volumev3 | True | internal | http://10.10.22.25:8776/v3 |
| 9ff94b75a93b4720a5a9c330fb64a589 | Test1 | cinderv3 | volumev3 | True | public | http://192.168.1.25:8776/v3 |
...
+---And just like that, another problem solved. With a simple rollout restart, openstack-cinder-csi-controllerplugin-5d966b8cd6-phcvh goes away, and a fresh new one drops right into place clean and clear.
Conclusion:
In today's IT world, we automate installations, management, configurations, and... as in our example, creating up Kubernetes clusters. Every single component we automate brings along other dependencies that are decoupled, different, and most of the time - fully unknown to us in terms of what they actually contain under the hood. All these layers operate completely abstracted away from us, and as long as they run without a hitch, we don't even notice a thing. 17 months have passed since the 1.32.0 version, that we were using; if it were working smoothly, we wouldn't care in the slightest, and we don't even know if we truly need 1.32.0 in the first place. If we tried to bump the version to 1.36.0 for a proper root-cause fix here, who knows what other dependencies would blow up and how much deeper we'd have to dive to fix things?
Anyway, let's keep automating our messes. By the way, I'm dropping a quick-read reference below around the theme "If you automate a mess, you get an automated mess."
References:
Thanks:
Main Photo: Brian Kelly on Unsplash