[{"content":"Kubernetes storage is one of those areas that looks simple from the outside — you create a PersistentVolumeClaim, a pod mounts it, done. But the moment you need to integrate your own storage backend, you\u0026rsquo;re staring at the Container Storage Interface spec, sidecar containers you\u0026rsquo;ve never heard of, and gRPC services that have to be wired together just right.\nI had the pleasure of writing and contributing to a production-grade CSI driver end-to-end. This post covers everything I wish I had in one place: what CSI actually is, how Kubernetes orchestrates it, and how to implement all three services in Go.\nWhat is CSI? CSI (Container Storage Interface) is a standardized gRPC API between Kubernetes and storage providers. Before CSI existed, storage drivers were compiled directly into Kubernetes — adding a new one meant patching the Kubernetes tree. If you remember a few Kubernetes versions ago where your non-Azure Kubernetes is actually logging non-stop that Azure is not working? Yeah, that kind of problem due to coupling.\nCSI moved drivers out-of-tree: your storage backend ships its own binary, Kubernetes talks to it over a Unix socket.\nThe spec defines three gRPC services:\nIdentityService — who are you, are you healthy? ControllerService — manage volumes at the infrastructure level (create, delete, attach, snapshot) NodeService — manage volumes on a specific node (format, mount, bind-mount into pods) Your driver implements these. Kubernetes provides sidecar containers that translate Kubernetes events into the right RPC calls.\nArchitecture Overview Two separate binaries run in the cluster:\ngraph TB subgraph CP[\"Controller Pod — Deployment, 1 replica\"] PROV[csi-provisioner] ATT[csi-attacher] SNAP[csi-snapshotter] RES[csi-resizer] CLIVE[liveness-probe] CDRV[\"Your Driver\\nControllerService + IdentityService\"] PROV --\u003e|gRPC unix socket| CDRV ATT --\u003e|gRPC unix socket| CDRV SNAP --\u003e|gRPC unix socket| CDRV RES --\u003e|gRPC unix socket| CDRV CLIVE --\u003e|gRPC unix socket| CDRV end subgraph NP[\"Node Pod — DaemonSet, 1 per node\"] REG[node-driver-registrar] NLIVE[liveness-probe] NDRV[\"Your Driver\\nNodeService + IdentityService\"] REG --\u003e|gRPC unix socket| NDRV NLIVE --\u003e|gRPC unix socket| NDRV end KBL[kubelet] --\u003e|gRPC unix socket| NDRV REG --\u003e|registers socket path| KBL The controller pod runs centrally and manages volume lifecycle against your storage backend API. The node pod runs on every machine and handles the actual OS-level work: formatting disks, mounting filesystems, bind-mounting into pods.\nThey communicate via Unix domain sockets, not TCP. Each sidecar and the driver share an emptyDir (controller) or hostPath (node) volume where the socket lives.\nHow Kubernetes Orchestrates CSI The sidecars are the glue. You don\u0026rsquo;t call your driver directly — the sidecars watch Kubernetes API resources and translate events into gRPC calls.\nSidecar Watches Triggers RPC csi-provisioner PersistentVolumeClaim CreateVolume / DeleteVolume csi-attacher VolumeAttachment ControllerPublishVolume / ControllerUnpublishVolume csi-snapshotter VolumeSnapshot CreateSnapshot / DeleteSnapshot csi-resizer PVC resize ControllerExpandVolume node-driver-registrar startup registers socket with kubelet kubelet Pod scheduling NodeStageVolume / NodePublishVolume kubelet talks to the node plugin directly after node-driver-registrar registers the socket path at /var/lib/kubelet/plugins_registry/.\nThe Full Volume Lifecycle Provision → Mount sequenceDiagram actor User participant K8s as Kubernetes API participant PROV as csi-provisioner participant ADC as AttachDetachController participant ATT as csi-attacher participant CTRL as ControllerService participant BACK as Storage Backend participant KBL as kubelet participant NODE as NodeService User-\u003e\u003eK8s: create PVC (16Gi, RWO) K8s-\u003e\u003ePROV: PVC event PROV-\u003e\u003eCTRL: CreateVolume(name, 16Gi, topology) CTRL-\u003e\u003eBACK: create volume CTRL-\u003e\u003eBACK: poll status → \"available\" CTRL--\u003e\u003ePROV: CreateVolumeResponse{volumeID, capacityBytes, topology} PROV-\u003e\u003eK8s: create PersistentVolume{volumeHandle=volumeID} K8s-\u003e\u003eK8s: bind PV ↔ PVC User-\u003e\u003eK8s: create Pod using PVC K8s-\u003e\u003eADC: Pod scheduled to node ADC-\u003e\u003eK8s: create VolumeAttachment{volumeID, nodeID} K8s-\u003e\u003eATT: VolumeAttachment event ATT-\u003e\u003eCTRL: ControllerPublishVolume(volumeID, nodeID) CTRL-\u003e\u003eBACK: attach volume to VM BACK--\u003e\u003eCTRL: attached, devicePath=/dev/vdb CTRL--\u003e\u003eATT: ControllerPublishVolumeResponse{PublishContext{DevicePath=/dev/vdb}} ATT-\u003e\u003eK8s: VolumeAttachment.status.attached=true KBL-\u003e\u003eNODE: NodeStageVolume(volumeID, stagingPath, PublishContext) NODE-\u003e\u003eNODE: mkfs.ext4 /dev/vdb NODE-\u003e\u003eNODE: mount /dev/vdb → stagingPath NODE--\u003e\u003eKBL: NodeStageVolumeResponse{} KBL-\u003e\u003eNODE: NodePublishVolume(volumeID, targetPath, stagingPath) NODE-\u003e\u003eNODE: mount --bind stagingPath → targetPath NODE--\u003e\u003eKBL: NodePublishVolumeResponse{} K8s--\u003e\u003eUser: Pod Running, /data mounted Three distinct paths to understand here:\nProvision (csi-provisioner → CreateVolume) — creates the volume in your backend, Kubernetes creates a PV. Attach (csi-attacher → ControllerPublishVolume) — attaches the block device to the VM running the pod. Returns the device path (e.g. /dev/vdb) in PublishContext. Stage + Publish (kubelet → NodeStageVolume + NodePublishVolume) — formats and mounts the device to a staging path, then bind-mounts that into the pod\u0026rsquo;s specific target path. The staging/publish split exists so multiple pods on the same node can share one formatted device via bind mounts, rather than formatting once per pod.\nUnmount → Delete sequenceDiagram actor User participant K8s as Kubernetes API participant PROV as csi-provisioner participant ADC as AttachDetachController participant ATT as csi-attacher participant CTRL as ControllerService participant BACK as Storage Backend participant KBL as kubelet participant NODE as NodeService User-\u003e\u003eK8s: delete Pod KBL-\u003e\u003eNODE: NodeUnpublishVolume(volumeID, targetPath) NODE-\u003e\u003eNODE: umount targetPath NODE--\u003e\u003eKBL: NodeUnpublishVolumeResponse{} KBL-\u003e\u003eNODE: NodeUnstageVolume(volumeID, stagingPath) NODE-\u003e\u003eNODE: umount stagingPath NODE--\u003e\u003eKBL: NodeUnstageVolumeResponse{} ADC-\u003e\u003eK8s: delete VolumeAttachment K8s-\u003e\u003eATT: VolumeAttachment deletion event ATT-\u003e\u003eCTRL: ControllerUnpublishVolume(volumeID, nodeID) CTRL-\u003e\u003eBACK: detach volume from VM CTRL--\u003e\u003eATT: ControllerUnpublishVolumeResponse{} User-\u003e\u003eK8s: delete PVC K8s-\u003e\u003ePROV: PVC deletion event PROV-\u003e\u003eCTRL: DeleteVolume(volumeID) CTRL-\u003e\u003eBACK: delete volume CTRL--\u003e\u003ePROV: DeleteVolumeResponse{} PROV-\u003e\u003eK8s: delete PersistentVolume Exact reverse order: unpublish → unstage → detach → delete. Each step must succeed before the next begins.\nSnapshots sequenceDiagram actor User participant K8s as Kubernetes API participant SNAP as csi-snapshotter participant PROV as csi-provisioner participant CTRL as ControllerService participant BACK as Storage Backend User-\u003e\u003eK8s: create VolumeSnapshot{source=PVC} K8s-\u003e\u003eSNAP: VolumeSnapshot event SNAP-\u003e\u003eCTRL: CreateSnapshot(name, sourceVolumeID) CTRL-\u003e\u003eBACK: create snapshot CTRL-\u003e\u003eBACK: poll status → \"available\" CTRL--\u003e\u003eSNAP: CreateSnapshotResponse{snapshotID, readyToUse=true} SNAP-\u003e\u003eK8s: create VolumeSnapshotContent{snapshotHandle=snapshotID} K8s--\u003e\u003eUser: VolumeSnapshot Ready=true User-\u003e\u003eK8s: create PVC{dataSource=VolumeSnapshot} K8s-\u003e\u003ePROV: PVC event PROV-\u003e\u003eCTRL: CreateVolume(name, size, contentSource={snapshotID}) CTRL-\u003e\u003eBACK: create volume from snapshot CTRL--\u003e\u003ePROV: CreateVolumeResponse{volumeID, resizeRequired=true} Note over PROV,K8s: NodeStageVolume will resize filesystem Volume Expansion sequenceDiagram actor User participant K8s as Kubernetes API participant RES as csi-resizer participant CTRL as ControllerService participant BACK as Storage Backend participant KBL as kubelet participant NODE as NodeService User-\u003e\u003eK8s: patch PVC 16Gi → 32Gi K8s-\u003e\u003eRES: PVC resize event RES-\u003e\u003eCTRL: ControllerExpandVolume(volumeID, 32Gi) CTRL-\u003e\u003eBACK: expand volume CTRL-\u003e\u003eBACK: poll → new size confirmed CTRL--\u003e\u003eRES: ControllerExpandVolumeResponse{nodeExpansionRequired=true} RES-\u003e\u003eK8s: update PV capacity = 32Gi KBL-\u003e\u003eNODE: NodeExpandVolume(volumeID, volumePath, 32Gi) NODE-\u003e\u003eNODE: rescan block device geometry NODE-\u003e\u003eNODE: resize2fs /dev/vdb NODE--\u003e\u003eKBL: NodeExpandVolumeResponse{} K8s--\u003e\u003eUser: PVC capacity = 32Gi Implementing the Driver in Go Project Structure cmd/ controller/main.go # wires ControllerService + IdentityService node/main.go # wires NodeService + IdentityService internal/ server/server.go # Unix socket listener + gRPC server driver/ identity.go controller.go # struct + capability declarations controller_volume.go controller_snapshot.go node.go mounter.go # interface wrapping k8s.io/mount-utils metadata.go # node instance ID + AZ Two binaries, one shared internal/driver package. The controller binary registers ControllerServer + IdentityServer. The node binary registers NodeServer + IdentityServer.\nThe gRPC Server func CreateListener(endpoint string) (net.Listener, error) { path := strings.TrimPrefix(endpoint, \u0026#34;unix://\u0026#34;) _ = os.Remove(path) return net.Listen(\u0026#34;unix\u0026#34;, path) } func CreateGRPCServer(logger *slog.Logger) *grpc.Server { return grpc.NewServer( grpc.ChainUnaryInterceptor(requestLogger(logger)), ) } // cmd/controller/main.go — error handling omitted for brevity. listener, _ := internal.CreateListener(cfg.CSIEndpoint) grpcServer := internal.CreateGRPCServer(logger) csiproto.RegisterControllerServer(grpcServer, controllerService) csiproto.RegisterIdentityServer(grpcServer, identityService) grpcServer.Serve(listener) IdentityService The simplest service. Declares what the driver supports and responds to health checks.\ntype IdentityService struct { csiproto.UnimplementedIdentityServer logger *slog.Logger ready atomic.Bool } func (is *IdentityService) GetPluginInfo(_ context.Context, _ *csiproto.GetPluginInfoRequest) (*csiproto.GetPluginInfoResponse, error) { return \u0026amp;csiproto.GetPluginInfoResponse{ Name: \u0026#34;csi.example.com\u0026#34;, VendorVersion: \u0026#34;1.0.0\u0026#34;, }, nil } func (is *IdentityService) GetPluginCapabilities(_ context.Context, _ *csiproto.GetPluginCapabilitiesRequest) (*csiproto.GetPluginCapabilitiesResponse, error) { return \u0026amp;csiproto.GetPluginCapabilitiesResponse{ Capabilities: []*csiproto.PluginCapability{ {Type: \u0026amp;csiproto.PluginCapability_Service_{ Service: \u0026amp;csiproto.PluginCapability_Service{ Type: csiproto.PluginCapability_Service_CONTROLLER_SERVICE, }, }}, }, }, nil } func (is *IdentityService) Probe(_ context.Context, _ *csiproto.ProbeRequest) (*csiproto.ProbeResponse, error) { return \u0026amp;csiproto.ProbeResponse{ Ready: \u0026amp;wrapperspb.BoolValue{Value: is.ready.Load()}, }, nil } Call is.SetReady(true) after everything is wired up, just before grpcServer.Serve.\nControllerService The controller manages your storage backend. The struct holds clients to your backend APIs:\ntype ControllerService struct { csiproto.UnimplementedControllerServer logger *slog.Logger volClient VolumeAPIClient vmClient VMAPIClient region string } CreateVolume must be idempotent — csi-provisioner will retry. Check by name first:\nfunc (cs *ControllerService) CreateVolume(ctx context.Context, req *csiproto.CreateVolumeRequest) (*csiproto.CreateVolumeResponse, error) { if req.GetName() == \u0026#34;\u0026#34; { return nil, status.Error(codes.InvalidArgument, \u0026#34;missing name\u0026#34;) } existing, err := cs.volClient.GetByName(ctx, req.GetName()) if err == nil { return buildCreateResponse(existing), nil } sizeGB := bytesToGB(req.GetCapacityRange().GetRequiredBytes()) vol, err := cs.volClient.Create(ctx, CreateVolumeParams{ Name: req.GetName(), SizeGB: sizeGB, AvailabilityZone: azFromTopology(req.GetAccessibilityRequirements()), }) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;create volume failed: %v\u0026#34;, err) } err = cs.waitForStatus(ctx, vol.ID, \u0026#34;available\u0026#34;) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;volume not ready: %v\u0026#34;, err) } return buildCreateResponse(vol), nil } The exponential backoff wait is critical — most storage APIs are asynchronous:\nfunc (cs *ControllerService) waitForStatus(ctx context.Context, volumeID, target string) error { backoff := wait.Backoff{ Duration: time.Second, Factor: 1.1, Steps: 15, } return wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) { vol, err := cs.volClient.Get(ctx, volumeID) if err != nil { return false, err } return vol.Status == target, nil }) } ControllerPublishVolume attaches the block device to the VM. The key output is PublishContext — this is how the device path travels from the controller to the node:\nfunc (cs *ControllerService) ControllerPublishVolume(ctx context.Context, req *csiproto.ControllerPublishVolumeRequest) (*csiproto.ControllerPublishVolumeResponse, error) { volumeID := req.GetVolumeId() nodeID := req.GetNodeId() err = cs.vmClient.AttachVolume(ctx, nodeID, volumeID) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;attach failed: %v\u0026#34;, err) } err = cs.waitUntilAttached(ctx, nodeID, volumeID) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;attach timeout: %v\u0026#34;, err) } devicePath, err := cs.volClient.GetDevicePath(ctx, volumeID, nodeID) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;device path: %v\u0026#34;, err) } return \u0026amp;csiproto.ControllerPublishVolumeResponse{ PublishContext: map[string]string{ \u0026#34;DevicePath\u0026#34;: devicePath, }, }, nil } NodeService The node runs with elevated privileges on every machine. It receives the PublishContext from the controller and turns it into real mounts.\nNodeStageVolume formats (if needed) and mounts to a shared staging path:\nfunc (ns *NodeService) NodeStageVolume(ctx context.Context, req *csiproto.NodeStageVolumeRequest) (*csiproto.NodeStageVolumeResponse, error) { stagingTarget := req.GetStagingTargetPath() devicePath := req.GetPublishContext()[\u0026#34;DevicePath\u0026#34;] notMnt, err := ns.mounter.IsLikelyNotMountPointAttach(stagingTarget) if err != nil { return nil, status.Error(codes.Internal, err.Error()) } if notMnt { fsType := \u0026#34;ext4\u0026#34; if mnt := req.GetVolumeCapability().GetMount(); mnt != nil \u0026amp;\u0026amp; mnt.GetFsType() != \u0026#34;\u0026#34; { fsType = mnt.GetFsType() } err = ns.mounter.FormatAndMount(devicePath, stagingTarget, fsType, nil) if err != nil { return nil, status.Error(codes.Internal, err.Error()) } } return \u0026amp;csiproto.NodeStageVolumeResponse{}, nil } NodePublishVolume bind-mounts the staging path into the pod\u0026rsquo;s specific directory:\nfunc (ns *NodeService) NodePublishVolume(ctx context.Context, req *csiproto.NodePublishVolumeRequest) (*csiproto.NodePublishVolumeResponse, error) { targetPath := req.GetTargetPath() stagingPath := req.GetStagingTargetPath() mountOptions := []string{\u0026#34;bind\u0026#34;} if req.GetReadonly() { mountOptions = append(mountOptions, \u0026#34;ro\u0026#34;) } else { mountOptions = append(mountOptions, \u0026#34;rw\u0026#34;) } err := ns.mounter.Mount(stagingPath, targetPath, \u0026#34;ext4\u0026#34;, mountOptions) if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;bind mount failed: %v\u0026#34;, err) } return \u0026amp;csiproto.NodePublishVolumeResponse{}, nil } NodeGetInfo is called by kubelet at startup to learn the node\u0026rsquo;s identity and topology. This is how topology.kubernetes.io/zone gets set:\nfunc (ns *NodeService) NodeGetInfo(ctx context.Context, _ *csiproto.NodeGetInfoRequest) (*csiproto.NodeGetInfoResponse, error) { instanceID, err := ns.metadata.GetInstanceID() if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;instance ID: %v\u0026#34;, err) } zone, err := ns.metadata.GetAvailabilityZone() if err != nil { return nil, status.Errorf(codes.Internal, \u0026#34;availability zone: %v\u0026#34;, err) } return \u0026amp;csiproto.NodeGetInfoResponse{ NodeId: instanceID, AccessibleTopology: \u0026amp;csiproto.Topology{ Segments: map[string]string{ \u0026#34;topology.kubernetes.io/zone\u0026#34;: zone, }, }, }, nil } The Mounter Interface Rather than calling mount directly, wrap k8s.io/mount-utils. This makes the node service testable with a fake mounter:\ntype Mounter interface { FormatAndMount(source, target, fsType string, options []string) error Mount(source, target, fsType string, options []string) error UnmountPath(mountPath string) error IsLikelyNotMountPointAttach(path string) (bool, error) GetDevicePath(ctx context.Context, volumeID string) (string, error) Execer() exec.Interface } func NewMounter() Mounter { return \u0026amp;mounter{ BaseMounter: mountutil.New(\u0026#34;\u0026#34;), exec: exec.New(), } } Deployment Controller Deployment apiVersion: apps/v1 kind: Deployment metadata: name: csi-controller spec: replicas: 1 template: spec: containers: - name: csi-provisioner image: registry.k8s.io/sig-storage/csi-provisioner:v5.2.0 args: [\u0026#34;--csi-address=/csi/csi.sock\u0026#34;, \u0026#34;--leader-election\u0026#34;] volumeMounts: - name: socket-dir mountPath: /csi - name: csi-attacher image: registry.k8s.io/sig-storage/csi-attacher:v4.8.0 args: [\u0026#34;--csi-address=/csi/csi.sock\u0026#34;, \u0026#34;--leader-election\u0026#34;] volumeMounts: - name: socket-dir mountPath: /csi - name: csi-driver image: your-registry/csi-controller:latest env: - name: CSI_ENDPOINT value: unix:///csi/csi.sock volumeMounts: - name: socket-dir mountPath: /csi volumes: - name: socket-dir emptyDir: {} # shared between all sidecars in this pod Node DaemonSet apiVersion: apps/v1 kind: DaemonSet metadata: name: csi-node spec: template: spec: containers: - name: node-driver-registrar image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0 args: - --csi-address=/csi/csi.sock - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.example.com/csi.sock volumeMounts: - name: socket-dir mountPath: /csi - name: registration-dir mountPath: /registration - name: csi-driver image: your-registry/csi-node:latest securityContext: privileged: true capabilities: add: [\u0026#34;SYS_ADMIN\u0026#34;] env: - name: CSI_ENDPOINT value: unix:///csi/csi.sock volumeMounts: - name: socket-dir mountPath: /csi - name: kubelet-dir mountPath: /var/lib/kubelet mountPropagation: Bidirectional # essential: propagates mounts to host - name: dev-dir mountPath: /dev volumes: - name: socket-dir hostPath: path: /var/lib/kubelet/plugins/csi.example.com/ type: DirectoryOrCreate - name: registration-dir hostPath: path: /var/lib/kubelet/plugins_registry/ - name: kubelet-dir hostPath: path: /var/lib/kubelet - name: dev-dir hostPath: path: /dev Two things that catch people out:\nmountPropagation: Bidirectional on the kubelet dir — without this, mounts made inside the node container aren\u0026rsquo;t visible to the host, so the pod never sees the volume. privileged: true — the node driver calls mount(2), which requires root. There\u0026rsquo;s no way around it. Error Handling and Idempotency The CSI spec requires every RPC to be idempotent. Kubernetes retries. You will receive CreateVolume twice with the same name. You will receive ControllerPublishVolume for an already-attached volume. Handle this by checking state before acting:\n// CreateVolume — return existing volume if found by name. existing, err := cs.volClient.GetByName(ctx, name) if err == nil { return buildCreateResponse(existing), nil } // ControllerPublishVolume — skip attach if already attached. if attached, _ := cs.isAttached(ctx, nodeID, volumeID); attached { devicePath, _ := cs.getDevicePath(ctx, volumeID, nodeID) return \u0026amp;csiproto.ControllerPublishVolumeResponse{ PublishContext: map[string]string{\u0026#34;DevicePath\u0026#34;: devicePath}, }, nil } Use gRPC status codes correctly:\ncodes.InvalidArgument // caller\u0026#39;s fault, missing/bad input codes.NotFound // resource doesn\u0026#39;t exist codes.AlreadyExists // for CreateVolume when name conflicts with different params codes.Internal // backend/driver error codes.Unavailable // transient, safe to retry Node Registration Sequence sequenceDiagram participant REG as node-driver-registrar participant DRV as NodeService participant KBL as kubelet REG-\u003e\u003eDRV: GetPluginInfo() → {name, version} REG-\u003e\u003eDRV: GetPluginCapabilities() REG-\u003e\u003eKBL: register at /var/lib/kubelet/plugins_registry/csi.example.com-reg.sock KBL-\u003e\u003eREG: NodeGetInfo request REG-\u003e\u003eDRV: NodeGetInfo() → {nodeID, topology} REG--\u003e\u003eKBL: {nodeID, driverSocket=/var/lib/kubelet/plugins/csi.example.com/csi.sock} Note over KBL: kubelet routes volume opsfor this driver to the node socket KBL-\u003e\u003eDRV: NodeStageVolume / NodePublishVolume / ... Common Pitfalls Volume size rounding. Many backends work in GiB, not bytes. req.GetCapacityRange().GetRequiredBytes() is in bytes. Convert carefully and round up, not down — returning a smaller volume than requested violates the spec.\nNodeStageVolume vs NodePublishVolume. Stage runs once per volume per node. Publish runs once per pod. If you skip staging and try to format in publish, two pods requesting the same volume will race to run mkfs on the same device.\nBlocking RPCs. All CSI RPCs are synchronous from the caller\u0026rsquo;s perspective, but backends are async. CreateVolume must not return until the volume is usable. Poll with backoff; don\u0026rsquo;t sleep for a fixed duration.\nxfs and duplicate UUIDs. If you restore a snapshot to create a new volume, the new volume has the same filesystem UUID as the original. xfs refuses to mount two volumes with the same UUID on the same node. Pass nouuid as a mount option for xfs volumes:\nif fsType == \u0026#34;xfs\u0026#34; { options = append(options, \u0026#34;nouuid\u0026#34;) } Block volumes. If you support VolumeCapability_AccessMode_SINGLE_NODE_WRITER for raw block, NodePublishVolume needs a different code path — create a file at targetPath and bind-mount the device file, not a directory.\nTesting Unit test the controller and node services with mock backends and a fake mounter:\nfunc TestCreateVolume_Idempotent(t *testing.T) { ctrl := NewControllerService(fakeVolClient, fakeVMClient, ...) req := \u0026amp;csiproto.CreateVolumeRequest{ Name: \u0026#34;test-vol\u0026#34;, CapacityRange: \u0026amp;csiproto.CapacityRange{RequiredBytes: 16 * 1024 * 1024 * 1024}, VolumeCapabilities: defaultCaps(), } resp1, err := ctrl.CreateVolume(ctx, req) require.NoError(t, err) resp2, err := ctrl.CreateVolume(ctx, req) require.NoError(t, err) assert.Equal(t, resp1.GetVolume().GetVolumeId(), resp2.GetVolume().GetVolumeId()) } For the node service, k8s.io/mount-utils ships a FakeMounter that records mount calls without touching the filesystem — use it.\nIntegration testing the full flow requires a real Kubernetes cluster. The Kubernetes CSI sanity test suite provides a standard set of conformance tests you can run against a live driver socket.\nWhat You End Up With After all of this, what you have is:\nA controller binary that speaks to your storage API and manages volume/snapshot lifecycle A node binary that runs on every node, formats disks, and manages mounts Two Kubernetes workloads (Deployment + DaemonSet) with the right sidecar containers StorageClass, RBAC, and a driver registration object pointing at your plugin name The CSI spec is verbose but consistent. Once you\u0026rsquo;ve implemented CreateVolume and NodeStageVolume, the pattern repeats. The hard parts are operational: idempotency under retries, async backend polling, and getting mount propagation right in the DaemonSet.\nThe official CSI spec and the kubernetes-csi examples repo are the two references worth keeping open while building. Note, even though the examples repo says this is not a good example, it\u0026rsquo;s actually a decent start to get the boilerplate and setup right.\n","permalink":"https://syndbg.github.io/posts/2026-04-24-writing-a-kubernetes-csi-driver/","summary":"\u003cp\u003eKubernetes storage is one of those areas that looks simple from the outside — you create a \u003ccode\u003ePersistentVolumeClaim\u003c/code\u003e, a pod mounts it, done. But the moment you need to integrate your own storage backend, you\u0026rsquo;re staring at the \u003ca href=\"https://github.com/container-storage-interface/spec\"\u003eContainer Storage Interface spec\u003c/a\u003e, sidecar containers you\u0026rsquo;ve never heard of, and gRPC services that have to be wired together just right.\u003c/p\u003e\n\u003cp\u003eI had the pleasure of writing and contributing to a production-grade CSI driver end-to-end. This post covers everything I wish I had in one place: what CSI actually is, how Kubernetes orchestrates it, and how to implement all three services in Go.\u003c/p\u003e","title":"Writing a Kubernetes CSI Driver: Controller and Node from Scratch"},{"content":"Go 1.25 just dropped with expected changes to GOMAXPROCS, which significantly change how Go applications behave in containerized environments. The runtime now automatically detects and respects container CPU limits when setting GOMAXPROCS. This isn\u0026rsquo;t just a minor improvement—it\u0026rsquo;s a shift that may dramatically improve performance for millions of containerized Go applications.\nAnd this isn\u0026rsquo;t the only amazing change, but this is the one I\u0026rsquo;ll focus on in this post.\nThe Problem That Plagued Go for Years Before Go 1.25, there was a fundamental mismatch between Go\u0026rsquo;s runtime and containerized environments:\n# Your Kubernetes pod with 2 CPU cores resources: limits: cpu: \u0026#34;2\u0026#34; # But Go saw the entire host machine runtime.NumCPU() // Returns 8 (host CPUs) runtime.GOMAXPROCS(0) // Also 8 - ignoring your 2-core limit! This led to:\nOver-scheduling: Go created 8 goroutines for CPU-bound work on a 2-core container Context switching overhead: Excessive goroutine switching degraded performance Resource contention: Multiple containers fighting for the same CPU cores Manual workarounds: Developers had to manually set GOMAXPROCS everywhere Popular libraries like uber-go/automaxprocs existed solely to fix this fundamental issue, showing how widespread the problem was.\nWhat Changed in Go 1.25 Go 1.25 introduces two features:\n1. Container-Aware GOMAXPROCS (Linux) The runtime now reads cgroup CPU bandwidth limits and automatically sets GOMAXPROCS accordingly:\n// On a container with 2 CPU cores limit: runtime.NumCPU() // Still returns 8 (host CPUs) runtime.GOMAXPROCS(0) // Now returns 2 (respects container limit!) 2. Dynamic GOMAXPROCS Updates The runtime periodically updates GOMAXPROCS if CPU availability changes:\n// Your application automatically adapts to: // - Kubernetes horizontal pod autoscaling // - Container CPU limit changes // - Node CPU availability changes Seeing It in Action Let me show you the difference with a practical example. Here\u0026rsquo;s a test program that reveals the new behavior:\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Printf(\u0026#34;Go version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34;Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34;GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) // Monitor for changes (Go 1.25 feature) for i := 0; i \u0026lt; 10; i++ { time.Sleep(5 * time.Second) fmt.Printf(\u0026#34;Time %ds: GOMAXPROCS = %d\\n\u0026#34;, (i+1)*5, runtime.GOMAXPROCS(0)) } } Kubernetes Results Comparison Go 1.24 and earlier:\n# Pod with 2 CPU cores limit resources: limits: cpu: \u0026#34;2\u0026#34; Go version: go1.24.0 Host CPUs: 8 GOMAXPROCS: 8 # ❌ Ignores container limit Go 1.25:\nGo version: go1.25.0 Host CPUs: 8 GOMAXPROCS: 2 # ✅ Respects container limit! Docker Results Running the same program in Docker with CPU limits:\n# Docker with 1.5 CPU cores docker run --cpus=\u0026#34;1.5\u0026#34; golang:1.25 go run main.go Go version: go1.25.0 Host CPUs: 8 GOMAXPROCS: 2 # ✅ Rounds up fractional CPU limits Complete Examples and Testing I\u0026rsquo;ve created comprehensive examples to test all scenarios. Here are the key ones:\nKubernetes Manifest Examples # Example 1: CPU-limited container apiVersion: apps/v1 kind: Deployment metadata: name: go-app-cpu-limited spec: template: spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;2\u0026#34; # GOMAXPROCS will be 2 memory: \u0026#34;256Mi\u0026#34; command: [\u0026#34;go\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;/app/main.go\u0026#34;] # Example 2: Manual override (disables auto-detection) apiVersion: apps/v1 kind: Deployment metadata: name: go-app-manual spec: template: spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;2\u0026#34; # CPU limit is 2 env: - name: GOMAXPROCS value: \u0026#34;8\u0026#34; # Manual override - will use 8 command: [\u0026#34;go\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;/app/main.go\u0026#34;] When the Magic Doesn\u0026rsquo;t Happen The automatic behavior is disabled in these cases:\n1. Manual GOMAXPROCS Setting // Environment variable GOMAXPROCS=8 go run main.go // Or in code runtime.GOMAXPROCS(8) 2. GODEBUG Overrides # Disable container awareness GODEBUG=containermaxprocs=0 go run main.go # Disable dynamic updates GODEBUG=updatemaxprocs=0 go run main.go # Disable both GODEBUG=containermaxprocs=0,updatemaxprocs=0 go run main.go 3. Non-Linux Platforms Currently, container-aware GOMAXPROCS only works on Linux (where cgroups exist).\nMigration Guide for Existing Applications 1. Remove Manual GOMAXPROCS Workarounds Before (Go \u0026lt; 1.25):\nimport ( _ \u0026#34;go.uber.org/automaxprocs\u0026#34; // Popular workaround library ) func init() { // Manual calculation based on container limits if cpuLimit := os.Getenv(\u0026#34;CPU_LIMIT\u0026#34;); cpuLimit != \u0026#34;\u0026#34; { if limit, err := strconv.Atoi(cpuLimit); err == nil { runtime.GOMAXPROCS(limit) } } } After (Go 1.25):\n// Just delete all the manual workarounds! // Go handles it automatically now 2. Test Thoroughly Since this changes fundamental runtime behavior, comprehensive testing is critical:\n# Test your application with various CPU limits docker run --cpus=\u0026#34;1\u0026#34; myapp:go1.25 docker run --cpus=\u0026#34;2\u0026#34; myapp:go1.25 docker run --cpus=\u0026#34;4\u0026#34; myapp:go1.25 # Monitor performance metrics kubectl top pods 3. Monitor Key Metrics Watch these metrics after upgrading:\nCPU utilization efficiency Response times Context switch rates Memory usage patterns Goroutine counts Cloud Platform Compatibility This change affects major cloud platforms:\nAWS ECS Fargate: ✅ Respects CPU limits EKS: ✅ Works with pod CPU limits Lambda: ✅ May improve performance in custom runtimes Google Cloud Cloud Run: ✅ Automatically optimizes for allocated CPU GKE: ✅ Full Kubernetes support Cloud Functions: ✅ Benefits custom Go runtimes Azure Container Instances: ✅ Respects CPU limits AKS: ✅ Full Kubernetes support Functions: ✅ Helps custom Go runtimes Performance Tuning Considerations When Automatic is Perfect Standard web applications: REST APIs, microservices Data processing pipelines: ETL jobs, stream processing Background workers: Queue processors, schedulers When Manual Tuning May Be Better CPU-intensive algorithms: May need fine-tuning for specific workloads Memory-bound applications: GOMAXPROCS might not be the bottleneck Legacy applications: With complex custom scheduling logic Testing Your Specific Workload func benchmarkWithDifferentGOMAXPROCS() { for _, procs := range []int{1, 2, 4, 8} { runtime.GOMAXPROCS(procs) start := time.Now() // Your workload here runYourWorkload() fmt.Printf(\u0026#34;GOMAXPROCS=%d: %v\\n\u0026#34;, procs, time.Since(start)) } } Troubleshooting Common Issues Issue 1: Performance Regression Symptoms: Application slower after upgrading to Go 1.25 Solution:\n# Temporarily disable to confirm GODEBUG=containermaxprocs=0 go run main.go # Or use manual setting GOMAXPROCS=8 go run main.go Issue 2: GOMAXPROCS Not Changing Check: Container actually has CPU limits set\n# In container, check cgroup limits cat /sys/fs/cgroup/cpu.max cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us Issue 3: Fractional CPU Confusion Remember: Go rounds UP fractional CPU limits\nCPU Limit: 1.5 → GOMAXPROCS: 2 CPU Limit: 2.7 → GOMAXPROCS: 3 CPU Limit: 3.9 → GOMAXPROCS: 4 The Bigger Picture: Why This Matters This change represents a fundamental shift in how Go applications integrate with modern infrastructure:\n1. Cloud-Native by Default Go applications now understand their containerized environment without additional configuration.\n2. Better Resource Efficiency Automatic optimization means better cost efficiency in cloud deployments.\n3. Simplified Operations DevOps teams no longer need to manually tune GOMAXPROCS for every deployment.\n4. Platform Consistency The same application binary adapts to different deployment environments automatically.\nFuture Implications This opens doors for even smarter runtime optimizations:\nMemory-aware garbage collection tuning Network buffer sizing based on container limits Dynamic goroutine pool sizing Automatic backpressure mechanisms Conclusion Go 1.25\u0026rsquo;s container-aware GOMAXPROCS is more than just a feature—it\u0026rsquo;s a fundamental improvement that makes Go applications truly cloud-native by default. The days of manual GOMAXPROCS tuning and third-party workaround libraries are finally over.\nKey takeaways:\n✅ Automatic container CPU limit detection on Linux ✅ Dynamic updates when limits change ✅ Zero configuration required ✅ Significant performance improvements in many cases ✅ Backward compatible with manual overrides If you\u0026rsquo;re running Go applications in containers (and who isn\u0026rsquo;t these days?), Go 1.25 should be at the top of your upgrade priority list. Test it thoroughly, but expect to see measurable performance improvements across your containerized Go workloads.\nWant to test this yourself? All the examples and code from this post are included below. Try them out in your own Kubernetes cluster or Docker environment to see the magic in action!\nComplete Test Code Examples Kubernetes Manifests Here are the complete Kubernetes manifests for testing different scenarios:\n# kubernetes-manifests.yaml --- # Example 1: CPU-limited container that will respect limits apiVersion: apps/v1 kind: Deployment metadata: name: go-app-cpu-limited-1core labels: app: go-gomaxprocs-test scenario: cpu-limited-1core spec: replicas: 1 selector: matchLabels: app: go-gomaxprocs-test scenario: cpu-limited-1core template: metadata: labels: app: go-gomaxprocs-test scenario: cpu-limited-1core spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;1\u0026#34; # GOMAXPROCS should be 1 memory: \u0026#34;256Mi\u0026#34; requests: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;128Mi\u0026#34; command: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;] args: - | cat \u0026gt; /tmp/main.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \u0026#34;fmt\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Printf(\u0026#34;=== Go 1.25 GOMAXPROCS Test (1 CPU limit) ===\\n\u0026#34;) fmt.Printf(\u0026#34;Go version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34;Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34;GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\u0026#34;Expected GOMAXPROCS: 1 (CPU limit)\\n\\n\u0026#34;) for i := 0; i \u0026lt; 6; i++ { time.Sleep(10 * time.Second) fmt.Printf(\u0026#34;Time %ds: GOMAXPROCS = %d\\n\u0026#34;, (i+1)*10, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run main.go --- # Example 2: CPU-limited container with 2 cores apiVersion: apps/v1 kind: Deployment metadata: name: go-app-cpu-limited-2core labels: app: go-gomaxprocs-test scenario: cpu-limited-2core spec: replicas: 1 selector: matchLabels: app: go-gomaxprocs-test scenario: cpu-limited-2core template: metadata: labels: app: go-gomaxprocs-test scenario: cpu-limited-2core spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;2\u0026#34; # GOMAXPROCS should be 2 memory: \u0026#34;512Mi\u0026#34; requests: cpu: \u0026#34;1\u0026#34; memory: \u0026#34;256Mi\u0026#34; command: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;] args: - | cat \u0026gt; /tmp/main.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \u0026#34;fmt\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Printf(\u0026#34;=== Go 1.25 GOMAXPROCS Test (2 CPU limit) ===\\n\u0026#34;) fmt.Printf(\u0026#34;Go version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34;Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34;GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\u0026#34;Expected GOMAXPROCS: 2 (CPU limit)\\n\\n\u0026#34;) for i := 0; i \u0026lt; 6; i++ { time.Sleep(10 * time.Second) fmt.Printf(\u0026#34;Time %ds: GOMAXPROCS = %d\\n\u0026#34;, (i+1)*10, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run main.go --- # Example 3: Manual GOMAXPROCS override (should ignore container limits) apiVersion: apps/v1 kind: Deployment metadata: name: go-app-manual-override labels: app: go-gomaxprocs-test scenario: manual-override spec: replicas: 1 selector: matchLabels: app: go-gomaxprocs-test scenario: manual-override template: metadata: labels: app: go-gomaxprocs-test scenario: manual-override spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;1\u0026#34; # CPU limit is 1 memory: \u0026#34;256Mi\u0026#34; env: - name: GOMAXPROCS value: \u0026#34;4\u0026#34; # Manual override - should use 4, not 1 command: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;] args: - | cat \u0026gt; /tmp/main.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \u0026#34;fmt\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Printf(\u0026#34;=== Go 1.25 GOMAXPROCS Test (Manual Override) ===\\n\u0026#34;) fmt.Printf(\u0026#34;Go version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34;Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34;GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\u0026#34;CPU Limit: 1, Manual GOMAXPROCS: 4\\n\u0026#34;) fmt.Printf(\u0026#34;Expected GOMAXPROCS: 4 (manual override)\\n\\n\u0026#34;) for i := 0; i \u0026lt; 6; i++ { time.Sleep(10 * time.Second) fmt.Printf(\u0026#34;Time %ds: GOMAXPROCS = %d\\n\u0026#34;, (i+1)*10, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run main.go --- # Example 4: Fractional CPU limits (should round up) apiVersion: apps/v1 kind: Deployment metadata: name: go-app-fractional-cpu labels: app: go-gomaxprocs-test scenario: fractional-cpu spec: replicas: 1 selector: matchLabels: app: go-gomaxprocs-test scenario: fractional-cpu template: metadata: labels: app: go-gomaxprocs-test scenario: fractional-cpu spec: containers: - name: go-app image: golang:1.25 resources: limits: cpu: \u0026#34;1.5\u0026#34; # GOMAXPROCS should be 2 (rounds up) memory: \u0026#34;256Mi\u0026#34; requests: cpu: \u0026#34;750m\u0026#34; memory: \u0026#34;128Mi\u0026#34; command: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;] args: - | cat \u0026gt; /tmp/main.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \u0026#34;fmt\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Printf(\u0026#34;=== Go 1.25 GOMAXPROCS Test (1.5 CPU limit) ===\\n\u0026#34;) fmt.Printf(\u0026#34;Go version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34;Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34;GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\u0026#34;Expected GOMAXPROCS: 2 (1.5 rounds up to 2)\\n\\n\u0026#34;) for i := 0; i \u0026lt; 6; i++ { time.Sleep(10 * time.Second) fmt.Printf(\u0026#34;Time %ds: GOMAXPROCS = %d\\n\u0026#34;, (i+1)*10, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run main.go Docker Compose Test Scenarios Complete Docker Compose setup for testing different CPU limits:\n# docker-compose.yaml version: \u0026#39;3.8\u0026#39; services: # Test 1: 1 CPU limit go-app-1cpu: image: golang:1.25 container_name: go-1.25-test-1cpu deploy: resources: limits: cpus: \u0026#39;1.0\u0026#39; memory: 256M reservations: cpus: \u0026#39;0.5\u0026#39; memory: 128M command: \u0026gt; sh -c \u0026#34; cat \u0026gt; /tmp/test.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \\\u0026#34;fmt\\\u0026#34; \\\u0026#34;runtime\\\u0026#34; \\\u0026#34;time\\\u0026#34; ) func main() { fmt.Printf(\\\u0026#34;=== Docker Test: 1.0 CPU Limit ===\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Go version: %s\\n\\\u0026#34;, runtime.Version()) fmt.Printf(\\\u0026#34;Host CPUs: %d\\n\\\u0026#34;, runtime.NumCPU()) fmt.Printf(\\\u0026#34;GOMAXPROCS: %d\\n\\\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\\\u0026#34;Expected: 1\\n\\n\\\u0026#34;) for i := 0; i \u0026lt; 12; i++ { time.Sleep(5 * time.Second) fmt.Printf(\\\u0026#34;[%02d] GOMAXPROCS = %d\\n\\\u0026#34;, i+1, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run test.go \u0026#34; # Test 2: 2.5 CPU limit (should round up to 3) go-app-2-5cpu: image: golang:1.25 container_name: go-1.25-test-2-5cpu deploy: resources: limits: cpus: \u0026#39;2.5\u0026#39; memory: 512M reservations: cpus: \u0026#39;1.0\u0026#39; memory: 256M command: \u0026gt; sh -c \u0026#34; cat \u0026gt; /tmp/test.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \\\u0026#34;fmt\\\u0026#34; \\\u0026#34;runtime\\\u0026#34; \\\u0026#34;time\\\u0026#34; ) func main() { fmt.Printf(\\\u0026#34;=== Docker Test: 2.5 CPU Limit ===\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Go version: %s\\n\\\u0026#34;, runtime.Version()) fmt.Printf(\\\u0026#34;Host CPUs: %d\\n\\\u0026#34;, runtime.NumCPU()) fmt.Printf(\\\u0026#34;GOMAXPROCS: %d\\n\\\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\\\u0026#34;Expected: 3 (2.5 rounds up)\\n\\n\\\u0026#34;) for i := 0; i \u0026lt; 12; i++ { time.Sleep(5 * time.Second) fmt.Printf(\\\u0026#34;[%02d] GOMAXPROCS = %d\\n\\\u0026#34;, i+1, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run test.go \u0026#34; # Test 3: Manual override go-app-manual: image: golang:1.25 container_name: go-1.25-test-manual environment: - GOMAXPROCS=8 deploy: resources: limits: cpus: \u0026#39;1.0\u0026#39; memory: 256M command: \u0026gt; sh -c \u0026#34; cat \u0026gt; /tmp/test.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \\\u0026#34;fmt\\\u0026#34; \\\u0026#34;runtime\\\u0026#34; \\\u0026#34;time\\\u0026#34; ) func main() { fmt.Printf(\\\u0026#34;=== Docker Test: Manual Override ===\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Go version: %s\\n\\\u0026#34;, runtime.Version()) fmt.Printf(\\\u0026#34;Host CPUs: %d\\n\\\u0026#34;, runtime.NumCPU()) fmt.Printf(\\\u0026#34;GOMAXPROCS: %d\\n\\\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\\\u0026#34;CPU Limit: 1.0, GOMAXPROCS env: 8\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Expected: 8 (manual override)\\n\\n\\\u0026#34;) for i := 0; i \u0026lt; 12; i++ { time.Sleep(5 * time.Second) fmt.Printf(\\\u0026#34;[%02d] GOMAXPROCS = %d\\n\\\u0026#34;, i+1, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run test.go \u0026#34; # Test 4: GODEBUG disable go-app-godebug: image: golang:1.25 container_name: go-1.25-test-godebug environment: - GODEBUG=containermaxprocs=0 deploy: resources: limits: cpus: \u0026#39;1.0\u0026#39; memory: 256M command: \u0026gt; sh -c \u0026#34; cat \u0026gt; /tmp/test.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; package main import ( \\\u0026#34;fmt\\\u0026#34; \\\u0026#34;runtime\\\u0026#34; \\\u0026#34;time\\\u0026#34; ) func main() { fmt.Printf(\\\u0026#34;=== Docker Test: GODEBUG Disabled ===\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Go version: %s\\n\\\u0026#34;, runtime.Version()) fmt.Printf(\\\u0026#34;Host CPUs: %d\\n\\\u0026#34;, runtime.NumCPU()) fmt.Printf(\\\u0026#34;GOMAXPROCS: %d\\n\\\u0026#34;, runtime.GOMAXPROCS(0)) fmt.Printf(\\\u0026#34;CPU Limit: 1.0, GODEBUG=containermaxprocs=0\\n\\\u0026#34;) fmt.Printf(\\\u0026#34;Expected: %d (same as host CPUs)\\n\\n\\\u0026#34;, runtime.NumCPU()) for i := 0; i \u0026lt; 12; i++ { time.Sleep(5 * time.Second) fmt.Printf(\\\u0026#34;[%02d] GOMAXPROCS = %d\\n\\\u0026#34;, i+1, runtime.GOMAXPROCS(0)) } } EOF cd /tmp \u0026amp;\u0026amp; go run test.go \u0026#34; Interactive Test Program An interactive program to test GOMAXPROCS behavior in different scenarios:\n// gomaxprocs-test.go package main import ( \u0026#34;fmt\u0026#34; \u0026#34;os\u0026#34; \u0026#34;runtime\u0026#34; \u0026#34;strconv\u0026#34; \u0026#34;time\u0026#34; ) func main() { fmt.Println(\u0026#34;🚀 Go 1.25 Container-Aware GOMAXPROCS Interactive Test\u0026#34;) fmt.Println(\u0026#34;=====================================================\u0026#34;) // Display current environment showEnvironmentInfo() // Test scenarios fmt.Println(\u0026#34;\\n🧪 Running test scenarios...\u0026#34;) testScenario1_DefaultBehavior() testScenario2_ManualOverride() testScenario3_MonitorChanges() } func showEnvironmentInfo() { fmt.Printf(\u0026#34;\\n📊 Environment Information:\\n\u0026#34;) fmt.Printf(\u0026#34; Go Version: %s\\n\u0026#34;, runtime.Version()) fmt.Printf(\u0026#34; GOOS: %s\\n\u0026#34;, runtime.GOOS) fmt.Printf(\u0026#34; GOARCH: %s\\n\u0026#34;, runtime.GOARCH) fmt.Printf(\u0026#34; Host CPUs: %d\\n\u0026#34;, runtime.NumCPU()) fmt.Printf(\u0026#34; Current GOMAXPROCS: %d\\n\u0026#34;, runtime.GOMAXPROCS(0)) // Check environment variables if gomaxprocs := os.Getenv(\u0026#34;GOMAXPROCS\u0026#34;); gomaxprocs != \u0026#34;\u0026#34; { fmt.Printf(\u0026#34; GOMAXPROCS env: %s\\n\u0026#34;, gomaxprocs) } if godebug := os.Getenv(\u0026#34;GODEBUG\u0026#34;); godebug != \u0026#34;\u0026#34; { fmt.Printf(\u0026#34; GODEBUG: %s\\n\u0026#34;, godebug) } // Check cgroup information (Linux only) if runtime.GOOS == \u0026#34;linux\u0026#34; { checkCgroupInfo() } } func checkCgroupInfo() { fmt.Printf(\u0026#34;\\n🐧 Linux cgroup Information:\\n\u0026#34;) // Check cgroup v2 CPU limits if data, err := os.ReadFile(\u0026#34;/sys/fs/cgroup/cpu.max\u0026#34;); err == nil { fmt.Printf(\u0026#34; cpu.max: %s\u0026#34;, string(data)) } // Check cgroup v1 CPU limits if data, err := os.ReadFile(\u0026#34;/sys/fs/cgroup/cpu/cpu.cfs_quota_us\u0026#34;); err == nil { fmt.Printf(\u0026#34; cfs_quota_us: %s\u0026#34;, string(data)) } if data, err := os.ReadFile(\u0026#34;/sys/fs/cgroup/cpu/cpu.cfs_period_us\u0026#34;); err == nil { fmt.Printf(\u0026#34; cfs_period_us: %s\u0026#34;, string(data)) } } func testScenario1_DefaultBehavior() { fmt.Printf(\u0026#34;\\n🔍 Test 1: Default Behavior\\n\u0026#34;) fmt.Printf(\u0026#34; Expected: GOMAXPROCS should respect container CPU limits\\n\u0026#34;) original := runtime.GOMAXPROCS(0) fmt.Printf(\u0026#34; Result: GOMAXPROCS = %d\\n\u0026#34;, original) if runtime.GOOS == \u0026#34;linux\u0026#34; { if original \u0026lt;= runtime.NumCPU() { fmt.Printf(\u0026#34; ✅ Looks good! GOMAXPROCS ≤ Host CPUs\\n\u0026#34;) } else { fmt.Printf(\u0026#34; ❓ Unexpected: GOMAXPROCS \u0026gt; Host CPUs\\n\u0026#34;) } } else { fmt.Printf(\u0026#34; ℹ️ Container awareness only works on Linux\\n\u0026#34;) } } func testScenario2_ManualOverride() { fmt.Printf(\u0026#34;\\n🔧 Test 2: Manual Override\\n\u0026#34;) fmt.Printf(\u0026#34; Testing manual GOMAXPROCS setting...\\n\u0026#34;) original := runtime.GOMAXPROCS(0) testValue := original + 2 fmt.Printf(\u0026#34; Setting GOMAXPROCS to %d\\n\u0026#34;, testValue) runtime.GOMAXPROCS(testValue) actual := runtime.GOMAXPROCS(0) fmt.Printf(\u0026#34; Result: GOMAXPROCS = %d\\n\u0026#34;, actual) if actual == testValue { fmt.Printf(\u0026#34; ✅ Manual override works correctly\\n\u0026#34;) } else { fmt.Printf(\u0026#34; ❌ Manual override failed\\n\u0026#34;) } // Restore original runtime.GOMAXPROCS(original) fmt.Printf(\u0026#34; Restored to original value: %d\\n\u0026#34;, original) } func testScenario3_MonitorChanges() { fmt.Printf(\u0026#34;\\n⏰ Test 3: Monitor for Dynamic Changes\\n\u0026#34;) fmt.Printf(\u0026#34; Monitoring GOMAXPROCS for 30 seconds...\\n\u0026#34;) fmt.Printf(\u0026#34; (In Go 1.25, this should update if container limits change)\\n\\n\u0026#34;) for i := 0; i \u0026lt; 6; i++ { time.Sleep(5 * time.Second) gomaxprocs := runtime.GOMAXPROCS(0) fmt.Printf(\u0026#34; [%02ds] GOMAXPROCS = %d\\n\u0026#34;, (i+1)*5, gomaxprocs) } fmt.Printf(\u0026#34;\\n ℹ️ In a real container environment with changing limits,\\n\u0026#34;) fmt.Printf(\u0026#34; you might see GOMAXPROCS change dynamically.\\n\u0026#34;) } Usage Instructions Running the Kubernetes Tests # Apply all manifests kubectl apply -f kubernetes-manifests.yaml # Check the results kubectl logs deployment/go-app-cpu-limited-1core kubectl logs deployment/go-app-cpu-limited-2core kubectl logs deployment/go-app-manual-override kubectl logs deployment/go-app-fractional-cpu # Clean up kubectl delete -f kubernetes-manifests.yaml Running the Docker Compose Tests # Run all test scenarios docker-compose up # Run individual tests docker-compose up go-app-1cpu docker-compose up go-app-2-5cpu docker-compose up go-app-manual docker-compose up go-app-godebug # Clean up docker-compose down Running the Interactive Tests # Save as gomaxprocs-test.go and run directly go run gomaxprocs-test.go # Or in a container with CPU limits docker run --cpus=\u0026#34;2\u0026#34; golang:1.25 sh -c \u0026#34; cat \u0026gt; test.go \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; [paste the gomaxprocs-test.go code here] EOF go run test.go \u0026#34; Related Resources Go 1.25 Release Notes Container-aware GOMAXPROCS Documentation cgroups Documentation ","permalink":"https://syndbg.github.io/posts/2025-01-13-go-1-25/","summary":"\u003cp\u003eGo 1.25 just dropped with expected changes to GOMAXPROCS, which significantly change how Go applications behave in containerized environments. The runtime now \u003cstrong\u003eautomatically detects and respects container CPU limits\u003c/strong\u003e when setting \u003ccode\u003eGOMAXPROCS\u003c/code\u003e. This isn\u0026rsquo;t just a minor improvement—it\u0026rsquo;s a shift that \u003cstrong\u003emay\u003c/strong\u003e dramatically improve performance for millions of containerized Go applications.\u003c/p\u003e\n\u003cp\u003eAnd this isn\u0026rsquo;t the only amazing change, but this is the one I\u0026rsquo;ll focus on in this post.\u003c/p\u003e\n\u003ch2 id=\"the-problem-that-plagued-go-for-years\"\u003eThe Problem That Plagued Go for Years\u003c/h2\u003e\n\u003cp\u003eBefore Go 1.25, there was a fundamental mismatch between Go\u0026rsquo;s runtime and containerized environments:\u003c/p\u003e","title":"Practically, Go 1.25's Container-Aware GOMAXPROCS: What You Need to Know"},{"content":"👋 Hi, I\u0026rsquo;m Anton 🏷️ Engineering Manager \u0026amp; Principal Software Engineer of Least Surprise Software, or so I like to think when it\u0026rsquo;s about Developer Experience\n🚀 About Me Go Practitioner for many years - Building high-performant distributed systems with Go, sometimes with RAFT consensus too. Kubernetes Practitioner — Focused on modern cluster ops, multi-cloud, and security (cert-manager \u0026amp; PKI automation). Developer Experience \u0026amp; Platform Engineering focused. I\u0026rsquo;ve built a few closed-source systems and open-sourced a few elements of them over the years. DevOps \u0026amp; Tooling — Author of goenv, taskporter, and contributor to open-source workflow and security utilities for the Go and DevOps communities. 🛠️ Skills \u0026amp; Tools By preference, solving problems with Go, CockroachDB, Redis, Terraform, and your favorite cloud or multi-cloud deployment environment.\n📦 Notable Projects Project Description goenv Go version manager for seamless multi-version workflows vaulted Multipurpose cryptography \u0026amp; secrets tool (AES256-GCM, etc.) gocat Like socat, but in Go! Multipurpose data relay for efficient data transfer and monitoring taskporter Flexible, programmable task runner for modern DevOps workflows terraform-provider-vaulted Terraform provider for encrypted Vault secrets in VCS webpack-gcs-plugin Webpack plugin for GCS asset uploads 🏆 GitHub Achievements I\u0026rsquo;m proud to be an Arctic Code Vault Contributor, which is due to goenv.\n📍 Location Based in Sofia, Bulgaria 🇧🇬\n🔗 Contact \u0026amp; Links GitHub: github.com/syndbg LinkedIn: linkedin.com/in/syndbg Blog: You\u0026rsquo;re reading it! 📖 ","permalink":"https://syndbg.github.io/about/","summary":"\u003ch1 id=\"-hi-im-anton\"\u003e👋 Hi, I\u0026rsquo;m Anton\u003c/h1\u003e\n\u003cp\u003e🏷️ \u003cstrong\u003eEngineering Manager \u0026amp; Principal Software Engineer\u003c/strong\u003e of Least Surprise Software, or so I like to think when it\u0026rsquo;s about Developer Experience\u003c/p\u003e\n\u003ch2 id=\"-about-me\"\u003e🚀 About Me\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGo Practitioner\u003c/strong\u003e for many years - Building high-performant distributed systems with Go, sometimes with RAFT consensus too.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eKubernetes Practitioner\u003c/strong\u003e — Focused on modern cluster ops, multi-cloud, and security (cert-manager \u0026amp; PKI automation).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeveloper Experience \u0026amp; Platform Engineering\u003c/strong\u003e focused. I\u0026rsquo;ve built a few closed-source systems and open-sourced a few elements of them over the years.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDevOps \u0026amp; Tooling\u003c/strong\u003e — Author of goenv, taskporter, and contributor to open-source workflow and security utilities for the Go and DevOps communities.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"-skills--tools\"\u003e🛠️ Skills \u0026amp; Tools\u003c/h2\u003e\n\u003cp\u003eBy preference, solving problems with \u003cstrong\u003eGo\u003c/strong\u003e, \u003cstrong\u003eCockroachDB\u003c/strong\u003e, \u003cstrong\u003eRedis\u003c/strong\u003e, \u003cstrong\u003eTerraform\u003c/strong\u003e, and your favorite cloud or multi-cloud deployment environment.\u003c/p\u003e","title":"About"},{"content":"","permalink":"https://syndbg.github.io/llms.txt","summary":"","title":"llms.txt"}]