NVIDIA Device Plugin Fails on Flatcar Container Linux

[kodeine]

NVIDIA Device Plugin Fails on Flatcar Container Linux with SystemD Extensions

Issue Summary

NVIDIA device plugin fails to detect GPUs on Flatcar Container Linux when NVIDIA drivers are installed via systemd-sysext, despite nvidia-smi working correctly.

Environment

• OS: Flatcar Container Linux 3815.2.5 (Oklo)
• Kubernetes: v1.32.0
• GPU: NVIDIA RTX 5090
• Driver: 570.86.10 (via systemd-sysext)
• Container Runtime: containerd 1.7.13
• NVIDIA Container Toolkit: 1.16.2 (via systemd-sysext)

Problem Description

1. NVIDIA drivers installed via systemd-sysext work correctly (nvidia-smi shows GPU)
2. NVIDIA device plugin pods remain in Pending state
3. Device plugin cannot detect GPU devices at /dev/nvidia*
4. Manual device node creation allows containers to see GPU, but device plugin still fails

Root Cause Analysis

1. Missing nvidia-container-runtime Binaries

The nvidia-container-toolkit sysext only provides wrapper scripts, not actual binaries:

$ ls -la /usr/bin/nvidia-container-*
-rwxr-xr-x 1 root root 73 /usr/bin/nvidia-container-cli
-rwxr-xr-x 1 root root 81 /usr/bin/nvidia-container-runtime
-rwxr-xr-x 1 root root 86 /usr/bin/nvidia-container-runtime-hook

$ cat /usr/bin/nvidia-container-runtime
#!/bin/sh
exec /usr/libexec/extensions/nvidia-container-toolkit/nvidia-container-runtime "$@"

# But the actual binary doesn't exist:
$ ls /usr/libexec/extensions/nvidia-container-toolkit/
# Directory not found

2. CDI (Container Device Interface) Issues

• CDI specification generation fails due to missing nvidia-ctk binary
• Even with manual CDI config, runc doesn't support CDI natively
• crun (which supports CDI) is not available on Flatcar

3. Device Node Creation Timing

• SystemD extensions are loaded early in boot
• Device nodes aren't created automatically by the sysext
• nvidia-modprobe doesn't work as expected with sysext installation

Reproduction Steps

1. Install Flatcar with NVIDIA sysext
2. Deploy Kubernetes with containerd
3. Apply NVIDIA device plugin
4. Observe pods stuck in Pending state
5. Check kubectl describe node | grep nvidia.com/gpu shows no GPU resources

Workaround Attempted (Partial Success)

# Manual device node creation
nvidia-modprobe -u -c=0
mknod -m 666 /dev/nvidia0 c 195 0
mknod -m 666 /dev/nvidiactl c 195 255
mknod -m 666 /dev/nvidia-uvm c 511 0
mknod -m 666 /dev/nvidia-uvm-tools c 511 0

# This allows containers to use GPU directly, but device plugin still doesn't work

Expected Behavior

NVIDIA device plugin should:

1. Detect GPU devices automatically
2. Create nvidia.com/gpu resources in Kubernetes
3. Allow pod scheduling with GPU resource requests

Actual Behavior

• Device plugin cannot find GPU devices
• No nvidia.com/gpu resources available
• Pods requesting GPUs remain pending

Potential Solutions

1. Flatcar: Provide complete nvidia-container-toolkit binaries in sysext, not just wrappers
2. Flatcar: Ensure device nodes are created during sysext activation
3. Flatcar: Include crun runtime for proper CDI support
4. NVIDIA: Document systemd-sysext compatibility requirements
5. Alternative: Use traditional package installation instead of sysext

Related Issues

• Flatcar's immutable filesystem makes traditional NVIDIA installation impossible
• SystemD extensions are the recommended approach but lack full compatibility
• Similar issues may affect other container runtimes relying on nvidia-container-runtime

Impact

This blocks GPU workload deployment on Flatcar Container Linux, forcing users to switch to traditional distributions for NVIDIA GPU support.

[jepio]

I don't think we published nvidia driver sysexts for 3815.2.5 - are you sure that's what you're running?

Where are you sourcing the nvidia-container-toolkit sysext from? We publish an nvidia-runtime sysext that is tested against the GPU operator (https://flatcar.github.io/sysext-bakery/nvidia_runtime/), and it doesn't have those wrapper scripts you mentioned.

As for the device nodes: we do the device node creation from our nvidia.service, check the journal for that service.

It would also be useful if you could provide a reproducer - how do you deploy the nvidia device plugin, how exactly is the node setup, etc.

[kodeine]

@jepio i am getting the following error, No space left on device, when downloading the nvidia driver, i re-wrote the configs to keep it minimal to test nvidia drivers.

this is how i install
./flatcar-install -d $DISK -C alpha -V 4372.0.1 -i ignition.json

erorrs im getting:

ruice - Ignition (files). ervice - Ignition record after-ignition.seruice - Root filesystem completion...
activated successfully.
ervice - Ignition (record completion).
: grep: /sysroot/oem/oem-release: No such file or directory : grep: /sysroot/usr/share/flatcar/enabled-sysext.conf: No such file or directory : Did not find /etc/flatcar/sysext/flatcar-nuidia-drivers-570-open-4372.0.1 .raw
ruice - Afterburn Initrd Setup Network Kernel Arguments was skipped because no trigger condition checks were met.
: Args ( output_dir:
"sysroot/ue-rs/",
target_filename: None,
input_ml: None, payload_url: Some ("https://update.release.flat>
Error: unable to verify signature
Caused by:
0: failed to get_data_blobs path ("/sysroot/ue-rs/ tmp/ue_data_blobs*)
1: failed to copy plain data at offset 183945087
2: No space left on device (os error 28)
rm: cannot remove
' /sysroot/ue-rs/': Is a directory mition service: Mailed uth resittitecited, status=1/PAILURE
nition.seruice: Failed with result up-root-after-ignition.seruice
Root filesystem completion.
tion complete .target
- Ignition Complete.
rd.target - Initrd Default Target. target/start failed with result
'dependency'.
OnFailure= dependencies.
Job ignition-complete.target/start failed with result 'dependency' .

variant: flatcar
version: 1.0.0

storage:
  files:
    # Enable built-in NVIDIA kernel drivers
    - path: /etc/flatcar/enabled-sysext.conf
      mode: 0644
      contents:
        inline: |
          nvidia-drivers-570-open

    # Download official sysext-bakery extensions
    - path: /etc/extensions/kubernetes.raw
      mode: 0644
      contents:
        source: https://extensions.flatcar.org/extensions/kubernetes-v1.33.3-x86-64.raw
    
    - path: /etc/extensions/nvidia-runtime.raw
      mode: 0644
      contents:
        source: https://extensions.flatcar.org/extensions/nvidia-runtime-v1.17.8-x86-64.raw
    
    - path: /etc/extensions/cilium.raw
      mode: 0644
      contents:
        source: https://extensions.flatcar.org/extensions/cilium-v0.18.5-x86-64.raw
    
    - path: /etc/extensions/ollama.raw
      mode: 0644
      contents:
        source: https://extensions.flatcar.org/extensions/ollama-v0.9.6-x86-64.raw

    # Static IP network configuration
    - path: /etc/systemd/network/00-enp5s0f0.network
      mode: 0644
      contents:
        inline: |
          [Match]
          Name=enp5s0f0
          
          [Network]
          DHCP=no
          Address=192.168.1.240/24
          Gateway=192.168.1.1
          DNS=8.8.8.8
          DNS=8.8.4.4
    
systemd:
  units:
    # Ensure systemd-networkd is enabled
    - name: systemd-networkd.service
      enabled: true
    
    # Disable automatic reboots
    - name: locksmithd.service
      enabled: false
      mask: true
    
    # Configure Ollama to listen on localhost only
    - name: ollama.service
      enabled: true
      dropins:
        - name: 10-ollama-env-override.conf
          contents: |
            [Service]
            Environment="OLLAMA_HOST=127.0.0.1:11434"
    
    # Initialize Kubernetes using Flatcar's recommended approach
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service
        After=containerd.service network-online.target
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        ExecStartPre=/usr/bin/kubeadm init --pod-network-cidr=10.244.0.0/16
        ExecStartPre=/usr/bin/mkdir -p /home/core/.kube
        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
        [Install]
        WantedBy=multi-user.target
    
    # Configure Cilium CNI
    - name: cilium.service
      enabled: true
      dropins:
        - name: 10-cilium-env-override.conf
          contents: |
            [Service]
            Environment="CILIUM_INSTALL_ARGS=--set kubeProxyReplacement=true --namespace=kube-system"
   

[jepio]

@kodeine I see the problem, the root partition is too small to handle all the sysexts, and it is only resized implicitly after ignition runs.

The only workaround I can offer you is force recreate the partition in ignition phase (which will size it correctly according to your disk), add this (caution; wipes filesystem)

storage:
  disks:
  - device: /dev/vda
    partitions:
    - number: 9
      label: ROOT
      resize: true
      size_mib: 0

  filesystems:
  - device: /dev/disk/by-partlabel/ROOT
    format: ext4
    label: ROOT
    wipe_filesystem: true