WhatBrokeToday
Today's Trends
mediumInvestigatingKubernetes

Kubernetes Device Manager Checkpoint State Desynchronizes on Admission Failure

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

During Pod admission, kubelet's Device Manager assigns physical device IDs and checkpoints the state to disk, but if container admission subsequently fails and the kubelet is restarted, the device checkpoint state can desynchronize.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

Container admission failure occurs after devicemanager.Allocate() has already updated and persisted the device allocation checkpoint state to disk without rolling back the checkpoint upon admission failure.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    [sig-node] Device Manager checkpoint state desynchronizes when container fails admission after Allocate()(opens GitHub issue in a new tab)

    /kind bug /sig node What happened? During Pod admission in kubelet, devicemanager.Allocate() assigns physical device IDs to a container, updates the in-memory device allocation state, and persists it to disk in /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint. If container admission subsequently fails (e.g., due to an image pull secret error, admission webhook rejection, pre-start hook failure, or runtime sandbox creation error) and kubelet is restarted or terminated before…

    by AyushSrivastava1818 0upvotes 2comments

Timeline

  1. First report on GitHub issue

    [sig-node] Device Manager checkpoint state desynchronizes when container fails admission after Allocate()

    Source