WhatBrokeToday
Today's Trends
highInvestigatingKubernetes

Kubernetes Default Preemption Plugin Evicts Partial Gang-Scheduling Members Causing Deadlock

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

When a high-priority batch workload triggers preemption, the default preemption plugin evaluates node-local priorities and evicts partial members of atomic gang-scheduled workloads. This results in cluster-wide task deadlocks for distributed jobs like multi-node PyTorch or MPI workloads.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

The scheduler identifies victim pods based purely on node-local priority evaluations, PDBs, and single-node disruption constraints without accounting for atomic gang-scheduling dependencies.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    [sig-scheduling] Default preemption plugin evicts partial gang-scheduling members causing cluster-wide task deadlock(opens GitHub issue in a new tab)

    /kind bug /sig scheduling What happened? When a high-priority batch workload triggers preemption in pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go, the scheduler identifies victim pods based purely on node-local priority evaluations, PDBs, and single-node disruption constraints. When victim pods belong to an atomic gang-scheduled workload (such as multi-node PyTorch distributed training or MPI jobs coordinated via PodGroup annotations or distributed operators like…

    by AyushSrivastava1818 0upvotes 2comments

Timeline

  1. First report on GitHub issue

    [sig-scheduling] Default preemption plugin evicts partial gang-scheduling members causing cluster-wide task deadlock

    Source