WhatBrokeToday
Today's Trends
mediumInvestigatingKubernetes

Kubernetes Concurrent Watch Decoder Goroutine Panic Handling Issues

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

An issue was discovered in Kubernetes where adding HandleCrash calls to the concurrent watch decoder's goroutines caused callers to wait indefinitely if a worker recovered from a panic without publishing a result. Recovery mechanisms must ensure failures are properly reported so callers can finish execution.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

Workers recovering from a panic via HandleCrash failed to publish results, causing callers to wait indefinitely without notification of failure.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Investigate goroutine panic handling issues(opens GitHub issue in a new tab)

    What happened? PR #143845 added HandleCrash calls to the concurrent watch decoder’s goroutines. During review, we found that adding a panic handler alone was not enough. If a worker recovers from a panic without publishing a result, its caller can wait indefinitely. Recovery must also ensure that failures are reported and callers can finish. We then used an AI agent to inspect the code for similar goroutine panic handling issues. Its findings are copied below. These findings are based on code…

    by p0lyn0mial 0upvotes 4comments

Timeline

  1. First report on GitHub issue

    Investigate goroutine panic handling issues

    Source