fix: retry transient RM allocation failures so power control block doesn't vanish

GSP transiently rejects NV0080 object allocation with status 0x1f when
busy — reliably on the first probe in a fresh server process and
intermittently right after RM operations (e.g. a power-limit write).
The probe surfaced that transient rejection as rm_power_supported=false,
making the Experimental Power Control block disappear in the UI until a
reload re-probed successfully.

- hal/rm_power: RmHandle.open() now retries the handle allocation
  (3 attempts, 100 ms apart) before reporting the interface unavailable;
  covers probe, set, and enable-check paths. Each attempt cleans up its
  own fds, so retries are safe and idempotent.
- PerformancePanel: confirm a negative rm_power_supported reading with
  one follow-up fetch before rendering it, so a single flaky probe can't
  hide the block or flip it into "interface unavailable"; the confirm
  has its own error boundary so a failed second request keeps the first
  (valid) read. Also added a monotonic fetch-seq guard so a slow
  response for a previously selected GPU cannot clobber fresher state.
- tests: cover transient-failure-retry and persistent-failure paths for
  RmHandle.open.

Verified live: first probe after restart 5/5 (was 0/3), 150 rapid
set+get cycles with 0 probe flips and 0 set failures.
This commit is contained in:
ARIA committed 2026-11-07 14:41:35 +01:00
1 parent 6afdd15d2a
commit 2e8e580d79
3 files changed
+137 -4

No files matched your search

@@ -1,4 +1,4 @@
import { useState, useEffect } from "react";
import { useState, useEffect, useRef } from "react";
import { Check, X, RotateCcw } from "lucide-react";
import { api } from "../../api/client.js";
import { useCurveStore } from "../../store/curveStore.js";
@@ -17,15 +17,45 @@ export function PerformancePanel() {
const [error, setError] = useState<string | null>(null);
const [confirmApply, setConfirmApply] = useState(false);
const [confirmReset, setConfirmReset] = useState(false);
// Monotonic fetch counter: a response may only commit state (or toast /
// clear loading) if it is still the latest fetch for this panel, so a slow
// response for a previously selected GPU cannot clobber fresher results.
const fetchSeq = useRef(0);
async function fetchLimits() {
const seq = ++fetchSeq.current;
try {
setLoading(true);
setLimits(await api.limits(selectedGpuIndex));
let next = await api.limits(selectedGpuIndex);
// The RM support probe can flake transiently (the driver briefly
// rejects RM object allocation when busy) and report
// rm_power_supported=false for one read. A single failed read must not
// make the Experimental Power Control block vanish — or flip it into
// "interface unavailable" — so confirm a negative reading with one
// follow-up fetch before trusting it. The confirm has its own error
// boundary: if it throws (network blip), keep the first result, which
// is still a valid read, rather than discarding it into the outer catch.
if (!next.rm_power_supported) {
try {
const again = await api.limits(selectedGpuIndex);
if (again.rm_power_supported) {
next = again;
}
} catch {
// keep first result
}
}
if (seq === fetchSeq.current) {
setLimits(next);
}
} catch {
toast.error("Failed to load performance limits");
if (seq === fetchSeq.current) {
toast.error("Failed to load performance limits");
}
} finally {
setLoading(false);
if (seq === fetchSeq.current) {
setLoading(false);
}
}
}