fix: retry transient RM allocation failures so power control block doesn't vanish #20

Merged
Pakobbix merged 1 commits from fix/rm-power-transient-allocation into main 2026-11-07 14:05:27 +00:00
Collaborator

Symptom: After applying a board power limit, the whole Experimental Power Control block vanished and only came back after reloading the tab.

Root cause: GSP transiently rejects RM NV0080 allocation with status 0x1f when busy — 100% on the first probe in a fresh server process, ~1/60 right after a power-limit write (verified by instrumenting the live server). The probe surfaced it as rm_power_supported=false; the panel renders behind (rm_power_supported || mode === "ioctl"), so one flaky probe hid the block until a reload re-probed.

Changes:

  • hal/rm_power.py: RmHandle.open() retries the handle allocation (3×, 100 ms apart) before reporting unavailable; original body moved to _open_once() which closes its fds per attempt, so retries are idempotent. Covers probe, set, and enable-check paths.
  • PerformancePanel.tsx: confirm a negative rm_power_supported reading with one follow-up fetch before rendering (own error boundary); plus a fetch-seq guard so a stale response for a previously selected GPU can't clobber fresher state.
  • test_rm_power.py: two new tests (transient retried until success; persistent still raises after the attempt count).

Verification: first probe after restart 5/5 (was 0/3); 150 rapid set+get cycles → 0 flips, 0 set failures. test_rm_power.py 76/76; full pytest green except one pre-existing unrelated wireview failure. tsc + vite clean, no new eslint findings. Passed an independent subagent review.

**Symptom:** After applying a board power limit, the whole Experimental Power Control block vanished and only came back after reloading the tab. **Root cause:** GSP transiently rejects RM NV0080 allocation with status `0x1f` when busy — 100% on the first probe in a fresh server process, ~1/60 right after a power-limit write (verified by instrumenting the live server). The probe surfaced it as `rm_power_supported=false`; the panel renders behind `(rm_power_supported || mode === "ioctl")`, so one flaky probe hid the block until a reload re-probed. **Changes:** - `hal/rm_power.py`: `RmHandle.open()` retries the handle allocation (3×, 100 ms apart) before reporting unavailable; original body moved to `_open_once()` which closes its fds per attempt, so retries are idempotent. Covers probe, set, and enable-check paths. - `PerformancePanel.tsx`: confirm a negative `rm_power_supported` reading with one follow-up fetch before rendering (own error boundary); plus a fetch-seq guard so a stale response for a previously selected GPU can't clobber fresher state. - `test_rm_power.py`: two new tests (transient retried until success; persistent still raises after the attempt count). **Verification:** first probe after restart 5/5 (was 0/3); 150 rapid set+get cycles → 0 flips, 0 set failures. `test_rm_power.py` 76/76; full pytest green except one pre-existing unrelated wireview failure. tsc + vite clean, no new eslint findings. Passed an independent subagent review.
ARIA added 1 commit 2026-11-07 13:44:12 +00:00
GSP transiently rejects NV0080 object allocation with status 0x1f when
busy — reliably on the first probe in a fresh server process and
intermittently right after RM operations (e.g. a power-limit write).
The probe surfaced that transient rejection as rm_power_supported=false,
making the Experimental Power Control block disappear in the UI until a
reload re-probed successfully.

- hal/rm_power: RmHandle.open() now retries the handle allocation
  (3 attempts, 100 ms apart) before reporting the interface unavailable;
  covers probe, set, and enable-check paths. Each attempt cleans up its
  own fds, so retries are safe and idempotent.
- PerformancePanel: confirm a negative rm_power_supported reading with
  one follow-up fetch before rendering it, so a single flaky probe can't
  hide the block or flip it into "interface unavailable"; the confirm
  has its own error boundary so a failed second request keeps the first
  (valid) read. Also added a monotonic fetch-seq guard so a slow
  response for a previously selected GPU cannot clobber fresher state.
- tests: cover transient-failure-retry and persistent-failure paths for
  RmHandle.open.

Verified live: first probe after restart 5/5 (was 0/3), 150 rapid
set+get cycles with 0 probe flips and 0 set failures.
Pakobbix requested review from Zephyre_Review 2026-11-07 13:55:04 +00:00
Pakobbix removed review request for Zephyre_Review 2026-11-07 13:55:37 +00:00
Pakobbix requested review from Zephyre_Review 2026-11-07 13:55:49 +00:00
Collaborator

✅ No issues found — changes look consistent with the stated intent. Ready to be merged.

The retry logic in RmHandle.open() is clean: bounded attempts, proper error propagation (raises the last RmPowerError), and the idempotency of _open_once closing its own fds per attempt is noted in the docstring. The frontend fetch-seq guard correctly prevents stale responses from clobbering state, and the transient-retry confirm for rm_power_supported has its own error boundary as described. Tests properly patch/restore the classmethod and verify both the retry-until-success and persistent-failure paths.

✅ No issues found — changes look consistent with the stated intent. Ready to be merged. The retry logic in `RmHandle.open()` is clean: bounded attempts, proper error propagation (raises the last `RmPowerError`), and the idempotency of `_open_once` closing its own fds per attempt is noted in the docstring. The frontend fetch-seq guard correctly prevents stale responses from clobbering state, and the transient-retry confirm for `rm_power_supported` has its own error boundary as described. Tests properly patch/restore the classmethod and verify both the retry-until-success and persistent-failure paths.
Collaborator

This PR looks good — the retry loop, fd cleanup on failed attempts, and the fetch-seq/confirm logic in the panel all check out (Python test suite itself couldn't run here without an NVIDIA driver, so the new retry logic was verified by executing the module in isolation; tsc, eslint, and vitest are clean). Nothing to fix, ready to merge.

This PR looks good — the retry loop, fd cleanup on failed attempts, and the fetch-seq/confirm logic in the panel all check out (Python test suite itself couldn't run here without an NVIDIA driver, so the new retry logic was verified by executing the module in isolation; tsc, eslint, and vitest are clean). Nothing to fix, ready to merge.
Pakobbix merged commit 8567eee475 into main 2026-11-07 14:05:27 +00:00
Pakobbix deleted branch fix/rm-power-transient-allocation 2026-11-07 14:05:27 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Pakobbix/nvcurve#20