fix: bound fleet rebuilds and enable OTLP exporter retries

This commit is contained in:
lda committed 2026-10-11 18:01:24 +07:00 Verified
1 parent 69691fda47
commit aff090a822
10 files changed
+271 -64

No files matched your search

+4 -3
View File
@@ -17,9 +17,10 @@ and reuse it. The helper checks the key again after building: if inputs changed
while database reads/grouping were in flight, it discards the result and retries.
Build errors are returned, not cached; cancelling a reader releases the lock.
The previous view is not served as a fallback after a rebuild error.
Retries currently have no fixed limit. Sustained mutations faster than rebuilding
can delay the rebuilding reader and readers waiting on its lock; bounding this
behavior is tracked as a reliability follow-up.
After three invalidated builds, the reader stops and releases the lock. Fleet
list/wake requests return HTTP 503 with code `fleet_busy`; clients can retry
later. No stale view or command is used as a fallback. This bounds the number
of rebuilds per reader, not database execution time or total concurrent requests.
Wake uses the same view, but command dispatch still checks the current session;
a cached route cannot grant access to a disconnected agent.
+2 -1
View File
@@ -41,6 +41,7 @@ of remembered identifiers based on observation expiry.
- [x] Lazy grouped fleet-view cache with store/connection invalidation and stale-build rejection. See [fleet cache](fleet-cache.md).
- [x] Separate console, OTLP log, and trace filters using per-layer filtering. Explicit signal config overrides `RUST_LOG`, then CLI verbosity supplies defaults. Unconfigured defaults remain info-level; exporter-failure logs stay local. Live validation pending.
- [x] Upgrade the matched OpenTelemetry stack to 0.33 with exporter-managed backoff/jitter retries. Controlled collector tests cover all-signal recovery, bounded retries, and no retry on authentication rejection; live outage validation remains pending.
- [ ] Record hostname-resolution success/failure without exporting hostnames or resolved addresses. Current spans show timing but do not explicitly classify the result.
- [ ] Investigate the slow resolver with a reproducible device-side check. Timing isolated the phase, not the cause; do not treat instrumentation as a fix.
- [x] Propagate optional W3C trace context for periodic/requested WS snapshots so agent collection and CC persistence can be inspected together. Wire compatibility and context linkage tested; live validation pending.
@@ -52,7 +53,7 @@ or current fleet size. Metric labels should remain bounded; no per-device labels
## Validation And Reliability
- [ ] Bound fleet-cache rebuild retries under sustained mutations; current stale-build rejection can keep a request waiting while inputs continuously change. Do not silently serve stale wake routes as a fallback.
- [x] Bound fleet-cache rebuilds to three attempts under sustained mutations; return retryable HTTP 503 `fleet_busy` rather than silently serving stale wake routes. Deterministic limit/recovery tests included.
- [ ] Reproduce/check Ctrl-C cleanup on the test device: process groups, PTYs, relay tasks, and telemetry shutdown. Leftovers were suspected, not confirmed.
- [ ] Exercise collector unavailability and recovery with running agent/CC instances. Verify normal operations continue and telemetry queues remain bounded; outage losses are expected under the current best-effort policy.
- [ ] Validate terminal recovery instrumentation live: generation timing/size, transport attach/resume, and backpressure episodes ending in drain or disconnect.
+12 -1
View File
@@ -153,9 +153,20 @@ dashboards are not part of this initial slice.
## Failure policy
Trace and log queues are capped at 256 records each, with export batches of 64
and a 3-second request timeout. Export happens off the application path. On an
and a 3-second request timeout. Export happens off the application path. The
OTLP exporters use their recommended retry policy: up to three retries after
the first attempt, exponential backoff and jitter, within the exporter timeout
budget. Temporary transport/receiver failures can recover; authentication
rejection and partial-success responses are not retried. Requests already in
flight can exceed the retry budget, so it is not a hard wall-clock deadline.
The application does not wrap exports in another retry loop. On an
outage, telemetry can be dropped rather than accumulating unbounded router RAM;
terminal output follows its existing lossless backpressure policy independently.
Automated receiver tests reject the first export of each signal and confirm
recovery, check the retry-attempt limit on a persistent outage, and confirm that
bad credentials cause only one attempt. These tests do not establish durable
delivery during a long outage; bounded queues can still drop telemetry.
The bound is a record count, not an absolute byte limit on arbitrary log messages.
Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while
Tokio remains alive. Only completed spans are exported; this is not graceful draining