fix: bound fleet rebuilds and enable OTLP exporter retries
This commit is contained in:
10 files changed
+271
-64
No files matched your search
+4
-3
@@ -17,9 +17,10 @@ and reuse it. The helper checks the key again after building: if inputs changed
|
||||
while database reads/grouping were in flight, it discards the result and retries.
|
||||
Build errors are returned, not cached; cancelling a reader releases the lock.
|
||||
The previous view is not served as a fallback after a rebuild error.
|
||||
Retries currently have no fixed limit. Sustained mutations faster than rebuilding
|
||||
can delay the rebuilding reader and readers waiting on its lock; bounding this
|
||||
behavior is tracked as a reliability follow-up.
|
||||
After three invalidated builds, the reader stops and releases the lock. Fleet
|
||||
list/wake requests return HTTP 503 with code `fleet_busy`; clients can retry
|
||||
later. No stale view or command is used as a fallback. This bounds the number
|
||||
of rebuilds per reader, not database execution time or total concurrent requests.
|
||||
|
||||
Wake uses the same view, but command dispatch still checks the current session;
|
||||
a cached route cannot grant access to a disconnected agent.
|
||||
|
||||
+2
-1
@@ -41,6 +41,7 @@ of remembered identifiers based on observation expiry.
|
||||
- [x] Lazy grouped fleet-view cache with store/connection invalidation and stale-build rejection. See [fleet cache](fleet-cache.md).
|
||||
|
||||
- [x] Separate console, OTLP log, and trace filters using per-layer filtering. Explicit signal config overrides `RUST_LOG`, then CLI verbosity supplies defaults. Unconfigured defaults remain info-level; exporter-failure logs stay local. Live validation pending.
|
||||
- [x] Upgrade the matched OpenTelemetry stack to 0.33 with exporter-managed backoff/jitter retries. Controlled collector tests cover all-signal recovery, bounded retries, and no retry on authentication rejection; live outage validation remains pending.
|
||||
- [ ] Record hostname-resolution success/failure without exporting hostnames or resolved addresses. Current spans show timing but do not explicitly classify the result.
|
||||
- [ ] Investigate the slow resolver with a reproducible device-side check. Timing isolated the phase, not the cause; do not treat instrumentation as a fix.
|
||||
- [x] Propagate optional W3C trace context for periodic/requested WS snapshots so agent collection and CC persistence can be inspected together. Wire compatibility and context linkage tested; live validation pending.
|
||||
@@ -52,7 +53,7 @@ or current fleet size. Metric labels should remain bounded; no per-device labels
|
||||
|
||||
## Validation And Reliability
|
||||
|
||||
- [ ] Bound fleet-cache rebuild retries under sustained mutations; current stale-build rejection can keep a request waiting while inputs continuously change. Do not silently serve stale wake routes as a fallback.
|
||||
- [x] Bound fleet-cache rebuilds to three attempts under sustained mutations; return retryable HTTP 503 `fleet_busy` rather than silently serving stale wake routes. Deterministic limit/recovery tests included.
|
||||
- [ ] Reproduce/check Ctrl-C cleanup on the test device: process groups, PTYs, relay tasks, and telemetry shutdown. Leftovers were suspected, not confirmed.
|
||||
- [ ] Exercise collector unavailability and recovery with running agent/CC instances. Verify normal operations continue and telemetry queues remain bounded; outage losses are expected under the current best-effort policy.
|
||||
- [ ] Validate terminal recovery instrumentation live: generation timing/size, transport attach/resume, and backpressure episodes ending in drain or disconnect.
|
||||
|
||||
+12
-1
@@ -153,9 +153,20 @@ dashboards are not part of this initial slice.
|
||||
## Failure policy
|
||||
|
||||
Trace and log queues are capped at 256 records each, with export batches of 64
|
||||
and a 3-second request timeout. Export happens off the application path. On an
|
||||
and a 3-second request timeout. Export happens off the application path. The
|
||||
OTLP exporters use their recommended retry policy: up to three retries after
|
||||
the first attempt, exponential backoff and jitter, within the exporter timeout
|
||||
budget. Temporary transport/receiver failures can recover; authentication
|
||||
rejection and partial-success responses are not retried. Requests already in
|
||||
flight can exceed the retry budget, so it is not a hard wall-clock deadline.
|
||||
The application does not wrap exports in another retry loop. On an
|
||||
outage, telemetry can be dropped rather than accumulating unbounded router RAM;
|
||||
terminal output follows its existing lossless backpressure policy independently.
|
||||
|
||||
Automated receiver tests reject the first export of each signal and confirm
|
||||
recovery, check the retry-attempt limit on a persistent outage, and confirm that
|
||||
bad credentials cause only one attempt. These tests do not establish durable
|
||||
delivery during a long outage; bounded queues can still drop telemetry.
|
||||
The bound is a record count, not an absolute byte limit on arbitrary log messages.
|
||||
Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while
|
||||
Tokio remains alive. Only completed spans are exported; this is not graceful draining
|
||||
|
||||
Reference in new issue
Block a user