Trace terminal recovery and isolate neighbor resolver latency

This commit is contained in:
lda committed 2026-10-10 04:11:41 +07:00 Verified
1 parent 74c9b9c33d
commit a2f39bd3c2
7 files changed
+84 -3

No files matched your search

+17
View File
@@ -103,6 +103,23 @@ sum by (job, instance) (increase(wakey_terminal_backpressure_total[15m]))
This counts encounters, not their duration and not dropped chunks. A burst of
image output can raise it while output still arrives intact through backpressure.
Average duration of completed backpressure episodes, in seconds:
```promql
sum by (job, instance, outcome) (
rate(wakey_operation_duration_seconds_sum{operation="terminal_backpressure_wait"}[5m])
)
/
sum by (job, instance, outcome) (
rate(wakey_operation_duration_seconds_count{operation="terminal_backpressure_wait"}[5m])
)
```
An episode starts when the output channel is full and ends when the pending
backlog drains or the relay disconnects. It is not per-frame network latency;
unfinished episodes do not appear until they end. Keep the outcome grouping so
disconnected episodes are not confused with successfully drained ones.
## Snapshots
Device entries in successfully sent snapshots over the last hour:
+6
View File
@@ -100,6 +100,12 @@ existing debug events too; inspect them before enabling it broadly.
with backoff and CC disconnect duration. Sending credentials is not an auth ACK;
CC's authentication event is the authoritative result. DNS diagnostics are a
separate lookup, not a measurement of the resolver inside WebSocket connect.
- Terminal recovery uses `terminal_snapshot_generate` (encoded byte count),
transport-attachment spans, and `terminal_snapshot_request`. These measure
local work, not browser restoration or end-to-end image replay. Resume operation
success means credentials were queued for the worker. `terminal_backpressure_wait`
operation durations cover one full-queue backlog episode until it drains (`ok`)
or disconnects (`disconnected`); cancellation can retain the default `error` outcome.
Instrument operation boundaries rather than individual rows or terminal chunks.
Use `skip_all`, explicitly allow count fields, and instrument async futures rather