Trace terminal recovery and isolate neighbor resolver latency
This commit is contained in:
7 files changed
+84
-3
No files matched your search
@@ -103,6 +103,23 @@ sum by (job, instance) (increase(wakey_terminal_backpressure_total[15m]))
|
||||
This counts encounters, not their duration and not dropped chunks. A burst of
|
||||
image output can raise it while output still arrives intact through backpressure.
|
||||
|
||||
Average duration of completed backpressure episodes, in seconds:
|
||||
|
||||
```promql
|
||||
sum by (job, instance, outcome) (
|
||||
rate(wakey_operation_duration_seconds_sum{operation="terminal_backpressure_wait"}[5m])
|
||||
)
|
||||
/
|
||||
sum by (job, instance, outcome) (
|
||||
rate(wakey_operation_duration_seconds_count{operation="terminal_backpressure_wait"}[5m])
|
||||
)
|
||||
```
|
||||
|
||||
An episode starts when the output channel is full and ends when the pending
|
||||
backlog drains or the relay disconnects. It is not per-frame network latency;
|
||||
unfinished episodes do not appear until they end. Keep the outcome grouping so
|
||||
disconnected episodes are not confused with successfully drained ones.
|
||||
|
||||
## Snapshots
|
||||
|
||||
Device entries in successfully sent snapshots over the last hour:
|
||||
|
||||
@@ -100,6 +100,12 @@ existing debug events too; inspect them before enabling it broadly.
|
||||
with backoff and CC disconnect duration. Sending credentials is not an auth ACK;
|
||||
CC's authentication event is the authoritative result. DNS diagnostics are a
|
||||
separate lookup, not a measurement of the resolver inside WebSocket connect.
|
||||
- Terminal recovery uses `terminal_snapshot_generate` (encoded byte count),
|
||||
transport-attachment spans, and `terminal_snapshot_request`. These measure
|
||||
local work, not browser restoration or end-to-end image replay. Resume operation
|
||||
success means credentials were queued for the worker. `terminal_backpressure_wait`
|
||||
operation durations cover one full-queue backlog episode until it drains (`ok`)
|
||||
or disconnects (`disconnected`); cancellation can retain the default `error` outcome.
|
||||
|
||||
Instrument operation boundaries rather than individual rows or terminal chunks.
|
||||
Use `skip_all`, explicitly allow count fields, and instrument async futures rather
|
||||
|
||||
Reference in new issue
Block a user