149 lines
5.4 KiB
Markdown
149 lines
5.4 KiB
Markdown
# Wakey Metrics Recipes
|
|
|
|
Use Grafana Explore with your Prometheus-compatible data source and the Code
|
|
query editor. These recipes assume standard OTLP-to-Prometheus name conversion.
|
|
Discover your actual series first:
|
|
|
|
```promql
|
|
count by (__name__, job) ({__name__=~"wakey[._].*"})
|
|
```
|
|
|
|
With Alloy's Prometheus exporter, service names normally become `job`, and agent
|
|
instance IDs become `instance`. Inspect the returned labels before adding filters.
|
|
For example, append `{job="wakey-agent"}` to a metric selector only if that is
|
|
the job value you actually see. Metrics export every 30 seconds; use a time range
|
|
that includes recent activity and allow at least two exports for rate queries.
|
|
|
|
## The Three Types
|
|
|
|
| Type | What it records | How to read it |
|
|
| --- | --- | --- |
|
|
| Counter | Cumulative events or bytes since process startup | `rate` for per-second throughput; `increase` for a window's estimated total |
|
|
| UpDownCounter | Adds and subtracts from a current quantity | Read the current value directly; do not use `rate` for active sessions |
|
|
| Histogram | Distribution of individual measurements | `_sum / _count` for an average; `_bucket` for estimated percentiles |
|
|
|
|
Counter resets on restart are handled by `rate` and `increase`. Apply those
|
|
functions **before** summing across instances so individual resets remain visible.
|
|
|
|
## Operations
|
|
|
|
Completed attempts per second, including errors and timeouts:
|
|
|
|
```promql
|
|
sum by (job, operation, outcome) (rate(wakey_operations_total[5m]))
|
|
```
|
|
|
|
Estimated attempts over the last hour:
|
|
|
|
```promql
|
|
sum by (job, operation, outcome) (increase(wakey_operations_total[1h]))
|
|
```
|
|
|
|
Average duration over the last five minutes:
|
|
|
|
```promql
|
|
sum by (job, operation) (rate(wakey_operation_duration_seconds_sum[5m]))
|
|
/
|
|
sum by (job, operation) (rate(wakey_operation_duration_seconds_count[5m]))
|
|
```
|
|
|
|
Set Grafana's unit to **seconds**. This is a weighted average, not an average of
|
|
instance averages. Both successful and unsuccessful attempts are included;
|
|
add `outcome="ok"` to both selectors for successful attempts only.
|
|
|
|
Estimated p95 duration:
|
|
|
|
```promql
|
|
histogram_quantile(
|
|
0.95,
|
|
sum by (le, job, operation) (
|
|
rate(wakey_operation_duration_seconds_bucket[5m])
|
|
)
|
|
)
|
|
```
|
|
|
|
This needs classic histogram bucket series. Keep `le` in the grouping: it is
|
|
the bucket boundary, in seconds. p95 is an estimate from those boundaries,
|
|
not an exact sort of recorded durations. Few operations make percentiles noisy.
|
|
|
|
CC relay duration includes waiting for the agent; agent execution duration
|
|
measures its dispatch separately. Do not sum both and call that user latency.
|
|
|
|
## Terminals
|
|
|
|
Active PTY workers by agent, including detached sessions:
|
|
|
|
```promql
|
|
sum by (job, instance) (wakey_terminal_active)
|
|
```
|
|
|
|
Read this directly as a count. It is not a count of browser tabs or attachments.
|
|
|
|
PTY output bytes per second:
|
|
|
|
```promql
|
|
sum by (job, instance) (rate(wakey_terminal_output_bytes_total[5m]))
|
|
```
|
|
|
|
Input bytes per second:
|
|
|
|
```promql
|
|
sum by (job, instance) (rate(wakey_terminal_input_bytes_total[5m]))
|
|
```
|
|
|
|
Set the unit to **bytes/sec**. Output is measured at the PTY read, even while
|
|
detached; it is not WebSocket wire traffic or confirmed browser delivery.
|
|
|
|
Relay queue-full encounters over the last fifteen minutes:
|
|
|
|
```promql
|
|
sum by (job, instance) (increase(wakey_terminal_backpressure_total[15m]))
|
|
```
|
|
|
|
This counts encounters, not their duration and not dropped chunks. A burst of
|
|
image output can raise it while output still arrives intact through backpressure.
|
|
|
|
Average duration of completed backpressure episodes, in seconds:
|
|
|
|
```promql
|
|
sum by (job, instance, outcome) (
|
|
rate(wakey_operation_duration_seconds_sum{operation="terminal_backpressure_wait"}[5m])
|
|
)
|
|
/
|
|
sum by (job, instance, outcome) (
|
|
rate(wakey_operation_duration_seconds_count{operation="terminal_backpressure_wait"}[5m])
|
|
)
|
|
```
|
|
|
|
An episode starts when the output channel is full and ends when the pending
|
|
backlog drains or the relay disconnects. It is not per-frame network latency;
|
|
unfinished episodes do not appear until they end. Keep the outcome grouping so
|
|
disconnected episodes are not confused with successfully drained ones.
|
|
|
|
## Snapshots
|
|
|
|
Device entries in successfully sent snapshots over the last hour:
|
|
|
|
```promql
|
|
sum by (job, instance) (increase(wakey_snapshot_devices_total[1h]))
|
|
```
|
|
|
|
Repeated snapshots count the same devices again. This is collection volume,
|
|
**not unique devices, current fleet size, or confirmed CC persistence**.
|
|
|
|
## Empty or Strange Results
|
|
|
|
- No series: generate the relevant activity, wait for export, and check the metrics pipeline.
|
|
- Different names: use the discovery query; collector translation can change suffixes and labels.
|
|
- `NaN` average: no operations occurred in the window, so the denominator is zero. Show a gap, not a fake zero latency.
|
|
- Empty `rate`: it needs multiple samples. Try a longer window and check the time range.
|
|
- Fractional `increase`: Prometheus extrapolates to the window boundaries; event counts can therefore appear fractional.
|
|
- Stale active count: the last exported value is not an immediate liveness check after a crash or network loss.
|
|
- No `instance` label: CC does not currently set a service instance ID; agent telemetry does.
|
|
|
|
## References
|
|
|
|
- [Wakey telemetry setup and limitations](telemetry.md)
|
|
- [Prometheus rate, increase, and histogram_quantile](https://prometheus.io/docs/prometheus/latest/querying/functions/)
|
|
- [Alloy OTLP-to-Prometheus conversion](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.prometheus/)
|