wakey/docs/telemetry.md

Telemetry

CC and the agent daemon share the optional wakey-core/telemetry module. Local logs remain enabled. Configure an OTLP gRPC receiver to additionally export traces, structured logs, and metrics; HTTP/4318 is not supported in this slice.

CC configuration:

[telemetry]
otlp_endpoint = "http://127.0.0.1:4317"
authorization_env = "WAKEY_OTLP_AUTHORIZATION"
service_name = "wakey-control-plane"
json_logs = true

Agent configuration (replace the example with the collector's WireGuard address):

[telemetry]
otlp_endpoint = "http://10.5.0.1:4317"
authorization_env = "WAKEY_OTLP_AUTHORIZATION"
service_name = "wakey-agent"
json_logs = false

Omit otlp_endpoint to disable exporting. Omit authorization_env for a receiver that requires no authentication. Existing configs need no changes. An empty agent service name uses wakey-agent; its enrolled agent ID becomes the resource's service.instance.id, not a metric label. Telemetry config changes require a daemon restart, not SIGHUP. Short-lived enroll/reload/observe commands only log locally and do not initialize exporters.

Receiver and filtering

The collector must enable OTLP receivers in traces, logs, and metrics pipelines. Loopback is sufficient for CC on the same host/network namespace. The router needs a collector listener reachable over WireGuard; do not expose an unauthenticated receiver on the public Internet. This initial setup assumes plaintext inside that tunnel. json_logs changes local output, not whether OTLP logs are exported.

Credentials and services

The OpenWrt agent uses /etc/config/wakey; see the UCI setup guide for commands and permissions. The CC uses a systemd environment file as described below.

authorization_env names an environment variable containing the complete header: Basic <base64(username:password)> or Bearer <token>. With exporting enabled, missing/empty/invalid values fail startup; credentials are never written to TOML or included in error messages. Metadata is marked sensitive for Debug output. Base64 is not encryption: use the WireGuard/SSH tunnel or another protected transport.

The systemd CC reads the optional /etc/wakey-control-plane/environment file.

Use this portable literal format, without quotes or export:

# Replace this placeholder with the actual encoded credential locally.
WAKEY_OTLP_AUTHORIZATION=Basic BASE64_OF_USERNAME_COLON_PASSWORD
RUST_LOG=info

Protect the file with owner root:root and mode 0600. systemd's manager reads it before switching to the service user. Use the simple format above, with LF endings and no inline comments or multiline values. An absent file is allowed; configuring auth without providing its variable is not.

After changing the file, restart the service. On systemd, run daemon-reload once after installing the updated unit. OpenWrt's HUP reload does not reload the process environment. For a foreground WSL test, export the variable in that shell and use http://127.0.0.1:4317 with the SSH tunnel; do not put credentials in CLI args.

Console logs, OTLP logs, and traces can have independent filters:

[telemetry]
console_filter = "info"
otlp_log_filter = "debug"
trace_filter = "info"

For each signal, an explicit filter wins over RUST_LOG, which wins over CLI verbosity (-v). With no filters, environment override, or verbosity flags, the application defaults remain info-level with SDK warnings. Metrics are unaffected. Explicit filters must be valid and nonempty; changing them requires a restart. Without explicit filters, RUST_LOG=warn suppresses informational spans and logs. Use wakey_agent=info,wakey=info,wakey_core=info,opentelemetry=warn for the agent and wakey_control_plane=info,wakey_core=info,opentelemetry=warn for CC. Exporter warnings remain local to prevent export-failure feedback loops. Both OTLP logs and traces exclude OpenTelemetry SDK targets and the h2, hyper, hyper_util, tonic and tower crate targets, even under broad debug filters. These transport crates also serve application traffic: their diagnostics stay console-only, while Wakey's command/storage/relay spans still export. Console filters remain independent. Increasing verbosity exports application debug events too; inspect them before enabling it broadly.

Signals

• relay_command on CC propagates W3C context in an optional command field. execute_command on the agent adopts it; older messages/agents remain compatible.
• collect_device_snapshot and store_device_snapshot measure collection and persistence as linked spans across the agent WebSocket. The snapshot carries optional W3C context from collection; CC adopts it after authentication checks. Periodic and requested syncs use the same path. Old messages and empty/invalid context remain valid but do not create a cross-process link. Both processes need tracing enabled and suitable filters/sampling to see both sides.
• Terminal create/attach/close/resume spans describe lifecycle operations, not output chunks. Trace-linked OTLP logs use the SDK's tracing-context bridge.
• Inventory source spans separate neighbor, DHCP lease, and hook-store reads from merging. db_* spans cover snapshot replacement, fleet reads, and manual identity changes, including transaction waits and result assembly where applicable. Snapshot replacement retains db_replace_agent_snapshot as its total duration, with four sequential child spans:
  • db_pool_acquire: obtaining a pooled connection, including any connection setup/checks; it is not exclusively queue wait.
  • db_write_transaction_begin: executing BEGIN IMMEDIATE, including SQLite writer-lock wait and worker scheduling, not a direct lock-wait measurement.
  • db_snapshot_write_rows: existing-key lookup, normalization, parent/child writes, and deletion of devices missing from the complete snapshot.
  • db_snapshot_commit: transaction commit, including its SQLite/worker work; this does not isolate filesystem flush time. Pool and transaction-begin spans come from the shared write helper, so they also appear under other write operations. No per-row spans or SQL values are recorded. A failed write rolls back through SQLx and has no commit span; fleet cache invalidation still happens only after a successful commit.
• Fleet spans cover list/load/grouping, refresh fan-out with per-agent outcomes, and wake dispatch. Counts distinguish grouped, filtered, and returned rows; wake events identify the chosen agent and whether the route was explicit, without recording device identifiers or addresses.
• Connectivity uses short agent_dns, agent_connect, agent_handshake_send, and CC agent_auth_verify spans, rather than connection-long spans. Connect attempts also contribute to operation metrics. Lifecycle events report retries with backoff and CC disconnect duration. Sending credentials is not an auth ACK; CC's authentication event is the authoritative result. DNS diagnostics are a separate lookup, not a measurement of the resolver inside WebSocket connect.
• Terminal recovery uses terminal_snapshot_generate (encoded byte count), transport-attachment spans, and terminal_snapshot_request. These measure local work, not browser restoration or end-to-end image replay. Resume operation success means credentials were queued for the worker. terminal_backpressure_wait operation durations cover one full-queue backlog episode until it drains (ok) or disconnects (disconnected); cancellation can retain the default error outcome.

Instrument operation boundaries rather than individual rows or terminal chunks. Use skip_all, explicitly allow count fields, and instrument async futures rather than holding entered span guards across awaits. Do not record query arguments, SQL parameters, device payloads, or credentials. Span names are fixed across requests.

Metrics are exported every 30 seconds and at orderly shutdown:

The table uses OpenTelemetry instrument names. Prometheus conversion typically replaces dots with underscores and adds type/unit suffixes; histograms produce multiple series. See the metrics recipe book for discovery queries, rates, averages, percentiles, and Grafana units.

Metric	Meaning
wakey.operations	Completed command/snapshot attempts, labeled by operation and outcome
wakey.operation.duration	Corresponding duration histogram in seconds
wakey.snapshot.devices	Cumulative devices in snapshots successfully sent by the agent
wakey.stored.devices	CC gauge of retained device rows, labeled by agent_id
wakey.stored.endpoints	CC gauge of retained endpoint rows, including inactive entries, labeled by agent_id
wakey.fleet.devices	CC gauge of unfiltered grouped identities, labeled by identity (saved or unsaved)
wakey.process.uptime	Linux gauge: seconds since this process's kernel-reported start, including suspend
wakey.process.memory.resident	Linux gauge: approximate resident memory in bytes, from VmRSS
wakey.process.cpu.time	Linux counter: cumulative user + system CPU seconds for all process threads, excluding children
wakey.process.open_fds	Linux gauge: current count of open process file descriptors
wakey.host.uptime	Linux gauge: uptime in seconds reported by the process's procfs view, including suspend
wakey.host.memory.total	Linux gauge: total usable RAM in bytes from the visible meminfo
wakey.host.memory.available	Linux gauge: estimated available RAM in bytes; omitted if MemAvailable is unavailable
wakey.host.load	Linux gauge: load average, with fixed period labels 1m, 5m, 15m
wakey.terminal.active	Agent PTY workers currently alive
wakey.terminal.input.bytes	Input bytes received by the terminal relay
wakey.terminal.output.bytes	Output bytes read from the PTY, including while detached
wakey.terminal.backpressure	Relay queue-full encounters, not dropped output

Metric labels do not include request/session IDs, MACs, or IPs. New spans use skip_all and explicitly select safe fields; terminal bytes and credentials are not attached. Profiles, host CPU-utilization metrics, connected-agent gauges, and export-loss dashboards are not part of this initial slice.

Process and host gauges are registered automatically for Linux agent/CC daemons when OTLP is enabled. wakey-core owns the OS-independent instruments and ProcessMetricsSource interface; wakey-linux supplies the procfs reader via its optional telemetry feature. Each daemon attaches that reader to its telemetry guard, so core has no Linux dependency or filesystem reads. Their small procfs reads run during the existing 30-second metric collection and final shutdown collection; no independent poller or task is started. Non-Linux builds omit these gauges. Missing/unreadable procfs data omits the affected observation instead of reporting zero or failing the daemon. No hostname, PID, command line or environment values are exported by this slice. Service/instance resource attributes identify each daemon; two daemons on the same host can report the same host uptime, so do not sum it across processes. CPU time is not elapsed time or CPU percentage: use its rate for consumption. Open-FD sampling can temporarily open descriptors counted by the measurement. Process metrics describe the daemon itself, not its child shells/PTY programs or the entire host. CPU/FD collection is also supplied by wakey-linux; unsupported platform readers may omit these optional measurements. Host memory/load come from the same optional Linux reader. MemAvailable estimates memory reusable without swapping and is not MemFree; no fallback is invented on older kernels. Load averages include runnable and uninterruptible tasks, so they are not CPU percentages. These are the visible kernel/procfs figures, not cgroup memory limits or container-specific load measurements; multiple daemons may export duplicate views of the same host.

RSS includes anonymous, file-backed and shared resident pages, not just heap, and is an approximate kernel measurement. Process age uses procfs start ticks and uptime, not wall-clock subtraction or telemetry initialization time. Container/time namespaces can change the visible host clock; this is the process's Linux view, not an assertion about a physical machine's boot time. The Linux parser is procfs, with optional default features disabled. Public /healthz remains unchanged.

Current-state fleet gauges sample independently every 30 seconds, with bounded revision retries to avoid mixed-state counts. Enrolled empty agents report zero; removed labels disappear when no enrollment or retained rows remain. Failed samples are omitted, not converted to zeros. No per-device labels are emitted. See the recipe book for scopes, queries, and the distinction between devices and endpoints.

Failure policy

Trace and log queues are capped at 256 records each, with export batches of 64 and a 3-second request timeout. Export happens off the application path. The OTLP exporters use their recommended retry policy: up to three retries after the first attempt, exponential backoff and jitter, within the exporter timeout budget. Temporary transport/receiver failures can recover; authentication rejection and partial-success responses are not retried. Requests already in flight can exceed the retry budget, so it is not a hard wall-clock deadline. The application does not wrap exports in another retry loop. On an outage, telemetry can be dropped rather than accumulating unbounded router RAM; terminal output follows its existing lossless backpressure policy independently.

Automated receiver tests reject the first export of each signal and confirm recovery, check the retry-attempt limit on a persistent outage, and confirm that bad credentials cause only one attempt. These tests do not establish durable delivery during a long outage; bounded queues can still drop telemetry. The bound is a record count, not an absolute byte limit on arbitrary log messages. Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while Tokio remains alive. Only completed spans are exported; this is not graceful draining of every in-flight request. SIGKILL/power loss cannot flush. This is not a persistent telemetry spool.

Validation

cargo test -p wakey-core --features telemetry --test telemetry

The test starts a temporary gRPC collector and verifies all three signals, command parent/child IDs, log correlation, and shutdown delivery without sleeps.