Add shared OTLP telemetry and propagate command traces across agents

docs/telemetry.md

@@ -0,0 +1,127 @@
+# Telemetry
+
+CC and the agent daemon share the optional `wakey-core/telemetry` module.
+Local logs remain enabled. Configure an OTLP **gRPC** receiver to additionally
+export traces, structured logs, and metrics; HTTP/4318 is not supported in this slice.
+
+CC configuration:
+
+```toml
+[telemetry]
+otlp_endpoint = "http://127.0.0.1:4317"
+authorization_env = "WAKEY_OTLP_AUTHORIZATION"
+service_name = "wakey-control-plane"
+json_logs = true
+```
+
+Agent configuration (replace the example with the collector's WireGuard address):
+
+```toml
+[telemetry]
+otlp_endpoint = "http://10.5.0.1:4317"
+authorization_env = "WAKEY_OTLP_AUTHORIZATION"
+service_name = "wakey-agent"
+json_logs = false
+```
+
+Omit `otlp_endpoint` to disable exporting. Omit `authorization_env` for a receiver
+that requires no authentication. Existing configs need no changes.
+An empty agent service name uses `wakey-agent`; its enrolled agent ID becomes
+the resource's `service.instance.id`, not a metric label. Telemetry config changes
+require a daemon restart, not SIGHUP. Short-lived enroll/reload/observe commands
+only log locally and do not initialize exporters.
+
+## Receiver and filtering
+
+The collector must enable OTLP receivers in **traces, logs, and metrics** pipelines.
+Loopback is sufficient for CC on the same host/network namespace. The router
+needs a collector listener reachable over WireGuard; do not expose an unauthenticated
+receiver on the public Internet. This initial setup assumes plaintext inside that
+tunnel. `json_logs` changes local output,
+not whether OTLP logs are exported.
+
+## Credentials and services
+
+The OpenWrt agent uses `/etc/config/wakey`; see the
+[UCI setup guide](openwrt-uci.md) for commands and permissions.
+The CC uses a systemd environment file as described below.
+
+`authorization_env` names an environment variable containing the complete header:
+`Basic <base64(username:password)>` or `Bearer <token>`. With exporting enabled,
+missing/empty/invalid values fail startup; credentials are never written to TOML
+or included in error messages. Metadata is marked sensitive for Debug output.
+Base64 is not encryption: use the WireGuard/SSH tunnel or another protected transport.
+
+The systemd CC reads the optional `/etc/wakey-control-plane/environment` file.
+
+Use this portable literal format, **without quotes or `export`**:
+
+```text
+# Replace this placeholder with the actual encoded credential locally.
+WAKEY_OTLP_AUTHORIZATION=Basic BASE64_OF_USERNAME_COLON_PASSWORD
+RUST_LOG=info
+```
+
+Protect the file with owner `root:root` and mode `0600`. systemd's manager reads it
+before switching to the service user. Use the simple format above, with LF endings
+and no inline comments or multiline values.
+An absent file is allowed; configuring auth without providing its variable is not.
+
+After changing the file, **restart** the service. On systemd, run `daemon-reload`
+once after installing the updated unit. OpenWrt's HUP reload does not reload the
+process environment. For a foreground WSL test, export the variable in that shell
+and use `http://127.0.0.1:4317` with the SSH tunnel; do not put credentials in CLI args.
+
+`RUST_LOG` overrides the verbosity defaults for both local output and exports.
+For example, `RUST_LOG=warn` suppresses informational command spans as well as logs.
+Use `wakey_agent=info,wakey=info,wakey_core=info,opentelemetry=warn` for the agent and
+`wakey_control_plane=info,wakey_core=info,opentelemetry=warn` for CC. Exporter warnings
+remain local to prevent export-failure feedback loops. Increasing verbosity exports the
+existing debug events too; inspect them before enabling it broadly.
+
+## Signals
+
+- `relay_command` on CC propagates W3C context in an optional command field.
+  `execute_command` on the agent adopts it; older messages/agents remain compatible.
+- `collect_device_snapshot` and `store_device_snapshot` measure collection and
+  persistence separately. Snapshot traces are not yet linked across the wire.
+- Terminal create/attach/close/resume spans describe lifecycle operations, not
+  output chunks. Trace-linked OTLP logs use the SDK's tracing-context bridge.
+
+Metrics are exported every 30 seconds and at orderly shutdown:
+
+| Metric | Meaning |
+| --- | --- |
+| `wakey.operations` | Completed command/snapshot attempts, labeled by operation and outcome |
+| `wakey.operation.duration` | Corresponding duration histogram in seconds |
+| `wakey.snapshot.devices` | Cumulative devices in snapshots successfully sent by the agent |
+| `wakey.terminal.active` | Agent PTY workers currently alive |
+| `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
+| `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
+| `wakey.terminal.backpressure` | Relay queue-full encounters, not dropped output |
+
+Metric labels do not include request/session IDs, MACs, or IPs. New spans use
+`skip_all` and explicitly select safe fields; terminal bytes and credentials are
+not attached. Profiles, host CPU/RSS metrics, connected-agent gauges, and export-loss
+dashboards are not part of this initial slice.
+
+## Failure policy
+
+Trace and log queues are capped at 256 records each, with export batches of 64
+and a 3-second request timeout. Export happens off the application path. On an
+outage, telemetry can be dropped rather than accumulating unbounded router RAM;
+terminal output follows its existing lossless backpressure policy independently.
+The bound is a record count, not an absolute byte limit on arbitrary log messages.
+Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while
+Tokio remains alive. Only completed spans are exported; this is not graceful draining
+of every in-flight request. SIGKILL/power loss cannot flush. This is not a persistent
+telemetry spool.
+
+## Validation
+
+```sh
+cargo test -p wakey-core --features telemetry --test telemetry
+```
+
+The test starts a temporary gRPC collector and verifies all three signals,
+command parent/child IDs, log correlation, and shutdown delivery without sleeps.