# Telemetry

CC and the agent daemon share the optional `wakey-core/telemetry` module.
Local logs remain enabled. Configure an OTLP **gRPC** receiver to additionally
export traces, structured logs, and metrics; HTTP/4318 is not supported in this slice.

CC configuration:

```toml
[telemetry]
otlp_endpoint = "http://127.0.0.1:4317"
authorization_env = "WAKEY_OTLP_AUTHORIZATION"
service_name = "wakey-control-plane"
json_logs = true
```

Agent configuration (replace the example with the collector's WireGuard address):

```toml
[telemetry]
otlp_endpoint = "http://10.5.0.1:4317"
authorization_env = "WAKEY_OTLP_AUTHORIZATION"
service_name = "wakey-agent"
json_logs = false
```

Omit `otlp_endpoint` to disable exporting. Omit `authorization_env` for a receiver
that requires no authentication. Existing configs need no changes.
An empty agent service name uses `wakey-agent`; its enrolled agent ID becomes
the resource's `service.instance.id`, not a metric label. Telemetry config changes
require a daemon restart, not SIGHUP. Short-lived enroll/reload/observe commands
only log locally and do not initialize exporters.

## Receiver and filtering

The collector must enable OTLP receivers in **traces, logs, and metrics** pipelines.
Loopback is sufficient for CC on the same host/network namespace. The router
needs a collector listener reachable over WireGuard; do not expose an unauthenticated
receiver on the public Internet. This initial setup assumes plaintext inside that
tunnel. `json_logs` changes local output,
not whether OTLP logs are exported.

## Credentials and services

The OpenWrt agent uses `/etc/config/wakey`; see the
[UCI setup guide](openwrt-uci.md) for commands and permissions.
The CC uses a systemd environment file as described below.

`authorization_env` names an environment variable containing the complete header:
`Basic <base64(username:password)>` or `Bearer <token>`. With exporting enabled,
missing/empty/invalid values fail startup; credentials are never written to TOML
or included in error messages. Metadata is marked sensitive for Debug output.
Base64 is not encryption: use the WireGuard/SSH tunnel or another protected transport.

The systemd CC reads the optional `/etc/wakey-control-plane/environment` file.

Use this portable literal format, **without quotes or `export`**:

```text
# Replace this placeholder with the actual encoded credential locally.
WAKEY_OTLP_AUTHORIZATION=Basic BASE64_OF_USERNAME_COLON_PASSWORD
RUST_LOG=info
```

Protect the file with owner `root:root` and mode `0600`. systemd's manager reads it
before switching to the service user. Use the simple format above, with LF endings
and no inline comments or multiline values.
An absent file is allowed; configuring auth without providing its variable is not.

After changing the file, **restart** the service. On systemd, run `daemon-reload`
once after installing the updated unit. OpenWrt's HUP reload does not reload the
process environment. For a foreground WSL test, export the variable in that shell
and use `http://127.0.0.1:4317` with the SSH tunnel; do not put credentials in CLI args.

`RUST_LOG` overrides the verbosity defaults for both local output and exports.
For example, `RUST_LOG=warn` suppresses informational command spans as well as logs.
Use `wakey_agent=info,wakey=info,wakey_core=info,opentelemetry=warn` for the agent and
`wakey_control_plane=info,wakey_core=info,opentelemetry=warn` for CC. Exporter warnings
remain local to prevent export-failure feedback loops. Increasing verbosity exports the
existing debug events too; inspect them before enabling it broadly.

## Signals

- `relay_command` on CC propagates W3C context in an optional command field.
  `execute_command` on the agent adopts it; older messages/agents remain compatible.
- `collect_device_snapshot` and `store_device_snapshot` measure collection and
  persistence separately. Snapshot traces are not yet linked across the wire.
- Terminal create/attach/close/resume spans describe lifecycle operations, not
  output chunks. Trace-linked OTLP logs use the SDK's tracing-context bridge.

Metrics are exported every 30 seconds and at orderly shutdown:

| Metric | Meaning |
| --- | --- |
| `wakey.operations` | Completed command/snapshot attempts, labeled by operation and outcome |
| `wakey.operation.duration` | Corresponding duration histogram in seconds |
| `wakey.snapshot.devices` | Cumulative devices in snapshots successfully sent by the agent |
| `wakey.terminal.active` | Agent PTY workers currently alive |
| `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
| `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
| `wakey.terminal.backpressure` | Relay queue-full encounters, not dropped output |

Metric labels do not include request/session IDs, MACs, or IPs. New spans use
`skip_all` and explicitly select safe fields; terminal bytes and credentials are
not attached. Profiles, host CPU/RSS metrics, connected-agent gauges, and export-loss
dashboards are not part of this initial slice.

## Failure policy

Trace and log queues are capped at 256 records each, with export batches of 64
and a 3-second request timeout. Export happens off the application path. On an
outage, telemetry can be dropped rather than accumulating unbounded router RAM;
terminal output follows its existing lossless backpressure policy independently.
The bound is a record count, not an absolute byte limit on arbitrary log messages.
Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while
Tokio remains alive. Only completed spans are exported; this is not graceful draining
of every in-flight request. SIGKILL/power loss cannot flush. This is not a persistent
telemetry spool.

## Validation

```sh
cargo test -p wakey-core --features telemetry --test telemetry
```

The test starts a temporary gRPC collector and verifies all three signals,
command parent/child IDs, log correlation, and shutdown delivery without sleeps.