Add shared OTLP telemetry and propagate command traces across agents

2026-10-09 23:16:21 +07:00
1 parent 53c274dbe8
commit 9814754e50
@@ -0,0 +1,169 @@
+# OpenWrt UCI for Wakey
+
+Wakey's OpenWrt init script reads `/etc/config/wakey` for the log filter and
+OTLP authorization header, then passes them to the agent through procd.
+The agent's application settings remain in `/etc/wakey-agent/config.toml`.
+The previous `/etc/wakey-agent/environment` file is no longer read.
+
+## The three names
+
+```sh
+uci set wakey.main.otlp_authorization='Basic BASE64_OF_USERNAME_COLON_PASSWORD'
+```
+
+| Part | Meaning |
+| --- | --- |
+| `wakey` | Package: `/etc/config/wakey` |
+| `main` | Named section inside that package |
+| `otlp_authorization` | Option inside the section |
+
+Quotes keep the complete `Basic ...` value together despite its space.
+The credential is the Base64 encoding of `username:password`.
+It is not the htpasswd hash, and Base64 is not encryption.
+
+## Create and save
+
+Run as root on the router. `touch` does not truncate an existing file:
+
+```sh
+touch /etc/config/wakey
+chown root:root /etc/config/wakey
+chmod 600 /etc/config/wakey
+
+uci set wakey.main=service
+uci set wakey.main.log_filter='info'
+uci set wakey.main.otlp_authorization='Basic BASE64_OF_USERNAME_COLON_PASSWORD'
+uci commit wakey
+chmod 600 /etc/config/wakey
+```
+
+`wakey.main=service` creates a named section of type `service`. The type is our
+chosen label, not a built-in Wakey feature. Existing options are not cleared.
+`set` stages changes; `commit wakey` writes that package to disk. Committing does
+not restart a service.
+
+The resulting file looks like:
+
+```text
+config service 'main'
+    option log_filter 'info'
+    option otlp_authorization 'Basic BASE64_OF_USERNAME_COLON_PASSWORD'
+```
+
+The example above uses a placeholder. Avoid typing real credentials into a
+command that your shell will save in history. On OpenWrt's ash, you can instead
+read the Basic header without echoing it and send it to UCI over stdin:
+
+```sh
+read -r -s -p 'Complete Basic authorization header: ' authorization
+printf '\n'
+printf "set wakey.main.otlp_authorization='%s'\ncommit wakey\n" "$authorization" | uci batch
+unset authorization
+chmod 600 /etc/config/wakey
+```
+
+Use a normal `Basic <base64>` header here, without quotes/newlines inside the value.
+The shell's builtin `printf` avoids passing the secret in `uci`'s command arguments.
+UCI is not a secret vault: committed files, staged changes, router backups, and
+root access can expose these credentials. Do not paste their output into logs/chat.
+
+## Inspect, change, delete
+
+Inspect the non-secret option:
+
+```sh
+uci get wakey.main.log_filter
+```
+
+`uci show wakey`, `uci export wakey`, `uci changes wakey`, and
+`uci get wakey.main.otlp_authorization` **print the credential**. Use them only
+when you deliberately want to see it.
+
+Change an option:
+
+```sh
+uci set wakey.main.log_filter='wakey_agent=debug,wakey=debug,opentelemetry=warn'
+uci commit wakey
+```
+
+Delete just the authorization option:
+
+```sh
+uci -q delete wakey.main.otlp_authorization
+uci commit wakey
+```
+
+If TOML still requires `authorization_env`, removing the variable makes startup
+fail. Remove that TOML field too only when the receiver no longer requires auth.
+
+Discard all uncommitted changes to this package:
+
+```sh
+uci revert wakey
+```
+
+## How the service reads it
+
+`config_load` and `config_get` are shell helpers from `/lib/functions.sh`, available
+in the normal `/etc/rc.common` init-script environment. They are not standalone
+commands to run directly in your SSH shell.
+
+```sh
+config_load wakey
+config_get authorization main otlp_authorization
+```
+
+`config_get` arguments are **destination variable, section, option, optional default**.
+The line above puts the option into the shell variable `authorization`. It does
+not export it to the agent automatically; procd must receive it explicitly.
+
+The checked-in [init script](../scripts/init/openwrt/wakey) uses these helpers.
+Its configuration handling is:
+
+```sh
+start_service() {
+    local authorization='' log_filter='info'
+    if [ -f /etc/config/wakey ]; then
+        config_load wakey || return 1
+        config_get authorization main otlp_authorization
+        config_get log_filter main log_filter 'info'
+    fi
+
+    procd_open_instance
+    procd_set_param command "$BIN" serve --config "$CONFIG"
+    procd_set_param file "$CONFIG"
+    [ ! -f /etc/config/wakey ] || procd_append_param file /etc/config/wakey
+    procd_set_param env "RUST_LOG=$log_filter"
+    [ -z "$authorization" ] || procd_append_param env "WAKEY_OTLP_AUTHORIZATION=$authorization"
+    procd_set_param respawn 5 1 0
+    procd_set_param stdout 1
+    procd_set_param stderr 1
+    procd_close_instance
+}
+```
+
+Keep these fields in the agent's TOML:
+
+```toml
+[telemetry]
+otlp_endpoint = "http://10.5.0.1:4317"
+authorization_env = "WAKEY_OTLP_AUTHORIZATION"
+```
+
+UCI stores the header; procd passes the environment variable; Rust reads it using
+the name in TOML. The application does not parse UCI itself.
+
+After installing the updated init script and committing settings:
+
+```sh
+/etc/init.d/wakey restart
+```
+
+Use **restart**, not the existing HUP-based reload: a running process cannot receive
+changed environment variables that way. UCI is only for the router; the CC's
+systemd environment file remains unchanged. See [telemetry.md](telemetry.md).
+
+## References
+
+- [OpenWrt UCI CLI source and command usage](https://github.com/openwrt/uci/blob/master/cli.c)
+- [OpenWrt config_load/config_get helpers](https://github.com/openwrt/openwrt/blob/main/package/base-files/files/lib/functions.sh)
@@ -0,0 +1,127 @@
+# Telemetry
+
+CC and the agent daemon share the optional `wakey-core/telemetry` module.
+Local logs remain enabled. Configure an OTLP **gRPC** receiver to additionally
+export traces, structured logs, and metrics; HTTP/4318 is not supported in this slice.
+
+CC configuration:
+
+```toml
+[telemetry]
+otlp_endpoint = "http://127.0.0.1:4317"
+authorization_env = "WAKEY_OTLP_AUTHORIZATION"
+service_name = "wakey-control-plane"
+json_logs = true
+```
+
+Agent configuration (replace the example with the collector's WireGuard address):
+
+```toml
+[telemetry]
+otlp_endpoint = "http://10.5.0.1:4317"
+authorization_env = "WAKEY_OTLP_AUTHORIZATION"
+service_name = "wakey-agent"
+json_logs = false
+```
+
+Omit `otlp_endpoint` to disable exporting. Omit `authorization_env` for a receiver
+that requires no authentication. Existing configs need no changes.
+An empty agent service name uses `wakey-agent`; its enrolled agent ID becomes
+the resource's `service.instance.id`, not a metric label. Telemetry config changes
+require a daemon restart, not SIGHUP. Short-lived enroll/reload/observe commands
+only log locally and do not initialize exporters.
+
+## Receiver and filtering
+
+The collector must enable OTLP receivers in **traces, logs, and metrics** pipelines.
+Loopback is sufficient for CC on the same host/network namespace. The router
+needs a collector listener reachable over WireGuard; do not expose an unauthenticated
+receiver on the public Internet. This initial setup assumes plaintext inside that
+tunnel. `json_logs` changes local output,
+not whether OTLP logs are exported.
+
+## Credentials and services
+
+The OpenWrt agent uses `/etc/config/wakey`; see the
+[UCI setup guide](openwrt-uci.md) for commands and permissions.
+The CC uses a systemd environment file as described below.
+
+`authorization_env` names an environment variable containing the complete header:
+`Basic <base64(username:password)>` or `Bearer <token>`. With exporting enabled,
+missing/empty/invalid values fail startup; credentials are never written to TOML
+or included in error messages. Metadata is marked sensitive for Debug output.
+Base64 is not encryption: use the WireGuard/SSH tunnel or another protected transport.
+
+The systemd CC reads the optional `/etc/wakey-control-plane/environment` file.
+
+Use this portable literal format, **without quotes or `export`**:
+
+```text
+# Replace this placeholder with the actual encoded credential locally.
+WAKEY_OTLP_AUTHORIZATION=Basic BASE64_OF_USERNAME_COLON_PASSWORD
+RUST_LOG=info
+```
+
+Protect the file with owner `root:root` and mode `0600`. systemd's manager reads it
+before switching to the service user. Use the simple format above, with LF endings
+and no inline comments or multiline values.
+An absent file is allowed; configuring auth without providing its variable is not.
+
+After changing the file, **restart** the service. On systemd, run `daemon-reload`
+once after installing the updated unit. OpenWrt's HUP reload does not reload the
+process environment. For a foreground WSL test, export the variable in that shell
+and use `http://127.0.0.1:4317` with the SSH tunnel; do not put credentials in CLI args.
+
+`RUST_LOG` overrides the verbosity defaults for both local output and exports.
+For example, `RUST_LOG=warn` suppresses informational command spans as well as logs.
+Use `wakey_agent=info,wakey=info,wakey_core=info,opentelemetry=warn` for the agent and
+`wakey_control_plane=info,wakey_core=info,opentelemetry=warn` for CC. Exporter warnings
+remain local to prevent export-failure feedback loops. Increasing verbosity exports the
+existing debug events too; inspect them before enabling it broadly.
+
+## Signals
+
+- `relay_command` on CC propagates W3C context in an optional command field.
+  `execute_command` on the agent adopts it; older messages/agents remain compatible.
+- `collect_device_snapshot` and `store_device_snapshot` measure collection and
+  persistence separately. Snapshot traces are not yet linked across the wire.
+- Terminal create/attach/close/resume spans describe lifecycle operations, not
+  output chunks. Trace-linked OTLP logs use the SDK's tracing-context bridge.
+
+Metrics are exported every 30 seconds and at orderly shutdown:
+
+| Metric | Meaning |
+| --- | --- |
+| `wakey.operations` | Completed command/snapshot attempts, labeled by operation and outcome |
+| `wakey.operation.duration` | Corresponding duration histogram in seconds |
+| `wakey.snapshot.devices` | Cumulative devices in snapshots successfully sent by the agent |
+| `wakey.terminal.active` | Agent PTY workers currently alive |
+| `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
+| `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
+| `wakey.terminal.backpressure` | Relay queue-full encounters, not dropped output |
+
+Metric labels do not include request/session IDs, MACs, or IPs. New spans use
+`skip_all` and explicitly select safe fields; terminal bytes and credentials are
+not attached. Profiles, host CPU/RSS metrics, connected-agent gauges, and export-loss
+dashboards are not part of this initial slice.
+
+## Failure policy
+
+Trace and log queues are capped at 256 records each, with export batches of 64
+and a 3-second request timeout. Export happens off the application path. On an
+outage, telemetry can be dropped rather than accumulating unbounded router RAM;
+terminal output follows its existing lossless backpressure policy independently.
+The bound is a record count, not an absolute byte limit on arbitrary log messages.
+Providers flush on orderly daemon exit (Ctrl-C/SIGTERM), on a blocking worker while
+Tokio remains alive. Only completed spans are exported; this is not graceful draining
+of every in-flight request. SIGKILL/power loss cannot flush. This is not a persistent
+telemetry spool.
+
+## Validation
+
+```sh
+cargo test -p wakey-core --features telemetry --test telemetry
+```
+
+The test starts a temporary gRPC collector and verifies all three signals,
+command parent/child IDs, log correlation, and shutdown delivery without sleeps.