feat(telemetry): report process memory and process and host uptime

This commit is contained in:
lda committed 2026-10-12 16:28:21 +07:00 Verified
1 parent b98b8717d1
commit 4a4acedbc3
16 files changed
+311 -8

No files matched your search

+2 -1
View File
@@ -55,7 +55,8 @@ or current fleet size. Metric labels should remain bounded; no per-device labels
## Process And Host Visibility
- [ ] Add shared process measurements for agent and CC: start time/uptime, resident memory, CPU time, and open file descriptors where supported. Keep version as resource metadata rather than repeating it on every sample.
- [x] Add Linux process age/RSS and host uptime gauges for agent and CC through opt-in OTLP, using the existing collection interval. Core owns OS-independent instruments; wakey-linux supplies the optional procfs reader. Missing data omits observations; no added labels, config fields, health routes or polling tasks. Collector coverage verifies units and values; router/live dashboard validation pending.
- [ ] Extend process measurements with CPU time and open file descriptors where supported, plus start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample.
- [ ] Add a small host summary: hostname, OS/kernel, CPU count, host uptime, load, and available/total memory. Distinguish host uptime from process uptime so router reboots and agent restarts are visible separately. Sources must tolerate missing Linux files; not every agent host is a router.
- [ ] Expose changing measurements through OTLP on a modest sampling interval; show selected process/host details on Fleet Health and agent details. Keep host information behind the protected operator interface, not public health endpoints. Avoid expensive reads on every request and unbounded labels.
- [ ] Keep `/healthz` as cheap liveness; consider a separate bounded `/readyz` database-access check for CC. Collector outages and disconnected fleet agents must not make a functioning CC unready.
+35
View File
@@ -26,6 +26,41 @@ that includes recent activity and allow at least two exports for rate queries.
Counter resets on restart are handled by `rate` and `increase`. Apply those
functions **before** summing across instances so individual resets remain visible.
## Process And Host
Discover the new gauges first (these are Linux-only):
```promql
count by (__name__, job, instance) ({__name__=~"wakey_(process|host)_.*"})
```
Process age and host uptime, in seconds:
```promql
wakey_process_uptime_seconds
```
```promql
wakey_host_uptime_seconds
```
Set Grafana's unit to a duration. Compare the two: process age resetting while
host uptime continues suggests a daemon restart; both resetting suggests a host
restart or a changed namespace/view. Missing samples are not a zero uptime or
proof of a restart. These are gauges: do not use `rate` or `increase` on them.
Resident process memory, in bytes:
```promql
wakey_process_memory_resident_bytes
```
Set the unit to **bytes (IEC)**. Keep `job` and `instance` in the legend to compare
agent and CC separately. RSS includes shared/file-backed pages and is not heap
allocation alone. A missing series can mean disabled telemetry, an unsupported
platform, unavailable procfs, or export trouble. No new health endpoint or host
inventory is provided here; do not sum host uptime from multiple daemons.
## Operations
Completed attempts per second, including errors and timeouts:
+25 -1
View File
@@ -157,6 +157,9 @@ queries, rates, averages, percentiles, and Grafana units.
| `wakey.stored.devices` | CC gauge of retained device rows, labeled by `agent_id` |
| `wakey.stored.endpoints` | CC gauge of retained endpoint rows, including inactive entries, labeled by `agent_id` |
| `wakey.fleet.devices` | CC gauge of unfiltered grouped identities, labeled by `identity` (`saved` or `unsaved`) |
| `wakey.process.uptime` | Linux gauge: seconds since this process's kernel-reported start, including suspend |
| `wakey.process.memory.resident` | Linux gauge: approximate resident memory in bytes, from VmRSS |
| `wakey.host.uptime` | Linux gauge: uptime in seconds reported by the process's procfs view, including suspend |
| `wakey.terminal.active` | Agent PTY workers currently alive |
| `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
| `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
@@ -164,9 +167,30 @@ queries, rates, averages, percentiles, and Grafana units.
Metric labels do not include request/session IDs, MACs, or IPs. New spans use
`skip_all` and explicitly select safe fields; terminal bytes and credentials are
not attached. Profiles, host CPU/RSS metrics, connected-agent gauges, and export-loss
not attached. Profiles, host CPU/total-memory metrics, connected-agent gauges, and export-loss
dashboards are not part of this initial slice.
Process and host gauges are registered automatically for Linux agent/CC daemons
when OTLP is enabled. `wakey-core` owns the OS-independent instruments and
`ProcessMetricsSource` interface; `wakey-linux` supplies the procfs reader via
its optional `telemetry` feature. Each daemon attaches that reader to its
telemetry guard, so core has no Linux dependency or filesystem reads.
Their small procfs reads run during the existing 30-second
metric collection and final shutdown collection; no independent poller or task
is started. Non-Linux builds omit these gauges. Missing/unreadable procfs data
omits the affected observation instead of reporting zero or failing the daemon.
No hostname, PID, command line or environment values are exported by this slice.
Service/instance resource attributes identify each daemon; two daemons on the
same host can report the same host uptime, so do not sum it across processes.
RSS includes anonymous, file-backed and shared resident pages, not just heap,
and is an approximate kernel measurement. Process age uses procfs start ticks
and uptime, not wall-clock subtraction or telemetry initialization time.
Container/time namespaces can change the visible host clock; this is the
process's Linux view, not an assertion about a physical machine's boot time.
The Linux parser is [procfs](https://docs.rs/procfs/0.18.0/procfs/), with optional
default features disabled. Public `/healthz` remains unchanged.
Current-state fleet gauges sample independently every 30 seconds, with bounded
revision retries to avoid mixed-state counts. Enrolled empty agents report zero;
removed labels disappear when no enrollment or retained rows remain. Failed samples