feat(telemetry): report process CPU and FDs plus host memory and load
This commit is contained in:
7 files changed
+301
-8
No files matched your search
+4
-2
@@ -56,8 +56,10 @@ or current fleet size. Metric labels should remain bounded; no per-device labels
|
||||
## Process And Host Visibility
|
||||
|
||||
- [x] Add Linux process age/RSS and host uptime gauges for agent and CC through opt-in OTLP, using the existing collection interval. Core owns OS-independent instruments; wakey-linux supplies the optional procfs reader. Missing data omits observations; no added labels, config fields, health routes or polling tasks. Collector coverage verifies units and values; router/live dashboard validation pending.
|
||||
- [ ] Extend process measurements with CPU time and open file descriptors where supported, plus start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample.
|
||||
- [ ] Add a small host summary: hostname, OS/kernel, CPU count, host uptime, load, and available/total memory. Distinguish host uptime from process uptime so router reboots and agent restarts are visible separately. Sources must tolerate missing Linux files; not every agent host is a router.
|
||||
- [x] Add cumulative process CPU time (counter) and current open-FD count (gauge) through the same Linux reader and OS-independent instruments. Exclude child processes, omit failed samples, and document one-core CPU-rate interpretation and sampling FD overhead. Extend collector/reader coverage rather than adding timing-based tests; live validation pending.
|
||||
- [ ] Add start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample.
|
||||
- [x] Add total/available host memory and 1/5/15-minute load gauges through the optional Linux reader, with byte conversion owned by procfs and fixed load-period labels. Omit unavailable MemAvailable instead of substituting free memory; document kernel views versus container limits. Collector/reader coverage included; live validation pending.
|
||||
- [ ] Complete a small host summary with hostname, OS/kernel and CPU count, plus protected UI presentation of collected measurements. Sources must tolerate missing Linux files; not every agent host is a router.
|
||||
- [ ] Expose changing measurements through OTLP on a modest sampling interval; show selected process/host details on Fleet Health and agent details. Keep host information behind the protected operator interface, not public health endpoints. Avoid expensive reads on every request and unbounded labels.
|
||||
- [ ] Keep `/healthz` as cheap liveness; consider a separate bounded `/readyz` database-access check for CC. Collector outages and disconnected fleet agents must not make a functioning CC unready.
|
||||
- [ ] Add application resource gauges where useful: DB pool usage, connected agents, pending commands, and active terminal sessions. Check existing instruments first to avoid duplicates; distinguish pool pressure from host/process pressure.
|
||||
|
||||
@@ -61,6 +61,59 @@ allocation alone. A missing series can mean disabled telemetry, an unsupported
|
||||
platform, unavailable procfs, or export trouble. No new health endpoint or host
|
||||
inventory is provided here; do not sum host uptime from multiple daemons.
|
||||
|
||||
CPU consumption as a percentage of **one CPU core**:
|
||||
|
||||
```promql
|
||||
100 * rate(wakey_process_cpu_time_seconds_total[5m])
|
||||
```
|
||||
|
||||
Set the unit to **percent (0-100)**. A multithreaded process can exceed 100%;
|
||||
this is not normalized by the host's CPU count. The counter includes daemon
|
||||
user/system CPU, not its child shells or commands. Allow at least two exports;
|
||||
`rate` handles counter resets on process restart.
|
||||
|
||||
Current open descriptors:
|
||||
|
||||
```promql
|
||||
wakey_process_open_fds
|
||||
```
|
||||
|
||||
Read directly as a count, not a rate. The sampling read can contribute temporary
|
||||
descriptors. Persistent growth can warrant investigation, but sockets, terminal
|
||||
sessions and pool connections legitimately change the count too.
|
||||
|
||||
Host usable/available RAM, in bytes:
|
||||
|
||||
```promql
|
||||
wakey_host_memory_total_bytes
|
||||
```
|
||||
|
||||
```promql
|
||||
wakey_host_memory_available_bytes
|
||||
```
|
||||
|
||||
Set units to **bytes (IEC)**. Estimated percentage not available for new work:
|
||||
|
||||
```promql
|
||||
100 * (1 - wakey_host_memory_available_bytes / wakey_host_memory_total_bytes)
|
||||
```
|
||||
|
||||
Available includes reclaimable memory and is not merely free RAM. Missing
|
||||
`MemAvailable` leaves a gap rather than substituting `MemFree`. Do not sum host
|
||||
RAM from multiple daemon instances on the same host, or interpret these kernel
|
||||
values as a container's cgroup limit.
|
||||
|
||||
Load averages by period:
|
||||
|
||||
```promql
|
||||
wakey_host_load
|
||||
```
|
||||
|
||||
Use `{{job}} / {{instance}} / {{period}}` as the legend. Fixed periods are `1m`,
|
||||
`5m`, and `15m`. Read directly as a load count, not a percentage or a rate:
|
||||
Linux includes runnable and uninterruptible tasks; a high load can include I/O
|
||||
waiting rather than CPU work. These values are not normalized by CPU count.
|
||||
|
||||
## Operations
|
||||
|
||||
Completed attempts per second, including errors and timeouts:
|
||||
|
||||
+17
-1
@@ -159,7 +159,12 @@ queries, rates, averages, percentiles, and Grafana units.
|
||||
| `wakey.fleet.devices` | CC gauge of unfiltered grouped identities, labeled by `identity` (`saved` or `unsaved`) |
|
||||
| `wakey.process.uptime` | Linux gauge: seconds since this process's kernel-reported start, including suspend |
|
||||
| `wakey.process.memory.resident` | Linux gauge: approximate resident memory in bytes, from VmRSS |
|
||||
| `wakey.process.cpu.time` | Linux counter: cumulative user + system CPU seconds for all process threads, excluding children |
|
||||
| `wakey.process.open_fds` | Linux gauge: current count of open process file descriptors |
|
||||
| `wakey.host.uptime` | Linux gauge: uptime in seconds reported by the process's procfs view, including suspend |
|
||||
| `wakey.host.memory.total` | Linux gauge: total usable RAM in bytes from the visible meminfo |
|
||||
| `wakey.host.memory.available` | Linux gauge: estimated available RAM in bytes; omitted if MemAvailable is unavailable |
|
||||
| `wakey.host.load` | Linux gauge: load average, with fixed `period` labels `1m`, `5m`, `15m` |
|
||||
| `wakey.terminal.active` | Agent PTY workers currently alive |
|
||||
| `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
|
||||
| `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
|
||||
@@ -167,7 +172,7 @@ queries, rates, averages, percentiles, and Grafana units.
|
||||
|
||||
Metric labels do not include request/session IDs, MACs, or IPs. New spans use
|
||||
`skip_all` and explicitly select safe fields; terminal bytes and credentials are
|
||||
not attached. Profiles, host CPU/total-memory metrics, connected-agent gauges, and export-loss
|
||||
not attached. Profiles, host CPU-utilization metrics, connected-agent gauges, and export-loss
|
||||
dashboards are not part of this initial slice.
|
||||
|
||||
Process and host gauges are registered automatically for Linux agent/CC daemons
|
||||
@@ -182,6 +187,17 @@ omits the affected observation instead of reporting zero or failing the daemon.
|
||||
No hostname, PID, command line or environment values are exported by this slice.
|
||||
Service/instance resource attributes identify each daemon; two daemons on the
|
||||
same host can report the same host uptime, so do not sum it across processes.
|
||||
CPU time is not elapsed time or CPU percentage: use its rate for consumption.
|
||||
Open-FD sampling can temporarily open descriptors counted by the measurement.
|
||||
Process metrics describe the daemon itself, not its child shells/PTY programs
|
||||
or the entire host. CPU/FD collection is also supplied by `wakey-linux`;
|
||||
unsupported platform readers may omit these optional measurements.
|
||||
Host memory/load come from the same optional Linux reader. `MemAvailable`
|
||||
estimates memory reusable without swapping and is not `MemFree`; no fallback
|
||||
is invented on older kernels. Load averages include runnable and uninterruptible
|
||||
tasks, so they are not CPU percentages. These are the visible kernel/procfs
|
||||
figures, not cgroup memory limits or container-specific load measurements;
|
||||
multiple daemons may export duplicate views of the same host.
|
||||
|
||||
RSS includes anonymous, file-backed and shared resident pages, not just heap,
|
||||
and is an approximate kernel measurement. Process age uses procfs start ticks
|
||||
|
||||
Reference in new issue
Block a user