feat(telemetry): report process CPU and FDs plus host memory and load

docs/roadmap.md

@@ -56,8 +56,10 @@ or current fleet size. Metric labels should remain bounded; no per-device labels
 ## Process And Host Visibility
 
 - [x] Add Linux process age/RSS and host uptime gauges for agent and CC through opt-in OTLP, using the existing collection interval. Core owns OS-independent instruments; wakey-linux supplies the optional procfs reader. Missing data omits observations; no added labels, config fields, health routes or polling tasks. Collector coverage verifies units and values; router/live dashboard validation pending.
-- [ ] Extend process measurements with CPU time and open file descriptors where supported, plus start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample.
-- [ ] Add a small host summary: hostname, OS/kernel, CPU count, host uptime, load, and available/total memory. Distinguish host uptime from process uptime so router reboots and agent restarts are visible separately. Sources must tolerate missing Linux files; not every agent host is a router.
+- [x] Add cumulative process CPU time (counter) and current open-FD count (gauge) through the same Linux reader and OS-independent instruments. Exclude child processes, omit failed samples, and document one-core CPU-rate interpretation and sampling FD overhead. Extend collector/reader coverage rather than adding timing-based tests; live validation pending.
+- [ ] Add start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample.
+- [x] Add total/available host memory and 1/5/15-minute load gauges through the optional Linux reader, with byte conversion owned by procfs and fixed load-period labels. Omit unavailable MemAvailable instead of substituting free memory; document kernel views versus container limits. Collector/reader coverage included; live validation pending.
+- [ ] Complete a small host summary with hostname, OS/kernel and CPU count, plus protected UI presentation of collected measurements. Sources must tolerate missing Linux files; not every agent host is a router.
 - [ ] Expose changing measurements through OTLP on a modest sampling interval; show selected process/host details on Fleet Health and agent details. Keep host information behind the protected operator interface, not public health endpoints. Avoid expensive reads on every request and unbounded labels.
 - [ ] Keep `/healthz` as cheap liveness; consider a separate bounded `/readyz` database-access check for CC. Collector outages and disconnected fleet agents must not make a functioning CC unready.
 - [ ] Add application resource gauges where useful: DB pool usage, connected agents, pending commands, and active terminal sessions. Check existing instruments first to avoid duplicates; distinguish pool pressure from host/process pressure.

docs/telemetry-recipes.md

@@ -61,6 +61,59 @@ allocation alone. A missing series can mean disabled telemetry, an unsupported
 platform, unavailable procfs, or export trouble. No new health endpoint or host
 inventory is provided here; do not sum host uptime from multiple daemons.
 
+CPU consumption as a percentage of **one CPU core**:
+
+```promql
+100 * rate(wakey_process_cpu_time_seconds_total[5m])
+```
+
+Set the unit to **percent (0-100)**. A multithreaded process can exceed 100%;
+this is not normalized by the host's CPU count. The counter includes daemon
+user/system CPU, not its child shells or commands. Allow at least two exports;
+`rate` handles counter resets on process restart.
+
+Current open descriptors:
+
+```promql
+wakey_process_open_fds
+```
+
+Read directly as a count, not a rate. The sampling read can contribute temporary
+descriptors. Persistent growth can warrant investigation, but sockets, terminal
+sessions and pool connections legitimately change the count too.
+
+Host usable/available RAM, in bytes:
+
+```promql
+wakey_host_memory_total_bytes
+```
+
+```promql
+wakey_host_memory_available_bytes
+```
+
+Set units to **bytes (IEC)**. Estimated percentage not available for new work:
+
+```promql
+100 * (1 - wakey_host_memory_available_bytes / wakey_host_memory_total_bytes)
+```
+
+Available includes reclaimable memory and is not merely free RAM. Missing
+`MemAvailable` leaves a gap rather than substituting `MemFree`. Do not sum host
+RAM from multiple daemon instances on the same host, or interpret these kernel
+values as a container's cgroup limit.
+
+Load averages by period:
+
+```promql
+wakey_host_load
+```
+
+Use `{{job}} / {{instance}} / {{period}}` as the legend. Fixed periods are `1m`,
+`5m`, and `15m`. Read directly as a load count, not a percentage or a rate:
+Linux includes runnable and uninterruptible tasks; a high load can include I/O
+waiting rather than CPU work. These values are not normalized by CPU count.
+
 ## Operations
 
 Completed attempts per second, including errors and timeouts:

docs/telemetry.md

@@ -159,7 +159,12 @@ queries, rates, averages, percentiles, and Grafana units.
 | `wakey.fleet.devices` | CC gauge of unfiltered grouped identities, labeled by `identity` (`saved` or `unsaved`) |
 | `wakey.process.uptime` | Linux gauge: seconds since this process's kernel-reported start, including suspend |
 | `wakey.process.memory.resident` | Linux gauge: approximate resident memory in bytes, from VmRSS |
+| `wakey.process.cpu.time` | Linux counter: cumulative user + system CPU seconds for all process threads, excluding children |
+| `wakey.process.open_fds` | Linux gauge: current count of open process file descriptors |
 | `wakey.host.uptime` | Linux gauge: uptime in seconds reported by the process's procfs view, including suspend |
+| `wakey.host.memory.total` | Linux gauge: total usable RAM in bytes from the visible meminfo |
+| `wakey.host.memory.available` | Linux gauge: estimated available RAM in bytes; omitted if MemAvailable is unavailable |
+| `wakey.host.load` | Linux gauge: load average, with fixed `period` labels `1m`, `5m`, `15m` |
 | `wakey.terminal.active` | Agent PTY workers currently alive |
 | `wakey.terminal.input.bytes` | Input bytes received by the terminal relay |
 | `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached |
@@ -167,7 +172,7 @@ queries, rates, averages, percentiles, and Grafana units.
 
 Metric labels do not include request/session IDs, MACs, or IPs. New spans use
 `skip_all` and explicitly select safe fields; terminal bytes and credentials are
-not attached. Profiles, host CPU/total-memory metrics, connected-agent gauges, and export-loss
+not attached. Profiles, host CPU-utilization metrics, connected-agent gauges, and export-loss
 dashboards are not part of this initial slice.
 
 Process and host gauges are registered automatically for Linux agent/CC daemons
@@ -182,6 +187,17 @@ omits the affected observation instead of reporting zero or failing the daemon.
 No hostname, PID, command line or environment values are exported by this slice.
 Service/instance resource attributes identify each daemon; two daemons on the
 same host can report the same host uptime, so do not sum it across processes.
+CPU time is not elapsed time or CPU percentage: use its rate for consumption.
+Open-FD sampling can temporarily open descriptors counted by the measurement.
+Process metrics describe the daemon itself, not its child shells/PTY programs
+or the entire host. CPU/FD collection is also supplied by `wakey-linux`;
+unsupported platform readers may omit these optional measurements.
+Host memory/load come from the same optional Linux reader. `MemAvailable`
+estimates memory reusable without swapping and is not `MemFree`; no fallback
+is invented on older kernels. Load averages include runnable and uninterruptible
+tasks, so they are not CPU percentages. These are the visible kernel/procfs
+figures, not cgroup memory limits or container-specific load measurements;
+multiple daemons may export duplicate views of the same host.
 
 RSS includes anonymous, file-backed and shared resident pages, not just heap,
 and is an approximate kernel measurement. Process age uses procfs start ticks

wakey-core/src/telemetry.rs

@@ -309,7 +309,7 @@ static METRICS: OnceLock<Instruments> = OnceLock::new();
 mod fleet;
 mod process;
 pub use fleet::{AgentDeviceCounts, FleetCounts, FleetMetrics};
-pub use process::ProcessMetricsSource;
+pub use process::{HostLoad, HostMemory, ProcessMetricsSource};
 struct Instruments {
     operations: opentelemetry::metrics::Counter<u64>,
     duration: opentelemetry::metrics::Histogram<f64>,

wakey-core/src/telemetry/process.rs

@@ -1,22 +1,55 @@
 //! OS-independent process/host instruments. Callers supply a platform reader.
 //! Failed reads omit a sample: zero must not masquerade as a restart or freed RAM.
 
-use opentelemetry::metrics::{Meter, ObservableGauge};
+use opentelemetry::KeyValue;
+use opentelemetry::metrics::{Meter, ObservableCounter, ObservableGauge};
 use std::sync::Arc;
 
-/// Values are seconds/bytes, or None when unsupported/unavailable. Readers run
+/// Host-visible memory in bytes; available is an estimate, not merely free RAM.
+pub struct HostMemory {
+    pub total_bytes: u64,
+    pub available_bytes: Option<u64>,
+}
+
+/// Host load averages, not CPU utilization percentages.
+pub struct HostLoad {
+    pub one_minute: f64,
+    pub five_minutes: f64,
+    pub fifteen_minutes: f64,
+}
+
+/// Values use the documented seconds/bytes/count units, or None when unavailable. Readers run
 /// on the metric collection thread: keep work small and local, never perform
 /// network requests or emit exporter-directed logs from these callbacks.
 pub trait ProcessMetricsSource: Send + Sync + 'static {
     fn process_uptime_seconds(&self) -> Option<f64>;
     fn resident_memory_bytes(&self) -> Option<u64>;
     fn host_uptime_seconds(&self) -> Option<f64>;
+    /// Cumulative user + system CPU seconds for the process, excluding children.
+    /// Optional additions default to unavailable for existing platform readers.
+    fn cpu_time_seconds(&self) -> Option<f64> {
+        None
+    }
+    fn open_file_descriptors(&self) -> Option<u64> {
+        None
+    }
+    fn host_memory(&self) -> Option<HostMemory> {
+        None
+    }
+    fn host_load(&self) -> Option<HostLoad> {
+        None
+    }
 }
 
 pub(super) struct ProcessMetrics {
     _uptime: ObservableGauge<f64>,
     _resident: ObservableGauge<u64>,
     _host_uptime: ObservableGauge<f64>,
+    _cpu_time: ObservableCounter<f64>,
+    _open_fds: ObservableGauge<u64>,
+    _host_total_memory: ObservableGauge<u64>,
+    _host_available_memory: ObservableGauge<u64>,
+    _host_load: ObservableGauge<f64>,
 }
 
 impl ProcessMetrics {
@@ -24,6 +57,11 @@ impl ProcessMetrics {
         let source = Arc::new(source);
         let uptime = source.clone();
         let resident = source.clone();
+        let host = source.clone();
+        let cpu = source.clone();
+        let fds = source.clone();
+        let total_memory = source.clone();
+        let available_memory = source.clone();
         Self {
             _uptime: meter
                 .f64_observable_gauge("wakey.process.uptime")
@@ -53,7 +91,7 @@ impl ProcessMetrics {
                 .with_unit("s")
                 .with_description("Host uptime reported by the platform reader")
                 .with_callback(move |observer| {
-                    if let Some(seconds) = source.host_uptime_seconds()
+                    if let Some(seconds) = host.host_uptime_seconds()
                         && seconds.is_finite()
                         && seconds >= 0.0
                     {
@@ -61,6 +99,69 @@ impl ProcessMetrics {
                     }
                 })
                 .build(),
+            _cpu_time: meter
+                .f64_observable_counter("wakey.process.cpu.time")
+                .with_unit("s")
+                .with_description("Cumulative user and system process CPU time, excluding children")
+                .with_callback(move |observer| {
+                    if let Some(seconds) = cpu.cpu_time_seconds()
+                        && seconds.is_finite()
+                        && seconds >= 0.0
+                    {
+                        observer.observe(seconds, &[]);
+                    }
+                })
+                .build(),
+            _open_fds: meter
+                .u64_observable_gauge("wakey.process.open_fds")
+                .with_description(
+                    "Current open file-descriptor count reported by the platform reader",
+                )
+                .with_callback(move |observer| {
+                    if let Some(count) = fds.open_file_descriptors() {
+                        observer.observe(count, &[]);
+                    }
+                })
+                .build(),
+            _host_total_memory: meter
+                .u64_observable_gauge("wakey.host.memory.total")
+                .with_unit("By")
+                .with_description("Total usable RAM reported by the platform reader")
+                .with_callback(move |observer| {
+                    if let Some(memory) = total_memory.host_memory() {
+                        observer.observe(memory.total_bytes, &[]);
+                    }
+                })
+                .build(),
+            _host_available_memory: meter
+                .u64_observable_gauge("wakey.host.memory.available")
+                .with_unit("By")
+                .with_description("Estimated RAM available for applications without swapping")
+                .with_callback(move |observer| {
+                    if let Some(memory) = available_memory.host_memory()
+                        && let Some(bytes) = memory.available_bytes
+                    {
+                        observer.observe(bytes, &[]);
+                    }
+                })
+                .build(),
+            _host_load: meter
+                .f64_observable_gauge("wakey.host.load")
+                .with_description("Host load average over the labeled period, not CPU percent")
+                .with_callback(move |observer| {
+                    if let Some(load) = source.host_load() {
+                        for (period, value) in [
+                            ("1m", load.one_minute),
+                            ("5m", load.five_minutes),
+                            ("15m", load.fifteen_minutes),
+                        ] {
+                            if value.is_finite() && value >= 0.0 {
+                                observer.observe(value, &[KeyValue::new("period", period)]);
+                            }
+                        }
+                    }
+                })
+                .build(),
         }
     }
 }

wakey-core/tests/telemetry.rs

@@ -210,6 +210,25 @@ async fn exports_all_signals_and_preserves_remote_command_parent() {
         fn host_uptime_seconds(&self) -> Option<f64> {
             Some(10_000.0)
         }
+        fn cpu_time_seconds(&self) -> Option<f64> {
+            Some(1.25)
+        }
+        fn open_file_descriptors(&self) -> Option<u64> {
+            Some(12)
+        }
+        fn host_memory(&self) -> Option<telemetry::HostMemory> {
+            Some(telemetry::HostMemory {
+                total_bytes: 2 * 1024 * 1024 * 1024,
+                available_bytes: Some(512 * 1024 * 1024),
+            })
+        }
+        fn host_load(&self) -> Option<telemetry::HostLoad> {
+            Some(telemetry::HostLoad {
+                one_minute: 0.5,
+                five_minutes: 1.0,
+                fifteen_minutes: 1.5,
+            })
+        }
     }
     let guard = telemetry::init(
         &TelemetryConfig {
@@ -376,6 +395,21 @@ async fn exports_all_signals_and_preserves_remote_command_parent() {
                 "s",
                 number_data_point::Value::AsDouble(10_000.0),
             ),
+            (
+                "wakey.process.open_fds",
+                "",
+                number_data_point::Value::AsInt(12),
+            ),
+            (
+                "wakey.host.memory.total",
+                "By",
+                number_data_point::Value::AsInt(2 * 1024 * 1024 * 1024),
+            ),
+            (
+                "wakey.host.memory.available",
+                "By",
+                number_data_point::Value::AsInt(512 * 1024 * 1024),
+            ),
         ] {
             let metric = latest
                 .resource_metrics
@@ -396,6 +430,46 @@ async fn exports_all_signals_and_preserves_remote_command_parent() {
             );
             assert_eq!(point.value, Some(expected));
         }
+        let cpu = latest
+            .resource_metrics
+            .iter()
+            .flat_map(|resource| &resource.scope_metrics)
+            .flat_map(|scope| &scope.metrics)
+            .find(|metric| metric.name == "wakey.process.cpu.time")
+            .expect("CPU time exported");
+        assert_eq!(cpu.unit, "s");
+        let Some(metric::Data::Sum(sum)) = &cpu.data else {
+            panic!("CPU time must be a counter, not a gauge");
+        };
+        assert!(sum.is_monotonic);
+        assert_eq!(sum.data_points.len(), 1);
+        assert_eq!(
+            sum.data_points[0].value,
+            Some(number_data_point::Value::AsDouble(1.25))
+        );
+        assert!(sum.data_points[0].attributes.is_empty());
+        let load = latest
+            .resource_metrics
+            .iter()
+            .flat_map(|resource| &resource.scope_metrics)
+            .flat_map(|scope| &scope.metrics)
+            .find(|metric| metric.name == "wakey.host.load")
+            .expect("host load exported");
+        let Some(metric::Data::Gauge(gauge)) = &load.data else {
+            panic!("host load must be a gauge");
+        };
+        assert_eq!(gauge.data_points.len(), 3);
+        for (period, expected) in [("1m", 0.5), ("5m", 1.0), ("15m", 1.5)] {
+            let point = gauge.data_points.iter().find(|point| point.attributes.iter().any(|attribute| {
+                attribute.key == "period" && attribute.value.as_ref().and_then(|value| value.value.as_ref())
+                    == Some(&opentelemetry_proto::tonic::common::v1::any_value::Value::StringValue(period.into()))
+            })).expect("load period exported");
+            assert_eq!(point.attributes.len(), 1, "only fixed period labels");
+            assert_eq!(
+                point.value,
+                Some(number_data_point::Value::AsDouble(expected))
+            );
+        }
     }
     for (name, label, expected) in [
         (

wakey-linux/src/telemetry.rs

@@ -1,6 +1,6 @@
 //! Linux procfs reader for core's OS-independent process/host instruments.
 use procfs::Current;
-use wakey_core::telemetry::ProcessMetricsSource;
+use wakey_core::telemetry::{HostLoad, HostMemory, ProcessMetricsSource};
 
 /// Reads only this process and the visible host clock; never scans other PIDs.
 /// Constructing the reader does no I/O, so disabled OTLP remains local-only.
@@ -16,6 +16,39 @@ impl ProcessMetricsSource for ProcfsProcessMetrics {
     fn host_uptime_seconds(&self) -> Option<f64> {
         host_uptime()
     }
+    fn cpu_time_seconds(&self) -> Option<f64> {
+        let stat = procfs::process::Process::myself().ok()?.stat().ok()?;
+        let ticks = procfs::ticks_per_second();
+        if ticks == 0 {
+            return None;
+        }
+        // utime/stime account for all process threads, not child processes.
+        Some(stat.utime.checked_add(stat.stime)? as f64 / ticks as f64)
+    }
+    fn open_file_descriptors(&self) -> Option<u64> {
+        let process = procfs::process::Process::myself().ok()?;
+        // procfs uses the kernel's count when available, otherwise enumerates
+        // directory entries without reading every FD's target. Sampling itself
+        // temporarily opens procfs descriptors, which can contribute to this count.
+        u64::try_from(process.fd_count().ok()?).ok()
+    }
+    fn host_memory(&self) -> Option<HostMemory> {
+        let memory = procfs::Meminfo::current().ok()?;
+        // Unlike Status::vmrss, Meminfo has already converted its fields to bytes.
+        // Do not substitute MemFree when an older kernel omits MemAvailable.
+        Some(HostMemory {
+            total_bytes: memory.mem_total,
+            available_bytes: memory.mem_available,
+        })
+    }
+    fn host_load(&self) -> Option<HostLoad> {
+        let load = procfs::LoadAverage::current().ok()?;
+        Some(HostLoad {
+            one_minute: f64::from(load.one),
+            five_minutes: f64::from(load.five),
+            fifteen_minutes: f64::from(load.fifteen),
+        })
+    }
 }
 
 fn host_uptime() -> Option<f64> {
@@ -56,5 +89,19 @@ mod tests {
         assert!(age.is_finite() && age >= 0.0);
         assert!(host.is_finite() && host >= age);
         assert!(source.resident_memory_bytes().expect("VmRSS available") > 0);
+        let cpu = source
+            .cpu_time_seconds()
+            .expect("process CPU ticks available");
+        assert!(cpu.is_finite() && cpu >= 0.0);
+        assert!(source.open_file_descriptors().expect("FD count available") > 0);
+        let memory = source.host_memory().expect("host memory available");
+        assert!(memory.total_bytes > 0);
+        if let Some(available) = memory.available_bytes {
+            assert!(available <= memory.total_bytes);
+        }
+        let load = source.host_load().expect("host load available");
+        for average in [load.one_minute, load.five_minutes, load.fifteen_minutes] {
+            assert!(average.is_finite() && average >= 0.0);
+        }
     }
 }