diff --git a/docs/roadmap.md b/docs/roadmap.md index 8cdccb7..08a0e10 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -56,8 +56,10 @@ or current fleet size. Metric labels should remain bounded; no per-device labels ## Process And Host Visibility - [x] Add Linux process age/RSS and host uptime gauges for agent and CC through opt-in OTLP, using the existing collection interval. Core owns OS-independent instruments; wakey-linux supplies the optional procfs reader. Missing data omits observations; no added labels, config fields, health routes or polling tasks. Collector coverage verifies units and values; router/live dashboard validation pending. -- [ ] Extend process measurements with CPU time and open file descriptors where supported, plus start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample. -- [ ] Add a small host summary: hostname, OS/kernel, CPU count, host uptime, load, and available/total memory. Distinguish host uptime from process uptime so router reboots and agent restarts are visible separately. Sources must tolerate missing Linux files; not every agent host is a router. +- [x] Add cumulative process CPU time (counter) and current open-FD count (gauge) through the same Linux reader and OS-independent instruments. Exclude child processes, omit failed samples, and document one-core CPU-rate interpretation and sampling FD overhead. Extend collector/reader coverage rather than adding timing-based tests; live validation pending. +- [ ] Add start-time/version metadata if needed. Keep version as resource metadata rather than repeating it on every sample. +- [x] Add total/available host memory and 1/5/15-minute load gauges through the optional Linux reader, with byte conversion owned by procfs and fixed load-period labels. Omit unavailable MemAvailable instead of substituting free memory; document kernel views versus container limits. Collector/reader coverage included; live validation pending. +- [ ] Complete a small host summary with hostname, OS/kernel and CPU count, plus protected UI presentation of collected measurements. Sources must tolerate missing Linux files; not every agent host is a router. - [ ] Expose changing measurements through OTLP on a modest sampling interval; show selected process/host details on Fleet Health and agent details. Keep host information behind the protected operator interface, not public health endpoints. Avoid expensive reads on every request and unbounded labels. - [ ] Keep `/healthz` as cheap liveness; consider a separate bounded `/readyz` database-access check for CC. Collector outages and disconnected fleet agents must not make a functioning CC unready. - [ ] Add application resource gauges where useful: DB pool usage, connected agents, pending commands, and active terminal sessions. Check existing instruments first to avoid duplicates; distinguish pool pressure from host/process pressure. diff --git a/docs/telemetry-recipes.md b/docs/telemetry-recipes.md index b2718c7..7d3f014 100644 --- a/docs/telemetry-recipes.md +++ b/docs/telemetry-recipes.md @@ -61,6 +61,59 @@ allocation alone. A missing series can mean disabled telemetry, an unsupported platform, unavailable procfs, or export trouble. No new health endpoint or host inventory is provided here; do not sum host uptime from multiple daemons. +CPU consumption as a percentage of **one CPU core**: + +```promql +100 * rate(wakey_process_cpu_time_seconds_total[5m]) +``` + +Set the unit to **percent (0-100)**. A multithreaded process can exceed 100%; +this is not normalized by the host's CPU count. The counter includes daemon +user/system CPU, not its child shells or commands. Allow at least two exports; +`rate` handles counter resets on process restart. + +Current open descriptors: + +```promql +wakey_process_open_fds +``` + +Read directly as a count, not a rate. The sampling read can contribute temporary +descriptors. Persistent growth can warrant investigation, but sockets, terminal +sessions and pool connections legitimately change the count too. + +Host usable/available RAM, in bytes: + +```promql +wakey_host_memory_total_bytes +``` + +```promql +wakey_host_memory_available_bytes +``` + +Set units to **bytes (IEC)**. Estimated percentage not available for new work: + +```promql +100 * (1 - wakey_host_memory_available_bytes / wakey_host_memory_total_bytes) +``` + +Available includes reclaimable memory and is not merely free RAM. Missing +`MemAvailable` leaves a gap rather than substituting `MemFree`. Do not sum host +RAM from multiple daemon instances on the same host, or interpret these kernel +values as a container's cgroup limit. + +Load averages by period: + +```promql +wakey_host_load +``` + +Use `{{job}} / {{instance}} / {{period}}` as the legend. Fixed periods are `1m`, +`5m`, and `15m`. Read directly as a load count, not a percentage or a rate: +Linux includes runnable and uninterruptible tasks; a high load can include I/O +waiting rather than CPU work. These values are not normalized by CPU count. + ## Operations Completed attempts per second, including errors and timeouts: diff --git a/docs/telemetry.md b/docs/telemetry.md index 47b5742..ac3e6c7 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -159,7 +159,12 @@ queries, rates, averages, percentiles, and Grafana units. | `wakey.fleet.devices` | CC gauge of unfiltered grouped identities, labeled by `identity` (`saved` or `unsaved`) | | `wakey.process.uptime` | Linux gauge: seconds since this process's kernel-reported start, including suspend | | `wakey.process.memory.resident` | Linux gauge: approximate resident memory in bytes, from VmRSS | +| `wakey.process.cpu.time` | Linux counter: cumulative user + system CPU seconds for all process threads, excluding children | +| `wakey.process.open_fds` | Linux gauge: current count of open process file descriptors | | `wakey.host.uptime` | Linux gauge: uptime in seconds reported by the process's procfs view, including suspend | +| `wakey.host.memory.total` | Linux gauge: total usable RAM in bytes from the visible meminfo | +| `wakey.host.memory.available` | Linux gauge: estimated available RAM in bytes; omitted if MemAvailable is unavailable | +| `wakey.host.load` | Linux gauge: load average, with fixed `period` labels `1m`, `5m`, `15m` | | `wakey.terminal.active` | Agent PTY workers currently alive | | `wakey.terminal.input.bytes` | Input bytes received by the terminal relay | | `wakey.terminal.output.bytes` | Output bytes read from the PTY, including while detached | @@ -167,7 +172,7 @@ queries, rates, averages, percentiles, and Grafana units. Metric labels do not include request/session IDs, MACs, or IPs. New spans use `skip_all` and explicitly select safe fields; terminal bytes and credentials are -not attached. Profiles, host CPU/total-memory metrics, connected-agent gauges, and export-loss +not attached. Profiles, host CPU-utilization metrics, connected-agent gauges, and export-loss dashboards are not part of this initial slice. Process and host gauges are registered automatically for Linux agent/CC daemons @@ -182,6 +187,17 @@ omits the affected observation instead of reporting zero or failing the daemon. No hostname, PID, command line or environment values are exported by this slice. Service/instance resource attributes identify each daemon; two daemons on the same host can report the same host uptime, so do not sum it across processes. +CPU time is not elapsed time or CPU percentage: use its rate for consumption. +Open-FD sampling can temporarily open descriptors counted by the measurement. +Process metrics describe the daemon itself, not its child shells/PTY programs +or the entire host. CPU/FD collection is also supplied by `wakey-linux`; +unsupported platform readers may omit these optional measurements. +Host memory/load come from the same optional Linux reader. `MemAvailable` +estimates memory reusable without swapping and is not `MemFree`; no fallback +is invented on older kernels. Load averages include runnable and uninterruptible +tasks, so they are not CPU percentages. These are the visible kernel/procfs +figures, not cgroup memory limits or container-specific load measurements; +multiple daemons may export duplicate views of the same host. RSS includes anonymous, file-backed and shared resident pages, not just heap, and is an approximate kernel measurement. Process age uses procfs start ticks diff --git a/wakey-core/src/telemetry.rs b/wakey-core/src/telemetry.rs index 2546157..ef566df 100644 --- a/wakey-core/src/telemetry.rs +++ b/wakey-core/src/telemetry.rs @@ -309,7 +309,7 @@ static METRICS: OnceLock = OnceLock::new(); mod fleet; mod process; pub use fleet::{AgentDeviceCounts, FleetCounts, FleetMetrics}; -pub use process::ProcessMetricsSource; +pub use process::{HostLoad, HostMemory, ProcessMetricsSource}; struct Instruments { operations: opentelemetry::metrics::Counter, duration: opentelemetry::metrics::Histogram, diff --git a/wakey-core/src/telemetry/process.rs b/wakey-core/src/telemetry/process.rs index 8d5af68..f4f8abf 100644 --- a/wakey-core/src/telemetry/process.rs +++ b/wakey-core/src/telemetry/process.rs @@ -1,22 +1,55 @@ //! OS-independent process/host instruments. Callers supply a platform reader. //! Failed reads omit a sample: zero must not masquerade as a restart or freed RAM. -use opentelemetry::metrics::{Meter, ObservableGauge}; +use opentelemetry::KeyValue; +use opentelemetry::metrics::{Meter, ObservableCounter, ObservableGauge}; use std::sync::Arc; -/// Values are seconds/bytes, or None when unsupported/unavailable. Readers run +/// Host-visible memory in bytes; available is an estimate, not merely free RAM. +pub struct HostMemory { + pub total_bytes: u64, + pub available_bytes: Option, +} + +/// Host load averages, not CPU utilization percentages. +pub struct HostLoad { + pub one_minute: f64, + pub five_minutes: f64, + pub fifteen_minutes: f64, +} + +/// Values use the documented seconds/bytes/count units, or None when unavailable. Readers run /// on the metric collection thread: keep work small and local, never perform /// network requests or emit exporter-directed logs from these callbacks. pub trait ProcessMetricsSource: Send + Sync + 'static { fn process_uptime_seconds(&self) -> Option; fn resident_memory_bytes(&self) -> Option; fn host_uptime_seconds(&self) -> Option; + /// Cumulative user + system CPU seconds for the process, excluding children. + /// Optional additions default to unavailable for existing platform readers. + fn cpu_time_seconds(&self) -> Option { + None + } + fn open_file_descriptors(&self) -> Option { + None + } + fn host_memory(&self) -> Option { + None + } + fn host_load(&self) -> Option { + None + } } pub(super) struct ProcessMetrics { _uptime: ObservableGauge, _resident: ObservableGauge, _host_uptime: ObservableGauge, + _cpu_time: ObservableCounter, + _open_fds: ObservableGauge, + _host_total_memory: ObservableGauge, + _host_available_memory: ObservableGauge, + _host_load: ObservableGauge, } impl ProcessMetrics { @@ -24,6 +57,11 @@ impl ProcessMetrics { let source = Arc::new(source); let uptime = source.clone(); let resident = source.clone(); + let host = source.clone(); + let cpu = source.clone(); + let fds = source.clone(); + let total_memory = source.clone(); + let available_memory = source.clone(); Self { _uptime: meter .f64_observable_gauge("wakey.process.uptime") @@ -53,7 +91,7 @@ impl ProcessMetrics { .with_unit("s") .with_description("Host uptime reported by the platform reader") .with_callback(move |observer| { - if let Some(seconds) = source.host_uptime_seconds() + if let Some(seconds) = host.host_uptime_seconds() && seconds.is_finite() && seconds >= 0.0 { @@ -61,6 +99,69 @@ impl ProcessMetrics { } }) .build(), + _cpu_time: meter + .f64_observable_counter("wakey.process.cpu.time") + .with_unit("s") + .with_description("Cumulative user and system process CPU time, excluding children") + .with_callback(move |observer| { + if let Some(seconds) = cpu.cpu_time_seconds() + && seconds.is_finite() + && seconds >= 0.0 + { + observer.observe(seconds, &[]); + } + }) + .build(), + _open_fds: meter + .u64_observable_gauge("wakey.process.open_fds") + .with_description( + "Current open file-descriptor count reported by the platform reader", + ) + .with_callback(move |observer| { + if let Some(count) = fds.open_file_descriptors() { + observer.observe(count, &[]); + } + }) + .build(), + _host_total_memory: meter + .u64_observable_gauge("wakey.host.memory.total") + .with_unit("By") + .with_description("Total usable RAM reported by the platform reader") + .with_callback(move |observer| { + if let Some(memory) = total_memory.host_memory() { + observer.observe(memory.total_bytes, &[]); + } + }) + .build(), + _host_available_memory: meter + .u64_observable_gauge("wakey.host.memory.available") + .with_unit("By") + .with_description("Estimated RAM available for applications without swapping") + .with_callback(move |observer| { + if let Some(memory) = available_memory.host_memory() + && let Some(bytes) = memory.available_bytes + { + observer.observe(bytes, &[]); + } + }) + .build(), + _host_load: meter + .f64_observable_gauge("wakey.host.load") + .with_description("Host load average over the labeled period, not CPU percent") + .with_callback(move |observer| { + if let Some(load) = source.host_load() { + for (period, value) in [ + ("1m", load.one_minute), + ("5m", load.five_minutes), + ("15m", load.fifteen_minutes), + ] { + if value.is_finite() && value >= 0.0 { + observer.observe(value, &[KeyValue::new("period", period)]); + } + } + } + }) + .build(), } } } diff --git a/wakey-core/tests/telemetry.rs b/wakey-core/tests/telemetry.rs index 9bcf931..9f95112 100644 --- a/wakey-core/tests/telemetry.rs +++ b/wakey-core/tests/telemetry.rs @@ -210,6 +210,25 @@ async fn exports_all_signals_and_preserves_remote_command_parent() { fn host_uptime_seconds(&self) -> Option { Some(10_000.0) } + fn cpu_time_seconds(&self) -> Option { + Some(1.25) + } + fn open_file_descriptors(&self) -> Option { + Some(12) + } + fn host_memory(&self) -> Option { + Some(telemetry::HostMemory { + total_bytes: 2 * 1024 * 1024 * 1024, + available_bytes: Some(512 * 1024 * 1024), + }) + } + fn host_load(&self) -> Option { + Some(telemetry::HostLoad { + one_minute: 0.5, + five_minutes: 1.0, + fifteen_minutes: 1.5, + }) + } } let guard = telemetry::init( &TelemetryConfig { @@ -376,6 +395,21 @@ async fn exports_all_signals_and_preserves_remote_command_parent() { "s", number_data_point::Value::AsDouble(10_000.0), ), + ( + "wakey.process.open_fds", + "", + number_data_point::Value::AsInt(12), + ), + ( + "wakey.host.memory.total", + "By", + number_data_point::Value::AsInt(2 * 1024 * 1024 * 1024), + ), + ( + "wakey.host.memory.available", + "By", + number_data_point::Value::AsInt(512 * 1024 * 1024), + ), ] { let metric = latest .resource_metrics @@ -396,6 +430,46 @@ async fn exports_all_signals_and_preserves_remote_command_parent() { ); assert_eq!(point.value, Some(expected)); } + let cpu = latest + .resource_metrics + .iter() + .flat_map(|resource| &resource.scope_metrics) + .flat_map(|scope| &scope.metrics) + .find(|metric| metric.name == "wakey.process.cpu.time") + .expect("CPU time exported"); + assert_eq!(cpu.unit, "s"); + let Some(metric::Data::Sum(sum)) = &cpu.data else { + panic!("CPU time must be a counter, not a gauge"); + }; + assert!(sum.is_monotonic); + assert_eq!(sum.data_points.len(), 1); + assert_eq!( + sum.data_points[0].value, + Some(number_data_point::Value::AsDouble(1.25)) + ); + assert!(sum.data_points[0].attributes.is_empty()); + let load = latest + .resource_metrics + .iter() + .flat_map(|resource| &resource.scope_metrics) + .flat_map(|scope| &scope.metrics) + .find(|metric| metric.name == "wakey.host.load") + .expect("host load exported"); + let Some(metric::Data::Gauge(gauge)) = &load.data else { + panic!("host load must be a gauge"); + }; + assert_eq!(gauge.data_points.len(), 3); + for (period, expected) in [("1m", 0.5), ("5m", 1.0), ("15m", 1.5)] { + let point = gauge.data_points.iter().find(|point| point.attributes.iter().any(|attribute| { + attribute.key == "period" && attribute.value.as_ref().and_then(|value| value.value.as_ref()) + == Some(&opentelemetry_proto::tonic::common::v1::any_value::Value::StringValue(period.into())) + })).expect("load period exported"); + assert_eq!(point.attributes.len(), 1, "only fixed period labels"); + assert_eq!( + point.value, + Some(number_data_point::Value::AsDouble(expected)) + ); + } } for (name, label, expected) in [ ( diff --git a/wakey-linux/src/telemetry.rs b/wakey-linux/src/telemetry.rs index 3960d66..83f7300 100644 --- a/wakey-linux/src/telemetry.rs +++ b/wakey-linux/src/telemetry.rs @@ -1,6 +1,6 @@ //! Linux procfs reader for core's OS-independent process/host instruments. use procfs::Current; -use wakey_core::telemetry::ProcessMetricsSource; +use wakey_core::telemetry::{HostLoad, HostMemory, ProcessMetricsSource}; /// Reads only this process and the visible host clock; never scans other PIDs. /// Constructing the reader does no I/O, so disabled OTLP remains local-only. @@ -16,6 +16,39 @@ impl ProcessMetricsSource for ProcfsProcessMetrics { fn host_uptime_seconds(&self) -> Option { host_uptime() } + fn cpu_time_seconds(&self) -> Option { + let stat = procfs::process::Process::myself().ok()?.stat().ok()?; + let ticks = procfs::ticks_per_second(); + if ticks == 0 { + return None; + } + // utime/stime account for all process threads, not child processes. + Some(stat.utime.checked_add(stat.stime)? as f64 / ticks as f64) + } + fn open_file_descriptors(&self) -> Option { + let process = procfs::process::Process::myself().ok()?; + // procfs uses the kernel's count when available, otherwise enumerates + // directory entries without reading every FD's target. Sampling itself + // temporarily opens procfs descriptors, which can contribute to this count. + u64::try_from(process.fd_count().ok()?).ok() + } + fn host_memory(&self) -> Option { + let memory = procfs::Meminfo::current().ok()?; + // Unlike Status::vmrss, Meminfo has already converted its fields to bytes. + // Do not substitute MemFree when an older kernel omits MemAvailable. + Some(HostMemory { + total_bytes: memory.mem_total, + available_bytes: memory.mem_available, + }) + } + fn host_load(&self) -> Option { + let load = procfs::LoadAverage::current().ok()?; + Some(HostLoad { + one_minute: f64::from(load.one), + five_minutes: f64::from(load.five), + fifteen_minutes: f64::from(load.fifteen), + }) + } } fn host_uptime() -> Option { @@ -56,5 +89,19 @@ mod tests { assert!(age.is_finite() && age >= 0.0); assert!(host.is_finite() && host >= age); assert!(source.resident_memory_bytes().expect("VmRSS available") > 0); + let cpu = source + .cpu_time_seconds() + .expect("process CPU ticks available"); + assert!(cpu.is_finite() && cpu >= 0.0); + assert!(source.open_file_descriptors().expect("FD count available") > 0); + let memory = source.host_memory().expect("host memory available"); + assert!(memory.total_bytes > 0); + if let Some(available) = memory.available_bytes { + assert!(available <= memory.total_bytes); + } + let load = source.host_load().expect("host load available"); + for average in [load.one_minute, load.five_minutes, load.fifteen_minutes] { + assert!(average.is_finite() && average >= 0.0); + } } }