wraps

CHECKPOINT_2026-04-12.md

@@ -0,0 +1,82 @@
+# Wakey Checkpoint (2026-04-12)
+
+## Snapshot
+
+This checkpoint captures progress after adding audit persistence, alert evaluation and transitions, control-plane alert APIs, and websocket timing diagnostics across agent and control-plane.
+
+## Major Changes Landed
+
+- Control-plane audit system implemented with persistent sled-backed events.
+- Audit emission wired into:
+  - enroll accept/reject
+  - token issue/list/revoke
+  - command dispatch/result/error/timeout
+  - websocket auth accept/reject and disconnect
+- Audit query API added:
+  - `GET /api/v1/control/audit/events`
+- Active alert engine added with deterministic rules over audit + live session state.
+- Alert APIs added:
+  - `GET /api/v1/control/alerts`
+  - `GET /api/v1/control/alerts/history`
+  - `GET /api/v1/control/alerts/ws`
+- Alert transition persistence added (open/resolve transitions tracked across evaluations).
+- Route classes split explicitly in runtime:
+  - public routes (enroll/ws/health)
+  - control routes (`/api/v1/control/*`)
+- Caddy template added for edge policy and Cloudflare Access boundary:
+  - `deploy/Caddyfile.control-plane.example`
+
+## Reliability and Diagnostics Improvements
+
+- Agent websocket connect diagnostics now include:
+  - DNS resolution timing (`dns_resolve_ms`)
+  - websocket connect timing (`ws_connect_ms`)
+- Control-plane websocket lifecycle logs include:
+  - connect-to-hello timing
+  - connect-to-auth timing
+  - hello-to-auth timing
+- Slow connect warnings are now emitted when timing thresholds are exceeded.
+
+## Root-Cause Findings Captured
+
+- Long agent websocket connect delays were reproduced and traced to hostname resolution path.
+- Switching agent `server_url` hostname to direct IP made connect immediate.
+- This confirms app-level relay logic was not the source of the startup delay.
+
+## Verification Status
+
+- `cargo check --workspace` passing after all changes.
+- Added and passing tests include:
+  - audit event append/filter in state store
+  - alert transition open/resolve persistence
+  - alert evaluator rule checks (offline + timeout, auth/enroll rejection spikes)
+
+## Current API Surface for UI Start
+
+- Agents and command execution:
+  - `GET /api/v1/control/agents`
+  - `POST /api/v1/control/agents/{agent_id}/command`
+- Audits:
+  - `GET /api/v1/control/audit/events`
+- Alerts:
+  - `GET /api/v1/control/alerts`
+  - `GET /api/v1/control/alerts/history`
+  - websocket subscribe: `GET /api/v1/control/alerts/ws`
+
+## Remaining Plan Items (Most Significant)
+
+1. UI implementation (`/ui` app shell and pages) is still open.
+2. Alert dedupe/cooldown persistence and tuning are still basic and need hardening.
+3. Audit retention pruning policy and long-run storage controls are not finalized.
+4. Edge auth enforcement tests and deployment rehearsals remain to be added.
+5. Multi-day soak drills and failure-injection validation remain open.
+
+## Suggested Next Actions
+
+1. Build minimal UI shell with three views:
+   - agents/commands
+   - audit timeline
+   - alerts panel (active + history + websocket stream)
+2. Add periodic retention task for audit and alert transition trees.
+3. Add proxy-level integration tests that assert private endpoints are blocked without Access headers.
+4. Run a 48-72h soak with hostname vs IP connect-path metrics collected.

README.md

@@ -172,7 +172,7 @@ During registration/enroll:
 During live connectivity:
 
 - control-plane: `agent websocket upgraded`, `agent authenticated`, `agent disconnected`
-- agent: `connecting agent websocket`, `agent websocket session authenticated`, `heartbeat sent` (debug)
+- agent: `connecting agent websocket`, `agent websocket dns resolved`, `agent websocket connected`, `agent websocket session authenticated`, `heartbeat sent` (debug)
 
 During command relay:
 
@@ -198,6 +198,10 @@ Control-plane admin API includes token management endpoints:
 If commands still appear silent, verify both processes are running with `-v`
 and that `RUST_LOG` is not overriding to a stricter level.
 
+If websocket connect feels delayed, compare `dns_resolve_ms` and `ws_connect_ms`
+from agent logs. Slow DNS is a common source of multi-second connection stalls
+when using hostnames; using a stable IP or local host mapping can avoid this.
+
 ## Edge Exposure (Caddy + Cloudflare Access)
 
 Control-plane is intended to run behind a reverse proxy with TLS termination.

wakey-agent/src/session.rs

@@ -1,5 +1,6 @@
 use anyhow::{Context, Result};
 use futures_util::{SinkExt, StreamExt};
+use std::net::IpAddr;
 use std::time::Instant;
 use tokio::time::{Duration, MissedTickBehavior, interval, sleep};
 use tokio_tungstenite::{connect_async, tungstenite::Message};
@@ -39,6 +40,22 @@ async fn run_once(config: &AgentConfig) -> Result<()> {
 
     let ws_url = websocket_url(&config.server_url)?;
     info!(%ws_url, agent_id = %config.agent_id, "connecting agent websocket");
+    if let Some((dns_resolve_ms, resolved_addrs)) = dns_resolution_diagnostics(&ws_url).await {
+        info!(
+            host = %ws_url.host_str().unwrap_or(""),
+            dns_resolve_ms,
+            resolved_addrs,
+            "agent websocket dns resolved"
+        );
+        if dns_resolve_ms > 5_000 {
+            warn!(
+                host = %ws_url.host_str().unwrap_or(""),
+                dns_resolve_ms,
+                resolved_addrs,
+                "agent websocket dns resolution was slow"
+            );
+        }
+    }
     let connect_started = Instant::now();
     let (stream, _) = connect_async(ws_url.as_str())
         .await
@@ -253,6 +270,23 @@ pub fn websocket_url(server_url: &str) -> Result<url::Url> {
     Ok(url)
 }
 
+async fn dns_resolution_diagnostics(ws_url: &url::Url) -> Option<(u64, usize)> {
+    let host = ws_url.host_str()?;
+    if host.parse::<IpAddr>().is_ok() {
+        return None;
+    }
+    let port = ws_url.port_or_known_default()?;
+
+    let started = Instant::now();
+    match tokio::net::lookup_host((host, port)).await {
+        Ok(addrs) => Some((started.elapsed().as_millis() as u64, addrs.count())),
+        Err(err) => {
+            warn!(host, port, error = %err, "agent websocket dns resolution failed");
+            None
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use std::collections::VecDeque;