refactor(state): trace snapshot write phases and strengthen refresh coverage

docs/roadmap.md

@@ -1,6 +1,6 @@
 # Wakey Roadmap
 
-Last updated: 2026-10-10. This records agreed work and follow-ups, not a release
+Last updated: 2026-10-12. This records agreed work and follow-ups, not a release
 promise. Checked items are implemented; live validation is called out separately.
 
 ## Completed Observability
@@ -44,16 +44,50 @@ of remembered identifiers based on observation expiry.
 - [x] Upgrade the matched OpenTelemetry stack to 0.33 with exporter-managed backoff/jitter retries. Controlled collector tests cover all-signal recovery, bounded retries, and no retry on authentication rejection; live outage validation remains pending.
 - [ ] Record hostname-resolution success/failure without exporting hostnames or resolved addresses. Current spans show timing but do not explicitly classify the result.
 - [ ] Investigate the slow resolver with a reproducible device-side check. Timing isolated the phase, not the cause; do not treat instrumentation as a fix.
-- [x] Propagate optional W3C trace context for periodic/requested WS snapshots so agent collection and CC persistence can be inspected together. Wire compatibility and context linkage tested; live validation pending.
+- [x] Propagate optional W3C trace context for periodic/requested WS snapshots so agent collection and CC persistence can be inspected together. Wire compatibility and context linkage tested; live Grafana inspection confirmed CC storage is a child of agent collection in the same trace. Storage can start after collection ends: this is an asynchronous handoff, not a persistence acknowledgement.
 - [x] Add current-state gauges: retained device/endpoint rows per agent and unfiltered grouped fleet identities split saved/unsaved. Sample independently every 30 seconds with bounded revision checks; replace labels and suppress failed samples. Collector and state lifecycle tests included; dashboard recipes ready.
 - [ ] Decide whether snapshots need persistence acknowledgements. The existing sent-device counter does not prove CC storage; any confirmed-storage metric needs an authoritative measurement point.
+- [x] Break down snapshot persistence timing into pool acquisition, transaction acquisition, row replacement, and commit child spans. Shared write acquisition retains SQLx ownership and `BEGIN IMMEDIATE`; snapshot row work stays in the caller's transaction. Existing rollback and cache-refresh tests cover behavior. Live trace screenshot confirmed all phases: about 83 microseconds acquiring a connection, 154 microseconds beginning the transaction, 12.8 ms replacing rows, and 18.94 ms committing. This sample does not justify changing SQLite durability or prove a persistent bottleneck. See [telemetry span semantics](telemetry.md#signals).
 
 The existing snapshot counter is cumulative collection volume, not unique devices
 or current fleet size. Metric labels should remain bounded; no per-device labels.
 
+## Process And Host Visibility
+
+- [ ] Add shared process measurements for agent and CC: start time/uptime, resident memory, CPU time, and open file descriptors where supported. Keep version as resource metadata rather than repeating it on every sample.
+- [ ] Add a small host summary: hostname, OS/kernel, CPU count, host uptime, load, and available/total memory. Distinguish host uptime from process uptime so router reboots and agent restarts are visible separately. Sources must tolerate missing Linux files; not every agent host is a router.
+- [ ] Expose changing measurements through OTLP on a modest sampling interval; show selected process/host details on Fleet Health and agent details. Keep host information behind the protected operator interface, not public health endpoints. Avoid expensive reads on every request and unbounded labels.
+- [ ] Keep `/healthz` as cheap liveness; consider a separate bounded `/readyz` database-access check for CC. Collector outages and disconnected fleet agents must not make a functioning CC unready.
+- [ ] Add application resource gauges where useful: DB pool usage, connected agents, pending commands, and active terminal sessions. Check existing instruments first to avoid duplicates; distinguish pool pressure from host/process pressure.
+
+## Device Diagnostics
+
+- [ ] Add agent-side ICMP ping through the existing command relay, with an explicit IP and selected agent/network route. Bound probe count, total duration, output and concurrent work; no shell interpolation. Account for missing tools and ICMP permissions, and support IPv4/IPv6 where available.
+- [ ] Return probe replies, loss and round-trip times; offer Ping alongside Wake in device details. MAC-only devices cannot be pinged without an IP. A reply means that IP responded from that agent's network, not proof of saved identity; no reply must not automatically mark a device offline or remove an identifier.
+
+## Optional Agent Sources And Capabilities
+
+These are exploration items, not a commitment to a router-only model or a new
+message bus. Sparse WSL inventory is legitimate; missing evidence is not failure.
+
+- [ ] Discover and advertise usable observation sources separately from supported commands and configured permissions. Extend the existing hello/capability mechanism where appropriate; CC/UI must tolerate missing sources and older agents. Do not infer router status from Linux alone.
+- [ ] Investigate read-only Wi-Fi station data on the actual router: signal in **dBm**, association/interface, rates and traffic counters where available. Evaluate driver-supported `iw`/netlink or local `ubus`/hostapd interfaces before choosing a dependency. Keep measurements scoped to agent/interface/station with source and capture time; they are not permanent identity or globally valid presence. UI may show a signal badge only when supported and sufficiently fresh.
+- [ ] Investigate interface lifecycle hooks for WAN/link changes, keeping scripts minimal and logic in the binary. Interface up/down does not prove Internet reachability; any active connectivity probe needs an explicit target and bounded policy. Existing DHCP/neighbor hooks remain independent evidence, not a way to enrich or overwrite other sources.
+- [ ] Keep source support, operator authorization and local agent policy distinct. Capability advertisement is not permission: CC checks authorization and the agent enforces configured limits. New diagnostic APIs should be typed, protected operations, not arbitrary shell, UCI or bus calls exposed to the browser.
+
+OpenWrt [hotplug](https://openwrt.org/docs/guide-user/base-system/hotplug) provides
+event hooks; [ubus](https://openwrt.org/docs/techref/ubus) is local IPC/RPC, and
+[UCI](https://openwrt.org/docs/guide-developer/ubus/uci) manages configuration.
+None is a Kafka-style retained observation history. Verify what the vendor
+firmware actually exposes before making any of them an agent requirement.
+
 ## Validation And Reliability
 
 - [x] Bound fleet-cache rebuilds to three attempts under sustained mutations; return retryable HTTP 503 `fleet_busy` rather than silently serving stale wake routes. Deterministic limit/recovery tests included.
+- [x] Separate fleet wake selection from dispatch; wake lookup uses the complete cached view rather than the presentation list's 1,000-row cap. Automated coverage checks explicit-route rejection and lookup beyond the cap; live checks confirm missing-device and invalid-route errors without dispatching wakes.
+- [x] Share device read assembly within a single SQLite read transaction. Query-plan inspection on the test database with Python SQLite confirmed all six per-agent reads use primary-key searches; full-fleet reads scan intentionally, with covering indexes for MAC/IP projections and no temporary sort. No additional indexes justified by this check.
+- [x] Separate fleet refresh selection, relay, decoding/storage, and aggregation. Preserve concurrency of eight and per-agent failures; tests cover disconnected agents, sorted partial results, invalid payload preservation, and valid empty snapshot replacement.
+- [x] Add a refresh integration regression: a simulated agent answers through the production command relay; its inventory reaches SQLite, invalidates a previously populated fleet cache, and appears in the next fleet read. A disconnected agent does not cancel success; duplicate selection sends only one command. This covers the CC application path, not a real agent process or WebSocket transport.
 - [ ] Reproduce/check Ctrl-C cleanup on the test device: process groups, PTYs, relay tasks, and telemetry shutdown. Leftovers were suspected, not confirmed.
 - [ ] Exercise collector unavailability and recovery with running agent/CC instances. Verify normal operations continue and telemetry queues remain bounded; outage losses are expected under the current best-effort policy.
 - [ ] Validate terminal recovery instrumentation live: generation timing/size, transport attach/resume, and backpressure episodes ending in drain or disconnect.

docs/telemetry.md

@@ -106,6 +106,20 @@ existing debug events too; inspect them before enabling it broadly.
 - Inventory source spans separate neighbor, DHCP lease, and hook-store reads
   from merging. `db_*` spans cover snapshot replacement, fleet reads, and manual
   identity changes, including transaction waits and result assembly where applicable.
+  Snapshot replacement retains `db_replace_agent_snapshot` as its total duration,
+  with four sequential child spans:
+  - `db_pool_acquire`: obtaining a pooled connection, including any connection
+    setup/checks; it is not exclusively queue wait.
+  - `db_write_transaction_begin`: executing `BEGIN IMMEDIATE`, including SQLite
+    writer-lock wait and worker scheduling, not a direct lock-wait measurement.
+  - `db_snapshot_write_rows`: existing-key lookup, normalization, parent/child
+    writes, and deletion of devices missing from the complete snapshot.
+  - `db_snapshot_commit`: transaction commit, including its SQLite/worker work;
+    this does not isolate filesystem flush time.
+  Pool and transaction-begin spans come from the shared write helper, so they
+  also appear under other write operations. No per-row spans or SQL values are
+  recorded. A failed write rolls back through SQLx and has no commit span;
+  fleet cache invalidation still happens only after a successful commit.
 - Fleet spans cover list/load/grouping, refresh fan-out with per-agent outcomes,
   and wake dispatch. Counts distinguish grouped, filtered, and returned rows;
   wake events identify the chosen agent and whether the route was explicit,