Crash Recovery

Overview

The recovery runtime handles two distinct failure scenarios: per-app crashes and full compositor crashes. Per-app monitoring tracks individual application health and automatically downgrades capabilities when an app becomes unstable. Compositor-level recovery reconstructs PCP state from the audit log after a process crash.

Both mechanisms share a common design goal: a crash must not permanently lose PCP state. The audit log on disk is the single source of truth for recovery.

Per-App Health Checks

Each Tier 1 app exposes a health check capability that returns its current state:

health.check -> {state: healthy|degraded|crashed, last_heartbeat, restart_count}

The state field reflects the app’s observed stability. last_heartbeat is the timestamp of the most recent successful health probe. restart_count tracks how many times the app has been restarted within the current observation window.

PCP queries health.check periodically for every registered Tier 1 app. The check is lightweight: a D-Bus call to the app’s systemd unit via the compositor daemon. Apps that do not respond within the timeout are marked as crashed.

Crash Detection

The daemon monitors app systemd units via D-Bus. When systemd reports a unit transition to failed or inactive, the daemon increments that app’s restart counter and notifies the recovery runtime.

The restart counter uses a sliding window. Only restarts within the last 5 minutes count toward the threshold. This prevents a single crash hours ago from triggering a downgrade.

Automatic Tier Downgrade

When an app’s restart counter hits three restarts within five minutes, the recovery runtime marks all of that app’s Tier 1 capabilities as degraded and falls back to Tier 2 (AT-SPI2).

3+ restarts in 5 minutes
    |
    v
Mark Tier 1 capabilities as CapabilityState::Degraded { reason }
    |
    v
Fall back to Tier 2 (AT-SPI2) for that app
    |
    v
After 5 minutes stable (no further crashes)
    |
    v
Upgrade back to Tier 1

The downgrade is per-app, not global. Other apps retain their Tier 1 capabilities. The downgrade reason is recorded in the audit log for debugging.

Recovery upgrades happen automatically after five minutes of stability. If the app crashes again during that window, the timer resets. There is no manual intervention required, though administrators can force a downgrade or upgrade through the admin capability recovery.admin.degrade.

Capabilities

The recovery runtime exposes three capabilities:

• recovery.health.check — query health status of a specific app or all apps
• recovery.restart.counter — read the restart counter for an app (useful for diagnostics)
• recovery.admin.degrade — manually trigger a tier downgrade or upgrade (admin-only)

Compositor Crash Recovery

When the compositor itself crashes and restarts via systemd, PCP must reconstruct its state. The audit log on disk is the authoritative source for this reconstruction.

What Survives a Crash

Data	Survives?	Storage
Audit log	Yes	Append-only file on disk (/var/log/portal/audit-*.jsonl)
Capability learning state	Yes	Written by Context Manager to persistent storage
User preferences	Yes	Written by Context Manager to persistent storage
Static detection cache (.desktop, ELF)	No	In-memory only (re-scanned on boot)
Capability registry	No	In-memory only (reconstructed from audit log)
Session context (focus, clipboard)	No	In-memory only (reconstructed from compositor state)
Event subscriptions	No	In-memory only (re-registered on boot)
Active transactions	No	In-memory only (marked as “interrupted” in audit)

Only the audit log and learning state survive. Everything else is in-memory and must be rebuilt.

Recovery Procedure

Compositor crashes (panic, OOM, segfault)
     |
     v
Compositor restarts (systemd automatic restart)
     |
     v
(1) PCP Core initializes with empty state
     |
     v
(2) AUDIT LOG REPLAY
     +- Read last N entries from /var/log/portal/audit-*.jsonl
     +- For each app that was active (last audit entry < 5 min ago):
     |   +- Mark as "needs re-validation" in registry
     +- Reconstruct approximate registry state:
     |   +- App IDs and capability sets from audit entries
     |   +- Tier classifications from audit metadata
     |   +- Adapter assignments from audit metadata
     +- Result: registry is approximately correct, not fully validated
     |
     v
(3) RE-CONNECT TO RUNNING APPS
     +- Check which Wayland clients survived the crash
     +- For each surviving app:
     |   +- Re-run AT-SPI2 probe (lightweight)
     |   +- Re-run adapter detection (if stale)
     |   +- Update registry entry status
     +- Apps that didn't survive: mark as "closed" in registry
     |
     v
(4) RE-REGISTER EVENT SUBSCRIPTIONS
     +- Re-subscribe to compositor events
     +- Re-subscribe to AT-SPI2 events
     +- Re-subscribe to system backend events
     |
     v
(5) REQUEST LEARNING STATE FROM CONTEXT MANAGER
     +- Load adapter effectiveness scores
     +- Load confirmation behavior data
     +- Load usage records
     |
     v
(6) NOTIFY USER
     +- If recovery time < 3s: no notification needed
     +- If 3-10s: brief overlay "System Intelligence reconnected"
     +- If > 10s: voice notification
     |
     v
PCP Core fully operational

Most Wayland clients survive a compositor restart because the protocol is designed for reconnection. Apps briefly lose their surface but reconnect to the new compositor instance. PCP uses this property to avoid a full system scan after a crash.

Warm Restart Path

A warm restart (post-crash recovery) is faster than a cold boot because the audit log provides approximate state:

Phase	Cold Boot	Warm Restart
System scan (.desktop files)	2s	Skipped
ELF probing	1s	Skipped
System backend probe	500ms	Skipped
AT-SPI2 connection	200ms	200ms
Audit log replay	Skipped	200ms
Re-validate running apps	Skipped	500ms
Event re-subscription	100ms	100ms
Learning state load	100ms	100ms
Total	~4s	~1s

The warm restart eliminates the slow scanning phases by trusting the audit log’s registry reconstruction. The tradeoff is that the registry is approximate rather than exact, but lightweight validation of surviving apps corrects any drift within the first few seconds of operation.

Interrupted Transactions

Transactions in progress when the crash occurred are marked as “interrupted” in the audit log:

{
  "timestamp": "2026-05-10T12:00:00.123Z",
  "transaction_id": "txn:email-move-042",
  "transaction_state": "interrupted",
  "completed_steps": 3,
  "total_steps": 5,
  "note": "Compositor crash during execution. Steps 1-3 may or may not have completed."
}

Interrupted transactions are never retried automatically. Duplicate actions (sending an email twice, moving a file that has already moved) are worse than leaving a transaction unfinished. On recovery, the system logs the interruption and notifies the user so they can decide whether to retry manually.