Agent Seat Proto
Can Linux and Unix-like desktops agree on one protocol for Computer Use? Agent Seat is a working proposal for a shared, display-server-neutral boundary between AI agents and the desktops people control.
This is an invitation: the pre-RFC draft is ready for independent review. Desktop environment, compositor, window-manager, accessibility, security, and agent-tooling maintainers can help turn one implementation's lessons into a protocol the wider ecosystem can support.
A Protocol the Desktop Ecosystem Can Own Together
Computer Use should not require every agent vendor to invent a private automation stack for every desktop. A common protocol could give agents one bounded semantic model while letting each desktop environment or window manager retain its own policy, object model, security boundary, and implementation.
The draft does not standardize X11, Wayland, MCP, systemd, or a particular compositor. It separates a display-server-neutral information model from independently testable backend profiles, so implementations can share semantics without pretending their guarantees are identical.
SHARED MEANING ≠ SHARED IMPLEMENTATION
Three Layers Keep the Core Portable
A desktop can adopt the common semantics without adopting the reference wire, MCP, systemd, X11, or someone else's internal architecture.
Information model
Shared semantics for sessions, authority, grants, scope, identity, freshness, operations, and outcomes—without assuming an encoding or display server.
Transport binding
Exact revision selection, encoding, framing, peer authentication, strict grammar, and finite resource limits for a concrete wire.
Backend profile
The desktop evidence, ordering, races, hostile tests, and prohibited claims behind one narrowly stated assurance level.
The Common Ground
Different desktops do not need identical internals, but Computer Use clients need a few reliable truths at the boundary.
Authenticate before granting
Peer descriptions are untrusted metadata. The transport peer must be authenticated before the provider returns any grant.
Keep authority local
The desktop provider—not the model, harness, MCP companion, or target application—owns policy, scope, and realization.
Use opaque, session-local identity
Targets never expose raw X11, Wayland, process, or application identifiers, and every reconnect invalidates prior handles.
Revalidate before mutation
The provider checks the live grant, scope, target generation, request freshness, and backend evidence immediately before acting.
Bound everything
Frames, strings, collections, waits, retries, concurrent work, and retained state all have published finite limits.
Report only what is known
Outcomes distinguish no-send, observed, sent-but-unobserved, target-lost, interrupted, and unavailable states.
Comparable Claims Without Flattening Real Differences
Conformance is not “these tools have the same names.” It combines an exact wire revision, a versioned backend profile, advertised features, and public evidence.
TIER 0
Standalone provider
Runs beside a desktop authority. It can provide bounded, revalidated observations and qualified requests, but must disclose foreign-authority limits and races it cannot close.
TIER 1
Integrated desktop authority
A compositor or window manager may prove stronger state and ordering from the event loop or object model it owns. It still must define policy, scope, application trust, and OS/session limits.
Tier 1 does not automatically mean secure, and Tier 0 does not automatically mean conformant. Each implementation advertises only the assurance its own evidence supports.
Where Maintainer Experience Matters
Desktop Semantics
Do the target, scope, freshness, observation, and mutation models fit real compositor and window-manager internals?
Security Boundaries
Are grants, revocation, human priority, sensitive observations, and fail-closed behavior strong and implementable?
Backend Profiles
How should X11, Wayland compositors, accessibility stacks, BSD desktops, and integrated providers state comparable—but non-transferable—assurance?
Governance
Where should the protocol live, who should steward its registries, and what review and change process could maintainers trust?
The draft has no IETF, freedesktop.org, or other external standards status today. The immediate goal is serious technical review, independent implementation experience, and a governance path the community can stand behind.
Conformance That Can Be Reviewed Outside the Codebase
The draft includes a closed, machine-readable conformance report format so another maintainer can answer five practical questions without reverse-engineering an implementation:
- What exact product, wire binding, and backend profile was tested?
- In which released environment did it run?
- Which stable public fixtures passed, failed, or could not run?
- What evidence shows each process lacks authority it should not possess?
- What limitations remain, and is the result pass, fail, or incomplete?
Logs alone are not proof. Reports bind public evidence by digest, record negative-authority enforcement, preserve limitations, and remain incomplete when a required gate has not been demonstrated.
A Reference Implementation, Not the Standard
The v0.2.0 X11 implementation proves that the proposed boundaries can support useful desktop work. It is evidence for the conversation—not a requirement that other desktops copy its architecture.
Observe Before Acting
Inspect only admitted windows and workspaces through structured metadata and filtered events.
Bounded Management
Freshness-checked EWMH operations cover activation, closing, workspaces, state, and geometry.
Controlled Launching
Applications launch shell-free from a bounded XDG catalog and an explicit admission policy.
Optional Profiles
Capture, pointer input, keyboard commands, and exact text transfer require separate grants and make narrower claims.
How the Reference Is Split
agent-seat-x11The standalone X11 provider owns policy, grants, launch admission, and desktop authority.
agent-seat-mcpAn authority-free companion translates MCP tools into one authenticated provider session.
agent-seat-settingsGTK and command-line controls keep saved policy separate from the live runtime seat.
agent-seat-protoDisplay-server-neutral, bounded wire types and framing define the portable contract.
activity-brokerAn optional research component explores stronger physical-activity evidence.
Read the Design at the Right Layer
Try the Current X11 Provider
Build with Rust 1.85 or newer. The generated policy starts disabled, and runtime access must be enabled explicitly for each provider instance.
git clone https://github.com/ZaguanLabs/agent-seat-proto.git
cd agent-seat-proto
cargo build --release --workspace
agent-seat-x11 --check-config
agent-seat-x11
agent-seat-x11 seat enableHelp Move the Draft Forward
The current release is proven with Openbox on X11. It does not claim physical-user priority, accessibility trees, proof that an application accepted an action, or whole-output capture. The compatibility matrix names those boundaries directly.
Read the draft from the perspective of your desktop, challenge assumptions that do not survive contact with your architecture, propose better semantics, or build an independent profile. The protocol only becomes useful if other maintainers can shape it and choose to support it.
Challenge the model
Comment on identity, scope, freshness, outcomes, interruption, extension rules, or anything that does not fit your desktop.
Describe your backend
Define the authority, evidence, ordering, races, lifecycle behavior, tests, and prohibited claims your implementation can support.
Bring independent evidence
A second implementation or independent black-box harness is what moves an experimental profile toward provisional maturity.