Developer preview · Linux (X11)

The in&space driver contract

Version 0.8.0 (2026-10-26). Normative. nish-driver manifest and the daemon's metadata report it as contract_version; nd_core::CONTRACT_VERSION is the source. The words MUST, MUST NOT, SHOULD and MAY are used as in RFC 2119.

This document is the contract between the in&space driver and its host apps. It covers the model-facing MCP tools, the host-facing doctor report, and the platform plans that shape them. Names such as nish-driver below are the driver's current working names in its own repository.

1. Versioning and compatibility

• MAJOR.MINOR.PATCH. A MINOR bump adds tools, arguments, result fields or error codes; it never changes the meaning of a field a host already reads. A MAJOR bump may. 0.7.0 is additive over 0.6.0 except for one deliberate safety default (§4): synthetic input now needs a takeover session. 0.8.0 is additive over 0.7.0: the takeover hand-back (§4.3), with one new error code (user_took_over), one new host-control verb (takeover.resume), two host events, the takeover.paused field, move_cursor {duration_ms}, the xinput2 registry entry, the takeover_handback capability and the xinput2_missing reason.
• Hosts MUST ignore result fields they do not know. Drivers ignore arguments they do not know. Hosts MUST send listed arguments with the types tools/list declares: 0.7 refuses a wrong input with invalid_arguments, but most other arguments of a wrong type are treated as absent (for example skeleton: "true" is a normal read), not refused.
• The doctor report has its own schema_version (2, §6.4). tools/list keeps tools_list_schema_version 1.
• Unchanged from 0.6.0: tool names; element tokens <snapshot_id>:<N> and their rules (design §3.1); effect, verified, path, took_cursor, escalation, diff, snapshot_id, settle on act results; error.code and error.message; the text of a failed call still starts with <code>: <message>.

2. The error envelope

2.1 Shape

Every failed tool call is an MCP result with isError: true whose structuredContent is:

{"ok": false,
 "error": {"code": "ambiguous_target",
           "message": "2 approved windows match app",
           "disposition": {"retry": "never"},
           "suggestion": "Pick one of the candidates and call again with its pid and window_id.",
           "candidates": [{"window_id": 41943047, "pid": 5120, "app": "Slack"},
                          {"window_id": 41943112, "pid": 5121, "app": "Slack"}]}}

• code: one of the table in §2.2. A driver MUST NOT emit a code missing from it.
• message: one line for people and logs. It MAY name bounded identifiers (pid, window id, token, a count). It SHOULD NOT quote application text.
• disposition.retry:
  • safe: nothing happened; the same call may be sent again (after a short wait).
  • unsafe: something may have happened (keys partly typed, an app that hung mid-action); observe the state before acting again.
  • after_refresh: read again first (get_window_state, list_windows or start_session), then retry with fresh tokens or targets.
  • never: the same call will fail again; change the arguments or ask the person.
• suggestion: exactly one actionable sentence for the model, fixed text per code (the table below). It is never built from application content, so an app cannot steer the model through an error.
• candidates (only with ambiguous_target and ambiguous_window_target): at most 8 {window_id, pid, app} of approved apps only; app is the app's own name, escaped and cut to 64 characters, and is untrusted app text. candidates_truncated: true says there were more.
• Other fields MAY ride along: escalation ({recommend, reason}), effect: "refused", typed_characters.

The MCP text content of a failed call is <code>: <message> [retry: <disposition>] <suggestion>. The daemon socket's own errors (unknown_method) carry the same error object.

2.2 Dispositions

Every code the core and the Linux backend can produce. The conformance test every_error_code_has_a_disposition scans the sources for codes and checks this table against nd_core::errors::ERRORS.

New in 0.7.0: ambiguous_target, capability_not_consented, requires_takeover, unsupported. New in 0.8.0: user_took_over (§4.3). unknown_method is the daemon socket's, not a tool's.

2.3 Ambiguous targets

get_window_state and every tool that takes a window target accept app (a name or title fragment) instead of pid/window_id. Only approved apps' visible windows are considered. One match is used; several give ambiguous_target with candidates; none gives window_target_not_found. A call with no target at all keeps the 0.6 code ambiguous_window_target and now also carries candidates. With pid alone the topmost window of that pid is used, as in 0.6.

3. Skeleton snapshots and drill-down

For dense apps (Slack, Teams, WhatsApp Desktop, Electron generally) a full tree spends the model's context on rows it does not need.

• get_window_state {skeleton: true} returns only containers and landmarks: the window; landmark roles (tool_bar, menu_bar, status_bar, tablist, list, list_box, tree, table, tree_table, document, split_pane, scroll_pane, landmark, navigation, form, article, header, footer, info_bar, notification, dialog) that have children; and generic containers (panel, filler, section, group(ing), layered_pane, internal_frame, viewport, tab) that have children and a name. Secure fields are never listed.
• Each skeleton row: element_index, element_token, role, label (≤ 80 characters), parent_index (the nearest skeleton ancestor), depth (skeleton depth), child_count, descendant_count, frame. Text lines read [N] role "label" children=K descendants=D.
• Bounded: max_elements (default 60, at most 200) rows and max_depth (default
  1. skeleton levels; truncated: true when cut. No screenshot unless include_screenshot: true (then the whole window is captured with every secure field blacked out and no marks).
• get_window_state {root: <token>} reads one region: the element and its descendants, with the normal limits counted from the root (max_elements default 250, max_depth default 25 levels below it), and root echoed.
• Tokens follow design §3.1 unchanged: skeleton, drill-down and full reads of the same window share stable row numbers; a token from any of them acts while the window's structure is unchanged; a structural change makes older tokens stale_element_token (after_refresh).
• Skeleton and drill-down reads read the whole window internally (5,000 elements, depth 200, the zoom budget) so their tokens share one structure. The savings are in the model's context, not in accessibility round trips.
• Measured: see CODEX-REPORT.md ("Skeleton token savings").

4. Input modes: headless and takeover

A session declares its input mode in start_session {input}.

4.1 headless (the default)

• Accessibility actions only (AT-SPI actions and EditableText on Linux; UIA patterns on Windows; AX actions on macOS).
• It never moves the pointer, never sends synthetic keys or buttons and never takes focus or raises a window.
• These are refused with requires_takeover before anything is sent: click/double_click/right_click/type_text with delivery_mode: "foreground"; press_key; hotkey; scroll; drag; bring_to_front; move_cursor with scope: "desktop" and foreground. The engine checks again right before every synthetic call.
• NishBot's background work uses this mode.

4.2 takeover

• Synthetic pointer and keyboard are allowed (XTEST on X11), with every M0 check still in force (scope, secure fields, focus re-checks, occlusion, no drag, no middle-click paste).
• It needs its own line in the native consent dialog: "ALSO allow VISIBLE TAKEOVER", unticked by default, ticked only by a mouse click. A session that asks for takeover and is not granted it stays headless (start_session returns input: "headless", takeover: {on: false, reason}).
• While on, a visible on-screen indicator is shown for the whole takeover: a red frame round the screen and a label "nish-driver: AI takeover active". It is click-through (empty input shape; the driver's own hit-testing skips it), it is left out of the driver's own captures (captures read a window's own contents), and it is removed on end_session, on the kill switch or revocation, on lease expiry, when the connection closes, and when start_session {input: "headless"} is called — within about 100 ms. If the indicator cannot be shown, takeover is not granted. If it goes away on its own (another X client unmaps or destroys it, or its display connection fails), the takeover ends at once and takeover.reason says so.
• A takeover is granted only while it is on: after a switch to headless or a lost indicator, start_session {input: "takeover"} shows the dialog (and a new indicator) again.
• start_session and get_session_state report input and takeover: {on, indicator, paused, paused_reason?, reason?} (paused is new in 0.8, §4.3).
• Vixy's visible work uses this mode.

4.3 Hand-back: the person can always take control back

New in 0.8.0. During a takeover the person's own pointer or key input stops the driver and hands the computer back to them. Work resumes only after the person clicks Continue; never by voice, text or the model.

Detection. The person's input is:

• the pointer more than 8 px [E] (either axis) from where the driver last put it (or, before the driver moved it, from where it was when the takeover began or resumed), read at each check; or
• a key or button press the driver did not send itself.

The driver logs every pointer target and every key and button press before it sends it; input that is not in the log is the person's. Checks happen before every synthetic call (between actions), before every step of a pointer move (at least every 20 ms; move_cursor {duration_ms} glides in 10 ms steps), and between typed characters. An input watcher also reports the person's input as it happens, so a backend call already in progress stops at its own next step.

Stopping. On detection the batch stops at the next check: nothing more is sent; every key and button the driver holds is released (the release path of the kill switch: releases are sent even after cancellation, with bounded I/O); and the call returns user_took_over (retry never, suggestion "The person took control. Wait for them to click Continue, then start again."), with effect: "stopped", paused_reason and, for typing, typed_characters. An act whose synthetic part already finished before the person's input returns its normal result. While paused, every act of the session (every tool that is not a read or a session tool) is refused before anything is sent with the same code and effect: "refused"; reads still work. The audit log records takeover.paused (session, paused_reason, time) and takeover.resumed (session, by, time); never key names, characters or positions.

paused_reason is one of pointer_moved, key_pressed, button_pressed.

The paused indicator. The label changes to "You took over · Continue when ready", the frame dims, and a small Continue button appears under the label. The button is the only part of the indicator that takes input, and it accepts only a real mouse click, by the same rule as the consent dialog: a left-button press and release on it, not a SendEvent, and no driver button press in the last 1.5 s; no keyboard. Hit-testing skips it (the driver never targets it) and captures never contain it. If another X client unmaps or destroys it, the takeover ends, as for the rest of the indicator.

Resuming, only by a person's click. A paused takeover resumes when:

• the person clicks the indicator's Continue; or
• the host sends takeover.resume on the inherited control pipe, after a click in its own UI (NishBot's run view, a Vixy character line): {"method": "takeover.resume", "params": {"session": "<id>"}} (without session, every paused session of this driver). The ack lists the sessions resumed: {"event": "ack", "method": "takeover.resume", "resumed": ["<id>"]}.

Nothing else resumes: no MCP tool, argument or text (takeover.resume is not a tool, and the daemon socket answers it with unknown_method), not voice or chat (they reach the driver only as MCP), not start_session, end_session or a new consent. Resuming does not extend the lease or widen the app scope. It makes every token issued before it stale (stale_element_token, after_refresh), so the next action reads the window again; row numbers of unchanged elements stay the same. If the lease ended during the pause the session needs a new consent (consent_expired), and it stays paused until the person's click.

With the other controls. The kill switch and session.revoke still stop everything, paused or not; a resume never undoes them. start_session {input: "headless"} while paused ends the takeover and removes the indicator at once; the session stays paused (its acts refused) until the host's takeover.resume, or until a new takeover's indicator, which comes up paused with its Continue button, is clicked.

State and events. start_session and get_session_state report takeover.paused (true|false) and, while paused, takeover.paused_reason. Hosts get events on the control output pipe, so they can show their own "You took over · Resume" without polling:

{"event": "takeover.paused", "session": "mcp:host:4242", "reason": "pointer_moved", "at_ms": 18234}
{"event": "takeover.resumed", "session": "mcp:host:4242", "by": "indicator", "at_ms": 21007}

by is indicator or host. at_ms is the driver's monotonic clock (as in the session.revoke ack).

5. Notifications and clipboard

Each command maps to a capability the backend reports (§6.2). Where the OS gives no API the command returns unsupported and the capability says unsupported; a driver MUST NOT fake one (freedesktop, for example, gives no list of other apps' notifications).

5.1 Notifications

Linux/X11: all three are unsupported (reason no_platform_api). Windows (M2) and macOS plan to use the OS notification APIs where they exist.

5.2 Clipboard

Linux/X11 implements both with the CLIPBOARD selection (ICCCM): read converts to UTF8_STRING, then STRING, and refuses incremental (INCR) transfers; write owns the selection from a thread that stops with the session. The audit log records only lengths, never clipboard text.

5.3 Credential-looking text

The secure-field rules applied to text without a field: private key blocks; password=, secret:, api_key=, token: and similar pairs; known token prefixes (GitHub, GitLab, Slack, OpenAI-style sk-, Stripe, Google AIza, AWS access key ids); JSON web tokens; and a clipboard holding a single 16–512 character word that mixes at least three character classes. The heuristic errs towards refusing.

6. Backend registry and the doctor report

6.1 Registry

A backend per OS part, each available, unavailable or planned, with a reason_code and an action sentence a person can act on. required parts must all be available for ready.

health_report (MCP) carries backends and capabilities; the doctor (§6.4) is the full report.

6.2 Capabilities

read_tree, act_semantic, capture_window, input_takeover, takeover_indicator, takeover_handback (new in 0.8: available on X11 with XInput 2.1, else unavailable with xinput2_missing; planned on macOS and Windows; input_takeover is available only with it), list_notifications, dismiss_notification, invoke_notification_action, clipboard_read, clipboard_write. Each has status (available, unavailable, unsupported, planned), reason_code, action, and consent: session (the app consent), takeover (its own line), separate (its own line plus host approval per call) or host_approval.

6.3 Reason codes

display_unreachable, display_not_local, x11_extension_missing, atspi_bus_unreachable, atspi_bus_not_advertised, xres_missing, composite_missing, xtest_missing, xinput2_missing (new in 0.8), no_platform_api, not_built, wayland_session_unsupported. Reserved for the Wayland task (§10): wayland_portal_not_built (used today by the planned entry), portal_unavailable, portal_screencast_denied, portal_remote_desktop_denied, portal_global_shortcuts_denied, compositor_unsupported.

6.4 nish-driver doctor --json

Versioned (schema: "nish-driver/doctor", schema_version: 2), network-free, and readable by a first-run wizard. Exit status 0 when ready, else 1. The JSON schema is nd_core::registry::doctor_schema(); the conformance tests validate the report against it, with and without a display.

{"schema": "nish-driver/doctor", "schema_version": 2,
 "driver_version": "0.1.0", "contract_version": "0.8.0",
 "platform": "linux", "session_type": "x11",
 "ready": false,
 "backends": [{"id": "x11", "status": "available", "required": true, "reason_code": null, "action": null},
              {"id": "atspi", "status": "unavailable", "required": true, "reason_code": "atspi_bus_unreachable",
               "action": "Turn on assistive technologies in your desktop's accessibility settings, then sign out and in again."},
              {"id": "wayland-portal", "status": "planned", "required": false, "reason_code": "wayland_portal_not_built",
               "action": "The Wayland portals backend is planned; use an X11 session for now."}],
 "capabilities": [{"name": "clipboard_read", "status": "available", "reason_code": null, "action": null, "consent": "separate"}],
 "permissions": [{"name": "accessibility", "status": "missing", "reason_code": "atspi_bus_unreachable", "action": "..."},
                 {"name": "screen_capture", "status": "not_applicable", "reason_code": null, "action": null}],
 "next_steps": [{"reason_code": "atspi_bus_unreachable", "backend": "atspi", "permission": "accessibility", "required": true, "action": "..."}],
 "network": {"code_paths": "none", "telemetry": false, "update_check": false},
 "ok": false,
 "probes": [{"label": "display server", "status": "ok", "message": "X11 :0"}]}

• session_type: x11, wayland, none (also mock, and aqua/desktop reserved for macOS and Windows).
• next_steps: one entry per distinct reason code, with the unavailable backend and/or missing permission it explains (a missing permission whose reason an unavailable backend already gave joins that entry); what the person must do, in plain words.
• ok and probes are the 0.6 doctor fields, kept for older readers (ok now equals ready).

7. Act-then-observe and safe readback

7.1 observe

Every act tool (click, double_click, right_click, type_text, set_value, press_key, hotkey, scroll, drag) accepts observe: {tree?: bool, screenshot?: bool} and returns today's result and diff plus observe:

• snapshot_id, truncated; with tree: elements and tree_text of the fresh read (the diff's settled read when there is one), with secure values redacted as in get_window_state, and at most what a default read shows (250 rows, depth 25, truncated: true when cut; for more, call get_window_state);
• with screenshot: image metadata and the image block, under the same rules as get_window_state (complete tree, known secure-field bounds, blacked out, capture rate limit 5/s);
• an observation that cannot be made is reported as observe.error or observe.screenshot_error (an envelope without ok) and never turns the act into an error: the act already happened.
• an act with observe is also a read: when the host approves reads one by one (approve_reads, the default), a pre-approved act that observes is sent to the host for approval, and the approval card's description carries observe: {tree, screenshot}.

7.2 set_value readback

set_value reads the value back and returns readback: match (then effect: "confirmed"), mismatch (never confirmed or verified, even when the diff shows the field changed), or unverified (the read failed, or the field is secure). Before reading, the driver re-checks the field's security live; a password or secure field is never read back. (Secure fields are refused before any write in the first place; the re-check covers a field that turns secure during the call.)

8. Tool schemas and conformance

• tools/list input schemas keep the 0.6 rules: root type: "object" with properties, no $ref, no root combinators, under 12,000 bytes each.
• Output schemas of every tool (structuredContent of a success) are nd_core::schemas::output_schema(tool); failures follow nd_core::schemas::error_schema(). They are not advertised as MCP outputSchema yet (some MCP clients reject results that do not match an advertised schema exactly); a later minor version may.
• crates/core/tests/contract.rs exercises every tool against the mock backend and validates each input and output; crates/core/tests/handback.rs covers §4.3 on the mock, so it runs on every OS; the Linux backend's X11 tests cover the deltas it implements (§4 indicator, headless pointer and hand-back, §5 clipboard).

9. Windows (M2) plan: app APIs before clicks

Design only; no Windows code exists yet.

9.1 Order of preference

1. The app's own automation API, when the app has one and the person approved that app: COM automation for Office documents (Word Documents/Range, Excel Workbooks/Range, PowerPoint Presentations/Slides/Shapes). These read and change documents exactly, in the background, without focus.
2. UI Automation patterns (Invoke, Value, Toggle, SelectionItem, ExpandCollapse, Text, Scroll) on the UIA tree, in the background.
3. Synthetic input (SendInput) only in a takeover session (§4).

The tree the model sees is the UIA tree, merged with the app's object model where one exists: a UIA document element gains app_object handles (for example an Excel range Sheet1!A1:C10, a Word paragraph index) so app actions can target what the model sees. The merge is by stable identifiers (the window, UIA RuntimeId and the object model's own ids), never by matching visible text. For custom-drawn controls with no UIA element, a later version may add visually detected regions to the tree; they are marked detected: true, carry no semantic actions (only takeover clicks), and their results are never verified without a readback.

9.2 App actions (contract shape, reserved)

{"name": "list_app_actions", "input": {"pid": 0, "window_id": 0}}
-> {"app": "excel", "actions": [
     {"action": "excel.set_range_values", "risk": "edit", "consequential": false,
      "params": {"type": "object", "required": ["range", "values"],
                 "properties": {"range": {"type": "string"}, "values": {"type": "array"}}}},
     {"action": "word.save_document", "risk": "write_file", "consequential": true, "params": {}}]}

{"name": "app_action", "input": {"pid": 0, "window_id": 0, "action": "excel.set_range_values",
                                 "params": {"range": "Sheet1!A1:B2", "values": [[1, 2], [3, 4]]}}}
-> act result with "path": "app_action", "effect", "verified", "readback", "diff"

Safety rules for app actions:

• Typed and declared. Only actions the driver declares, each with a JSON schema for its parameters; no free-form scripts, no VBA, no Run/Execute of macros, no arbitrary COM method names from the model.
• Scoped. Only on documents of apps the person approved in the consent dialog, re-checked by process identity on every call, as for UIA.
• Host-approved when consequential. Every action declares consequential; consequential ones (save, save as, send, print, delete sheet/slide, close without saving, anything that leaves the document) need the host's approval on each call and cannot be pre-approved. Edits inside an open document follow normal act approval.
• No file system or network reach. No path arguments from the model (save uses the document's own path), no external data connections, no add-in loading, no hyperlinks followed.
• Secure content stays secret. Password-protected documents and protected ranges are not unlocked; cells or fields that look like credentials are read as redacted, as in §5.3.
• Readback. Every edit reads the changed object back (readback as in §7.2) and returns a diff in object-model terms ({"range": "A1", "from": 1, "to": 2}).
• Errors use the §2 envelope; a missing or unregistered COM server is unsupported with the registry entry com-office unavailable.

10. Wayland (separate task)

Wayland is a separate L-sized task. This contract only reserves:

• the registry entry wayland-portal (planned today, reason wayland_portal_not_built);
• the doctor reasons wayland_session_unsupported, portal_unavailable, portal_screencast_denied, portal_remote_desktop_denied, portal_global_shortcuts_denied, compositor_unsupported;
• the expectation that on Wayland, headless mode is AT-SPI only, takeover uses the RemoteDesktop portal, and capture uses the ScreenCast portal with the person's OS-level grant shown in the doctor's permissions.

11. What a host changes for 0.8

• Handle user_took_over: stop the plan, tell the person the driver stopped because they took over, and wait.
• Listen for takeover.paused / takeover.resumed on the control output pipe and show "You took over · Resume" (NishBot's run view, a Vixy character line). On the person's click on Resume, send {"method": "takeover.resume", "params": {"session": ...}}; never send it on the model's, a voice or a chat request.
• After takeover.resumed, read the window again before acting (older tokens are stale).

12. What a host changes for 0.7

• Call start_session {input: "takeover"} before any foreground or synthetic action; handle requires_takeover. Background work needs no change.
• Read error.disposition.retry and error.suggestion instead of parsing messages; handle ambiguous_target candidates.
• Offer clipboard reading only with start_session {clipboard_read: true} and approve each clipboard_read on the host's approval channel.
• Use skeleton: true then root for dense apps; use observe to save a round trip after acting; read readback after set_value.
• Read doctor --json (schema_version 2) in the first-run wizard.