Architecture

A developer-oriented overview of how LCXL Remote Desk is put together. For a gentler introduction, see Core Concepts.

Connection & Media Path

The browser and remote device exchange SDP / ICE through the signaling service and use STUN/TURN to gather candidates. They prioritize a direct WebRTC P2P connection and fall back to TURN relay only when NAT traversal fails. Signaling and TURN are built into the server. The terminal is intentionally separate from WebRTC data channels and uses its own authenticated WebSocket.

Process Model (service-daemon)

The ServiceDaemon (SYSTEM / root) owns the WebRTC connection, signaling, and child processes; it spawns a SessionWorker per desktop session for capture, encoding, input, files, and clipboard. The peer connection lives in the daemon, so workers can restart during user switching without dropping the browser connection.

Daemon and worker use three independent transports: a bidirectional event pipe for signaling and control, a one-way media pipe for encoded audio/video, and a bidirectional file pipe for file commands and chunks. Portable default and desk-server modes reuse the same logical daemon/worker path with in-process channels.

AI Diagnostic Pipeline

The central orchestrator runs collect → redact → model → render and fails closed on edge redaction. A model may suggest a command, but only the authenticated owner can approve the exact preview; the host receives a sealed plan and revalidates it before execution. The MCP service remains read-only. See the AI Security Model.

Tech Stack

Backend — Rust (Edition 2024, 1.90+), Actix-Web 4.11, webrtc-rs 0.17, Actix-Session, Utoipa 5 (OpenAPI), turn 0.17, Prometheus.

Frontend — React 19, TailwindCSS + Shadcn UI (Radix), Vite 7, Kubb (OpenAPI → React Query / TS), TypeScript 5.9, xterm.js 5.5, TanStack Query v5.

Multimedia — capture via DXGI / WGC (Windows), X11 / Wayland + PipeWire (Linux); encode via X264 / OpenH264 / VP8 / VP9 / AV1; audio via WASAPI / ALSA / PipeWire + Opus.

See the Module Map for crate-level detail.

AI Assistant tool identities and parameters

A subagent task_id is the compact native task ID: t_ followed by 22 URL-safe Base64 characters. Creation, status, waits, result reads, cancellation and adjustments use that same value, without allocating t1/t2 aliases or resolving a separate UUID.

The model selects observed objects and results by ID instead of copying complete references, protocol versions or SHA-256 values. Text updates/deletions use file_version_id; Gmail attachments use artifact_result_id; exact Gmail/Slack sending uses handoff_result_id. Document operations use target_id, and Office selection reads use document_id. A new document copy restores its destination from approved output.directory_request_id, which can be omitted when exactly one approved directory is available. Word reports select web_search_result_id and web_source_ids from that search; the server restores original titles and URLs.

Short references locate original evidence within the current conversation and do not grant permission. The server restores the exact object, complete digest, original message content and fixed defaults before freezing approval input. The device still checks versions, identity and scope. References never silently select newer results. Expired, removed, wrong-type, ambiguous or unknown references reject execution. Sending still needs separate one-shot confirmation, and unknown outcomes are never retried automatically. Original receipts, full digests and recovery information remain saved; externalized JSON attachments use the same reduced model view.

Natural-language goals, delegated tasks and schedule prompts count Unicode characters. Files, message bodies, commands and native input retain separately stated UTF-8 byte limits. Format errors identify the field and constraint and provide a structural example generated from the current tool definition. The server fixes defaults before approval; examples never replace observed IDs. Prefer named scroll directions and positive amount: semantic UI uses up/down/left/right with 1 (small) or 2 (large) increments; background input uses up/down with window-pixel distances and still requires position. Numeric axes remain available for diagonal scrolling and cannot be mixed with direction parameters.

Native identities and object references

Tasks, messages, attachments, approval/directory requests, previews and scheduled tasks created by the assistant use compact native IDs. Queries, waits, cancellation and retries reuse the same value without adding a short-ID-to-UUID alias. Identities that must remain unique across conversations retain their full identity strength, usually as a type prefix and 22 encoded characters. Web sources use stable positions within their original search result.

Device-issued UI, browser, file/document references and opaque paging cursors retain their complete original identity. One conversation reference state supplies concise selectors for the model. Copy the IDs actually returned by tools; do not infer them from names, positions or numbers in another conversation. A reference never changes its target. Obtain a fresh reference from the corresponding resource reader when it expires. Business text, paths, URLs, user-supplied business IDs and model-protocol correlation IDs are not globally rewritten.

Inline results, attachment pages, historical arguments, permission requests and AI reviews use the same representation. Summarizing context does not delete attachment contents: references with retained, valid sources remain resolvable, while deleted or evicted contents cannot be restored. Paging binds the original contents, query and relevant projection; unrelated new references do not invalidate the current cursor. Deleting a conversation removes its mappings.

AI Assistant context and execution recovery

Owner-selected terminal and window attachments store only bounded display metadata, opaque device references and source labels, not their contents. Selection does not grant execution authority. Open-source Signal atomically saves each new metadata operation and its first receipt in SQLite. Retrying the same request returns that original changed value, including after reconnecting or removing model configuration; it does not modify the attachment, switch its model destination, extend expiry or reactivate a detached selection. Reusing the request ID with different operation data is rejected. Detaching an existing attachment needs no configured model. New mutations are refused while a turn is active; a historical receipt can still be replayed after the current owner and target checks. A receipt is not proof of the attachment's current state: refresh the authoritative conversation snapshot. The original device still validates each reference when data is read; actual reads are determined by device results.

If retained selection metadata has no original receipt, Signal cannot reconstruct the first result from it and rejects that request ID. Refresh the snapshot and submit a new explicit operation with a new request ID; existing attachments and permissions are not rewritten during upgrade.

Live capability selection uses UpdateAiAssistantContext (638) → AiAssistantContextUpdated (639) on both Signal and Manager. A selection saves bounded metadata tied to the current worker incarnation, model destination and readiness expiry; it grants no tools and contains no screen or document content. CurrentScreen remains an explicit one-turn input and cannot be saved by 638. The same complete request returns its first changed result, including false, without refreshing metadata; another selection under the same request ID is rejected. Clearing live selection needs no configured model and does not remove file or terminal attachments. New selection updates refuse active turns even after their lease expires; normal recovery must settle those turns first. Success and failure preserve transport and client request correlation. Always refresh the session snapshot for current state rather than treating a historical receipt as an active selection.

For new AI Assistant inputs, Signal saves the exact read-tool and object selection atomically with the message. Reusing a message ID with different text, source labels, subject or selection is rejected. Permission continuations restore that original selection and deadline; later attachments or newly ready read tools do not expand it. Legacy inputs without an original object selection do not acquire one automatically. Before sending an object read and after receiving its result, Signal rechecks the current input and attachment state. File, spreadsheet, terminal and iWork batch reads use the original references, the smaller request/attachment limits, and original model destination and source labels; display names are not file-type authority. Expired, withdrawn or superseded selections are rejected, and oversized or image-bearing object results are not relabeled for model use. These checks do not replace capability grants or device-side reference validation.

When a trusted runtime supplies explicit input-source labels, the shared Assistant loop preserves those sources instead of adding unrelated active attachments from the conversation. Model-output and owner-message provenance links remain attached; this bookkeeping does not authorize another read or export.

Permission continuations recover the conversation identity and original question from the authoritative session, including when a mobile client supplies only its session ID. The decision must still belong to that input. Before replaying the question to a model, the runtime validates its original source label and preserves sensitivity, destination and the stricter retention deadline; it cannot relabel expired, Secret or differently bound content as a new control message. For owner-attached objects, read grants on both Signal and Manager are limited to the original selection and read deadline, not attachments added while confirmation was pending or ambient device references. Approved conversation directories and verified file-result references are separate explicit selection routes: they require current directory consent, exact input, valid source evidence and a separately approved operation. An old input with no selector evidence cannot receive a new object-read grant, but its request can still be denied. Readiness changes do not refresh an existing grant.

Signal permission-decision retries return the original decision after current caller and connected-target checks, even if readiness is now unavailable, grants expired or were revoked, or a newer question superseded the input. The complete decision must match the original items and limits; item order may differ. A retry does not mint grants, restore uses, change the session, or start another model continuation. Missing or inconsistent request/decision records cannot be reconstructed from the current approval status. The returned state is a historical decision receipt, not current authority or proof that a continuation ran.

New decisions also persist a continuation fence in the same SQLite transaction. Signal periodically discovers unclaimed continuations without requiring the phone or original HTTP request to remain connected. It waits for an unambiguous authenticated target and fresh readiness, then revalidates the original input, model destination and current grants. Session claim and the transition to started commit together; duplicate wakeups cannot start the same continuation again. A newer question supersedes an unclaimed continuation. An already-started continuation is never automatically repeated after a crash: its original lease and action ledger are reconciled instead. These recovery checks do not renew references, grants or source retention.

When AI Assistant accepts an in-flight follow-up, the shared loop rechecks the input revision at both the initial and final save boundaries so an old answer cannot become the result for a new requirement. Ordinary storage failures remain errors rather than being reported as supersession. Retracting an old answer does not confirm cancellation of a device action that already started.

The shared execution interface distinguishes runtime-confirmed non-execution from operator rejection. If an execution backend confirms an action was closed before dispatch, the loop records why it was not executed and stops further mutations in that turn, without calling it user cancellation. This internal outcome grants no execution authority and does not imply identical durable recovery support in every backend.

An execution backend may also supply a verified original result label. The shared loop checks that it matches the result bytes and preserves it without regenerating lineage or extending expiry at delivery time. A mismatch rejects the result without acknowledging delivery. The label grants no implicit model export permission. Open-source Signal persists original completion receipts for its bound, side-effecting Computer Action paths, including late results received after the foreground waiter exits. Foreground execution, active waiting and completion publishing use the same validated result projection. Legacy command results and inline browser reads retain their existing label paths.

With an explicit executor acceptance and a frozen background policy, a Computer Action can leave the foreground budget while remaining the same original task. Its owner snapshot reads conversation, grants and task state together; task changes advance the presentation sequence without invalidating a running turn's lease. Completion is saved before delivery is acknowledged, and expiry becomes unknown until a real result arrives. A bounded, tool-free model explanation requires the original, still-valid model/context export selection; Secret data, changed input/context/model, expired labels or unavailable replay cannot be silently reauthorized. The owner still retains the original result when automatic interpretation is denied. These original OSS tasks now support durable owner cancellation: only the original connection receives stop retries, never a repeated action or replacement connection. A stop acknowledgement remains nonterminal; only an original native pause receipt projects as cancelled, and a genuine late success or failure remains authoritative.

A terminal receipt does not mean success. When a backend confirms failure, the shared loop preserves the original result label and acknowledges delivery only after saving, while reporting tool failure and stopping subsequent calls in the group. An actively awaited failure likewise retains the original action's call identity, not the wait call's. If that tool group is still open, the receipt remains unconsumed until a subsequent standalone wait retrieves it. Unknown effects remain unknown rather than becoming success or failure merely because a message arrived.

A transactional execution backend may return a version receipt bound to the original input, lease, call, and post-save session version. The shared loop accepts only its executor's consecutive version advances, including when a later step fails, so the error can be saved against the correct version. It never refreshes to bypass concurrent revocation or new input, and the receipt is not tool authorization. Open-source Signal defaults to its existing execution entry and requires no Manager shared storage for this interface.

Personal AI Assistant shares a compiled policy revision with Manager, independent of unrelated organization policy counters. Older decisions cannot be upgraded automatically and resumed. A new explicit request starts a turn under the current policy without relabeling history, reissuing old grants, or restoring their uses. Automatic continuation and grant issuance require the current revision; a match does not replace target readiness or grant revocation checks. Ordinary Terminal AI Assistant policy semantics remain unchanged.

Browser Providers use shared closed-input preflight: input size is bounded, the selected BrowserSurface must be unexpired, and the server derives action scope and risk. Generic form filling and element activation remain InputFallback; Gmail/Slack draft handoffs do not become send actions. Passing preflight is not execution authority: the existing grant, egress, durable dispatch, and target-device checks still apply.

Semantic desktop execution, permission requests, and grant issuance share strict input decoding: objects and actions must match the current tool definition and device capabilities. An application grant allowing Toggle does not imply every native control supports it. The original target still checks reference expiry, session identity, and actual supported actions. Central validation replaces neither native Accessibility/UIA execution nor local permissions, writer leases, or input takeover.

A centrally dispatched Computer Action does not require a browser to keep a remote-desktop connection open. Portable/DeskServer paths with an existing single worker can hand the action to that worker; session-targeted routing still requires an explicit binding and never guesses a desktop. Rejection inside the action handler before worker handoff reports definitely-not-started through ComputerActionCompleted correlated to the original request, rather than waiting for a timeout. This neither cancels other actions nor relaxes authorization, readiness, or local capability checks.

A trusted central service can request an original Computer Action stop with CancelComputerAction. The daemon validates authorization and forwards the approver plus the complete action/work/generation identity to the original worker; ambiguous sessions are never guessed. CancelRequested in ComputerActionStateReported only acknowledges invalidating the matching writer, not undoing an OS call or proving non-execution. Subsequent adapter steps are blocked, while the original completion or unknown result remains authoritative. Cancellation, preemption, and expiry retain the writer slot until the original operation releases it. Daemon and worker must use the same build: the cancellation IPC approver field cannot be mixed with the previous layout.

AI Assistant live Office, Numbers, Pages and Keynote reads now retain the selected device-issued document reference with the original input. Permission continuation does not select a newer foreground document. Reads recheck the original input, model destination, current worker/reference and original expiry before dispatch and after the result arrives; a changed or expired target is rejected without exporting its content. Office permissions use the same exact-object resource contract as live iWork permissions. Result provenance identifies the frozen target without exposing its executable token, and output retention cannot extend its deadline. Batch iWork resolves source files through conversation directories and authorized read results. A live input also requires the matching active durable 638 selection at intake and before read. Withdrawing that selection while permission is pending prevents resume without consuming the grant; selecting it again later cannot revive the old input. Android and iOS strictly correlate 639 and retry the same client request identity after reconnect.