soundcraft/docs/control-protocol.md

Control protocol

SoundCraft can be driven completely by other programs: scripts, test harnesses and AI agents. Every menu item, button, mouse gesture and keystroke maps to a command (or to synthetic input), and every command is reachable from:

Surface	How
Desktop app control channel	soundcraft --control <port> (0 = pick a free port; printed as SOUNDCRAFT_CONTROL_PORT=…)
CLI against a running app	soundcraft-cli app --port <port> <method or command id> [JSON]
Headless CLI	soundcraft-cli run --demo --cmd 'mix.volume={"track":"Kick","db":-6}' --bounce out.wav
MCP (headless)	soundcraft-cli mcp [--demo]
MCP (bridged to the app)	soundcraft-cli mcp --connect <port>

Isolated automation profiles

Set SOUNDCRAFT_CONFIG_DIR to a fresh directory to isolate UI preferences, Autosave, Presets, and logs. The nonempty value is used as the complete settings directory. Unset retains the normal platform paths; an explicit empty value disables the settings directory instead of falling back to a personal profile. Save synthetic sessions to explicit scratch paths. Do not use SOUNDCRAFT_NO_PREFS when testing preference persistence across restarts.

Run UI acceptance with --no-audio. This disables the output player; it does not disable recording input opened by arm/monitor commands. An automated UI smoke run must never arm a track, enable input monitoring, or start recording. Control input injects egui events; it does not establish operating-system input delivery or screen-reader acceptance. Native screenshots use the presented viewport frame.

Wire format

Newline-delimited JSON over TCP on 127.0.0.1. One request per line, one reply per line:

{"id": 1, "method": "engine.execute", "params": {"command": "track.new", "params": {"count": 2, "format": "Stereo"}}}
{"id": 1, "ok": true, "result": {"tracks": [12, 13]}}

Errors: {"id": 1, "ok": false, "error": "bad parameters for mix.volume: no track named Nope"}.

Methods

Method	Params	What it does
engine.execute (alias command)	{command, params}	Run a command programmatically. Never opens a dialog; empty params use defaults or the current selection.
engine.commands	{filter?}	Every command: id, label, menu path, shortcut, params doc, enabled + reason. Includes UI-layer commands (window.*).
engine.parity	{}	Parity report against the incumbent's menu catalog.
session.inspect	`{detail?: "summary"	"full"}`
ui.inspect	{}	UI state (window, panels, dialogs), window size, playing/position, audio device, and edit_layout (timeline rect and every track row's rect, for clicking).
ui.set	any UiState fields	e.g. {"window": "Mix", "narrow_mix": true}.
ui.menu.list	{}	Every catalog menu path and the command that implements it (or null).
ui.menu.invoke	{path} or {id}	Like clicking the menu item: may open its dialog (e.g. "Track > New...").
ui.click	{x, y, button?, count?, cmd?, shift?, alt?, ctrl?}	Synthetic click (window points).
ui.drag	{x, y, to_x, to_y, steps?}	Synthetic drag with the primary button.
ui.scroll	{x, y, dx?, dy?, unit?: "point"|"line", steps?, cmd?, shift?, alt?, ctrl?}	A two-finger trackpad swipe over (x, y): dx/dy points spread over steps frames (default 8). Positive values move the content right/down (natural scrolling). unit: "line" sends mouse-wheel notches instead. With cmd it zooms; with shift it scrolls sideways.
ui.zoom	{x, y, factor, steps?}	A trackpad pinch around (x, y): factor > 1 spreads the fingers (zooms in).
ui.move	{x, y}	Move the pointer.
ui.key	{key, cmd?, shift?, alt?, ctrl?}	Key press (Space, Enter, A, F7, =…).
ui.text	{text}	Type text into the focused field.
ui.screenshot	{path?}	PNG of the window (base64 in png_base64 when no path).
app.quit	{}	Quit.

Positions and references

• Tracks: id (number) or name ("Kick").
• Clips: id (from session.inspect).
• Positions: samples (96000), seconds ({"seconds": 2.5}), or a time string in any timebase, optionally prefixed: "bars_beats:5|1|000", "min_secs:0:12.500", "timecode:00:00:10:00", "samples:48000".
• Ranges: start + end (or start + length); omitted = the current edit selection.

Examples

P=$(soundcraft --demo --control 0 | sed -n 's/SOUNDCRAFT_CONTROL_PORT=//p')   # or read stderr
soundcraft-cli app --port $P mix.solo '{"track":"Bass"}'
soundcraft-cli app --port $P transport.play
soundcraft-cli app --port $P ui.screenshot '{"path":"/tmp/shot.png"}'
soundcraft-cli app --port $P ui.menu.invoke '{"path":"Track > New..."}'

Offscreen (no window): cargo run -p soundcraft-ui-egui --example ui_shot -- out.png script.jsonl, where each script line is a request ({"method": …, "params": …}), {"shot": "path.png"} or {"steps": n}. Use --system-theme light|dark|none to supply the initial OS appearance, and script events such as {"system_theme": "dark"} to change it on subsequent frames.

Appearance defaults to Dark. ui.theme {"mode":"system"} follows the appearance supplied by the native or web integration at launch and on later frames, while the saved choice stays System. When the integration cannot report an appearance, System uses Dark. Manual Light and Dark stay fixed when the OS appearance changes. An offscreen render without --system-theme supplies no OS appearance and therefore resolves System to Dark.

Panel docking

Tracks and Groups, Clips, MIDI Editor, Memory Locations, and Undo History use movable panels. Commands target the active Edit or Mix view; floating panels stay inside the current viewport. Edit and Mix retain separate arrangements; Clips and MIDI Editor belong to Edit. Transport, plug-in, and specialist utility windows retain their existing hosts. The document/session view is protected. Layouts, active tabs, panel visibility, and floating rectangles are saved with the app's existing UI preferences. These operations do not edit document contents.

Panel IDs: tracks, clips, midi, memory, history. The protected center is canvas.

ui.dock accepts operation and panel. Operations are open, close, activate, float (with finite positive rect: [x, y, width, height]), or move (with target and zone: center|left|right|top|bottom). Center groups with the target; side zones split beside it. ui.dock.reset restores the default arrangement. Invalid moves return an error without changing the layout.

Invoke these UI commands through engine.execute, for example {"command":"ui.dock","params":{"operation":"float","panel":"tracks","rect":[100,100,320,440]}}.

Advanced ui.dock operations use the same validated transaction as pointer gestures: moveFloating updates rect, resizeSplit accepts a boolean child path and size ({"Ratio":0.5}, {"FixedFirst":280}, or {"FixedSecond":280}), setStackOpen takes open, and resizeStack takes a point height or null. A center move can include before (panel ID or null to append) for tab order. The equivalent {"action": ...} form accepts a serialized shared docking action; this is also the form emitted by the renderer. Protected-canvas and app-specific panel rules apply equally to both forms.