# Control protocol & MCP

## Desktop control channel
`filmcraft --control 9876` (or `FILMCRAFT_CONTROL_PORT`) listens on `127.0.0.1:<port>` (loopback only).
One JSON request per line → one JSON reply per line. The app opts out of macOS App Nap
(`apps/filmcraft/src/app_nap.rs`): a hidden window would otherwise drop the whole process to
background priority, and on a busy machine it would stop answering.

**Only requests are read.** Every line must be a JSON object with a string `method` (`id` and `params`
are optional; blank lines are skipped). Anything else gets one error reply
(`{"ok": false, "error": "… closing the connection"}`) and the server **closes the connection**, so
nothing sent after it on that connection runs. That covers text that isn't JSON, a JSON array or number,
an object without `method`, invalid UTF-8, and a line longer than 4 MiB. An HTTP request (for example a
web page's cross-origin `fetch` to `127.0.0.1:<port>`) therefore can't smuggle a command in its body:
its request line is rejected first. At most 16 connections are served at once; further ones get an error
line and are closed. Clients that get an error reply should reconnect. The port has no authentication,
so only enable it while you use it. Transport: `apps/filmcraft/src/control_server.rs`.

```json
{"id": 1, "method": "engine.execute", "params": {"command": "sequence.addEdit", "params": {"seconds": 3}}}
{"id": 1, "ok": true, "result": {"cuts": 2}}
```

Methods (handlers in `crates/ui-egui/src/control.rs`):

| Method | Params | |
|---|---|---|
| `engine.execute` / `ui.menu.invoke` | `{command, params}` | any engine or UI command id |
| `engine.commands` / `ui.menu.list` | – | command registry / menu tree |
| `ui.inspect` | – | UI state (tool, workspace, dock, timeline view, playback, fps…) |
| `ui.elements` | `{prefix?}` | every on-screen interactive element: id, label, rect |
| `ui.set` | `{tool, workspace, mode, theme, focused, playbackRes, timeline:{pps,scroll,fit}, program:{…}, source:{…}, clipDialog:{param: value}, menuDialog:{param: value}, panels:{scopes, timecode, reference, events, progress}, export:{…}, keyframeSelection:[…]}` | `keyframeSelection` replaces the Effect Controls keyframe selection (`[{clip, effect, param, mask?, time}]`, `time` in media ticks; read back as `ui.inspect` → `ui.keyframe_selection`; a click on a lane keyframe selects just that one); `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `panels` merges into the panel settings (`ui.inspect` → `ui.panels`; see `panels::panel_state`): Lumetri Scopes `{shown: ["vectorscopeYuv"|"vectorscopeHls"|"histogram"|"parade"|"waveform"], waveformType, paradeType, colorSpace, brightness, scale, clamp, targets}`, Timecode `{rows: [{source, mode, display}], showName}`, Reference Monitor `{ganged, time, display: "composite"|"scopes", scopes}`, Events `{level, selected}`, Progress `{showFinished}` `export` deep-merges into Export mode state (`preset` — applied first —, `settings` (ExportSettings JSON), `fileName`, `location`, `range`, `customStart`, `customEnd`, `openSections`, `manager` (Preset Manager: `query`, `favoritesOnly`, `selected`, `saveName`), `quickOpen`, `quickPath`, `quickPreset`); `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `projectPanel` merges into the Project panel's view state (`bin` shown in place, `tabs` (bins in a tab or window: `{bin, floating, view, iconSize, nav}`), `activeTab`, `selectedBin`, `rename: {item|bin, text}`, `dialog` (`{"metadataDisplay": {columns, filter}}`, `{"savePresetAs": {name}}`, `{"managePresets": {selected, name}}`, `{"freeformOptions": {options}}`, `{"saveArrangement": {name, bin}}`)); `mediaBrowser` into the Media Browser's (`expanded`, `pathEdit`, `editColumns`, `treeWidth`). View settings, columns and presets are engine commands (`project.view.set`, `project.columns.set`, `project.viewPreset.*`, `mediaBrowser.settings`) |
| `ui.panel.show` / `ui.panel.close` | `{panel}` | |
| `ui.click` / `ui.move` | `{id}` or `{x,y}`, `button`, `count`, `modifiers` | synthetic pointer input |
| `ui.drag` | `{from, to, steps, modifiers}` | press–move–release |
| `ui.scroll` | `{id|x,y, dx, dy, modifiers}` | wheel / trackpad |
| `ui.key` / `ui.type` | `{key}` (`Cmd+K`, `Space`…; off macOS `Ctrl+K` is the same key) / `{text}` | keyboard |
| `ui.timeline.hit` / `ui.timeline.locate` | `{x,y,modifiers?}` / `{clip, edge?}` | timeline hit-testing. `hit` reports a clip's `edge` (`In`/`Out`) within 7 px of it (a third of a narrow clip); just left of a cut that is the left clip's Out, from the cut rightwards the right clip's In. `kind` is the trim a press of the current tool with `modifiers` starts there (`trim`, `ripple`, `roll`, `rateStretch`, `remix`), `volume` on an audio clip's Volume line (Selection and Pen tools), or null |
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
| `ui.resize`, `app.quit` | `{width,height}` / `{}` | resize or close the native window; `app.quit` quits without the save prompt (unsaved changes stay in the recovery journal), while the command `app.quit` asks first |
| `ui.focus` | `{}` / `{id}` | empty parameters activate the native window; a widget ID requests egui focus without OS activation |
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), `liveDecoders` (sources holding a decoder now), `cacheMB` / `cacheBudgetMB` (decoded frames all sources hold, and the shared budget), `planePoolMB` / `planesReused` (idle buffers of the frame pool, the planes of evicted frames and the compositor's float images, which a standalone export frees when it ends; and buffers taken from it instead of allocated), decoder ms (total and per sample), `framesDecoded`, `hardware` (Settings ▸ Playback ▸ Hardware decoding: `enabled`, pictures from hardware decoders `frames` vs `softwareFrames`, `sessions`, `declined` streams, mid-stream `fallbacks`); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding), `audio` (desktop: sound is mixed 200 ms ahead of the device; device `callbacks`, `underruns` and `missingFrames` played as silence because the mixer fell behind, `minLeadMs` the least sound left buffered); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `monitor`: the `frame` and `revision` of the picture the Program monitor shows against `playheadFrame` and `projectRevision` (how far a drag or a scrub is ahead of the picture); `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |

**Timeline thumbnails.** `ui.set {timeline: {thumbnails}}` sets the Timeline wrench menu's Show Video
Thumbnails: `off`, `head`, `headAndTail` or `continuous`.

**Selection or explicit targets.** A command that acts on the selection is "not available right now" when
nothing is selected. When `params` name the targets under a key the command documents (`clips` /
`clip`: timeline clips of the active sequence, `items` / `item`: project items), enablement is
checked against those instead and the selection is left as it is; the command's other conditions
still apply. `engine.commands` and the menus always show the selection's enablement.

**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
without activating itself. UI requests and screenshots that need a rendered frame bring the window
forward without making it key (macOS `orderFrontRegardless`), so the user's typing keeps going to
the app they are in. Only an explicit `ui.focus {}` activates FilmCraft. Injected keys work on
focused widgets even when the native window is in the background.

`ui.focus {id}` targets an enabled, visible widget registered with a real egui response. It rejects
unknown IDs, non-widget regions and hidden tool aliases. The reply acknowledges the request;
`ui.inspect.keyboardFocus` reports the actual widget ID, matching element IDs, label and active
focus. This is separate from `ui.focused`, which identifies the active panel. Tools, dock tabs,
tab menus and divider handles expose widget IDs through `ui.elements`. Tab/Shift-Tab move widget
focus; Space/Enter activate focused buttons and tabs, with visible focus outlines.

For unattended validation, start with `--control <port> --control-input-only`. This opt-in mode
accepts widget input through control requests and excludes native keyboard, pointer, file-drop
and menu commands. Window close/quit and screenshots remain available. Ordinary launches and
`--control` alone keep native input enabled. `ui.inspect.input` distinguishes `mode`, the observed
`nativeWindowFocused`, and egui's `effectiveWindowFocused`; effective focus does not activate the
OS window or prove physical-keyboard delivery.

Element ids are stable, e.g. `timeline.clip.<id>`, `timeline.track.V1.lock`, `tools.Razor`,
`project.item.<id>`, `effects.item.gaussian_blur`, `panel.tab.Timeline`, `program.transport.playback.toggle`.

Audio mixing: `mixer.*` commands (strips `"A1"`, `"S1"`, `"Mix"` or ids; lanes `volume`, `pan`,
`mute`, `send.<i>.level`, `fx.<slot>.<param>`): `mixer.inspect`, `mixer.setStrip` (volume, pan, mute,
solo, record arm, solo safe, mode Off/Read/Latch/Touch/Write, output, input map, channels),
`mixer.setValue`, `mixer.touch` / `mixer.release` (a fader gesture; recorded during an automation pass),
`mixer.recordStart` / `mixer.recordStop` (playback runs them), `mixer.addSubmix`, `mixer.deleteSubmix`,
`mixer.addInsert` / `removeInsert` / `setInsert`, `mixer.addSend` / `setSend` / `removeSend`,
`mixer.setKeyframe` / `deleteKeyframe` / `moveKeyframe` / `clearLane`, `mixer.writeAutomation`;
`clipMixer.set`; `clip.audioGain {clips?, mode: set|adjust|normalizeMax|normalizeAll, db}`, `clip.audioPeak`;
`effects.setDefaultTransition`. UI ids: `mixer.<A1|S1|Mix>.<fader|value|pan|panValue|mode|mute|solo|
record|soloSafe|output|input|fx.<n>|send.<n>|meter|name>` (popup entries below them, e.g.
`mixer.A1.mode.Touch`, `mixer.A1.fx.0.studio_reverb`), `mixer.showEffects`, `mixer.transport.<cmd>`,
`clipMixer.A1.<fader|pan|mute|solo|keyframe|value>`, `timeline.track.A1.keyframes[.<lane>|.clip]`,
`timeline.track.A1.lane[.kf.<n>]`, `audioGain.<set|adjust|normalizeMax|normalizeAll|ok|cancel|peak>`.

Essential Sound: `essentialSound.inspect`, `essentialSound.setType {type: dialogue|music|sfx|ambience}`,
`essentialSound.clearType`, `essentialSound.set {key, value}` or `{values: {key: value}}` (keys such as
`repair.noise.on`, `repair.noise.amount` (0–10), `repair.humHz`, `clarity.eqPreset`, `clarity.enhanceTone`,
`creative.reverbPreset`, `ducking.against`, `ducking.reduceDb`, `ducking.fadeS`, `pan.value`,
`volume.levelDb`, `mute`, section switches `<section>.enabled`), `essentialSound.applyPreset {preset, type?}`,
`essentialSound.savePreset {name}`, `essentialSound.deletePreset {name}`, `essentialSound.autoMatch {target?}`,
`essentialSound.generateDucking`; all take `clips` (default: the selected audio clips). UI ids:
`essentialSound.tab.<Browse|Edit>`, `essentialSound.type.<Dialogue|Music|SFX|Ambience>`,
`essentialSound.clearType`, `essentialSound.preset[.save|.delete|.name|.ok]`,
`essentialSound.section.<Name>[.toggle]`, a row per setting key (`essentialSound.repair.noise.on`,
`essentialSound.repair.noise.amount`, `essentialSound.repair.humHz.<i>`), `essentialSound.autoMatch`,
`essentialSound.ducking.against.<Type>`, `essentialSound.generateDucking`, `essentialSound.volume.on`,
`essentialSound.volume.levelDb`, `essentialSound.mute`, `essentialSound.browse.<Type>.<preset>`.

Project files, auto-save, crash recovery and preferences commands (`file.recover`, `prefs.set`, …) and their
automation ids are listed in [project-files.md](project-files.md). That file also lists the media
management commands and dialog ids: offline media and relinking (`media.findMissing`,
`media.relink`, `media.autoRelink`, `media.search`, `media.makeOffline`, `media.status`, `linkMedia.*`),
proxies (`media.createProxies`, `media.attachProxies`, `media.toggleProxies`, `proxies.*`), ingest
(`project.ingestSettings`) and the Project Manager (`file.projectManager`, `pm.*`).

**Moving clips**: `timeline.move {moves: [{clip, track, time}], insert?, linked?}` → `{moved: [clip ids]}`.
With Linked Selection on (the default; `linked` overrides the toggle for one call) the linked
partners of a listed clip that are not listed themselves move by the same time offset and stay on
their own tracks, so picture and sound stay in sync. Like a listed clip, a partner overwrites
what is at its destination (or inserts, with `insert`). A partner on a locked track stays where it
is, and so does one whose position would not change (the listed clip only changed track). `moved`
lists every clip that moved, partners included. No clip is moved before the sequence
start: if one would be, all clips of the call land later by the same amount (their spacing is
kept). A `time` beyond the representable range is an "invalid parameters" error. The timeline
panel's drag passes `linked: false`, because it already lists exactly the clips it moves. With
`copy: true` (Option-drag in the timeline) the clips stay where they are and copies land at the
destinations instead (linked partners are copied too, unless `linked: false`); the result lists
the new clips as `copied`, they are selected, and copies of linked clips are linked to each other.

**Trim mode** (`crates/engine/src/trim.rs`): `trim.selectEditPoint` / `trim.selectNearest` enter trim
mode (the Program monitor becomes the Trim Monitor, ids `trimMonitor.*`); `trim.monitor` returns what
it shows. Dynamic trimming takes an explicit clock (seconds) so it is deterministic: `trim.shuttle
{direction, slow?, clock}` (J/L), `trim.tick {clock}`, `trim.shuttleStop {clock?}` (K, one undo step),
`trim.cancelDynamic` (Esc), `trim.playAround {clock, loop?}` (Space / Shift+K).

**Replace With Clip**: `clip.replaceFromSource`, `clip.replaceFromSourceMatchFrame` and
`clip.replaceFromBin {clips?, item?, keepSourceIn?}` swap the media of the clips and return
`{clips: [ids], item, short: [{clip, shortByFrames}]}`. `short` lists the replaced clips that now
run past the end of their media (the renderer holds the last frame there) with the number of
sequence frames that have no media, at most the clip's length; it is empty when the new media
covers every clip. The edit is the same either way. From Bin starts each clip at the new item's In point; with
`keepSourceIn: true` each clip keeps its own source In instead. A subclip that restricts trims has
no media outside its range, so there a kept In is moved to the subclip's first or last frame, and
the result's `sourceInClamped: [ids]` (present with `keepSourceIn`) names the clips it was moved for.

**Colour** (`crates/engine/src/color.rs`): `sequence.colorSettings {workingSpace: rec709|rec2100-pq|
rec2100-hlg, wideGamut, autoToneMap}`, `clip.interpretFootage {items?, colorSpace?: auto|<id>, pixelAspect?: [num, den]|file}`,
`color.spaces`, `media.colorInfo {item}`; LUTs: `lut.import {path, name?}`, `lut.list`,
`lut.remove {id}`, `lut.export {lut, path, format?}`, `lumetri.setInputLut` / `lumetri.setLook
{clip?, lut: lib:<id>|builtin:<id>|"", path?}`, `lumetri.setSection {clip?, section, on?}`,
`lumetri.applyMatch {clip?, referenceTime|referenceFrame|referenceTimecode, faceDetection?}`. Without
params the menu entries open dialogs (ids `colorDialog.space.<id>`, `colorDialog.working.<id>`,
`colorDialog.wideGamut`, `colorDialog.autoToneMap`, `colorDialog.ok|cancel`). Lumetri panel ids:
`lumetri.input_lut`, `lumetri.look_lut`, `lumetri.switch.<section>`, `lumetri.section.<section>`,
`lumetri.match.*`; HDR scopes: `scopes.hdrWaveform`. `file.exportMedia {sdr: true}` exports an HDR
sequence as tone-mapped SDR.

**Keyboard shortcuts** (`crates/engine/src/shortcuts.rs`): `shortcuts.list {query?, panel?}`,
`shortcuts.get`, `shortcuts.set {command, keys, panel?, add?, keepConflicts?}`, `shortcuts.clear`,
`shortcuts.undo` / `shortcuts.redo`, `shortcuts.conflicts {platform?}`, `shortcuts.forKey {key}`,
`shortcuts.resolve {keys, panel?}`, `shortcuts.presets`, `shortcuts.loadPreset` / `savePreset` /
`deletePreset {name}`, `shortcuts.export` / `import {path}`, `shortcuts.audit`. Keys use `Cmd` (⌘ /
Ctrl), `Ctrl` (macOS ⌃), `Alt`, `Shift`. The dialog (Edit ▸ Keyboard Shortcuts…, ⌥⌘K) uses ids
`shortcuts.*` (`shortcuts.key.K`, `shortcuts.cell.<command>`, `shortcuts.ok`, …).

## MCP
`filmcraft-cli mcp` serves MCP on stdio: headless (in-process session; `--demo` / `--project p.fcproj`)
or `--bridge 127.0.0.1:9876` to drive the running app. Tools: `command_list`, `command_run`,
`command_batch`, `doc_inspect`, `render_preview`, `project_inspect`, `sequence_inspect`, `media_import`,
`render_frame`, `ui_inspect`, `ui_elements`, `ui_click`, `ui_drag`, `ui_key`, `ui_type`, `ui_screenshot`,
`ui_control`; resources `filmcraft://document` and `filmcraft://commands`. `.mcp.json` registers both.
Annotations, argument checking, errors and export progress / cancellation:
[agents.md § Conventions](agents.md#conventions).

Time is in ticks (254 016 000 000 per second); commands also accept `seconds`, `frame` or `timecode`.

## Panel docking commands

`ui.menu.invoke` and `engine.execute` route these frontend commands:

| Command | Parameters | Result |
| --- | --- | --- |
| `window.floatPanel` | `{panel?, rect?: [x,y,width,height]}` | Move an existing panel into a floating group. Omitted panel uses the focused panel; omitted rect uses a centered 420×320 group. |
| `window.dockPanel` | `{panel?, anchor?, zone?: "center"\|"left"\|"right"\|"top"\|"bottom", before?}` | Move into the anchor's tab group or a side split; `before` names an existing destination tab for precise ordering. Floating destinations accept tabs only. Without `anchor`, a side `zone` docks the panel along that edge of the whole workspace, at a quarter of its width or height. |
| `window.closePanel` | `{panel?}` | Close a docked or floating panel without closing its project documents. |

Invalid source/target identities and non-finite or out-of-range geometry return an error with
the layout unchanged. `ui.inspect.ui.floating` reports floating groups and their logical-point
rectangles. `panel.drag.<Panel>` locates each panel's header grip; `panel.float.<index>.move`
and `.resize` locate floating geometry handles. Named workspace saves include these groups.

`window.dockAction` accepts `{action}` using the shared docking action's JSON form. It
covers tab moves and ordering, split and floating geometry, activation, and closing.
For example, `{"action":{"ResizeSplit":{"path":[],"size":{"FixedFirst":360}}}}`
resizes the root split; path entries use `false` for the first child and `true` for
the second. `{"action":{"MoveFloating":{"panel":"Info","rect":[45,65,420,320]}}}`
repositions and resizes the floating group containing Info. Invalid actions leave the
layout unchanged. Pointer gestures dispatch through this same catalog command.