filmcraft/docs/control-protocol.md

Control protocol & MCP

Desktop control channel

filmcraft --control 9876 (or FILMCRAFT_CONTROL_PORT) listens on 127.0.0.1:<port> (loopback only). One JSON request per line → one JSON reply per line. The app opts out of macOS App Nap (apps/filmcraft/src/app_nap.rs): a hidden window would otherwise drop the whole process to background priority, and on a busy machine it would stop answering.

Only requests are read. Every line must be a JSON object with a string method (id and params are optional; blank lines are skipped). Anything else gets one error reply ({"ok": false, "error": "… closing the connection"}) and the server closes the connection, so nothing sent after it on that connection runs. That covers text that isn't JSON, a JSON array or number, an object without method, invalid UTF-8, and a line longer than 4 MiB. An HTTP request (for example a web page's cross-origin fetch to 127.0.0.1:<port>) therefore can't smuggle a command in its body: its request line is rejected first. At most 16 connections are served at once; further ones get an error line and are closed. Clients that get an error reply should reconnect. The port has no authentication, so only enable it while you use it. Transport: apps/filmcraft/src/control_server.rs.

{"id": 1, "method": "engine.execute", "params": {"command": "sequence.addEdit", "params": {"seconds": 3}}}
{"id": 1, "ok": true, "result": {"cuts": 2}}

Methods (handlers in crates/ui-egui/src/control.rs):

Method	Params
engine.execute / ui.menu.invoke	{command, params}	any engine or UI command id
engine.commands / ui.menu.list	–	command registry / menu tree
ui.inspect	–	UI state (tool, workspace, dock, timeline view, playback, fps…)
ui.elements	{prefix?}	every on-screen interactive element: id, label, rect
ui.set	{tool, workspace, mode, theme, focused, playbackRes, timeline:{pps,scroll,fit}, program:{…}, source:{…}, clipDialog:{param: value}, menuDialog:{param: value}, panels:{scopes, timecode, reference, events, progress}, export:{…}, keyframeSelection:[…]}	keyframeSelection replaces the Effect Controls keyframe selection ([{clip, effect, param, mask?, time}], time in media ticks; read back as ui.inspect → ui.keyframe_selection; a click on a lane keyframe selects just that one); program / source merge fields into the monitor view state (res, paused_res, high_quality, display, zoom, pan, show_rulers, show_guides, lock_guides, snap, guides, compare_ref, …; see monitors.md); clipDialog sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); menuDialog those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see panels::menu_dialogs); panels merges into the panel settings (ui.inspect → ui.panels; see panels::panel_state): Lumetri Scopes `{shown: ["vectorscopeYuv"
ui.panel.show / ui.panel.close	{panel}
ui.click / ui.move	{id} or {x,y}, button, count, modifiers	synthetic pointer input
ui.drag	{from, to, steps, modifiers}	press–move–release
ui.scroll	`{id	x,y, dx, dy, modifiers}`
ui.key / ui.type	{key} (Cmd+K, Space…; off macOS Ctrl+K is the same key) / {text}	keyboard
ui.timeline.hit / ui.timeline.locate	{x,y,modifiers?} / {clip, edge?}	timeline hit-testing. hit reports a clip's edge (In/Out) within 7 px of it (a third of a narrow clip); just left of a cut that is the left clip's Out, from the cut rightwards the right clip's In. kind is the trim a press of the current tool with modifiers starts there (trim, ripple, roll, rateStretch, remix), volume on an audio clip's Volume line (Selection and Pen tools), or null
ui.playback	`{action: play	stop
ui.screenshot	{path?, panel?}	PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep)
ui.resize, app.quit	{width,height} / {}	resize or close the native window; app.quit quits without the save prompt (unsaved changes stay in the recovery journal), while the command app.quit asks first
ui.focus	{} / {id}	empty parameters activate the native window; a widget ID requests egui focus without OS activation
perf.stats	–	performance counters (also the command perf.stats, so engine.execute and MCP command_run reach it; headless sessions return the engine part). decode: GOP-cache requests / hits / cacheHitRate, decoder seeks, samples decoded and skipped while catching up, draftFrames (decoded in draft mode), h264Threads (frame-threading workers per H.264 decoder), liveDecoders (sources holding a decoder now), cacheMB / cacheBudgetMB (decoded frames all sources hold, and the shared budget), planePoolMB / planesReused (idle buffers of the frame pool, the planes of evicted frames and the compositor's float images, which a standalone export frees when it ends; and buffers taken from it instead of allocated), decoder ms (total and per sample), framesDecoded, hardware (Settings ▸ Playback ▸ Hardware decoding: enabled, pictures from hardware decoders frames vs softwareFrames, sessions, declined streams, mid-stream fallbacks); playback: playing, shown / dropped frames and dropRate of the current or last play, resolution, draftDecode (Settings ▸ Playback ▸ Draft decoding), audio (desktop: sound is mixed 200 ms ahead of the device; device callbacks, underruns and missingFrames played as silence because the mixer fell behind, minLeadMs the least sound left buffered); frames: frame-worker jobs, cancelled, requestHitRate, decodeMs / renderMs (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; monitor: the frame and revision of the picture the Program monitor shows against playheadFrame and projectRevision (how far a drag or a scrub is ahead of the picture); ui: fps and frame ms; process: CPU seconds; media, jobs. Counters are cumulative: diff two readings to measure an interval (performance.md)

Timeline thumbnails. ui.set {timeline: {thumbnails}} sets the Timeline wrench menu's Show Video Thumbnails: off, head, headAndTail or continuous.

Selection or explicit targets. A command that acts on the selection is "not available right now" when nothing is selected. When params name the targets under a key the command documents (clips / clip: timeline clips of the active sequence, items / item: project items), enablement is checked against those instead and the selection is left as it is; the command's other conditions still apply. engine.commands and the menus always show the selection's enablement.

Focus. Driving the app never steals the user's keyboard. Started with --control, the app opens without activating itself. UI requests and screenshots that need a rendered frame bring the window forward without making it key (macOS orderFrontRegardless), so the user's typing keeps going to the app they are in. Only an explicit ui.focus {} activates FilmCraft. Injected keys work on focused widgets even when the native window is in the background.

ui.focus {id} targets an enabled, visible widget registered with a real egui response. It rejects unknown IDs, non-widget regions and hidden tool aliases. The reply acknowledges the request; ui.inspect.keyboardFocus reports the actual widget ID, matching element IDs, label and active focus. This is separate from ui.focused, which identifies the active panel. Tools, dock tabs, tab menus and divider handles expose widget IDs through ui.elements. Tab/Shift-Tab move widget focus; Space/Enter activate focused buttons and tabs, with visible focus outlines.

For unattended validation, start with --control <port> --control-input-only. This opt-in mode accepts widget input through control requests and excludes native keyboard, pointer, file-drop and menu commands. Window close/quit and screenshots remain available. Ordinary launches and --control alone keep native input enabled. ui.inspect.input distinguishes mode, the observed nativeWindowFocused, and egui's effectiveWindowFocused; effective focus does not activate the OS window or prove physical-keyboard delivery.

Element ids are stable, e.g. timeline.clip.<id>, timeline.track.V1.lock, tools.Razor, project.item.<id>, effects.item.gaussian_blur, panel.tab.Timeline, program.transport.playback.toggle.

Audio mixing: mixer.* commands (strips "A1", "S1", "Mix" or ids; lanes volume, pan, mute, send.<i>.level, fx.<slot>.<param>): mixer.inspect, mixer.setStrip (volume, pan, mute, solo, record arm, solo safe, mode Off/Read/Latch/Touch/Write, output, input map, channels), mixer.setValue, mixer.touch / mixer.release (a fader gesture; recorded during an automation pass), mixer.recordStart / mixer.recordStop (playback runs them), mixer.addSubmix, mixer.deleteSubmix, mixer.addInsert / removeInsert / setInsert, mixer.addSend / setSend / removeSend, mixer.setKeyframe / deleteKeyframe / moveKeyframe / clearLane, mixer.writeAutomation; clipMixer.set; clip.audioGain {clips?, mode: set|adjust|normalizeMax|normalizeAll, db}, clip.audioPeak; effects.setDefaultTransition. UI ids: mixer.<A1|S1|Mix>.<fader|value|pan|panValue|mode|mute|solo| record|soloSafe|output|input|fx.<n>|send.<n>|meter|name> (popup entries below them, e.g. mixer.A1.mode.Touch, mixer.A1.fx.0.studio_reverb), mixer.showEffects, mixer.transport.<cmd>, clipMixer.A1.<fader|pan|mute|solo|keyframe|value>, timeline.track.A1.keyframes[.<lane>|.clip], timeline.track.A1.lane[.kf.<n>], audioGain.<set|adjust|normalizeMax|normalizeAll|ok|cancel|peak>.

Essential Sound: essentialSound.inspect, essentialSound.setType {type: dialogue|music|sfx|ambience}, essentialSound.clearType, essentialSound.set {key, value} or {values: {key: value}} (keys such as repair.noise.on, repair.noise.amount (0–10), repair.humHz, clarity.eqPreset, clarity.enhanceTone, creative.reverbPreset, ducking.against, ducking.reduceDb, ducking.fadeS, pan.value, volume.levelDb, mute, section switches <section>.enabled), essentialSound.applyPreset {preset, type?}, essentialSound.savePreset {name}, essentialSound.deletePreset {name}, essentialSound.autoMatch {target?}, essentialSound.generateDucking; all take clips (default: the selected audio clips). UI ids: essentialSound.tab.<Browse|Edit>, essentialSound.type.<Dialogue|Music|SFX|Ambience>, essentialSound.clearType, essentialSound.preset[.save|.delete|.name|.ok], essentialSound.section.<Name>[.toggle], a row per setting key (essentialSound.repair.noise.on, essentialSound.repair.noise.amount, essentialSound.repair.humHz.<i>), essentialSound.autoMatch, essentialSound.ducking.against.<Type>, essentialSound.generateDucking, essentialSound.volume.on, essentialSound.volume.levelDb, essentialSound.mute, essentialSound.browse.<Type>.<preset>.

Project files, auto-save, crash recovery and preferences commands (file.recover, prefs.set, …) and their automation ids are listed in project-files.md. That file also lists the media management commands and dialog ids: offline media and relinking (media.findMissing, media.relink, media.autoRelink, media.search, media.makeOffline, media.status, linkMedia.*), proxies (media.createProxies, media.attachProxies, media.toggleProxies, proxies.*), ingest (project.ingestSettings) and the Project Manager (file.projectManager, pm.*).

Moving clips: timeline.move {moves: [{clip, track, time}], insert?, linked?} → {moved: [clip ids]}. With Linked Selection on (the default; linked overrides the toggle for one call) the linked partners of a listed clip that are not listed themselves move by the same time offset and stay on their own tracks, so picture and sound stay in sync. Like a listed clip, a partner overwrites what is at its destination (or inserts, with insert). A partner on a locked track stays where it is, and so does one whose position would not change (the listed clip only changed track). moved lists every clip that moved, partners included. No clip is moved before the sequence start: if one would be, all clips of the call land later by the same amount (their spacing is kept). A time beyond the representable range is an "invalid parameters" error. The timeline panel's drag passes linked: false, because it already lists exactly the clips it moves. With copy: true (Option-drag in the timeline) the clips stay where they are and copies land at the destinations instead (linked partners are copied too, unless linked: false); the result lists the new clips as copied, they are selected, and copies of linked clips are linked to each other.

Trim mode (crates/engine/src/trim.rs): trim.selectEditPoint / trim.selectNearest enter trim mode (the Program monitor becomes the Trim Monitor, ids trimMonitor.*); trim.monitor returns what it shows. Dynamic trimming takes an explicit clock (seconds) so it is deterministic: trim.shuttle {direction, slow?, clock} (J/L), trim.tick {clock}, trim.shuttleStop {clock?} (K, one undo step), trim.cancelDynamic (Esc), trim.playAround {clock, loop?} (Space / Shift+K).

Replace With Clip: clip.replaceFromSource, clip.replaceFromSourceMatchFrame and clip.replaceFromBin {clips?, item?, keepSourceIn?} swap the media of the clips and return {clips: [ids], item, short: [{clip, shortByFrames}]}. short lists the replaced clips that now run past the end of their media (the renderer holds the last frame there) with the number of sequence frames that have no media, at most the clip's length; it is empty when the new media covers every clip. The edit is the same either way. From Bin starts each clip at the new item's In point; with keepSourceIn: true each clip keeps its own source In instead. A subclip that restricts trims has no media outside its range, so there a kept In is moved to the subclip's first or last frame, and the result's sourceInClamped: [ids] (present with keepSourceIn) names the clips it was moved for.

Colour (crates/engine/src/color.rs): sequence.colorSettings {workingSpace: rec709|rec2100-pq| rec2100-hlg, wideGamut, autoToneMap}, clip.interpretFootage {items?, colorSpace?: auto|<id>, pixelAspect?: [num, den]|file}, color.spaces, media.colorInfo {item}; LUTs: lut.import {path, name?}, lut.list, lut.remove {id}, lut.export {lut, path, format?}, lumetri.setInputLut / lumetri.setLook {clip?, lut: lib:<id>|builtin:<id>|"", path?}, lumetri.setSection {clip?, section, on?}, lumetri.applyMatch {clip?, referenceTime|referenceFrame|referenceTimecode, faceDetection?}. Without params the menu entries open dialogs (ids colorDialog.space.<id>, colorDialog.working.<id>, colorDialog.wideGamut, colorDialog.autoToneMap, colorDialog.ok|cancel). Lumetri panel ids: lumetri.input_lut, lumetri.look_lut, lumetri.switch.<section>, lumetri.section.<section>, lumetri.match.*; HDR scopes: scopes.hdrWaveform. file.exportMedia {sdr: true} exports an HDR sequence as tone-mapped SDR.

Keyboard shortcuts (crates/engine/src/shortcuts.rs): shortcuts.list {query?, panel?}, shortcuts.get, shortcuts.set {command, keys, panel?, add?, keepConflicts?}, shortcuts.clear, shortcuts.undo / shortcuts.redo, shortcuts.conflicts {platform?}, shortcuts.forKey {key}, shortcuts.resolve {keys, panel?}, shortcuts.presets, shortcuts.loadPreset / savePreset / deletePreset {name}, shortcuts.export / import {path}, shortcuts.audit. Keys use Cmd (⌘ / Ctrl), Ctrl (macOS ⌃), Alt, Shift. The dialog (Edit ▸ Keyboard Shortcuts…, ⌥⌘K) uses ids shortcuts.* (shortcuts.key.K, shortcuts.cell.<command>, shortcuts.ok, …).

MCP

filmcraft-cli mcp serves MCP on stdio: headless (in-process session; --demo / --project p.fcproj) or --bridge 127.0.0.1:9876 to drive the running app. Tools: command_list, command_run, command_batch, doc_inspect, render_preview, project_inspect, sequence_inspect, media_import, render_frame, ui_inspect, ui_elements, ui_click, ui_drag, ui_key, ui_type, ui_screenshot, ui_control; resources filmcraft://document and filmcraft://commands. .mcp.json registers both. Annotations, argument checking, errors and export progress / cancellation: agents.md § Conventions.

Time is in ticks (254 016 000 000 per second); commands also accept seconds, frame or timecode.

Panel docking commands

ui.menu.invoke and engine.execute route these frontend commands:

Command	Parameters	Result
window.floatPanel	{panel?, rect?: [x,y,width,height]}	Move an existing panel into a floating group. Omitted panel uses the focused panel; omitted rect uses a centered 420×320 group.
window.dockPanel	{panel?, anchor?, zone?: "center"|"left"|"right"|"top"|"bottom", before?}	Move into the anchor's tab group or a side split; before names an existing destination tab for precise ordering. Floating destinations accept tabs only. Without anchor, a side zone docks the panel along that edge of the whole workspace, at a quarter of its width or height.
window.closePanel	{panel?}	Close a docked or floating panel without closing its project documents.

Invalid source/target identities and non-finite or out-of-range geometry return an error with the layout unchanged. ui.inspect.ui.floating reports floating groups and their logical-point rectangles. panel.drag.<Panel> locates each panel's header grip; panel.float.<index>.move and .resize locate floating geometry handles. Named workspace saves include these groups.

window.dockAction accepts {action} using the shared docking action's JSON form. It covers tab moves and ordering, split and floating geometry, activation, and closing. For example, {"action":{"ResizeSplit":{"path":[],"size":{"FixedFirst":360}}}} resizes the root split; path entries use false for the first child and true for the second. {"action":{"MoveFloating":{"panel":"Info","rect":[45,65,420,320]}}} repositions and resizes the floating group containing Info. Invalid actions leave the layout unchanged. Pointer gestures dispatch through this same catalog command.