mirror of
https://github.com/storytold/artcraft.git
synced 2026-11-15 22:32:13 +00:00
478 lines
41 KiB
Markdown
478 lines
41 KiB
Markdown
# CLAUDE.md — pagescene (3D Scene Editor)
|
||
|
||
Guidance for working in the `pagescene` library — the 3D editor (a.k.a. "pagescene" / "Stage3D").
|
||
|
||
## ⚠️ Engine-canvas mounting invariant (footgun)
|
||
|
||
`EngineProvider`'s editor-construction effect depends on three DOM nodes: `sceneContainer`,
|
||
`editorCanvas`, and **`camViewCanvas`** (owned by `PreviewEngineCamera` → `CameraViewCanvas`).
|
||
If ANY of these unmounts, the effect tears down + recreates the whole `Editor` (serializing to
|
||
cache, then reloading) — which wipes a fresh/unsaved scene. **Never conditionally unmount the
|
||
components that host these canvases** (`SceneContainer`, `EditorCanvas`, `PreviewEngineCamera`)
|
||
for mode/chrome gating — hide them with CSS (`hidden`) instead. (Regression fixed: record mode
|
||
had gated `PreviewEngineCamera` behind `!isRecord`, wiping the scene on record→build toggle.)
|
||
|
||
## Behavior notes (fixes)
|
||
|
||
- **Record mode is fully immutable.** The viewport lock (`CameraController.locked`, set by the
|
||
record effect) gates every camera-move path: FreeCam drag/wheel/keyboard (`useFreeCam`),
|
||
FreeCam integration (`tickPerFrame`), AND `MouseControls` mouse-look + `focus()` (gated via
|
||
`MouseControlsDeps.isViewportLocked`). If you add a new camera-move path, gate it on `locked`.
|
||
- **Default camera framing** (`DEFAULT_CAMERAS.main`): a Blender-style pulled-back/elevated 3/4
|
||
view (`pos (-4.5,4,6)`, `lookAt (0,0.5,0.6)`) so the whole render-camera (`cam2`) frustum is
|
||
visible on a blank scene.
|
||
- **Auto-keyframe**: mutating an **already-keyframed** object (gizmo drag-end or inspector commit)
|
||
calls `TimelineController.autoKeyIfTracked(uuid)` → `addKeyframe` at the playhead, which
|
||
replace-or-creates a keyframe at the current scrub point. No-op for never-keyframed objects
|
||
(first keyframe is always explicit).
|
||
|
||
## Architecture (existing, do not rebuild)
|
||
|
||
Strict one-way data flow. Do not violate it.
|
||
|
||
```
|
||
Three.js engine → typed event bus → EngineStoreBridge → Zustand store → React → actions → engine methods
|
||
```
|
||
|
||
- **`Stage3D.tsx`** — public entry component; takes an `adapter` + optional slots/callbacks.
|
||
- **`Stage3DBody.tsx`** — layout composition (canvas, panels, toolbars, modals). Mounts the external `PromptBox3D`.
|
||
- **`PageSceneStore.ts`** — the single Zustand store (~60 fields). All UI state. The engine never imports this store.
|
||
- **`adapter.ts`** — platform-abstraction interface. All host-specific logic (HTTP, upload, auth, navigation, modals) flows through `PageSceneAdapter`. Hosts (Tauri app, web app) inject it at mount.
|
||
- **`comps/Controls3D/Controls3D.tsx`** — top-center toolbar: `+` add, Move/Rotate/Scale (`transformMode`), World/Local (`transformSpace`).
|
||
- **`comps/Outliner/Outliner.tsx`** — left "Scene" panel (code name: Outliner). Driven by `outlinerItems`; each `OutlinerRow` has lock + visibility toggles.
|
||
- **`engine/editor.ts`** — main `Editor` class; owns subsystems (GizmoController, CameraController, HistoryManager, SelectionBridge) and the prompt text (`editor.positive_prompt`).
|
||
- **Undo/redo** — `HistoryManager` + command-pattern `UndoableAction` classes. Every mutation that should be undoable records an action.
|
||
|
||
Key boundary: the **bottom prompt bar is NOT in this library**. It is `PromptBox3D` from the external `@storyteller/ui-promptbox` package, mounted by `Stage3DBody`. Anything that changes the prompt bar crosses the pagescene ↔ ui-promptbox boundary.
|
||
|
||
---
|
||
|
||
## Feature: Scene Builder (IN PROGRESS — branch `feature/scene-builder`)
|
||
|
||
Status: **spec / in development.** This section describes intended behavior; not all of it is built yet. Images referenced are from Figma design frames.
|
||
|
||
### Top-level mode pill (net-new)
|
||
|
||
A segmented toggle at top-center (above the `Controls3D` toolbar) with two modes:
|
||
|
||
- **Build** — default. Scene composition/layout. Used unless the user switches.
|
||
- **Record** — read-only playback/output mode (see below).
|
||
|
||
Rule: a scene with **no animations/keyframes (static)** defaults to **Build**.
|
||
|
||
This mode does not exist in code today. It needs a new store field (e.g. `sceneMode: "build" | "record"`) and a new component rendered in `Stage3DBody`. Distinct from the existing `editorState` enum (`EDIT` | `CAMERA_VIEW`), which is a different axis — don't overload it.
|
||
|
||
### Launch re-scope — drop promptbox, lightbox handoff, editable duration
|
||
|
||
- **Promptbox removed**: `SceneBuilderPromptBox` deleted (MCP not ready). The **animation
|
||
timeline** is the sole build-mode bottom UI: collapsed `TimelineBar` by default, expanded
|
||
`TimelineEditor` via chevron. An empty timeline is auto-created on editor init
|
||
(`editor.timelineController.create()`), so the bar is always functional (no "Add timeline"
|
||
button). Auto-expands once on load when `timelineTracks.length > 0` or
|
||
`editor.sceneHasAnimation()` (imported THREE clips).
|
||
- **Editable max duration**: `TimelineController.setDuration(s)` (clamped `[1,60]`, covers 5–30s) +
|
||
`comps/Timeline/DurationLabel.tsx` (click the total-time → number input). Persists via `getTimeline`.
|
||
- **Capture/Record → review modal (appears first) → app Lightbox**:
|
||
- `CompletionModal` opens **immediately** for both kinds (`producedArtifact`) with a **local
|
||
preview** (object URL, works offline) + upload status. Images **auto-upload** on open; videos
|
||
show a manual **Upload** button (large). On success → `adapter.openMediaLightbox(token, kind)`
|
||
(destinations) + close; on failure → **Retry** (so the modal is always visible even when the
|
||
upload API is unreachable in dev). The modal — not `RenderOverlay` — owns the upload.
|
||
- `RecordControls`: Capture → produce image `producedArtifact`; Record → encode (overlay) →
|
||
produce video `producedArtifact`.
|
||
- Adapter: `openMediaLightbox(token, kind)` (host resolves cdnUrl via `MediaFilesApi` and renders
|
||
the app `<Lightbox>`); `uploadMedia` kept; `openImageInEditor`/`openVideoInEditor` **removed**
|
||
(the Lightbox owns destination routing — Edit-on-Canvas/Make-Video/Recreate/Share/Download).
|
||
- Uploading first means the Lightbox gets a real token + cdnUrl → its actions route correctly
|
||
(retires the earlier blob/data-URL handoff fragility).
|
||
|
||
### Spec revision — 3D delegates generation to 2D/Video (branch `feature/scene-builder`)
|
||
|
||
The 3D editor no longer generates. It composes; 2D (pagedraw) and the video experiences generate.
|
||
|
||
- **Build promptbox is now the scene-builder/MCP tool only** — `comps/SceneBuilderPromptBox/`
|
||
replaced `PromptBox3D` in `Stage3DBody` (the Manual/Prompted toggle + `buildMode` store field are
|
||
gone; `PromptBox3D` is no longer mounted by 3D). Update is still an MCP stub (records prompt +
|
||
history on the engine). Hosts the collapsed `TimelineBar` + Add-animation-timeline affordance.
|
||
- **Record mode = Capture (still) / Record (timeline → video)**:
|
||
- `engine/recording/TimelineRecorder.ts` — deterministic `seekTo`→`renderScene`→`CanvasSource.add`
|
||
encode via **mediabunny** (already a dep). Capture reuses `snapShotOfCurrentFrame(false)`.
|
||
- `comps/RenderOverlay/` (opaque `LoadingDots` + progress) covers the viewport while encoding.
|
||
- Output cached locally as `producedArtifact` in the store (`{kind, blob, objectUrl, ...}`) — no
|
||
upload from 3D.
|
||
- **`comps/CompletionModal/`** — preview hub: image full-preview / video playback + **Delete**
|
||
(revokes URL), **Upload** (`adapter.uploadMedia` → library token), **Edit** → 2D for images,
|
||
Create-Video / Video-editor for videos.
|
||
- **Adapter handoff (new)** in `adapter.ts`: `openImageInEditor`, `openVideoInEditor(target)`,
|
||
`uploadMedia` + `PageSceneArtifact`. Host-wired in `apps/artcraft-webapp/.../pagescene` :
|
||
image → eager-upload + `applyEditOnCanvasFromImage` (`/edit-image`); video "generate" →
|
||
`useCreateVideoStore.setPendingRecreate` reference-video (lazy upload at generate, `/create-video`);
|
||
video "edit" → upload + `/video-editor` (TODO: auto-drop on timeline).
|
||
- **Camera-view regression fixes** (Phase 0): Esc + viewport double-click exit; double-click the
|
||
frustum to enter; robust enter/exit pose handling (removed the dead `hot_items` toggle);
|
||
frustum **pick-proxy mesh** (line raycasting disabled) so selection matches the wireframe;
|
||
`CameraStatusPill` (top-left) shows Viewport vs render-camera and toggles.
|
||
|
||
Deferred: real MCP backend; pagedraw file-only (lazy) base image; video-editor timeline
|
||
auto-import; audio in recordings; `snapShotOfCurrentFrame` render-camera reconciliation.
|
||
|
||
### Persistence + undo (timeline & split camera)
|
||
|
||
- **Timeline persists in scene JSON**: `save_manager.getSceneJson` now writes `timeline:
|
||
getTimeline() ?? null` (was a `""` stub); `loadFromJson` restores via `loadTimeline` (guards
|
||
legacy `""`/null). Flows through both backend save and the unmount→cache→remount roundtrip.
|
||
Loaded objects keep their saved `object_uuid` (proxy load, `obj.uuid = json_object.object_uuid`),
|
||
so timeline tracks (keyed by uuid) resolve.
|
||
- **Render camera persists**: at save, the render-camera entry in `cameras[]` is overridden with the
|
||
**live `::CAM::` transform** (`getRenderCameraTransform` dep) so it reflects gizmo moves (the store
|
||
config otherwise goes stale). The load path strips `::CAM::` and never recreated it — fixed:
|
||
`loadFromJson` calls `recreateCameraObject()` (→ `Scene._create_camera_obj()`, now idempotent)
|
||
AFTER `CamerasReplacedEvent`, so the frustum lands at the restored transform. This also fixes the
|
||
standalone "load loses the render camera" bug.
|
||
- **Undo/redo**: timeline is **transactional** — `Save` is one undo step (`SaveTimelineAction`;
|
||
`loadTimeline` re-seeks + emits on undo/redo); `Cancel` reverts to last saved (not history).
|
||
Render-camera gizmo moves were already undoable (`TransformAction` on `::CAM::`, no special-casing);
|
||
because save reads the live `cam_obj`, a save after an undo persists the undone position.
|
||
Per-edit timeline undo remains deferred.
|
||
|
||
### Implementation status (branch `feature/scene-builder`)
|
||
|
||
- **DONE — Build/Record pill**: `comps/SceneModePill/` (reuses `@storyteller/ui-tab-selector`), mounted above `Controls3D` in `Stage3DBody`. Store: `sceneMode` (default `build`) + `setSceneMode`.
|
||
- **DONE — Manual/Prompted toggle + prompted layout**: store `buildMode: "manual" | "prompted"` (default `manual`) + `setBuildMode`. The toggle + prompted layout live in `PromptBox3D` (`@storyteller/ui-promptbox`), gated by a new optional `buildMode` / `onBuildModeChange` / `onAddTimeline` prop trio passed from `Stage3DBody`. Toggle = a `ButtonIconSelect` (Manual = pointer icon, Prompted = wand-sparkles) in a floating glass pill above the prompt card. Prompted layout hides the manual toolbar and shows: textarea + "Add animation timeline" (`onAddTimeline`, stubbed) + prompt-history popover (`usePrompt3DStore.promptHistory`) + **Update**. `handlePromptedUpdate` is a **stub**: records the prompt on the engine + pushes to history + toasts; it does NOT call the MCP backend yet.
|
||
- **DONE — Animation timeline (in-memory)**: keyframe system + playback + full UI.
|
||
- Data model: `engine/timeline/types.ts` (`Keyframe`/`TimelineTrack`/`TimelineData`, `EasingSpec` = cubic-bezier ctrl pts, presets) + `interpolation.ts` (`cubicBezierYForX`, `sampleTrackAt`). Reuses `TransformSnap`/`snapshotTransform`/`writeTransform`.
|
||
- `engine/editor/TimelineController.ts` owns the timeline; ticked in `editor.ts` `renderSingleFrame()` after `entranceAnimator`. **Writes transforms only while playing or on seek** (idle never touches objects, so manual editing is safe).
|
||
- Events `TimelineChangedEvent`/`TimelinePlayheadEvent` → `EngineStoreBridge` → store fields (`timelineExists/Expanded/Playhead/IsPlaying/Duration/Tracks/SelectedKeyframeId`). Dispatchers in `actions/timeline.ts`.
|
||
- Undo: `engine/editor/actions/SaveTimelineAction.ts`; `Save` records it, `Cancel` reverts to last saved.
|
||
- UI in `comps/Timeline/`: `TimelineBar` (collapsed, via promptbox `aboveStackSlot`), `TimelineEditor` (expanded, replaces the prompt box; `Stage3DBody` hides `PromptBox3D` when `timelineExpanded`), `TimelineTrackRow` (diamonds, drag, add), `MotionPopover` (presets + draggable cubic-bezier). `onAddTimeline` → `createTimeline` + expand; `showAddTimelineButton={!timelineExists}` hides the promptbox button once a timeline exists.
|
||
- **In-memory only** — not persisted to scene JSON yet (reload clears). Rotation is euler-lerped (quaternion slerp is a possible refinement). Camera track is absent until the camera-split slice creates `::CAM::`.
|
||
- **DONE — Camera split (single camera) + camera view**: the render camera is now a selectable
|
||
wireframe-frustum placeholder in the scene.
|
||
- `scene.ts` `_create_camera_obj()` rewritten to build a `THREE.Group` named `"::CAM::"` (line
|
||
frustum, layer 1 = editor-only/excluded from render), positioned at the first non-`main`
|
||
store camera; called in `initialize()`/`clear()`; `refreshCamObj` resolves it. No
|
||
`userData.media_id` → the save path (`saveToScene`, which only serializes `media_id` children)
|
||
skips it, so scenes aren't corrupted.
|
||
- Enter/exit camera view reuses the existing `CameraController.switchCameraView()` +
|
||
`tickPerFrame` sync (canvas renders through the viewport camera, which is moved to the render
|
||
camera — no render-path change). Trigger: new `CameraViewToggleRequestedEvent` → `editor.ts`
|
||
subscription → `switchCameraView()`. Emitted by (a) `scene_manager_api.ts` `double_click()`
|
||
when the selected object is `::CAM::` (covers viewport + outliner-row double-click), and
|
||
(b) `actions/cameraView.ts` `toggleCameraView` (outliner icon + exit button).
|
||
- Outliner: `OutlinerItem.isCamera` (set in `convert_object`) → `Outliner.tsx` renders a
|
||
view-from-camera button (`faArrowRightToBracket`) left of the lock icon.
|
||
- `comps/ExitCameraView/` shows the "EXIT CAMERA VIEW" pill while `editorState === CAMERA_VIEW`.
|
||
- **DONE — Record mode**: `sceneMode === "record"` is read-only playback through the render camera.
|
||
- `Stage3DBody` effect on `isRecord`: `cameraController.enterCameraView()` + `setLocked(true)` on
|
||
entry; `setLocked(false)` + `exitCameraView()` on exit. `switchCameraView()` was refactored into
|
||
idempotent `enterCameraView()`/`exitCameraView()`; new `CameraController.locked` gates the
|
||
FreeCam integration in `tickPerFrame` (viewport can't be flown in record).
|
||
- Build chrome hidden in record (ControlsTopButtons, Controls3D, top-right cluster, Outliner,
|
||
PreviewEngineCamera, ControlPanelSceneObject, PromptBox3D, TimelineEditor). The **mode pill
|
||
stays**. `ExitCameraView` is suppressed in record (the pill leaves record).
|
||
- `comps/RecordControls/`: bottom-center read-only `TimelineBar` (`readOnly` hides the expand
|
||
chevron — playback only, no editing) + **Capture** (`editor.snapShotOfCurrentFrame(true)` — still
|
||
image download) and **Record** (red; **stub** — no video pipeline in pagescene yet).
|
||
- **NOT STARTED**: timeline persistence to scene JSON; natural-language motion clips; multi-camera;
|
||
render-camera transform round-trip to store `cameras[]`; **video render behind the Record button**;
|
||
the record-mode "expand timeline into a grayed-out read-only editor" (currently the expand chevron
|
||
is simply hidden in record).
|
||
|
||
### Feature: Mixamo animation clips on the timeline (IN PROGRESS)
|
||
|
||
Characters can carry skeletal (Mixamo) animation clips on the timeline, alongside the transform-
|
||
keyframe tracks. Answers locked with the user: retarget onto the character's skeleton; clips +
|
||
keyframes coexist; default clip source = the 37 curated demo clips. **Playback is sequential, one
|
||
clip at a time** — a character's clips live on a **single row** and cannot overlap (see the overlap
|
||
guard below). The earlier "stacked lanes" idea was dropped: stacking only pays off for *additive
|
||
layering* (e.g. a base walk + an upper-body wave playing together), which needs per-lane weights +
|
||
bone masking and is out of scope. If layering is ever wanted, that's the feature to build; until
|
||
then one row per character is the right model.
|
||
|
||
- **Data model (phase A, done)**: `engine/timeline/types.ts` — `ClipStrip` (`sourceMediaId`,
|
||
`startTime`, `duration`, `loop`) + `ClipLane` (`objectUuid`, `strip`); `TimelineData.clipLanes[]`.
|
||
Only the reference + placement is serialized; the `AnimationClip` is resolved at runtime.
|
||
`TimelineController` has `addClipLane` / `moveClipLane` / `removeClipLane` / `clipLanesFor`;
|
||
`TimelineChangedEvent` carries `clipLanes` → store `timelineClipLanes`. Persists via the existing
|
||
`getTimeline`/`loadTimeline` (rides `TimelineData`).
|
||
- **Playback engine (phase B, done — UNVERIFIED at runtime)**: `engine/animation/CharacterAnimationManager.ts`.
|
||
One `THREE.AnimationMixer` per character; each lane → an `AnimationAction`. Playhead-driven +
|
||
**deterministic**: on every `TimelineController.evaluate()` it sets each action's absolute `time`
|
||
from the playhead (paused actions) then `mixer.update(0)` — scrub, play, and frame-accurate record
|
||
all pose identically. `syncClipLanes()` reconciles the runtime on every clip-lane change; a
|
||
monotonic load token guards against superseded async clip loads. Clips are loaded via
|
||
`Scene.loadRawGlb(media_id)` (raw GLTF, not added to the scene) and bound **by node name** —
|
||
relies on the shared `mixorig:*` naming so a clip's tracks resolve against the character's bones
|
||
directly (direct-bind first; `SkeletonUtils.retargetClip` is the documented fallback if a rig uses
|
||
different bone names — **not yet wired**, pending the runtime spike).
|
||
- **Trigger UI (done — moved)**: the old right-docked `comps/AnimationsDrawer/` is retired. The
|
||
clip source is now the **"Animations" tab in `comps/AssetMenu/AssetModal.tsx`**: server-curated
|
||
animations (`list_featured` with `engine_category=animation`, GLB/GLTF only) REPLACE the 37
|
||
`demoAnimationItems` when the API returns any; the demos are only the empty-API fallback.
|
||
Animation cards are **click-to-add** (earliest free slot on the selected character's row; toast
|
||
if no character is selected or the row is full) **and pointer-draggable** via the shared
|
||
`DndAsset` singleton (`ANIMATION` branch): drop on a character in the 3D scene (earliest free
|
||
slot) or on a character's timeline row (pointer-x → time via the `data-timeline-ruler` rect;
|
||
rows tag `data-clip-drop-uuid`). While dragging, the shared `DragGhost` shows a green/red
|
||
**compatibility badge** (`PageSceneStore.animationDropState`): DndAsset preloads the clip's
|
||
track node names (`loadRawGlb`, cached per media id) and checks them against the hovered
|
||
character's nodes — the same direct-bind rule `CharacterAnimationManager` plays by; no
|
||
retargeting. Incompatible or slot-less drops are rejected with a toast. A drop that beats the
|
||
clip's name-inspection fetch is **queued** and completes (fully validated) when the names
|
||
resolve — never punted back to the user (`completeAnimationDrop`; a re-drop of the SAME clip
|
||
supersedes its earlier queued drop via a per-clip token, while queued drops of other clips
|
||
complete independently). The old HTML5 DnD path (`ANIMATION_CLIP_MIME`) is removed. A full row
|
||
(`resolveFreeStart` → null) **rejects** the add instead of overlapping.
|
||
- **Standalone AnimationsModal (done)**: `comps/AssetMenu/AnimationsModal.tsx` — an
|
||
animations-only sibling of `AssetModal` (same `@storyteller/ui-modal` floating-panel props,
|
||
same `ItemElements` grid, same drag-under semantics) with All / Presets / Uploaded tabs.
|
||
Opened by the **"Add Animation" button** next to "Enter Pose Mode" (`PoseModeSelector`,
|
||
visible when a character is selected and not posing) and by the **"Animations" entry in the
|
||
Controls3D "+" popover** (not auth-gated, like "presets"), both via
|
||
`actions/openAnimationsModal.ts`.
|
||
The two library modals are **mutually exclusive** (each open action closes the other), so
|
||
`DndAsset.endDrag` and `ItemElement`'s click-add close whichever is visible (reopen-off
|
||
path). Animations data (uploads-first + featured-else-demos fallback) lives in the shared
|
||
`comps/AssetMenu/hooks/useAnimationLibrary.tsx`, consumed by both modals.
|
||
**Upload from the modal**: an "Upload animation" button (sidebar bottom, above "Reopen
|
||
after adding") plus an empty-state CTA on the Uploaded tab open the host uploader through
|
||
the same `renderAssetUploader` adapter slot AssetModal uses. The slot (and `UploadModal3D`)
|
||
gained an optional `initialCategory?: "animation" | "character"` (literal strings — the
|
||
pagescene and upload-modal `FilterEngineCategories` enums aren't cross-assignable): it
|
||
pre-checks the matching "Upload as ..." toggle and suppresses the My-Library popup on
|
||
success (the panel shows the result itself). On success the modal refetches user
|
||
animations and jumps to the Uploaded tab (only if the category stayed Animation).
|
||
Anonymous users get `promptSignup("upload-3d")` instead, same as the Controls3D entries.
|
||
- **Timeline reveal on add (done)**: a successful `addClipToCharacter` (click or drop, either
|
||
modal) sets `timelineExpanded` + `timelineRevealObjectUuid`; an effect in `TimelineEditor`
|
||
scrolls the row list to `[data-timeline-row-uuid]` (every row is tagged) then clears the
|
||
field. A store field because `TimelineEditor` is unmounted while collapsed — expand + scroll
|
||
can't happen in the caller's tick.
|
||
- **Timeline focus toggle (done)**: crosshair button in the `TimelineEditor` transport bar
|
||
(`timelineFocusSelected`, store-held so it survives the collapse/expand remount) narrows the
|
||
row list to the selected object's track. Falls back to all rows when nothing is selected (or
|
||
the selection has no outliner row), and the reveal effect switches focus off when a clip
|
||
lands on a non-selected object so the new strip is never hidden.
|
||
- **Timeline clip UI (phase C, done)**: `comps/Timeline/TimelineClipRow.tsx` renders **one row per
|
||
character** (rendered even when empty, as a drop hint) holding all that character's strips
|
||
end-to-end under its keyframe row in `TimelineEditor`. Strip body drags to **move**
|
||
(`moveClipLane`), the right edge drags to **trim** length (`resizeClipLane`), a loop chip toggles
|
||
repeat (`setClipLoop`), the × (selected strip only) removes it (`removeClipLane`, undoable — see
|
||
the selection bullet below). Clip-eligible rows are drop targets via `data-clip-drop-uuid` (see
|
||
the DnD bullet). Other clip edits (move/trim/loop) ride the timeline **Save/Cancel** session
|
||
exactly like keyframes (`cancel()` restores + re-syncs).
|
||
- **Overlap guard (single row)**: `TimelineController.resolveFreeStart` snaps add/move to the nearest
|
||
gap so a character's strips never overlap; `nextStartAfter` bounds trim (`resizeClipLane`) and
|
||
auto-length (`resolveClipDuration`) against the following clip. `evaluateAt` then has at most one
|
||
active clip at any playhead, so playback is an unambiguous sequence.
|
||
- **Runtime revalidation (loop-free)**: mixers/rest poses hold object-INSTANCE refs, but
|
||
delete→undo and in-session scene reloads recreate objects under the same uuids. Every lane
|
||
runtime lands in a TERMINAL state — bound, missing-object (`boundRoot: null`), or unbindable
|
||
(rig-less/clip-less target; one warn, no fetch) — and `CharacterAnimationManager.revalidate()`
|
||
(subscribed to `OutlinerRefreshedEvent` alongside the skeleton-helper sync) reassesses a lane
|
||
ONLY when the live top-level root instance for its uuid differs from `boundRoot` (object
|
||
appeared/vanished/replaced). One map lookup per lane per refresh; no traverses, timers, fetch
|
||
retries, or per-refresh re-adds. A uuid-resolvable root is always fully loaded (all load paths
|
||
stamp the uuid post-load), which is what makes rig-lessness a terminal verdict. Skeleton-helper `sync()` self-heals the same two cases: it purges and
|
||
recreates helpers that are orphaned (scene reloads strip children; the map survives) or whose
|
||
`helper.root` no longer matches the live object for that uuid.
|
||
- **Rest-pose in gaps** (was bind-pose): a disabled three.js action leaves the skeleton frozen on
|
||
its last frame, so `evaluateAt` resets any character with no clip under the playhead to a **rest
|
||
pose captured at mixer creation** (`captureRestPose` — local transforms of every node BELOW the
|
||
root; the root is excluded so gizmo/keyframe placement never snaps back). `Skeleton.pose()` was
|
||
abandoned: it rebuilds bone locals from inverse bind matrices and mis-scales rigs whose armature
|
||
carries a baked scale (mixamo cm-rigs), which made characters visually VANISH when a strip moved
|
||
off the playhead. `pruneMixers` also restores the rest pose before dropping a character's mixer.
|
||
- **Drag preview**: animation drags ride the same pointer-based `DndAsset` pipeline as every
|
||
other asset card, so the shared `DragGhost` (tilt card) works unchanged — plus the
|
||
compatibility badge described above.
|
||
- **Clip-row eligibility (widened)**: rows are no longer characters-only. Any **skinned** object
|
||
(creatures, rigged uploads — `OutlinerItem.hasSkeleton`, computed in `convert_object`) accepts
|
||
animation drags (bind check still gates), and any object with **baked clips**
|
||
(`OutlinerItem.bakedClips`, from `object.animations`) gets a clip row even without a skeleton.
|
||
- **Baked-in clips**: `ClipStrip.bakedClipIndex` marks a strip sourced from the object's own
|
||
`animations[]` (with `sourceMediaId: ""`); `CharacterAnimationManager.addLane` resolves it
|
||
synchronously from the model instead of `loadRawGlb`, and **CLONES the clip per lane** —
|
||
`mixer.clipAction` caches one action per (clip, root), so two strips of the same raw baked
|
||
clip would fight over a single action and `uncacheClip` on removal would kill the sibling's. Added via the **"+" picker** on the
|
||
object's clip row (`TimelineClipRow`, `addBakedClipToObject`, earliest free slot, real length
|
||
known up front so no `autoDuration`). INVARIANT: baked clips are **never removed from the THREE
|
||
model** — removing a baked strip only unschedules it (mixer uncache is mixer-state only); the
|
||
clip stays on `object.animations` and in the picker. Library-sourced strips remain fully
|
||
removable. ⚠️ Runtime-unverified: whether `object.animations` survives the save/load roundtrip
|
||
(the stash in `scene.ts` is gated on `auto_add`), and that the outliner refresh timing surfaces
|
||
`bakedClips` after async GLB loads.
|
||
- **Strip width = play window (revised)**: fresh strips default to a compact
|
||
`DEFAULT_CLIP_DURATION` (1 s) instead of adopting the clip's natural length — long clips were
|
||
eating the whole row. The strip's on-timeline width is the **authoritative play window**
|
||
(`LaneRuntime.stripDuration` in `evaluateAt`); the clip's natural length only drives the loop
|
||
modulo / final-frame clamp inside it, so trimming a strip shorter genuinely cuts playback.
|
||
Strips default to **loop: true** — dragging a strip wider than the clip keeps the motion
|
||
cycling (a walk keeps walking, per tester expectation/old-studio behavior); the loop chip opts
|
||
a strip into play-once/hold-last-frame instead. `resolveClipDuration` only SHRINKS an
|
||
`autoDuration` strip when the clip is shorter than the default.
|
||
- **Gap transitions (opt-in blending)**: `ClipStrip.transitionEasing` on the LEADING strip
|
||
(presence = enabled; mirrors keyframes storing easing "into the next") cross-fades the strip's
|
||
exit pose into the next strip's entry pose across the gap between them. `evaluateAt` enables
|
||
BOTH actions pinned at their exit/entry frames with weights `(1−w, w)` from
|
||
`cubicBezierYForX` — weights sum to 1 so the mixer computes a proper lerp; this is the one
|
||
deliberate exception to the one-winner rule, and it's continuous at both gap edges (w=0 at the
|
||
leading strip's end, w=1 at the next strip's start). Gaps without a transition keep the rest
|
||
pose; leading/trailing row gaps never blend. UI: a `data-transition-chip` button at EVERY
|
||
consecutive-strip boundary (ghost = off, solid = on) — auto-placement packs strips flush, so
|
||
enabling on a flush/too-tight boundary first calls `ensureTransitionGap`
|
||
(`DEFAULT_TRANSITION_GAP` = 0.3s: shift the next strip right, bounded by its follower/timeline
|
||
end, then trim the leading strip if shifting isn't enough; a too-packed row rejects with a
|
||
toast and mutates nothing). Click enables with `DEFAULT_EASING` and opens the generalized
|
||
`MotionPopover` (`easing`/`onChange`/`footer` props — keyframes use the same component) with a
|
||
"Remove transition" footer; popover selection rides `timelineEasingClipLaneId` — bridge-pruned
|
||
both when the lane vanishes AND when it loses its transition with the id intact (Cancel /
|
||
undo-of-Save revert the easing wholesale; an open popover would otherwise re-create it on the
|
||
next curve drag) — and the keyframe popover takes render priority so the two never coexist.
|
||
Edits ride the Save/Cancel session; serialization needs no special casing (`transitionEasing`
|
||
rides `TimelineData` through every wholesale-clone path: scene JSON save/load, Save/Cancel
|
||
snapshots, SaveTimelineAction, RemoveClipLaneAction restore).
|
||
- **Strip selection + delete UX**: strips are click-to-select
|
||
(`timelineSelectedClipLaneId` store field, `data-clip-strip` exempts the pointerdown from the
|
||
click-away deselect). The **× renders only on the selected strip** (always-visible was too easy
|
||
to hit); **Del/Backspace** removes the selected strip. Removal (either path) is **individually
|
||
undoable** via `RemoveClipLaneAction` — the one exception to "clip edits ride Save/Cancel" —
|
||
and to keep that honest it is **mirrored into the `saved` snapshot** (removeClipLane strips the
|
||
lane from `saved` too, so Cancel can't resurrect a deleted strip and Save doesn't double-record
|
||
the removal); the action captures both the live lane and its saved-snapshot entry so undo
|
||
restores both worlds exactly (`restoreClipLane(lane, savedLane?)`).
|
||
- **Rig-mismatch diagnostic**: `CharacterAnimationManager.clipBindsToCharacter` warns (console) when
|
||
a clip's tracks resolve to **0** nodes on the character — the signal that `SkeletonUtils.retargetClip`
|
||
is needed. Retarget itself is still **not wired** (direct-bind first).
|
||
- **⚠️ Runtime unknown to verify**: whether the demo clips and character rigs share `mixamorig:*`
|
||
bone names (→ direct bind works) or need retargeting. Drop/scrub a clip and watch for motion (and
|
||
the console warn) to decide whether to wire the retarget fallback.
|
||
|
||
### ⚠️ Unverified — compiles but NOT runtime-tested
|
||
|
||
Everything in slices 1–3 has been validated **only** by `tsc -b` (both `pagescene` and
|
||
`promptbox` exit 0). The app has **not** been launched; no visual, behavioral, lint, or test
|
||
verification has been done. Outstanding items to verify live:
|
||
|
||
- **Slice 1 (pill)**: placement/spacing above `Controls3D` (possible double `pt-3`); white-active
|
||
styling vs Figma; that clicking actually flips `sceneMode`.
|
||
- **Slice 2 (prompted)**: Manual/Prompted toggle renders/positions correctly above the card;
|
||
prompted layout hides the toolbar and shows textbox + add-timeline + history + Update; Update
|
||
stub toasts + records history; history popover lists/restores prompts; toggle only appears when
|
||
the host wires `onBuildModeChange`.
|
||
- **Slice 3 (timeline)**: nothing behaviorally verified — playback actually interpolates objects;
|
||
scrub/seek updates transforms; add/move/delete keyframes; Save+undo / Cancel; collapsed bar
|
||
positioning via `aboveStackSlot`; expanded editor replacing the prompt box; **playhead-line
|
||
alignment** (the `LANE_LEFT=8.75rem`/`LANE_RIGHT=2.5rem` offsets are hand-computed guesses);
|
||
`MotionPopover` bezier drag coordinate math + popover anchoring; range-scrubber styling; euler
|
||
rotation lerp quality; the assumption that the render loop stays alive during playback
|
||
(`shouldRender`); "selected object gets an empty track row" behavior.
|
||
|
||
- **Slice 4 (camera split)**: nothing runtime-verified — that the frustum appears at the render
|
||
camera and is excluded from the render; that selecting/moving it moves the render framing;
|
||
double-click (viewport + outliner row) and the outliner icon enter camera view; the aspect
|
||
frame + focal label show in camera view; EXIT returns to the prior free-cam position; the
|
||
frustum is/ isn't viewport-click-selectable (**may need `raycaster.layers.enable(1)`** in
|
||
MouseControls — outliner selection works regardless); `::CAM::` really is skipped by save.
|
||
|
||
- **Slice 5 (record mode)**: nothing runtime-verified — that switching the pill to Record enters a
|
||
locked render-camera view (viewport truly can't be flown), hides all build chrome but keeps the
|
||
pill, and shows the read-only playback bar + Capture/Record; that Capture downloads the composed
|
||
frame; that switching back to Build restores the prior free-cam view and chrome; interaction
|
||
between record and a manually-entered camera view; whether `switchCameraView` refactor (idempotent
|
||
enter/exit) preserved the original toggle behavior. Record button is a **no-op stub**.
|
||
|
||
Treat all of the above as needing a live pass before shipping.
|
||
|
||
### Build mode has two usability sub-modes
|
||
|
||
- **Manual** — identical to current behavior. User manually moves/places/transforms objects with the gizmo and toolbar. This is what exists today.
|
||
- **Prompted (Builder)** — the prompt bar changes to show only:
|
||
- a text box,
|
||
- an **"Add animation timeline"** button,
|
||
- a **history** button (previous prompts),
|
||
- an **"Update"** primary button.
|
||
|
||
Prompted mode drives a **JSON-style, MCP-esque backend integration** (in development). The flow: an LLM (e.g. Claude) reads the scene's **scene-descriptor data** + the user's prompt, makes edits, **outputs a new scene descriptor**, which is then applied to the 3D scene.
|
||
|
||
The **"Add animation timeline"** button only appears when the scene is **static** (no keyframes/animations on any object). Clicking it reveals the animation timeline bar (see below).
|
||
|
||
### Animation timeline
|
||
|
||
**Collapsed state** (a slim bar that sits above the prompt box, replacing the "Add animation timeline" button once a timeline exists):
|
||
- ▶ play/pause · current-time readout (`0:00`) · scrubber (blue fill to playhead) · total duration (`0:10`) · **chevron ▾**.
|
||
- The **chevron ▾ expands the timeline downward, collapsing the prompt box** (they trade vertical space).
|
||
|
||
**Expanded state** (full multi-track keyframe editor):
|
||
- **Transport (top-left):** skip-to-start ⏮ · play ▶ · skip-to-end ⏭.
|
||
- **Time ruler** with a draggable **playhead** (`0:00 … 0:04 … 0:10`).
|
||
- **One track per object.** A track can hold either:
|
||
- **Diamond keyframes** — each diamond stores the object's **full transform (position, rotation, scale)** at that moment. Tapping a diamond jumps the playhead to it. Each row has an "add keyframe" diamond button at its right edge.
|
||
- **Natural-language motion clip** — a text clip spanning a time range (e.g. *"walks to car - opens door - gets in"*) that drives generated motion.
|
||
- **Any object can have either type.** For simplicity, an early version may show only raw keyframes and drop the natural-language grouping.
|
||
- **Motion popover** — appears when a keyframe is selected; edits the **easing/interpolation curve into the next keyframe**. Contains ~4 **easing presets** at top plus a large **editable bezier curve** with draggable **handles**; a **dotted line** shows the linear reference.
|
||
- **Cancel / Save** (bottom-right):
|
||
- **Cancel** — drops all changes back to the last saved timeline state.
|
||
- **Save** — commits the changes. **This is an undoable action** (record it via `HistoryManager` / an `UndoableAction`).
|
||
|
||
### Camera model change (Blender-style)
|
||
|
||
Introduce a distinction between:
|
||
|
||
- **Viewport (scene) camera** — what the user flies around with; the editing view.
|
||
- **Render camera(s)** — the camera(s) that define the final render output.
|
||
|
||
Render cameras become **objects within the 3D scene**: visible/selectable in the editor (with a camera gizmo/frustum, as in the Figma), but **excluded from the final render**. This mirrors professional 3D software (Blender).
|
||
|
||
Render cameras appear in the **Outliner** (left "Scene" panel). A camera row shows a **"view from camera"** icon positioned **right of the camera item's name and left of the lock icon** (visible in the reference frames). Clicking it **switches the viewport camera to the selected render camera** and lets the user control it with the usual navigation keybinds.
|
||
|
||
**This is the big architectural change** — but the engine is **already partially scaffolded** for it. Verify these anchors before building on them:
|
||
|
||
- **Two cameras already exist** in `engine/editor/CameraController.ts`: `camera` (viewport/free-fly, sees layer 1 incl. gizmos) and `render_camera` (layer 0 only, clean output, its own aspect ratio). Created in `editor.ts` `initialize()` from the store's camera configs.
|
||
- **`Camera` type** lives in `frontend/libs/common/src/lib/interfaces/Camera.ts`: `{ id, label, focalLength, position, rotation, lookAt }`. Store seeds two: `"main"` (Main View, viewport) and `"cam2"` (Camera 2, render). So the store models multiple camera *configs*, but only **one** `render_camera` THREE object is currently instantiated (first non-main).
|
||
- **Placeholder object `"::CAM::"`** is referenced throughout (`scene.getObjectByName("::CAM::")`, `CameraController.refreshCamObj`, layer 1 so it's editor-only/excluded from render) — **but `_create_camera_obj()` is commented out** in `scene.ts`, so no camera mesh is actually in the scene graph today. The Outliner already knows to render a `::CAM::` object as a "Camera" row (`scene_manager_api.ts` `convert_object`) — it just never appears because the object isn't created.
|
||
- **`CAMERA_VIEW` state** (`enums/EditorState.ts`) + `CameraController.switchCameraView()` already implement enter/exit: on enter it copies `cam_obj` transform → viewport camera, hides `cam_obj`/objects/gizmo; on exit it restores. **Caveat:** today it *repositions the single viewport camera* rather than truly swapping to render through a separate scene camera. `PreviewEngineCamera.tsx` + an unused `camViewCanvasEl` exist as a secondary preview surface.
|
||
- **Letterbox/aspect overlay** already built: `comps/SceneContainer/Letterbox.tsx` reads `cameraAspectRatio`, draws mattes, toggled via `editorLetterBox`. `CameraAspectRatio` enum + `getRenderDimensions()` map to pixel sizes.
|
||
- **Focal length** is per-camera (`Camera.focalLength`), converted via `CameraController.focalLengthToFov()`, applied per-frame in `tickPerFrame()`; `focalLengthDragging` tracks the slider. The `35mm` overlay label maps to this.
|
||
- **Render path**: `editor.ts` `renderScene()` renders `activeScene.scene` through `render_camera`; but `snapShotOfCurrentFrame()` currently snapshots through the **viewport** `camera`, not `render_camera` — reconcile so exports use the render camera.
|
||
|
||
**Gaps to close for the true split:** (1) actually create camera placeholder meshes as scene objects (uncomment/rework `_create_camera_obj`); (2) map `cameras: Camera[]` 1:1 to scene objects; (3) make `CAMERA_VIEW` swap which camera the viewport renders through (not just reposition one); (4) support N render cameras, not just the first non-main; (5) make cameras keyframeable timeline targets. Reconcile all of this with the existing `cameras[]` + `selectedCameraId` store state.
|
||
|
||
### Camera view
|
||
|
||
Entered by **double-clicking the camera placeholder in the viewport** or the **view-from-camera button in the outliner**. In this state the **viewport camera becomes the selected render camera** — the user flies it with the usual navigation keybinds, and that motion is what gets keyframed.
|
||
|
||
- **Framing overlay**: the render **aspect frame** with **rule-of-thirds** guides; everything **outside the frame is dimmed** (won't be captured).
|
||
- **Focal-length label** (e.g. `35mm`) shown top-left inside the frame. Editable; may become a keyframable value in the future.
|
||
- The camera's frustum is not drawn while looking through it.
|
||
- **Timeline filters to just the selected camera's track** while in camera view.
|
||
- **Selected keyframe** renders as a **blue diamond** (at the playhead); other keyframes are white.
|
||
- **Exit** via the **EXIT CAMERA VIEW** button (bottom-right), or **Esc** / **double-click out**. (The text button is a dev placeholder; a proper icon comes later.)
|
||
|
||
Camera keyframes store the camera's transform just like object keyframes (and later, potentially focal length).
|
||
|
||
### Record mode
|
||
|
||
Record mode is the **read-only output/preview** mode — practically **immutable**. It plays back the render timeline **as seen from the main render camera** only.
|
||
|
||
- **All build chrome is hidden**: no outliner, no transform toolbar, no prompt box, no gizmos.
|
||
- The **viewport is locked to the render camera** (no free-fly, no object/camera editing). Framing overlay (aspect frame, dimmed exterior) is shown.
|
||
- **The only interactive controls are play/pause and scrubber position.** Nothing in the scene can be mutated.
|
||
- The **chevron can still expand the timeline**, but it renders **grayed out / read-only** — no keyframe editing.
|
||
- Two actions:
|
||
- **Capture** — export the **current frame as a still image**.
|
||
- **Record** — render the **full animation to video**.
|
||
|
||
### Build ↔ Record relationship (summary of axes)
|
||
|
||
- **Top pill**: `sceneMode` = `build | record` (net-new).
|
||
- **Within Build**: a usability sub-mode = `manual | prompted`.
|
||
- **Camera view** is an orthogonal viewport state (looking through a render camera) available while editing — not a top-level mode.
|
||
- Keep these distinct from the pre-existing `editorState` (`EDIT | CAMERA_VIEW`) and `poseMode` (`select | pose`) fields; do not overload them.
|
||
|
||
### Notes / deferred
|
||
|
||
- **JSON scene descriptor / MCP backend contract** is out of scope for the UI work — the UI and other features can be built independently. The one thing that won't function until the backend lands is the **Update button in Prompted mode** (acceptable for dev).
|
||
- The **view-from-camera icon** is already shown in the reference frames (outliner, right of the camera name, left of the lock icon) — no separate spec needed.
|
||
- **EXIT CAMERA VIEW** text button is a dev placeholder; a proper icon comes later.
|