screencastgen

Architecture

High-level design of the screencastgen system.

Layer Diagram

flowchart TB
    CLI[CLI] --> Pipelines
    Web[FastAPI application] --> Tasks[Celery pipeline tasks]
    Tasks --> Pipelines
    Server[Inference server] <--> Pipelines

    subgraph Pipelines[Shared pipelines]
        Audio[Audio]
        Highlight[Highlight]
        LipSync[Lip-sync]
        Visualization[Visualization]
    end

    Pipelines --> Core[Document, audio, alignment, and video core]
    Core --> Providers[Provider registries]
    Providers --> TTS[TTS]
    Providers --> Alignment[Alignment]
    Providers --> LipProvider[Lip-sync]
    Providers --> VisualProvider[Visualization]

Design Principles

Deferred Imports

Heavy ML dependencies (torch, whisperx, moviepy, qwen-tts) are imported inside functions, never at module top level. This lets screencastgen --help and Doctor run without loading every optional model dependency. Every new provider must follow this pattern.

Provider Registries

TTS, alignment, lip-sync, and visualization backends are pluggable via registries/factories in screencastgen/providers/. Heavy providers use deferred imports so modules are only loaded when they are actually created.

TTS Registry — create_backend(name, **kwargs)
Alignment Registry — align_with_provider(provider, audio_path, text, ...)
Lipsync Registry — run_lipsync_provider(provider, video_path, audio_path, ...)
Visualization Registry — create_renderer(name)

Resumable Processing

The Tracker (ProcessingTracker) persists chunk-level state to a JSON file. Chunks are keyed by chunk_number + MD5 hash, so content changes invalidate stale entries. The tracker also stores alignment results and video rendering state.

Byte-Based Sizing

All chunk and sentence sizing uses UTF-8 byte length (not character count). Each TTS backend declares its own max_chunk_bytes via the TTSBackend protocol. Constants provides system-wide defaults.

CPU/GPU Split

The remote TTS backend (Remote TTS) delegates ML work to a GPU inference server over HTTP. The Remote GPU Client handles alignment and lip-sync offloading too. Remote lip-sync now uses a job-handle protocol: submit work, poll elapsed/status, optionally cancel, download the result, and ask the server to delete temporary output.

Managed Environment Profiles

Setup Script and Doctor share the auto, local-gpu, remote-client, and dev profile model. Automatic selection uses local-gpu only on Linux or WSL2 when nvidia-smi can access a GPU; other platforms default to remote-client. Setup mutates the managed environment, while doctor only reports active capabilities and exits nonzero for required failures.

Pluggable Storage

The Storage Service uses a StorageBackend ABC with local, GCS, and S3 implementations. Pipelines always work against local directories; remote backends handle uploads/downloads to buckets. Cloud SDKs (google-cloud-storage, boto3) are deferred imports, following the same pattern as ML deps. Configured via P2A_STORAGE_BACKEND env var.

Structured Events

PipelineReporter emits both human-readable console lines and machine-parseable PipelineEvent objects. Optional structured data carries lip-sync page start/progress/completion details and timings. The web app’s Progress Reporter persists completed timing state and publishes events to Redis pubsub for SSE delivery to the browser. Hosts can also supply a cooperative should_cancel callback.

Partial Lip-Sync Completion

The web API stores a confirmed lip-sync stop request in Redis. Remote runs forward it to the GPU job; local runs observe it between provider calls. When pages have already completed, the pipeline builds the output from that prefix and records stopped-early counts/timings in result metadata. An empty partial run fails instead of creating an unusable artifact.

Reader-First Lip-Sync Output

The web lip-sync default is the LipSync Reader. The pipeline concatenates per-page presenter clips into presenter.mp4, builds shared reader assets with Reader Assets, and lets the browser combine document text/page images with a draggable/resizable presenter. The standalone offline reader ZIP is the recommended portable output.

Module Dependency Graph

cli.py
├── pipelines/audio.py
│   ├── pipelines/common.py
│   │   ├── extractor.py
│   │   ├── text_processing.py
│   │   ├── tracker.py
│   │   ├── aligner.py
│   │   │   └── providers/align/
│   │   ├── providers/tts/
│   │   └── remote_gpu.py
│   ├── concatenator.py
│   └── pipelines/events.py
├── pipelines/highlight.py
│   ├── (all of audio dependencies)
│   ├── highlight_renderer.py  (fallback for non-PDF)
│   ├── page_renderer.py       (PDF page-image rendering)
│   ├── word_matcher.py        (WhisperX words → PDF bboxes)
│   ├── video_composer.py
│   └── epub_builder.py
├── pipelines/lipsync.py
│   ├── (all of highlight dependencies)
│   ├── lipsync.py (facade)
│   │   └── providers/lipsync/
│   ├── reader_assets.py
│   └── remote_gpu.py
├── pipelines/visualization.py
│   └── providers/visualization/
├── doctor.py
│   └── environment, model-cache, sidecar, and remote-server checks
└── models.py
    └── providers/tts/ + providers/lipsync/

Web Application Stack

flowchart LR
    Browser[React browser application] <-->|HTTP and SSE| API[FastAPI routers]
    API <--> DB[(PostgreSQL)]
    API -->|dispatch| Worker[Celery worker]
    Worker <--> Redis[(Redis broker and pubsub)]
    Worker --> Pipelines[screencastgen pipelines]
    Pipelines --> Redis

See Web Overview for the full web architecture breakdown.