Upload a song, add a background, and turn the lyrics into a video. That sounds like a fairly straightforward application. Getting the words to appear at the right moment, in the right font, while keeping long-running processing out of the web server takes a little more work.
Verseframe is my lyric-video studio for musicians / artists. Artists can upload media, synchronize lyrics, review the timing, and render an MP4. Behind that workflow are Python, speech recognition, GPU workers, object storage, and a surprising amount of font handling.
Let’s look at how the pieces fit together.
1. The Stack: Flask Meets Media Processing
Verseframe uses Flask and server-rendered Jinja templates for the web application, with JavaScript for interactions such as timing review and progress updates.
The main pieces are:
- Flask and Jinja: application routes, forms, and the artist’s workspace.
- PostgreSQL, SQLAlchemy, and Alembic: persistent job metadata, lyric timing, and schema migrations.
- Railway: hosting for the web application, database, and private S3-compatible storage.
- Modal: separate workers for lyric analysis and video rendering, including GPU-backed transcription.
- faster-whisper: speech recognition with word timestamps.
- FFmpeg and libass: media processing and animated subtitle rendering.
- Clerk: account authentication, with ownership checks enforced by the application.
The web application handles requests and coordinates work. The workers handle the expensive media operations. PostgreSQL records the job state and timing, while object storage holds uploaded media and generated files.
That separation matters when a song takes longer to process than a normal web request should stay open.
2. Turning a Song into Editable Timing
A lyric video needs more than a transcript. It needs to know when each word starts and ends.
Verseframe exposes Standard and Precision timing options, using the small and large-v3 models respectively. The visible Precision option transcribes the original audio directly on a GPU worker.
Artists can also supply their own lyrics. Those lyrics become the canonical display text, with detected words providing timing information. This helps preserve the spelling the artist intended.
Singing still creates difficult cases: held notes, repeated choruses, and vocals buried in the mix. A model can also return overlapping timestamps or words with no duration at all.
Before creating the review data, Verseframe normalizes the raw word timestamps. It gives words positive intervals, keeps them ordered, and bounds them to the track’s duration. Invalid input that cannot support those constraints produces an error.
Here is an illustrative example of the structured timing data:
{ "text": "Stay with me", "start": 12.4, "end": 14.1, "words": [ { "text": "Stay", "start": 12.4, "end": 12.9 }, { "text": "with", "start": 12.9, "end": 13.3 }, { "text": "me", "start": 13.3, "end": 14.1 } ]}
Normalization makes that data usable by the editor and renderer. It does not prove that the words match the performance perfectly. The timing editor gives the artist a place to listen, review, and correct the result before rendering.
3. Saving Timing Separately from the Video
Timing is useful beyond a single export. Once someone has corrected a chorus, changing the background should not require doing that work again.
Verseframe stores structured lyric timing separately from the rendered MP4. A generation with saved timing can be duplicated with new media or style settings, and timing can also be exported as reusable karaoke ASS.
ASS stands for Advanced SubStation Alpha. It is a subtitle format that supports positioning, styling, and timing effects, making it useful for animated lyrics.
This separation also helps when rendering fails. If synchronization succeeded and its timing was saved, the artist can still recover that work without needing a successful video export.
4. Background Jobs Need More Than a Status Label
Moving processing into a worker introduces another problem: the application and worker can disagree about what is happening.
A user might cancel a job while a worker is still running. A callback might arrive after the application has moved on. Allowing a retry too early could start a second execution against the same generation.
Verseframe tracks an execution ID alongside the job’s visible status. Worker updates must match the current execution, and execution ownership continues to reserve capacity until the worker has stopped.
Cancellation therefore has an intermediate state: cancelling. Requesting cancellation does not immediately mean the underlying process has exited.
Admission checks also run through the database so concurrent requests cannot independently claim the same available capacity. A job looking old is insufficient evidence that it is safe to release its execution ownership.
These details are less visible than an animated chorus, but they make retries and cancellation much safer to operate.
5. Rendering Lyrics with FFmpeg and libass
Once the timing is ready, Verseframe generates ASS subtitles and uses FFmpeg with libass to render them into the video. Effects include karaoke fills, word and letter reveals, typewriter text, and fading movement.
Fonts turned out to deserve their own set of checks.
A font file can contain internal family names that differ from its filename. Multiple fonts can also share a family name. If the renderer selects a fallback, the video may finish successfully while looking different from what the artist selected.
Verseframe gives a temporary rendering copy of the selected font an unambiguous family name. The original font remains intact. It checks character coverage and validates libass’s font selection, reporting an actionable error when the requested font cannot render the lyrics correctly.
Sizing introduces another wrinkle. Font metrics affect how ASS interprets size and positioning. The renderer converts the requested em-pixel size using the font’s metrics and accounts for visible glyph bounds when placing text.
Choosing a font in a dropdown is easy. Making that choice survive the entire rendering pipeline takes more care.
6. Keeping Artist Media Private
Uploaded songs, backgrounds, and custom fonts need access controls throughout the workflow.
Verseframe stores media in private object storage and checks ownership when serving workspace data and downloads. Authenticated browser mutations also require CSRF protection. Worker communication uses separate service authentication.
Workers receive a constrained job contract rather than arbitrary filesystem paths from the browser. Saved custom fonts are copied into a generation’s inputs, preserving the exact font used even if the artist later removes it from their library.
Persistence has a clear boundary too: a configured database must be available and migrated. The application fails rather than silently switching to in-memory jobs and making saved work appear to disappear.
Try Verseframe
Building Verseframe has involved connecting model output, editable data, rendering, and worker lifecycle management into one workflow. Each part has to preserve the artist’s choices as the job moves through the system.
You can try it at verseframe.studio. Upload a track, review its lyrics, and turn the timing into a video.
Hope you enjoyed this look behind the scenes. Until next time, stay nerdic!
Leave a Reply