Rescuing a production LMS without stopping the classes

ACADEMUN came to us the way rescue projects usually do: servers failing, the previous team gone, and a hard constraint — classes were running and could not stop. A takeover like this begins below the features. Nobody cares about your roadmap while the video pipeline is dropping lectures.

First: make it reproducible

The system's real configuration lived in someone's SSH history. Step one was archaeology — reading the running servers and writing down what they actually did. Step two was moving every service into Docker with Nginx in front as reverse proxy and static edge, configuration in the repository. The rule we enforced: nothing survives that can't be rebuilt from a Dockerfile.

Then: the pipeline

The FFmpeg pipeline was one monolithic MP4 encode per lecture, launched by hand when it failed. We rebuilt it around a real job queue — one worker per encode, HLS ladders instead of a single file, presets tuned to the source. Three properties did most of the work:

• Idempotent jobs — a re-run overwrites cleanly instead of doubling output.
• Bounded retries — a failed encode retries a fixed number of times, then lands in a queue a human actually reads.
• Segment caching at the Nginx layer — a popular lecture stops punishing the transcoder.

Review where it already failed

Inherited code gets beaten twice. We concentrated review on the paths with production scar tissue: encodes killed mid-segment, corrupt sources, disks filling during a transcode, deploys landing during a live class. Each one became a test and a runbook entry instead of a memory.

ACADEMUN now uploads, transcodes and streams without an engineer watching. The teams that walked away weren't wrong about the difficulty — only about whether it could be held. It could. It just had to be made boring first.