Importing 51 Million Songs on a Budget: MusicBrainz + Elasticsearch

Free-text song requests are a recipe for typos and guessing games, so GrooveShouter needed real search. The MusicBrainz dataset has over 51 million tracks and is free to use — the catch is that importing it affordably, on self-hosted hardware, is genuinely hard.

The pipeline that emerged: download the ~15GB MusicBrainz dump weekly, build an optimized SQLite database as a fast intermediate store, then stream-batch that into Elasticsearch using under 4GB of RAM and about 20GB of temp storage, with zero-downtime index swaps so search never goes offline mid-import. Getting there meant fighting several out-of-memory crashes, tuning batch sizes and consumer concurrency, and eventually scheduling the whole thing as a recurring CI job with Ofelia. It's the single most technically demanding piece of the entire project — and it runs quietly, once a week, without anyone noticing.