Splitzo

One messy conversation in, one clean track per voice out.

Splitzo takes any audio or video file, works out who is talking when, and writes one audio track per speaker. Every track is the full length of the original and sits silent while that person is not talking, exactly like a real mic in a call. Play them all at once and you hear the original conversation back.

Optionally it also renders a lip-synced cartoon avatar video per speaker, so you get a whole fake video call out of one podcast file.

Open full size

Why I built it

I was building video chat into another app and needed to test it with a room full of people. Scheduling real humans for every run is not testing, it is a favour you can only ask so many times. Playing one recorded file into the app does not work either: a chat client does not want a single mixed conversation, it wants one stream per participant, and each of those streams has to go quiet when that person stops talking. Otherwise you never exercise the parts that actually break, like active speaker detection, mute states, talk-over, and mixing.

Podcasts and panel shows are already the conversation I wanted. They are just glued into one track. So Splitzo pulls them back apart into per-participant packages: one same-length audio file per speaker, all sample aligned, plus an optional avatar video per speaker. Point N fake clients at N packages, press play, and the app under test sees a real multi-party call, with interruptions, pauses and cross-talk, without a single human in the room.

The guarantee

Every slice of audio is assigned to exactly one speaker. Overlapping speech goes to the dominant speaker, and non-speech is silence in every track. The tracks are a strict partition of the source, so summing them reproduces the original sample for sample. --verify proves that in code, on the raw PCM, before any lossy encode touches it.

This is diarization plus lossless cutting, not source separation. Splitzo never tries to un-mix two voices that are talking at the same instant, it just decides who owns that instant.

How it works

input.(mp4 | mov | wav | m4a | mp3 | ...)
   |
   +-- ffmpeg --> source PCM at native rate + channels        <- what we cut from
   |
   +-- ffmpeg --> mono 16 kHz WAV --> pyannote --> who speaks when
                                          |
                                          v
                            per-sample owner timeline
                                          |
             +----------------------------+----------------------------+
             v                                                         v
   speaker_0.mp3, speaker_1.mp3 ...                    speaker_0.mp4, speaker_1.mp4 ...
   (same length as source, sample aligned)             (optional lip-synced avatars)

1. Ingest (ingest.py). ffmpeg decodes the input twice: once to float32 PCM at the native sample rate and channel count, which is the audio we actually cut, and once to mono 16 kHz, which is what the diarizer eats. Any trim is applied to both decodes identically so the timestamps stay aligned.
2. Diarize (diarize.py). pyannote's speaker-diarization-community-1 pipeline returns speech segments tagged with a speaker label. This is the slow step, and the only one that wants a GPU.
3. Partition (partition.py). Segments become an owner value per sample. Overlaps are resolved by the chosen policy (dominant speaker, first to start, drop, or give it to everyone), then each speaker gets a boolean mask. For the first three policies the masks are disjoint, which is what makes the reconstruction exact.
4. Render (render.py). Per speaker: copy the source where the mask is true, zeros everywhere else, then encode. Because every track is built from the same buffer length, they all end up sample aligned. With --verify, the summed tracks are compared against the source before encoding.
5. Avatars (avatar.py, optional). A Node sidecar bakes one unique character sprite atlas per speaker (a tile per mouth shape plus a blink). Python derives a per-frame mouth shape from that speaker's own audio using a cheap spectral analysis, composites the right tile with head bob, sway, blink and an active-speaker ring, and pipes raw frames into ffmpeg to mux with that speaker's track. No per-frame Node, no ML, no model download.

The web UI

Double-click start-webui.cmd (or run .\.venv\Scripts\python.exe webui.py) and it opens http://127.0.0.1:8765. Drop a file in, and it reads the length so you can trim to just the slice you care about before spending any GPU time on it.

Open full size

When the run finishes you get a player per speaker, a timeline showing when that voice is active, live level meters, and a scrub bar that seeks every track together. Play all together is the ear version of --verify: it should sound just like the source.

Open full size

Each voice gets a random friendly name that you can rename inline. The name becomes that track's download filename (spaces are saved as underscores) and it carries through to the .zip and to the avatar videos. Untick a speaker to leave them out of the zip.

The screenshots use a sample job. The avatars in them are genuine output from the character baker in avatar_bake/.

Quick start

Built and run on Windows with an RTX 4070. Nothing in the Python is Windows specific apart from start-webui.cmd, so the commands below should translate directly to macOS and Linux (brew install ffmpeg or your package manager, then source .venv/bin/activate), but I have not tested those platforms.

1. ffmpeg (system):

   winget install Gyan.FFmpeg
   

   Reopen the shell, confirm ffmpeg -version.

2. Python env (3.13 works, 3.12 is fine too):

   py -3.13 -m venv .venv
   .\.venv\Scripts\Activate.ps1
   # GPU build of torch FIRST. pyannote 4.x needs torch>=2.8, and cu126 has it:
   pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu126
   pip install -r requirements.txt
   

3. Hugging Face token (free, one-time):

   • Create a token at https://hf.co/settings/tokens.
   • While logged in, accept the user conditions for the diarization model: https://huggingface.co/pyannote/speaker-diarization-community-1.
   • copy .env.example .env and put the token in HF_TOKEN.

   Set AUDIOSPLITTER_DIAR_MODEL if you want a different pipeline. The first run downloads about 31 MB of model weights, after which it is offline.

4. Something to split. No audio ships with this repo. Point it at any recording with more than one voice in it: a podcast episode, a recorded meeting, an interview, a panel. Audio or video, whatever ffmpeg can open.

Command line

python split_speakers.py conversation.wav -o out --verify
# just a slice of a long recording:
python split_speakers.py long_call.mp4 -o out --start 1:30 --end 3:00 --verify

Run against a 45 second, three-way clip:

[1/4 Ingesting] Reading conversation.wav
[2/4 Diarizing] Identifying speakers (the slow step)
[3/4 Partitioning] Building timeline for 3 speaker(s)
[4/4 Rendering] Writing 3 track(s)

=== Summary ===
input    : conversation.wav
duration : 44.7s @ 22050 Hz, 1ch
device   : cuda
speakers : 3
  speaker_0 (SPEAKER_00):    10.6s (23.7%)  -> out\speaker_0.mp3
  speaker_1 (SPEAKER_01):    10.8s (24.2%)  -> out\speaker_1.mp3
  speaker_2 (SPEAKER_02):     9.0s (20.2%)  -> out\speaker_2.mp3
overlap  :    0.0s ( 0.0%) -> dominant
silence  :   14.2s (31.8%) -> silent in all tracks
verify   : max|sum(tracks) - source| = 0.00e+00  (EXACT partition OK)
format   : mp3

Common options:

Option	What it does
--num-speakers N	exact speaker count if you know it, otherwise auto-detected
--min-speakers / --max-speakers	bounds when the count is uncertain
--format mp3|wav|flac	output format (default mp3, wav and flac are bit-exact)
--bitrate 192k	mp3 bitrate
--sample-rate 48000 --mono	match what your chat client ingests
--overlap dominant|first|none|all	how overlapping speech is assigned (default dominant)
--start / --end	trim, as ss, mm:ss or hh:mm:ss
--device auto|cuda|cpu	auto picks the GPU if there is one
--verify	assert the tracks sum back to the source
--avatars	also render the avatar videos

Video avatars

Each speaker gets a unique animated character whose mouth follows their track, idle when silent, with an active-speaker ring while talking. Play them together and you have a reconstructed multi-party video call to drive a thin video client. Characters are derived deterministically from the speaker index, so the same input always produces the same cast.

The character art is rendered by a small Node sidecar in avatar_bake/, an avatar-tuned fork of the ink renderer from my MarkLikey game project. It runs once per job to bake the sprite atlases, and everything after that is numpy and ffmpeg.

One-time setup (needs Node.js):

cd avatar_bake
npm install        # @napi-rs/canvas, prebuilt, no build tools needed
cd ..

Then:

python split_speakers.py conversation.mp4 -o out --avatars
#   --avatar-size 360   square video size in px (default 360)
#   --avatar-fps 20     frames per second (default 20)

In the web UI, tick Generate video avatars. The status bar tells you whether Node and the sidecar deps are ready, and the checkbox stays disabled until they are. The audio-only path never touches any of this.

Speed: cost scales with duration x fps x size x speakers. Speakers render in parallel and composited frames are cached, but the biggest lever by far is trimming to the slice you need, since cost is linear in duration.

Repo layout

audiosplitter/
  ingest.py           ffmpeg decode + trim
  diarize.py          pyannote pipeline, device selection
  partition.py        segments -> per-sample owner -> per-speaker masks
  render.py           cut, encode, verify
  avatar.py           viseme track, frame compositing, mux
  pipeline.py         orchestration shared by CLI and web
  cli.py              argparse front end
  web.py              Flask API behind the web UI
  webui/index.html    the whole single-page UI
avatar_bake/          Node sidecar: bakes per-speaker character sprite atlases
split_speakers.py     CLI entry point
webui.py              web UI entry point

Notes

• --verify sums the per-speaker PCM and compares it to the source with non-speech zeroed. It runs before MP3 encoding, so the partition check is exact.
• MP3 is lossy, so the encoded files are not bit-for-bit. Use --format wav or flac if you need that. Ambient sound during pauses is dropped, since non-speech is silenced in every track. That is fine for true silence, which is the common case.
• --overlap all gives every speaker their own words back during cross-talk, so nobody gets clipped. The tracks then overlap by design, so playing them together is louder during cross-talk and no longer an exact match of the source.
• If auto-detection over- or under-splits, pass --num-speakers.
• No GPU? It still works on CPU, just slower, roughly a couple of minutes for 30 minutes of audio.
• Everything runs locally. Nothing is uploaded, and the only network call is the one-time model download from Hugging Face.

Troubleshooting

• ffmpeg not found: install it and reopen the shell.
• pyannote 401, or "returned no pipeline": the token is missing, or you did not accept the model conditions while logged in.
• torch / CUDA mismatch: reinstall the matching CUDA wheel from the pytorch index URL, or run with --device cpu.
• Avatars checkbox disabled: Node is not on PATH, or npm install has not been run in avatar_bake/.

Credits and license

Splitzo is MIT licensed, see LICENSE. It leans on other people's work:

• pyannote.audio and the speaker-diarization-community-1 pipeline do the actual speaker diarization, under their own licenses and model conditions.
• ffmpeg does every decode and encode.
• avatar_bake/ is an avatar-tuned fork of the ink character renderer from my MarkLikey game project, rendered through @napi-rs/canvas.

Whatever you feed it stays on your machine. The only thing that leaves is the one-time model download from Hugging Face.