feat: DelaySum beamform plus hop process latency

Keep C920 stereo through SpeechBrain GCC-PHAT Delay-and-Sum, then
MetricGAN, and publish mono to LiveKit. Record last_dt_ms and log
process_ms/capture_ms every 50 hops for the latency plan.
This commit is contained in:
root
2026-08-28 06:16:25 +00:00
parent abeea6256c
commit 950128deec
8 changed files with 252 additions and 56 deletions
+19 -9
View File
@@ -71,13 +71,22 @@ re-run `--write` unless you intend to remap slots.
## Audio cleanup
The mic audio is cleaned per hop (default 200 ms) by
`speechbrain/metricgan-plus-voicebank` (16 kHz, CPU) over a 1 s rolling
window (`ENHANCE_CHUNK_S=1.0`, `ENHANCE_HOP_S=0.2`). One worker process
per camera loads the model once and reuses it. `enhance_batch` must be
called with `lengths=torch.tensor([1.0])` (relative, not sample count).
`ENHANCE_MODE=off` (or a failed model load with `auto`) falls back to a
lightweight high-pass + soft-clip DSP path so streams still work.
C920 mics are 16 kHz **stereo**. Each hop (default 200 ms) is kept
stereo through a SpeechBrain preprocessor, then published as mono:
1. `AudioNormalizer(mix="keep")` at 16 kHz
2. Delay-and-Sum beamform (GCC-PHAT TDOA) — the live path from the
SpeechBrain [multi-microphone beamforming tutorial](https://speechbrain.readthedocs.io/en/latest/tutorials/preprocessing/multi-microphone-beamforming.html)
(`speechbrain.lobes.beamform_multimic.DelaySum_Beamformer`)
3. `speechbrain/metricgan-plus-voicebank` neural enhancement on a 1 s
rolling window (`ENHANCE_CHUNK_S=1.0`, `ENHANCE_HOP_S=0.2`)
One worker process per camera loads the modules once and reuses them.
`enhance_batch` must be called with `lengths=torch.tensor([1.0])`
(relative, not sample count). `ENHANCE_MODE=off` (or a failed model
load with `auto`) skips MetricGAN; `BEAMFORM_MODE=off` averages L/R
instead of Delay-and-Sum. DSP high-pass + soft-clip is the last-resort
fallback so streams still work.
Until the 1 s window is full, hops use the DSP fallback so LiveKit is
not silent on startup.
@@ -116,8 +125,9 @@ systemctl restart livekit-cameras # pick up newly plugged cameras
- Video uses MJPG directly from the UVC interface (`ffmpeg -f v4l2
-input_format mjpeg`) — cheap, and converts to BGRA for LiveKit.
- Audio: ALSA mics on these C920s deliver 16 kHz **stereo** only; the worker
downmixes to mono for LiveKit.
- Audio: ALSA mics on these C920s deliver 16 kHz **stereo** only; the
worker beamforms (Delay-and-Sum) then enhances, and publishes **mono**
to LiveKit.
- Camera identity is USB serial via `slots.json` / udev; USB port is
fallback only for cameras not yet in the map.
- Workers auto-restart (up to 10 times) if a camera drops.