feat: DelaySum beamform plus hop process latency
Keep C920 stereo through SpeechBrain GCC-PHAT Delay-and-Sum, then MetricGAN, and publish mono to LiveKit. Record last_dt_ms and log process_ms/capture_ms every 50 hops for the latency plan.
This commit is contained in:
@@ -71,13 +71,22 @@ re-run `--write` unless you intend to remap slots.
|
||||
|
||||
## Audio cleanup
|
||||
|
||||
The mic audio is cleaned per hop (default 200 ms) by
|
||||
`speechbrain/metricgan-plus-voicebank` (16 kHz, CPU) over a 1 s rolling
|
||||
window (`ENHANCE_CHUNK_S=1.0`, `ENHANCE_HOP_S=0.2`). One worker process
|
||||
per camera loads the model once and reuses it. `enhance_batch` must be
|
||||
called with `lengths=torch.tensor([1.0])` (relative, not sample count).
|
||||
`ENHANCE_MODE=off` (or a failed model load with `auto`) falls back to a
|
||||
lightweight high-pass + soft-clip DSP path so streams still work.
|
||||
C920 mics are 16 kHz **stereo**. Each hop (default 200 ms) is kept
|
||||
stereo through a SpeechBrain preprocessor, then published as mono:
|
||||
|
||||
1. `AudioNormalizer(mix="keep")` at 16 kHz
|
||||
2. Delay-and-Sum beamform (GCC-PHAT TDOA) — the live path from the
|
||||
SpeechBrain [multi-microphone beamforming tutorial](https://speechbrain.readthedocs.io/en/latest/tutorials/preprocessing/multi-microphone-beamforming.html)
|
||||
(`speechbrain.lobes.beamform_multimic.DelaySum_Beamformer`)
|
||||
3. `speechbrain/metricgan-plus-voicebank` neural enhancement on a 1 s
|
||||
rolling window (`ENHANCE_CHUNK_S=1.0`, `ENHANCE_HOP_S=0.2`)
|
||||
|
||||
One worker process per camera loads the modules once and reuses them.
|
||||
`enhance_batch` must be called with `lengths=torch.tensor([1.0])`
|
||||
(relative, not sample count). `ENHANCE_MODE=off` (or a failed model
|
||||
load with `auto`) skips MetricGAN; `BEAMFORM_MODE=off` averages L/R
|
||||
instead of Delay-and-Sum. DSP high-pass + soft-clip is the last-resort
|
||||
fallback so streams still work.
|
||||
|
||||
Until the 1 s window is full, hops use the DSP fallback so LiveKit is
|
||||
not silent on startup.
|
||||
@@ -116,8 +125,9 @@ systemctl restart livekit-cameras # pick up newly plugged cameras
|
||||
|
||||
- Video uses MJPG directly from the UVC interface (`ffmpeg -f v4l2
|
||||
-input_format mjpeg`) — cheap, and converts to BGRA for LiveKit.
|
||||
- Audio: ALSA mics on these C920s deliver 16 kHz **stereo** only; the worker
|
||||
downmixes to mono for LiveKit.
|
||||
- Audio: ALSA mics on these C920s deliver 16 kHz **stereo** only; the
|
||||
worker beamforms (Delay-and-Sum) then enhances, and publishes **mono**
|
||||
to LiveKit.
|
||||
- Camera identity is USB serial via `slots.json` / udev; USB port is
|
||||
fallback only for cameras not yet in the map.
|
||||
- Workers auto-restart (up to 10 times) if a camera drops.
|
||||
|
||||
Reference in New Issue
Block a user