root
950128deec
Keep C920 stereo through SpeechBrain GCC-PHAT Delay-and-Sum, then MetricGAN, and publish mono to LiveKit. Record last_dt_ms and log process_ms/capture_ms every 50 hops for the latency plan.
135 lines
5.3 KiB
Markdown
135 lines
5.3 KiB
Markdown
# livekit-cameras
|
|
|
|
Publishes 20 Logitech C920 webcam streams (video + cleaned audio) to a
|
|
LiveKit room.
|
|
|
|
Each camera becomes one LiveKit participant (`cam-01` .. `cam-20`) that
|
|
publishes:
|
|
- a camera video track (1280x720 @ 30 fps by default, MJPG from V4L2)
|
|
- a microphone audio track (16 kHz mono, cleaned with speechbrain)
|
|
|
|
Identity is the USB serial, pinned by udev (`/dev/camNN` + `slots.json`).
|
|
`--cameras 1,5` means participants `cam-01` and `cam-05`, not discovery order.
|
|
|
|
## Layout
|
|
|
|
```
|
|
run.py orchestrator: discovery + per-camera worker processes
|
|
run_worker.py one camera: ffmpeg capture -> LiveKit track publish
|
|
discovery.py serial-based camera <-> ALSA card discovery
|
|
audio_cleanup.py speechbrain SpectralMaskEnhancement (16 kHz) + fallback
|
|
tokens.py LiveKit JWT generation
|
|
config.py .env / environment configuration
|
|
prefetch_model.py one-time speechbrain model download
|
|
generate_udev_rules.py (re)generate udev pinning + slots.json from live state
|
|
slots.json stable slot map: cam-NN -> USB serial, /dev/camNN, ALSA card
|
|
smoke_test.py one-camera LiveKit publish with enhancement ON
|
|
scripts/bench_enhance.py RTF bench (1 s window / 200 ms hop)
|
|
deploy/livekit-cameras.service optional systemd unit (do not enable blindly)
|
|
```
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
uv venv --python 3.12
|
|
uv pip install -p .venv/bin/python -r pyproject.toml # or: uv sync
|
|
cp .env.example .env # then fill in LIVEKIT_URL / API_KEY / API_SECRET
|
|
.venv/bin/python prefetch_model.py # download speechbrain model once
|
|
```
|
|
|
|
## Stable camera identity (cam-01 is always cam-01)
|
|
|
|
Each C920 has a unique built-in USB serial — that is the camera's UUID.
|
|
udev rules (`/etc/udev/rules.d/99-c920-pin.rules`, generated from the live
|
|
state) bind every slot to a serial, so identity survives reboots, hub
|
|
reordering, and any connect sequence:
|
|
|
|
- video: `/dev/cam01` .. `/dev/cam20` (primary UVC node; `.meta` = metadata)
|
|
- audio: fixed ALSA card per slot, `hw:2,0` .. `hw:21,0` (SOUND_CARD_INDEX)
|
|
|
|
`discovery.py` assigns identities from `slots.json` (serial lookup, with a
|
|
port-based fallback for cameras not yet in the map). To re-map slots (e.g.
|
|
after physically swapping cameras), regenerate:
|
|
|
|
```bash
|
|
.venv/bin/python generate_udev_rules.py --write
|
|
sudo udevadm control --reload-rules
|
|
sudo udevadm trigger --subsystem-match=video4linux --action=add
|
|
sudo udevadm trigger --subsystem-match=sound --action=add
|
|
```
|
|
|
|
The generator bakes the **current physical wiring** as canonical. Do not
|
|
re-run `--write` unless you intend to remap slots.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
.venv/bin/python run.py # all 20 cameras
|
|
.venv/bin/python run.py --cameras 1,5 # subset
|
|
.venv/bin/python run.py --once # status report after timeout
|
|
```
|
|
|
|
## Audio cleanup
|
|
|
|
C920 mics are 16 kHz **stereo**. Each hop (default 200 ms) is kept
|
|
stereo through a SpeechBrain preprocessor, then published as mono:
|
|
|
|
1. `AudioNormalizer(mix="keep")` at 16 kHz
|
|
2. Delay-and-Sum beamform (GCC-PHAT TDOA) — the live path from the
|
|
SpeechBrain [multi-microphone beamforming tutorial](https://speechbrain.readthedocs.io/en/latest/tutorials/preprocessing/multi-microphone-beamforming.html)
|
|
(`speechbrain.lobes.beamform_multimic.DelaySum_Beamformer`)
|
|
3. `speechbrain/metricgan-plus-voicebank` neural enhancement on a 1 s
|
|
rolling window (`ENHANCE_CHUNK_S=1.0`, `ENHANCE_HOP_S=0.2`)
|
|
|
|
One worker process per camera loads the modules once and reuses them.
|
|
`enhance_batch` must be called with `lengths=torch.tensor([1.0])`
|
|
(relative, not sample count). `ENHANCE_MODE=off` (or a failed model
|
|
load with `auto`) skips MetricGAN; `BEAMFORM_MODE=off` averages L/R
|
|
instead of Delay-and-Sum. DSP high-pass + soft-clip is the last-resort
|
|
fallback so streams still work.
|
|
|
|
Until the 1 s window is full, hops use the DSP fallback so LiveKit is
|
|
not silent on startup.
|
|
|
|
## Appliance boot / power button
|
|
|
|
This box is meant to behave like an appliance:
|
|
|
|
- Power button while **off** → BIOS/OS boot → `livekit-cameras` waits for
|
|
USB cameras + LiveKit, then publishes.
|
|
- Power button while **on** → full `poweroff` (not sleep). Suspend/hibernate
|
|
targets are masked. Drop-in:
|
|
`/etc/systemd/logind.conf.d/50-power-button.conf`
|
|
|
|
## systemd (enabled)
|
|
|
|
- `livekit-server.service` — **local TEST** LiveKit on :7880 (autostart for now)
|
|
- `livekit-cameras.service` — publishers. `Wants=` the test server, does not
|
|
`Require` it. Destination is `LIVEKIT_URL` in `.env` (currently
|
|
`ws://127.0.0.1:7880`).
|
|
|
|
When the real/remote LiveKit exists, change `.env` `LIVEKIT_URL` and:
|
|
|
|
```bash
|
|
systemctl disable --now livekit-server
|
|
systemctl restart livekit-cameras
|
|
```
|
|
|
|
```bash
|
|
systemctl status livekit-cameras livekit-server
|
|
journalctl -u livekit-cameras -f
|
|
systemctl restart livekit-cameras # pick up newly plugged cameras
|
|
```
|
|
|
|
## Notes
|
|
|
|
- Video uses MJPG directly from the UVC interface (`ffmpeg -f v4l2
|
|
-input_format mjpeg`) — cheap, and converts to BGRA for LiveKit.
|
|
- Audio: ALSA mics on these C920s deliver 16 kHz **stereo** only; the
|
|
worker beamforms (Delay-and-Sum) then enhances, and publishes **mono**
|
|
to LiveKit.
|
|
- Camera identity is USB serial via `slots.json` / udev; USB port is
|
|
fallback only for cameras not yet in the map.
|
|
- Workers auto-restart (up to 10 times) if a camera drops.
|
|
- Local smoke: `.venv/bin/python smoke_test.py` (needs livekit-server on 7880).
|