Lingvo

01 Problem

Live cross-language conversation and meeting intelligence share a backend but have very different latency profiles. One needs realtime STT → MT → TTS on the audio path, the other needs structured summaries, action items and decisions extracted after the call. Both had to run on the same Django core with shared billing and identity.

02 What I built

A single Django + DRF + Celery backend serving two Next.js frontends (Translator and Meetings). WebRTC media runs through a LiveKit SFU. A separate Pipecat agent on a GPU box joins each room over WSS and runs the realtime pipeline, dispatched via Redis pub/sub and reporting usage back to the backend. Meetings add a note-taker pipeline plus Celery tasks that generate summaries and extract action items / decisions. A Beat task builds a pre-meeting brief via RAG over recent meetings (pgvector embeddings). Payments sit behind Cryptomus and YooKassa adapters with idempotent, monotone webhook processing.

• ▸One Django backend serves two Next.js products (translator + meetings) with shared identity and billing
• ▸Realtime STT → MT → TTS via a Pipecat GPU agent over a LiveKit SFU, dispatched through Redis pub/sub
• ▸Meeting summaries, action items and decisions extracted by Celery; pre-meeting briefs via RAG over pgvector
• ▸Idempotent, monotone payment webhooks behind Cryptomus / YooKassa adapters

03 The realtime audio path

Speech recognition runs on faster-whisper large-v3 on CUDA. A phrase ends after about 0.7 seconds of silence or 25 seconds of continuous speech, and Silero VAD drops fragments that are too short or too quiet to be words. Translation goes through OpenRouter, with Claude Haiku as the default model.

Speech synthesis is routed by language: Kokoro for English, XTTS-v2 for Russian and other languages it supports, and Edge-TTS as the fallback for Thai and the rest. Kokoro returns its first audio in about 50–100 ms against 500–700 ms for Edge, which by our estimate shortens the whole chain from about 2 seconds to about 1.5. The models share one RTX 3070 Ti and use about 6.9 of its 8 GB.

04 Control plane and resilience

Django and the agent talk through Redis channels. The backend publishes room start, stop and reconfiguration events. The agent joins the room as a bot and reports usage every 30 seconds with a service token. A participant can switch language mid-call without leaving the room. When both people speak the same language, the room only transcribes and skips translation and synthesis.

A dropped connection does not end the room. Recordings are kept as separate tracks in numbered segments. A Celery task stitches them into one file per session, so the only gap is the time a participant was really offline.

Other call features follow the same approach. Participants can share files inside the call, and each person can turn the AI voiceover on or off for themselves without affecting the rest of the room. Batch voiceover of a prepared text goes through a separate REST endpoint and runs as a Celery job, apart from the live call pipeline.

05 Meeting memory

In Meetings a note-taker transcribes the call without speaking. When the room stops, a Celery task writes the summary and extracts action items and decisions. Every minute a Beat task looks for meetings that start in 10–11 minutes and prepares a brief from recent meetings of the same project.

Transcript messages and knowledge entries are embedded into 768-dimensional vectors and stored in pgvector with an HNSW cosine index; Gemini text-embedding-004 is the default model. The same recognition model also serves a file transcription API, sharing three worker slots between live calls and offline jobs.

06 Accounts, billing and delivery

Users sign in with the Telegram Login Widget and get an HttpOnly session cookie, while LiveKit access tokens live for an hour at most. Plans are tied to a product, translator or meetings. Payments come through Cryptomus (USDT TRC-20) and YooKassa (SBP and cards). The webhook signature is verified and the payment row is locked. The external ID is unique per provider, amount and currency are cross-checked, and a status can only move forward.

Every push to main runs Forgejo Actions and builds four images: Django, the Pipecat agent and the two frontends. Spikes are written up too. One showed that an iOS web app cannot keep a call alive after the screen locks, so background calls on iPhone need a native app.

07 Result

One backend cleanly carries two products with different realtime constraints. The audio path stays on LiveKit while control runs over a Tailscale mesh, and meeting knowledge is searchable via vector embeddings. Between May and July 2026 the project went through 18 sprints and 216 commits, with 13 ADRs and more than 440 automated tests across the backend and the agent. Both products run in a development environment on a GPU server; the public launch is still ahead.