Doctors in India spend roughly 30-40% of their consultation time on documentation. In a busy OPD where a doctor sees 60 patients in four hours, that overhead is not a UX inconvenience — it's patient time lost. This app started as a solution to this specific problem: record the conversation, generate the SOAP note, let the doctor get back to the patient.
Here is what we built, how we built it, and what we got wrong the first time.
The Architecture
The core pipeline is simple in concept:
- Flutter app on a tablet at the doctor's desk streams audio via WebSocket
- FastAPI backend feeds chunks to OpenAI Whisper for transcription
- A second LLM call structures the transcript into a SOAP note
- The completed note appears in the app before the patient leaves the room
In practice, each of those steps had failure modes we did not anticipate.
Real-Time Audio Streaming
WebSockets were the right call. HTTP polling introduced 2–3 second latency which made the live transcript feel broken. With WebSockets, we stream 100ms audio chunks from the Flutter app to the backend and send transcription segments back as they complete.
The non-obvious problem: silence. A doctor pausing to examine a patient would cause the transcription model to emit hallucinated filler text. We added a voice activity detector on the client side — simple energy-threshold based, not ML — that suppresses chunks below a noise floor. This eliminated about 90% of hallucinations in silent segments.
# Simplified chunk handler on the backend
async def handle_audio_chunk(websocket: WebSocket, session: Session):
async for chunk in websocket.iter_bytes():
session.buffer.append(chunk)
if len(session.buffer) >= FLUSH_THRESHOLD:
transcript = await transcribe(session.buffer)
session.buffer.clear()
await websocket.send_json({"type": "transcript", "text": transcript})
Structuring the SOAP Note
The transcription gives you a messy conversation. Turning that into a structured SOAP note — Subjective, Objective, Assessment, Plan — requires a second LLM call with a carefully crafted prompt.
The first version of the prompt asked the model to "write a SOAP note from this conversation." The output was medically plausible but inconsistent: sometimes the Assessment and Plan merged, sometimes findings appeared in the wrong section. Experienced doctors flagged it within a week.
The fix was to decompose the task. Instead of one large prompt, we run four focused extractions — one per SOAP section — and assemble the result. Parallel calls keep latency acceptable. The consistency improvement was significant: structured output compliance went from around 70% to over 95%.
Handling Indian Medical Context
This one surprised us. Indian clinical consultations mix English with Hindi (and occasionally regional languages). Terms like bukhaar (fever), dard (pain), and khoon (blood) appear naturally in conversations. Early Whisper models transcribed these inconsistently or dropped them.
Two mitigations helped:
- Passing a glossary of common Hinglish medical terms as the Whisper
promptparameter (it biases decoding toward known vocabulary) - Post-processing the transcript to normalize common phonetic variants before the SOAP extraction step
Neither is a clean solution. The right fix is a fine-tuned transcription model on Indian medical speech — that is on the roadmap.
What We Got Wrong
Session management. The first version held the entire audio buffer in memory for the session lifetime. With 50 concurrent consultations, this caused OOM crashes during OPD peaks. We moved to a chunked processing model where buffers are flushed and discarded after each transcription call.
Feedback loops. Doctors would correct the generated note in the app, but those corrections went nowhere. We were throwing away the most valuable training signal we had. We now log corrections (with consent) and use them to improve the prompt templates.
Assuming connectivity. Rural hospitals have unreliable internet. A WebSocket drop in the middle of a consultation meant the transcript was lost. We added a local audio buffer on the Flutter side that replays missed chunks on reconnect. Simple, but it should have been there from day one.
The Metric That Actually Mattered
We measured a lot of things — transcription accuracy, SOAP completeness, API latency. The number that changed how we thought about the product was simpler: time saved per consultation.
Before ai scribe app, average documentation time in our pilot hospitals was 4.2 minutes per patient. After, it was under 45 seconds. For a doctor seeing 60 patients a day, that is roughly three hours returned.
That is the kind of number that gets a hospital to renew.
If you are building something similar — real-time audio pipelines, clinical NLP, or multilingual transcription — I would be happy to compare notes. Reach out via the connect page.