Every sales team I've worked with has the same quiet leak: reps spend a fifth of their day retyping what they just said on a call into a CRM. The conversation happened. The commitment was made. And then somebody has to remember it well enough, four hours later, to put it in the right HubSpot field.
Zen AI was my attempt to close that gap. I came in as the founding engineer with an empty repository and the following brief: listen to the call, understand the deal, update the CRM, and never make a rep think about it.
Here's how that actually got built, including the parts that broke.
The shape of the system
I split it into two services from the first commit, and I'd do it again:
zen-api— a FastAPI service handling audio upload streams, auth, tenant boundaries, and webhook subscriptions. Stateless, fast I/O, never does long work.zen-worker— a Node/Express queue worker doing transcription dispatch, LLM schema parsing, and CRM connectors. Slow, CPU-hungry, allowed to fail and retry.
The split isn't architectural vanity. A 60-minute call is a 100MB+ file and several minutes of transcription. If that work lives in the request path, one customer uploading three calls at once makes the API unresponsive for everyone else. Separating "accept the work" from "do the work" is the cheapest reliability decision available to you at this stage.
Two databases, on purpose
I used PostgreSQL and DynamoDB, which looks like over-engineering until you look at the access patterns.
Postgres holds the things that have relationships and need transactions: users, accounts, deal states, permission trees. DynamoDB holds transcript chunks — append-only, high-volume, temporally ordered, queried by call ID and nothing else.
Putting transcript text in Postgres works fine for a week. Then your main relational tables are mostly blobs, your backups take twenty minutes, and every VACUUM is a small outage. The rule I've settled on: relationships go in the relational database, firehoses go somewhere that likes firehoses.
Making an LLM produce something you can write to a database
The naive version of this is "ask the model to summarize the call." That's a demo, not a product. A product needs specific fields — budget, authority, need, timeline, competitor mentions, promised follow-ups — landing in specific CRM properties.
Three things made extraction reliable enough to write to a customer's CRM unattended:
- A strict schema with real parsing. Pydantic models on the way out, not string inspection. If it doesn't parse, it isn't an answer.
- Defensive retries. Models will occasionally wrap JSON in markdown fences or invent a field key that never existed. Catch the parse failure, re-prompt with the validation error, and cap the attempts.
- Validation against the tenant's actual CRM schema. A field the customer doesn't have is a hallucination no matter how confident the model sounds.
That combination got extraction consistency to ~99.4% across 10,000+ call minutes. The remaining fraction isn't "the model was wrong" — it's audio that no human could have transcribed either.
Idempotency is not optional when you write to someone else's system
CRM APIs have strict concurrency limits, and webhooks get delivered more than once. It is extremely easy to write a system that creates three copies of the same deal because a retry fired while the first request was still in flight.
The fix is boring and total: a deterministic hash of the call ID plus the target record becomes an idempotency key, and every write goes through an exponential-backoff queue keyed on it. Repeated delivery becomes a no-op. Across all production and test traffic, zero duplicate CRM writes.
If you take one thing from this post: when you write into a system you don't control, assume every request will happen twice.
What went wrong
During an early test, two 45-minute calls came in simultaneously and the worker was killed by the OS OOM killer, dropping three active jobs.
The cause was embarrassingly simple. The worker was holding the raw audio buffer in memory and the full transcript text arrays at the same time. One call was fine. Two calls, both long, overlapping — and the process crossed the memory ceiling and died with no ceremony at all.
Two changes fixed it:
- Stream audio from S3 to disk-backed temp files with cleanup handled by context managers, so nothing large lives on the heap.
- Redis-backed job retries with a dead-letter queue and alerting, so a dropped job surfaces as a page instead of a customer noticing next Tuesday.
The lesson wasn't "add more RAM." It was that a worker processing media should treat memory as the scarce resource and disk as the working surface.
Results
- Sub-3s time-to-sync from transcription completion to CRM update
- Zero duplicate CRM writes across 10,000+ call minutes
- Fully automated extraction of action items and stakeholder sentiment
- Roughly 4 hours per rep per week returned to selling
What I'd change today
I'd stream transcription to the frontend over Server-Sent Events so users watch the call get processed instead of staring at a spinner — perceived latency is a product feature. And I'd evaluate hosting an open-weight Whisper variant on Modal or RunPod; at the volumes this hit, self-hosting transcription looked like a 60% cost cut with a weekend of work.