hrutik.site
← All writing

How I Built My First Production AI Workflow

August 14, 2026·4 min read·By Hrutik

Demos are easy. A quick script that takes a user query, feeds it to an LLM, and prints the answer looks magical on a Tuesday afternoon.

Production is where the illusion shatters.

When I started building Zen AI — an automated deal-intelligence system that takes 45-minute sales recordings, extracts structured deal commitments, and writes them directly into HubSpot and Salesforce — I quickly discovered that 90% of the engineering has nothing to do with prompt engineering and everything to do with distributed system design.

Here are the hard lessons I learned taking an AI workflow from prototype to production.

1. Never Trust an LLM to Output Valid JSON Naively

In an early prototype, the system asked the LLM to format extraction results as JSON:

Extract BANT criteria and return ONLY a JSON object with keys: budget, authority, need, timeline.

In 94 out of 100 runs, it worked. In the remaining 6 runs:

  • It wrapped the JSON in triple backticks: \``json ... ````
  • It appended commentary: "Here is the extracted data: {...}"
  • Or it silently renamed "timeline" to "estimated_timeline"

If this JSON is piped directly into a database or an external CRM webhook, your parser crashes.

The Fix: Schema Enforcement and Defensive Normalization

We switched to strict Pydantic schemas with structured output guarantees (JSON mode / tool calling) and placed a defensive validation step in front of the database:

from pydantic import BaseModel, Field, ValidationError

class DealExtraction(BaseModel):
    budget: str = Field(..., description="Mentioned budget or currency figures")
    authority: str = Field(..., description="Decision maker identification")
    need: str = Field(..., description="Core business pain point")
    timeline: str = Field(..., description="Target implementation timeframe")

def parse_llm_payload(raw_text: str) -> DealExtraction:
    # 1. Clean accidental markdown backticks
    cleaned = raw_text.strip().removeprefix("```json").removesuffix("```").strip()
    # 2. Strict validation
    return DealExtraction.model_validate_json(cleaned)

If parsing fails, we don't throw an unhandled 500 error: we send the failure to an automated single-turn repair prompt with the exact validation error message.

2. Decouple Ingestion from AI Processing

A 45-minute audio call cannot be processed in an HTTP request cycle. Web clients will time out, mobile connections drop, and proxy servers (like Cloudflare or Nginx) have 60-second execution caps.

The Architecture:

  1. Client initiates upload: The frontend requests a pre-signed S3 upload URL from FastAPI.
  2. Stateless API ack: The file uploads directly to S3. Once complete, an S3 event triggers a lightweight queue job.
  3. Dedicated worker execution: A background worker (Express / Node.js) picks up the audio file, streams it to local disk, runs Whisper transcription, and calls the LLM.
  4. Optimistic polling / Webhooks: The frontend polls job status or receives a push notification when the status transitions from PROCESSING to READY.

This separation guarantees that a surge in large incoming audio files never exhausts web server connection pools.

3. The Context Window Trap

Feeding an entire 45-minute raw transcript (often 12,000+ words) into a single prompt is tempting now that models offer 128k+ token windows.

Don't do it.

Two things happen when you dump giant transcripts into an LLM:

  1. "Lost in the middle" degradation: Models pay significantly more attention to the beginning and end of huge prompts, missing crucial objections raised at minute 28.
  2. Cost & Latency explode: Time-to-first-token spikes drastically, and running full-window prompts across hundreds of calls per day destroys your margin.

The Solution: Semantic Chunking & Map-Reduce

Instead of raw dumps:

  1. Split the transcript by speaker turns and time intervals.
  2. Filter out filler chitchat ("Can you hear me?", "Let me share my screen").
  3. Run targeted extractors on relevant conversational chunks.
  4. Merge partial extractions with a final reconciliation pass.

4. Idempotency is Non-Negotiable

When integrating with external CRMs like Salesforce, HubSpot, or Slack, webhooks will inevitably retry due to network blips.

If your worker retries an LLM job without an idempotency key:

  • A sales rep gets 3 identical Slack alerts.
  • A deal record is duplicated in HubSpot.
  • Your LLM bill doubles.

We hash the combination of (call_id, target_crm, prompt_version) into a deterministic SHA-256 key. Before dispatching any write to an external API, we acquire a Redis-backed lock. If a duplicate arrives, it is safely dropped.

Summary

Moving an AI workflow from demo to production requires treating the model as an unreliable, high-latency external microservice.

Surround it with:

  • Strict schema validation
  • Decoupled asynchronous worker queues
  • Bounded memory buffers
  • Deterministic idempotency keys

That is what turns an AI novelty into a dependable product that founders and enterprise teams can rely on every single day.

← Read more posts