Building a voice AI agent that books meetings, and the three bugs that nearly stopped it

A voice AI agent is the most unforgiving AI product you can ship. There is no draft, no preview, no undo: the first output goes straight into a stranger's ear, and every mistake is a phone call somebody remembers. We built one that filters a national business directory, researches each lead, and cold-calls it with an evidence-based pitch. Schema to dialer took forty days.

What the agent actually does on a call

1. 01Discloses that it is AI on the first turn, before anything else.
2. 02Opens with a finding from the lead's own website, not a generic pitch.
3. 03Qualifies the prospect against the tenant's services.
4. 04Captures an email and books a calendar slot inside the call.
5. 05Honours an opt-out immediately and writes it to a global do-not-call list.

Disclosure first is not a compliance checkbox for us, it is a design constraint that improves the product: an agent that has already said what it is can spend the rest of the call being useful instead of being evasive.

The opener has to be falsifiable

Before the call, a worker fetches the lead's homepage and pulls deterministic signals. Those go to a model constrained to fifteen weakness tags, each requiring one sentence of evidence. "No viewport meta tag was found" passes. "Your mobile experience could be improved" does not.

This is what makes the call defensible. The agent is not guessing at a pain point, it is repeating something that was true on the page thirty seconds ago, and the recommended service is intersected against what the tenant actually sells.

Bug one: every call was voicemail, and none of them were

Calls to international numbers connected and ended within seconds, marked as voicemail. The prospect never heard the agent speak. Transcripts were empty, so the first two theories were a text-to-speech failure and a bad voice id. Both were discarded because the identical configuration completed calls to US numbers normally.

The cause was the carrier. Some carriers put a "press 1 to accept" screen in front of an incoming call. To an answering-machine detector working from the audio envelope, that is voice followed by silence, which is exactly the signature of a recorded greeting and a beep. Every screened call was classified as a machine and hung up before the first turn.

We removed answering-machine detection and raised the silence timeout from twelve to thirty seconds so the screen has room to clear. Tuning the detector was the obvious alternative and we rejected it: the screen is indistinguishable from a greeting on the signal the detector uses, so any threshold that passes one passes the other. The accepted cost is occasionally speaking into a real voicemail, which a hard duration cap bounds.

"The only variable separating the failing calls from the working ones was the destination country, not anything in the configuration."

Bug two: the dialer stopped, and nothing was wrong

Campaigns placed a handful of calls and went quiet. Nothing logged, campaign still active, leads still pending, restarting the worker changed nothing.

Global concurrency was a single counter: incremented before placing a call, decremented when the end-of-call event arrived. Any path that skipped the decrement left it permanently high, and once it hit the ceiling the guard refused every call indefinitely. Because it held a number rather than a set of call identities, nothing in the system had enough information to correct it.

# before: a number that can only drift
INCR calls:inflight            # place
DECR calls:inflight            # on webhook, maybe

# after: a set that heals itself
ZADD calls:inflight <expiry> <call_id>
ZREMRANGEBYSCORE calls:inflight -inf <now>   # every read
ZREM calls:inflight <call_id>                # idempotent release

Every read prunes entries whose expiry has passed, so a dropped webhook or a killed worker releases its own slot without intervention. We rejected the cheaper fix, wrapping the placement call in a compensating decrement, because it repairs the failure paths you anticipated and none of the ones you did not.

Bug three: bookings worked, emails arrived, the dashboard read zero

A test call booked a real calendar slot and both confirmation emails were delivered, yet the dashboard reported zero calls and zero meetings for that tenant.

The calendar API returns a booking start time as an ISO string. The Postgres driver's binary protocol requires a datetime object for a timestamptz column and will not coerce it. The meeting insert raised, and because the lead, call and meeting rows were written in one transaction, the rollback discarded the two inserts that had already succeeded. The booking and the emails were side effects outside the transaction, so they survived, which is why every visible signal said the flow had worked.

A coercion helper now parses ISO strings, including a trailing Z, into timezone-aware datetimes at both insert sites. Widening the column to text was rejected because it turns every downstream time comparison into a string comparison.

The integration that cost the most

Not the telephony. The calendar. The v1 API was retired partway through the build and began answering new keys with 410 Gone, so the integration was rewritten in place against v2. That was not a URL change: auth moved from a query parameter to a bearer header, every endpoint gained a date-pinned contract version header, the booking payload gained a nested attendee block, responses arrived wrapped in an envelope, and event ids came back as integers where they had been strings.

Availability then had its own layer: querying real slots across a fourteen-day window, trying each in order, retrying only on the two status codes that mean the slot is taken, and bailing immediately on anything else so an expired key is not mistaken for a full calendar.

If you are building one

• Assume the telephony layer is lying to you about call state, and verify against provider logs before you touch the agent.
• Never let concurrency be a number. Make it a set of identities with expiries.
• Keep side effects (bookings, emails) and database writes in the same failure domain, or accept that one can succeed while the other vanishes.
• Cache the expensive research per domain, not per lead.
• Write disclosure and opt-out into the call flow, not into the prompt.

The model was never the risky part. The risky parts were a carrier's call screen, a Redis counter and a timestamp format, and all three looked like something else entirely until the logs said otherwise.

Mahad Ali

Co-founder & CEO · Principal Architect

The tech guy. Architecture, data models, pipelines, and the production incidents at three in the morning. Most of the engineering behind the sixteen products, and the first reply to your message.

Direct Dispatch →