How to configure voice agent termination?

Looking for some help or guidance on a voice agent problem I'm stuck on. I'm building a voice agent that needs to guide users through specific milestones in a fixed sequence. The issue: the agent keeps diverging into open-ended conversations instead of completing its turn and handing off cleanly. I want the agent to know when it's done - semantically and programmatically - so the next step can begin without overlap. Tried Deepgram (didn't work out), then OpenAI Realtime API with function calling and time bounds. Tried all kinds of prompt-based instructions too - nothing reliably forces the agent to wrap up and signal completion. The agent either keeps talking and doesn't converge towards the next handoff milestone. I need bounded sessions that feel natural but end predictably. Anyone worked on voice agent termination patterns or seen approaches that solve this? Would appreciate pointers.