With a multitude of off-the-shelf, no-code solutions, why aren’t we actually seeing solid agents everywhere? Great for demos, great for the happy path, great for short-horizon work. Everything breaks down at the edges. And when it does, there’s a phrase waiting - you’ve heard it, and maybe you’ve said it too: “well, LLMs are inherently non-deterministic.”
That phrase is doing a lot of work. It sounds like an explanation - it’s actually a confession, covering for systems that have no observability, lack appropriate guardrails, defer way too much logic to an LLM, or (most often) all of the above.
After building these systems for a while, the position I’ve landed on is simple: use the LLM for what it’s good at - reasoning, data extraction, summarization, and yes… generating prose. Use code for everything that needs to work the same way every single time. The agent is the sum of the parts - intelligence at the borders, deterministic logic at the core. Get that split right and the ‘non-deterministic’ excuse mostly disappears, because the parts that fail are parts you can actually inspect.
Defensive coding, idempotent retries, schema validation - these aren’t meaningless rituals, they’re examples of what helps make a system reliable and maintainable (especially by someone other than the person who built it). Being able to observe and understand why an agentic system ‘worked’ matters just as much as whether it worked, because there’s rarely one business outcome or use case. Understanding the operations is the gateway to understanding failure.
No-code frameworks don’t give you any of that by default, and when you try to bolt it on, the visual flow that started out manageable becomes a spiderweb of intelligence, functionality, and rules that only the creator has any idea how to reason about. More often than not though, the complexity just hides in the prompt - LLM instructions that are overloaded, contradictory, with sprawling objectives. Visually clean, but hiding the complexity.
My own approach has evolved and in some ways come full circle. LLMs are great, but the instructions we accumulate aren’t always, and as the context grows what was a great demo becomes an unreliable black box. So I shrink the non-deterministic universe as far as I can, and push everything else into software systems - where a whole industry has been wrangling complexity for decades. Would you rather trust your cost-effective production model to follow instructions perfectly through every novel situation? Or point your best model and your engineer at the problem once, and codify the answer forever? That’s not to say there can’t be bugs, but they’re observable and can reliably be fixed and maintained vs. just another ‘prompt tweak’ you hope solves that edge case.
None of this is a knock on no-code for the class of problems it actually fits, like wiring up a personal agent for a daily flow, an ad hoc task, some research. You build quickly, don’t need to host any infrastructure or deal with a software development lifecycle, and get real value. Completely different requirements than a production system with real stakes, where failure means lost revenue and trust.
When the stakes grow and the scale increases, that’s where software engineering earns its keep. Customer trust doesn’t come from the demo. It comes from being able to answer why, and “because LLMs are non-deterministic” is not an answer.