LLM software development has to design for three properties of large language models: probabilistic output, knowledge limited to the context window, and fluent output that can be confidently wrong. Each property breaks a traditional engineering assumption, so reliable teams add evals, context engineering, and layered verification. Skipping these practices produces features that pass demos and fail in production.
When a large language model (LLM) feature misbehaves or a coding agent breaks a build, the usual fix is a sharper prompt and another code review. Over time, that fix stops holding, and 66% of software developers in the 2025 Stack Overflow Developer Survey named almost-correct AI output as their biggest frustration. Trust stays low in turn, with only 6.6% of respondents to the 2026 edition willing to rely on AI for important decisions.
The cause is that teams use traditional software engineering practices with LLMs. Those practices were built for components that behave the same way every time, and LLMs do not. LLMs differ from traditional components in three ways: their output is probabilistic, they know only what is in their context window, and they sound just as confident when they are wrong. Each of these properties breaks an assumption that traditional practices rely on, so each needs a practice of its own, whether the LLM runs inside the product or helps the team build it.
Why Do LLMs Break the Assumptions of Traditional Software Development?
LLMs generate output by predicting likely text rather than executing fixed logic, and that difference breaks three assumptions traditional software development relies on:
- Traditional components return the same output for the same input, yet LLMs are probabilistic and can answer one prompt differently on every run.
- Traditional components know the system they belong to, while an LLM knows only what sits in its context window.
- Broken code traditionally fails visibly, whereas an LLM fails fluently, so wrong output reads as convincingly as right output.
Because of these, LLM software development needs its own engineering practices on top of standard software development processes.
These three properties matter wherever a team uses LLMs, and the Software Engineering Institute at Carnegie Mellon separates two such places:
- LLM features inside the product, the territory of LLM application development and generative AI development services.
- LLMs in the development process, from code completion in coding assistants to autonomous coding agents.
The same properties and the same practices apply to both product features and AI-augmented software development. The table below maps each property to the assumption it breaks and the practice that replaces it.
Traditional assumption | LLM property | What breaks | Practice that replaces it |
|---|---|---|---|
Same input, same output | Probabilistic output | Assertion-only tests, reproducible bugs | Evals alongside tests |
The component knows the system | Knowledge limited to the context window | Consistency with codebase, domain rules, recent decisions | Context engineering |
Wrong output fails visibly | Fluent output that can be confidently wrong | Trust in plausible output; review at generation speed | Layered verification |
Traditional assumption
Same input, same output
The component knows the system
Wrong output fails visibly
LLM property
Probabilistic output
Knowledge limited to the context window
Fluent output that can be confidently wrong
What breaks
Assertion-only tests, reproducible bugs
Consistency with codebase, domain rules, recent decisions
Trust in plausible output; review at generation speed
Practice that replaces it
Evals alongside tests
Context engineering
Layered verification
Teams work with AI coding assistants and agents in different interaction patterns, and the patterns differ mainly in how much they account for the three properties. At one end sits vibe coding, a low-structure, prompt-and-accept style in which a developer describes the goal and accepts the code generated in reply. It skips the practices the three properties call for, which is acceptable for prototypes, technical exploration, and simple tasks.
At the other end, agentic engineering puts AI to work inside human-defined specs, tests, evals, and feedback loops, so those practices are built into every step. That structure is the basis of a mature agentic development workflow.
Since both sit on one spectrum, a team moves along it as the stakes of its software engineering tasks rise. The three properties decide where production work belongs, so any system that touches real users, real data, or real money needs the structure of agentic engineering at every step.
Property 1: Why Are LLMs Probabilistic, Not Deterministic?
LLMs are probabilistic because they generate output through next-token prediction, sampling likely tokens one at a time, so the same input can produce different outputs. A low temperature narrows that variation, but it does not guarantee identical results. Model updates, context changes, and serving infrastructure also shift behavior between runs.
This non-determinism changes what a passing result means. A green run becomes one sample among many, a reported bug may refuse to reproduce, and a model upgrade can change behavior with no code change at all. For example, a support assistant can answer a refund question correctly on Monday and cite the wrong policy on Tuesday, and a coding agent asked twice to refactor the same module can return two different structures.
A practical check is to ask the same question more than once and compare how much the answers vary. It also helps to avoid asking an LLM for any answer that ordinary code can compute, although asking the model to write that code remains a sound use of code generation.
What changes: from tests to evals (LLM evaluation)
LLM evaluation replaces the single pass or fail check with scoring across many runs.
Tests | Evals |
|---|---|
Checks | Deterministic code: given X, return Y |
Result | Pass/fail |
Runs | Once per change |
Built from | Specs and edge cases |
Still needed? | Yes, for all deterministic code around the model |
Tests
Checks
Result
Runs
Built from
Still needed?
Evals
Deterministic code: given X, return Y
Pass/fail
Once per change
Specs and edge cases
Yes, for all deterministic code around the model
Within an eval suite, output evaluation judges the final answer a model or agent produced. Trajectory evaluation goes a step further by judging the sequence of steps and tool calls an agent took, which is the core of knowing how to evaluate AI agents before production.
Many eval suites also rely on LLM-as-judge scoring, where a second model grades outputs against a rubric. Since the judge is probabilistic too, its scores earn trust only after calibration against human-labeled examples.
Serhii Leleko
AI & ML Engineer at SPD Technology
“A green test run on an LLM feature tells you that one sample passed. We set the release threshold on the eval pass rate across repeated runs. Every production failure also becomes a new eval case, because a failure that is not in the set tends to come back with the next model upgrade.”
In practice, a working LLM evaluation framework runs evals in CI on every prompt, model, or context change, with a threshold that blocks release when the pass rate drops. The eval set starts from golden datasets and grows with every failure seen in production, which is the central discipline of eval engineering for an early product.
Property 2: Why Does the LLM Context Window Limit What a Model Knows?
LLMs only know what is in their context window because every call is stateless. On each call, the model works from its training data, which stops at a training cutoff, plus whatever sits in the current LLM context window.
Anything outside those two sources is invisible to the model, so gaps in code context show up quickly. A coding agent, for example, writes an endpoint with its own error-handling style and data format because nobody told it the existing code already defines both. A product assistant answers a pricing question from outdated training data and behaves differently between sessions, because each one starts empty.
Supplying more information has limits too, because the LLM context window is finite and its effectiveness is uneven across its length. Long or noisy context degrades output quality, a pattern practitioners call context rot. It happens because the model’s attention spreads across everything loaded, so irrelevant files and stale instructions dilute the signal the task needs. Every extra token also adds cost and latency per request, which makes up a large part of what AI-assisted development really costs.
What changes: context engineering replaces prompt engineering
Most teams begin with prompt engineering, which tunes the wording of a request, often through reusable prompt templates.
Context engineering widens the scope to everything the model sees for a task, so prompt wording becomes one small part of a larger design that replaces ad hoc context construction.
Context type | What it holds | Static or dynamic |
|---|---|---|
Instructions | Rules, conventions, role | Static (rule files such as AGENTS.md or CLAUDE.md) |
Knowledge | Docs, specs, code, domain data | Dynamic (retrieval, RAG) |
Memory | Decisions and state across sessions | Static or dynamic |
Examples | Reference implementations, good outputs | Dynamic |
Tools | APIs, search, test runners the model can call | Dynamic (loaded when needed) |
Guardrails | Constraints and forbidden actions | Static |
Context type
Instructions
Knowledge
Memory
Examples
Tools
Guardrails
What it holds
Rules, conventions, role
Docs, specs, code, domain data
Decisions and state across sessions
Reference implementations, good outputs
APIs, search, test runners the model can call
Constraints and forbidden actions
Static or dynamic
Static (rule files such as AGENTS.md or CLAUDE.md)
Dynamic (retrieval, RAG)
Static or dynamic
Dynamic
Dynamic (loaded when needed)
Static
Static context loads on every call, so it should stay short and stable. Dynamic context, by comparison, arrives per task through retrieval and RAG, so its value depends on being findable and accurate when the task asks for it.
For coding agents, the static layer usually lives in rule files such as AGENTS.md or CLAUDE.md, which hold team conventions, architectural constraints, forbidden patterns, and the commands for running unit tests. Those files form one part of the harness, meaning the rule files, tools, sandboxes, hooks, and evals that surround a coding agent, and designing that whole setup is the work of agent harness engineering. Instructions for specific programming tasks can sit in separate skills that load only when needed, and every one of these context assets deserves versioning, review, and an eval run before it changes.
Property 3: Why Can LLMs Be Fast but Confidently Wrong?
LLMs can be fast but confidently wrong because they produce the most plausible continuation of their input with no built-in check of correctness. As a result, wrong output arrives with the same fluency as right output, which makes it one of the costliest LLM limitations.
In coding tasks, this behavior shows up as invented APIs or functions, code snippets that compile and still miss edge cases and business logic, and hallucinated package names. Product features show the same pattern, for example when a support answer cites a return policy the company never had and delivers it in the same calm tone as a correct one.
Across a coding project, these small errors add up to an imbalance in effort. AI produces the boilerplate code and most of a feature quickly, while the remaining work on edge cases, integration, error handling, and correctness takes disproportionate time and human expertise. That imbalance shapes how teams should plan and review AI-generated code.
Hallucinated package names also open a supply-chain attack called slopsquatting, in which attackers register the invented names, and it ranks among the main AI-generated code security risks. A USENIX Security 2025 study shows how wide that opening is, because across 576,000 code samples generated with 16 LLMs in two programming languages, Python and JavaScript, hallucinated-package rates averaged at least 5.2% for commercial models and 21.7% for open-source models. Taken together, those samples contained 205,474 unique invented package names that an attacker could register.
Agents add a further exposure called the lethal trifecta, in which an agent that combines private data, untrusted input, and a way to send data out can be steered by prompt injection into leaking that data. That risk is set by what the agent can reach, so the fix is to break the trifecta: scope its data access, treat external input as untrusted, and gate any outbound action.
What changes: layered verification sets the pace
When advanced code generation capabilities produce output in seconds, the time needed to establish correctness sets the limit on delivery speed.
Layered verification responds by stacking cheap automated checks beneath scarce human judgment, so each layer catches what the one before it lets through.
Layer | What it catches | Examples |
|---|---|---|
Deterministic checks | Syntax, types, regressions, known vulnerabilities, unknown dependencies | Tests, type checks, linters, SAST, dependency allow-lists |
Evals | Quality and behavior drift in LLM output | Eval suite in CI with thresholds |
Guardrails and hooks | Forbidden actions, policy breaches, unsafe tool use | Pre-commit hooks, scoped permissions, output filters |
Human-in-the-loop by risk | Wrong architecture, money, personal data, security | Required approval for high-risk changes; lighter review for low-risk ones |
Observability | Failures that only appear in production | Logged prompts, outputs, tool calls, and quality signals |
Layer
Deterministic checks
Evals
Guardrails and hooks
Human-in-the-loop by risk
Observability
What it catches
Syntax, types, regressions, known vulnerabilities, unknown dependencies
Quality and behavior drift in LLM output
Forbidden actions, policy breaches, unsafe tool use
Wrong architecture, money, personal data, security
Failures that only appear in production
Examples
Tests, type checks, linters, SAST, dependency allow-lists
Eval suite in CI with thresholds
Pre-commit hooks, scoped permissions, output filters
Required approval for high-risk changes; lighter review for low-risk ones
Logged prompts, outputs, tool calls, and quality signals
At the policy layer, well-designed guardrails stop forbidden actions and unsafe tool use at the moment an agent attempts them. Production observability then closes the loop by logging prompts, outputs, and tool calls and tracking quality signals alongside platform health metrics.
Across all layers, review effort follows risk. Human-in-the-loop approval covers expensive decisions about money, personal data, architecture, and security, while automated gates handle low-risk changes and routine code suggestions. That split lets review keep pace with generation and shifts the core engineering skill from manual coding toward verification and judgment.
Want a practical method for deciding when generated code can ship? Read our guide on how to trust AI-generated code.
How Do the Three Properties Reshape the Software Development Lifecycle?
The three properties reshape the software development lifecycle by moving work from writing code toward specifying intent, curating context, and verifying output. Each stage keeps its purpose, so existing software engineering processes adapt as their inputs, from natural language descriptions to generated code, change.
SDLC stage | Before LLMs | With LLMs, designed for the three properties |
|---|---|---|
Requirements | Tickets and conversations | Specs with acceptance criteria an agent and an eval can use (spec-driven development) |
Design | Architecture documents | Architecture plus context assets: rule files, examples, constraints |
Implementation | Engineers write code | Agents implement inside a harness; engineers direct and decompose |
Testing | Unit and integration tests | Tests for deterministic code plus evals for LLM behavior |
Review | Line-by-line review of everything | Risk-based review; automated gates handle volume |
Operations | Monitoring uptime and errors | Plus observability of model behavior, cost, and drift; controlled model upgrades |
SDLC stage
Requirements
Design
Implementation
Testing
Review
Operations
Before LLMs
Tickets and conversations
Architecture documents
Engineers write code
Unit and integration tests
Line-by-line review of everything
Monitoring uptime and errors
With LLMs, designed for the three properties
Specs with acceptance criteria an agent and an eval can use (spec-driven development)
Architecture plus context assets: rule files, examples, constraints
Agents implement inside a harness; engineers direct and decompose
Tests for deterministic code plus evals for LLM behavior
Risk-based review; automated gates handle volume
Plus observability of model behavior, cost, and drift; controlled model upgrades
The biggest shift in the table starts at the requirements stage, where spec-driven development has an explicit specification and acceptance criteria ready before any agent implements. That way, the agent and the eval work from the same definition of done. Applied ticket by ticket, that discipline becomes spec-first development, which keeps implementation details tied to agreed intent.
The engineer’s role shifts along with the process, and IBM principal research scientist Kaoutar El Maghraoui describes engineers moving from producing code to curating it. In practice, engineers and software architects become system designers and quality arbiters, which is the shift that lets AI-native development teams ship faster. Under this factory model, the team builds and maintains the system that produces the software, stopping well short of fully automated software engineering.
Operations change in a similar way, since model upgrades become controlled releases with evals run on the candidate model before any switch. Teams that self-host an LLM also gain control over exactly when that switch happens.
What Should Engineering Leaders Change First in LLM Software Development?
Engineering leaders should start where unreliability costs the most, which in LLM software development usually means missing evals and missing context assets. Risk-based verification comes next, once the team can measure whether an LLM-dependent behavior improved or degraded after a change.
Beyond the feature itself, leaders also need to measure delivery from the first week. Google’s DORA 2025 State of AI-assisted Software Development report links AI adoption to higher delivery throughput and to higher delivery instability at the same time. A team that tracks speed or developer productivity alone therefore sees the first effect and learns about the second only through incidents.
In practice, the first rollout comes down to a short checklist, roughly in this order:
- Build a first eval set of 50 to 100 cases for one LLM feature or agent workflow and run it in CI.
- Write a short, versioned rule file for coding agents and move task-specific instructions into retrievable context.
- Define which changes require human approval, starting with money, personal data, architecture, and security.
- Add dependency allow-lists and automated checks for LLM-generated code.
- Log prompts, outputs, and tool calls for LLM features in production.
- Treat model upgrades like dependency upgrades by running evals before switching.
- Measure delivery outcomes such as change failure rate, rework, and cost of change, and set aside the share of AI-written code as a success metric, since the ROI of AI-assisted software engineering shows up in those outcomes.
Progress shows up in delivery metrics, which is how AI-enhanced development velocity can be tracked over time. For teams bringing in a partner, the same sequence also helps define scope and success criteria when starting an AI-assisted engineering engagement.
Ready to test a release against these practices? Run it through our AI production-ready checklist.
Why Choose SPD Technology for LLM Software Engineering
SPD Technology builds LLM software that survives production by designing for all three properties. Evals, context, and verification are part of our AI/ML development services from the first sprint. That approach draws on 20 years of custom software delivery, with one team accountable for both software architecture and execution.
In client work, that approach shows up in several ways:
- Evals as the release gate. Every LLM-dependent behavior is scored against a threshold before release, and a named senior engineer approves each release.
- Context and harness the client owns. The iconic US landmark project’s monorepo now holds 47 custom skills, 6 subagents, and 19 slash commands that carry the team’s conventions, a setup described in the AI-engineering case for an iconic US landmark. With those conventions codified for coding agents, the repeat client’s output in 2026 rose 89% against the 2025 baseline at constant headcount, while code-review throughput doubled and human-reported bugs fell 81%. Comparable harness work runs through our agentic AI development services.
- Verification matched to risk. Incident response time fell from more than 60 minutes to under 30 minutes to a PR-ready fix once SPD Technology built the AI incident pilot for a US fintech/SaaS platform. Because the AI resolves up to 70% of incidents correctly, the platform needs less 24/7 on-call engineering, and every fix still arrives as a pull request for human review.
- Production-proven LLM features. A French B2C fashion retailer needed a support chatbot that would hold up under peak load. Running on LangChain with a Mistral LLM and RAG grounding on AWS, the chatbot answers 99% of queries in under 10 seconds and kept 100% uptime during peak hours at 30 requests per second.
- Claude expertise. SPD Technology is an Anthropic partner with Claude-certified architects. That expertise feeds directly into building production AI with Claude, where the same three practices apply.
Deciding who should help you with an AI-assisted project? See our guide on how to choose an AI development partner.
Key Takeaways
- Because LLMs have probabilistic output, a single passing test run proves only that one sample passed, so LLM-dependent behavior needs evals that score many runs against a release threshold.
- Knowledge limited to the context window means missing conventions and domain rules produce wrong output that prompt rewording alone cannot repair, so context engineering has to supply the missing knowledge.
- Fluent output that can be confidently wrong makes verification the limit on delivery speed once generation becomes fast, which is why layered verification decides how fast a team can safely ship.
- Tests verify the deterministic code around the model and evals verify the model’s behavior, so production LLM software needs both suites running in CI.
- When review effort follows risk, human approval stays on money, personal data, architecture, and security, and review keeps pace with generated code.
- Running evals before every model upgrade prevents silent behavior changes in production, the same way tests gate dependency upgrades.
- One set of practices covers both LLM features in a product and coding agents in a team, because the same three properties apply in both places.
In short: software built on LLMs becomes dependable when teams design for probabilistic output with evals, for knowledge limited to the context window with context engineering, and for fluent output that can be confidently wrong with layered verification.
FAQ
Why are LLMs non-deterministic?
LLMs generate text by sampling likely next tokens, so the same input can produce different outputs. Lowering the temperature reduces that variation but does not guarantee identical results. Model updates, changes in context, and differences in serving infrastructure can also shift behavior between runs. Software that uses an LLM therefore needs evals that run each case many times and score the results against a threshold, so one lucky run never decides a release.
What are the main limitations of LLMs in software development?
The main limitations come from three properties: probabilistic output, knowledge limited to training data and the current context window, and fluent output that can be confidently wrong. In day-to-day work, they show up as inconsistent results between runs, code that ignores project conventions, invented APIs and packages, and missed edge cases. Each limitation has an engineering answer: evals, context engineering, and layered verification. Human judgment still decides high-risk changes involving money, personal data, architecture, and security.
How do you test software that uses an LLM?
Software that uses an LLM needs two kinds of checks. Unit and integration tests cover the deterministic code around the model, while evals cover the model’s behavior by running a representative set of cases many times and scoring the results against a rubric or reference answers. For agents, trajectory evaluation also judges the steps and tool calls, not only the final output. Both suites run in CI on every prompt, model, or context change, and every production failure becomes a new eval case.
What is the difference between tests and evals?
Tests check deterministic behavior: given a specific input, the code must return a specific output, and each case passes or fails. Evals measure non-deterministic behavior, such as the quality or correctness of LLM output, and return a score or pass rate across many runs. Eval sets are built from golden datasets, rubrics, and real failures, and are often scored with LLM-as-judge. Production LLM software needs both.
What is context engineering, and how is it different from prompt engineering?
Context engineering designs everything an LLM sees for a task: instructions, knowledge, memory, examples, tools, and guardrails. Static context, such as rule files with team conventions, loads on every call, so it should stay short and stable. Dynamic context, such as retrieved documents and tool results, arrives per task. Prompt engineering covers only the wording of the request, which is one part of that design, and since many production failures trace back to missing or noisy context rather than poor wording, context engineering has the larger effect on reliability and token cost.
What is LLM-as-judge?
LLM-as-judge is an evaluation method in which one model scores another model’s output against a rubric. It makes qualities such as relevance, correctness, or tone practical to evaluate at scale. The judge is probabilistic too, so its scores should be calibrated against human-labeled examples and rechecked periodically. It complements deterministic checks and human review rather than replacing them.
What is the difference between vibe coding and agentic engineering?
Vibe coding is low-structure, prompt-and-accept use of AI tools with little specification or verification, which suits prototypes and technical exploration where failures are cheap. Agentic engineering puts AI inside human-defined specs, tests, evals, guardrails, and review, so its output can be verified before it ships. The two are ends of one spectrum rather than separate tools. The more a system touches real users, real data, or real money, the closer it needs to sit to agentic engineering.
How long does it take to put evals, context engineering, and verification in place for an existing LLM feature?
For a single existing LLM feature, putting all three practices in place typically takes 4 to 8 weeks with one or two engineers, plus a domain expert who supplies business rules part-time. A first eval set of 50 to 100 cases, built from real usage and known failures and wired into CI, takes about 2 to 3 weeks. A versioned rule file and retrieval for context add 1 to 3 weeks, and risk-based approval gates with production logging take another 1 to 2. Teams without existing logging, labeled examples, or mature CI should plan for 10 to 12 weeks.
How much does it cost to make LLM software reliable?
Making one LLM feature reliable usually takes about 4 to 16 engineer-weeks of initial work, in line with the timeline above, covering eval sets, context assets, approval gates, and logging. Inference spend comes on top: 100 eval cases sampled 5 times each means 500 model calls per CI run, and LLM-as-judge scoring roughly doubles that number. Ongoing eval maintenance typically takes 10 to 20% of one engineer’s time as the product and models change. That spend pays back through less rework and fewer production incidents, which cost more once users see them.