What LLMs Are Bad At: Engineering Failure Modes and When Deterministic Systems Win

What are LLMs bad at in production software?

LLMs are bad at any task requiring binary correctness, exact mathematical calculation, stateful transaction processing, temporal awareness, and strict schema compliance under edge conditions. They cannot guarantee factual consistency across consecutive calls, nor can they verify their own outputs. Because they operate on probabilistic token prediction rather than symbolic execution, they generate plausible-sounding falsehoods rather than halting on uncertainty.

Engineering teams routinely burn months attempting to prompt-engineer their way around five foundational limitations:

• Arithmetic and quantitative calculation: Models do not compute numbers; they guess the next numerical token based on training distribution. Asking an LLM to balance a general ledger, amortize a loan, or aggregate line items results in silent arithmetic drift.
• Long-horizon state tracking: An LLM has no persistent memory stack. When orchestrating an eight-step business process, the model frequently forgets intermediate variables, mutates payload shapes, or drops parameters established in step two.
• Rigid output parsing without failure: Even with constrained generation (JSON mode or grammar-guided sampling), temperature anomalies or rare token sequences can produce malformed JSON, truncated keys, or escaped characters that crash downstream parsers.
• Temporal and factual ground truth: A model has no inherent sense of "now." It cannot distinguish between a policy that was active in 2021 and one passed yesterday without exhaustive, deterministic Retrieval-Augmented Generation (RAG) context injected into the prompt.
• Negative constraint adherence: Instructions like "do not mention the pricing structure" or "never include customer identifiers" degrade as context grows. Generative models focus attention on semantic tokens, which paradoxically increases the odds of invoking the forbidden token.

The Tokenization Trap: Why Models Cannot Count or Do Math

LLMs process text through sub-word tokenization algorithms like Byte-Pair Encoding (BPE). When an engineer feeds 349 + 872 into a model, the model does not see the integers 349 and 872. It sees arbitrary token identifiers—perhaps token 4281 and token 19832.

Because mathematical operations require precise, positional digit-by-digit alignment, tokenization blinds the underlying attention heads to individual character boundaries. The model is forced to learn an addition table through associative memory rather than algorithmic calculation.

## What an engineer expects the LLM to do internally:
def add_integers(a: int, b: int) -> int:
    return a + b

## What the LLM actually does:
## P(next_token | "What is 349 + 872? The answer is")
## Picks highest probability token based on web text, not an ALU computation.

If you ask an LLM to count the letters in a word, reverse a string, or compute tax across an array of floating-point values, you are misusing the architecture. Solving this requires handing the computation to an external environment via function calling or an Abstract Syntax Tree (AST) evaluator. An engineer who writes a prompt to calculate sales commissions has shipped a liability. An engineer who writes a Python script called by an LLM tool-call has shipped a feature.

When should you use deterministic code instead of an LLM?

You should use deterministic code whenever an operation has a provably correct answer, requires sub-50-millisecond latency, demands strict auditability, or carries regulatory liability for incorrect outputs. If your business logic can be written as a series of if-then statements, a database query, or a finite state machine, deploying an LLM introduces latency, cost, and catastrophic failure modes with zero added value.

Engineering teams should enforce deterministic solutions over generative approaches across four distinct boundaries:

• Authorization and access control: Never allow an LLM to decide whether a user has permission to view a record. Role-based access control (RBAC) and attribute-based access control (ABAC) must execute in deterministic application code before data enters an LLM context window.
• Data transformation between known schemas: Converting a CSV to a PostgreSQL table or mapping a Stripe webhook payload to an internal billing model should be done with standard serializers, validation libraries (like Pydantic or Zod), and typed interfaces.
• Financial and inventory transactions: Deducting balance, reserving warehouse stock, and calculating shipping tariffs require ACID guarantees. Probabilistic systems cannot provide transactional rollbacks or atomic isolation.
• High-volume classification with static taxonomies: If you have 10 static categories and clean training data, a fine-tuned small model (like a BERT variant) or a simple logistic regression pipeline will run in 5 milliseconds for a fraction of a cent, outperforming a frontier LLM on both speed and operational cost.

For projects navigating these boundaries, our LLM development services focus on building strict boundaries between generative features and the deterministic backends that support them.

The Compounding Error Rate in Multi-Step Workflows

Autonomous agent architectures look compelling in slide decks. In production, they reveal why probabilistic systems fail at compound execution: the math of compounding errors destroys multi-step reliability.

Consider an agent pipeline designed to parse a customer ticket, query an ERP, assess customer tier, select a remediation path, and execute a refund. Each step relies on the model making an accurate tool call and extracting the correct parameters.

If the LLM operates with a 95% accuracy rate per individual step—an optimistic benchmark in complex enterprise environments—the pipeline's overall success rate degrades exponentially with each sequential decision:

• Step 1: 0.95 = 95.0% system reliability
• Step 3: 0.95 * 0.95 * 0.95 = 85.7% system reliability
• Step 5: 0.95^5 = 77.4% system reliability
• Step 8: 0.95^8 = 66.3% system reliability

By step eight, one out of every three executions fails. The failure modes are rarely loud crashes; they are subtle, silent errors: calling an API with an inverted boolean, dropping an order ID, or looping infinitely because the model misinterprets an API error message as a fresh user instruction.

Production-grade engineering avoids unbounded agent loops. Resilient systems use hard-coded directed acyclic graphs (DAGs) where deterministic code manages the state, routing, and transitions, invoking the LLM only for isolated, stateless extractions.

Deterministic Systems vs. LLMs: Architectural Tradeoffs

Choosing the right runtime engine requires mapping technical requirements directly against model capabilities:

How much does fixing LLM failure modes actually cost?

Fixing LLM failure modes typically costs between $120,000 and $250,000 in engineering time and infrastructure over a four-to-six-month remediation cycle. Teams routinely discover that while spinning up an initial prototype takes two weeks, building the deterministic guardrails, evaluation suites, and validation layers required to make the model enterprise-ready consumes 80% of the total project budget.

This cost breaks down across five concrete engineering investments:

1. Deterministic Validation Harnesses ($30,000 - $60,000): Writing custom Pydantic/Zod schemas, repair loops, AST parsers, and fallback handlers to catch and correct malformed outputs before they hit internal services.
2. Offline Evaluation Frameworks ($35,000 - $70,000): Developing automated CI/CD test suites with curated golden datasets to measure precision, recall, and regression every time a prompt or underlying base model changes.
3. Retrieval-Augmented Generation Architecture ($40,000 - $80,000): Building hybrid search systems (combining dense vector search with sparse BM25 indexing), rerankers, and chunking strategies to eliminate hallucinated context.
4. Tool-Use Integration Middleware ($15,000 - $40,000): Isolating the model behind sandboxed, rate-limited APIs so an erroneous token sequence cannot execute unauthorized database writes or corrupt third-party systems.

Teams that fail to allocate budget for these systems usually abandon their generative projects in staging, having built a prototype that delights internal stakeholders on 20 hand-picked demos but fails on 15% of real-world user queries. Building enterprise-grade generative systems requires balancing the strengths of probabilistic transformers with traditional software design; our team details this approach across our broader AI development services.

What this means for your team

If you are an engineering director, CTO, or product lead designing modern software, stop treating LLMs like drop-in replacements for application code. They are not databases, they are not logic gates, and they are not calculation engines. They are reasoning overlays that excel at bridging messy human communication with structured computing environments.

Architectural hygiene starts with four decisions:

• Isolate the LLM to the edges: Use models to extract structured entities from unstructured inputs, draft human-facing messages, or synthesize diverse text. Move those outputs immediately into strongly typed data models.
• Never let a model calculate: If your prompt contains the words "calculate," "sum," "total," or "average," rewrite the system to extract the raw variables and execute the math in code.
• Treat every model completion like untrusted user input: Run every output through strict schema validation, length boundaries, and security sanitization before passing it to internal services.
• Own your state deterministically: Build state machines in standard frameworks (temporal workers, step functions, or relational state stores). Never rely on an LLM to remember what it did five minutes ago.

If you have a stalled AI initiative, a model running wild in production, or a complex modernization project that needs senior engineering execution, talk to our team.

Schedule an engineering review with NextGen Coding Company to design a system that pairs probabilistic AI with deterministic software that does not break.