Over a quarter, additions accumulate into a file found in nearly every LLM project I have seen: utils.py, helpers.py, or llm_utils.py. It starts with two or three functions and reaches forty a year later. Nobody owns or fully understands it. Removing anything feels risky because something may depend on it.
That's the glue code problem. It isn't unique to LLM development, but LLM work produces it faster and more densely than nearly any other application work. If you build LLM features and haven't encountered it, you will.
Why LLM glue code grows so fast
The usual explanation is that fast-moving teams skip refactoring. True, but incomplete for LLM work. LLM projects have a structural reason for producing so much glue.
LLM work means frequent, slightly varied operations that existing libraries don't cover well. You must inject variables into prompts, parse structured output that is almost valid JSON, count tokens before requests to stay within context limits, retry with exponential backoff, log requests and responses for debugging, and truncate overlong document context.
None is large. Each takes a twenty-line function, but you need them repeatedly across features. So you write each once, place it in a shared file, and continue. Nobody writes the same twenty-line function again when a close match exists. Variants multiply: format_prompt, format_prompt_v2, format_prompt_with_context, and format_prompt_safe. Each differs slightly, serves different locations, and lacks documentation.
The problem's three phases
Glue code tends to follow a predictable path.
Phase 1: Useful scaffolding. You're moving quickly, and the utility functions truly help. The codebase is small enough for any engineer to understand. It feels good. The cost stays hidden because the debt has not compounded.
Phase 2: Dependency creep. Production features depend on your utilities. Some have subtle behavior that application code now requires, not by design, but because it existed and worked. Changing one safely means checking every call site. Refactoring costs too much, so you add variants instead.
Phase 3: Load-bearing infrastructure. Glue code now serves production traffic, handling retries, token counting, structured output parsing, and context management. It was not built for this and has no tests of its own. Changes require knowing how every feature uses it, which nobody fully does. Incidents are difficult to debug when requests pass through ownerless code.
When we were building Monogram, most teams I spoke with were in phase 2 or early phase 3. They weren't in crisis, but mysterious slowdowns were appearing. Looking through the utility layer usually removed the mystery.
What is really at stake
Glue code's true cost is not maintenance. It is the eval blindspot it creates.
When token counting, context truncation, and prompt assembly sit inside opaque utilities, evals struggle to mirror what the model receives at inference time. An eval may build a request differently from production. Then eval results and production behavior measure different things, and regressions you expect to catch pass through.
This shaped much of our early product thinking. We saw eval suites that looked sound but failed to predict production quality. In cases where evals passed and production failed, the cause was usually request assembly: the eval path built requests one way, the production utility another, and no one saw the divergence.
Fixing it without rewriting
The answer is neither abandoning utility functions nor rewriting everything. Use practices that block the worst failure modes.
Name the canon. Choose the official version of each utility, and document it briefly, even in a comment. Archive or remove variants. If one serves a real purpose, state that purpose.
Separate request assembly and execution. The function building the prompt and the function sending it to the model should be independent and separately callable. Then each can be tested alone, and the same assembly logic can serve evals and production.
Assign utility code owners. This needn't be heavy: an owner is accountable for understanding the function and reviewing changes. Without an owner, code has no visible change cost and can quietly become load-bearing.
Treat context management as first-class code. Token counting, truncation strategy, and context window management are most likely to create subtle differences between eval and production. Give them the same care as business logic.
Where tooling helps and doesn't
A prompt management platform can handle structured pieces of this work: prompt versioning, variable injection, structured output parsing, and request logging. Moving them out of the utility layer removes its most change-sensitive functions.
Tooling cannot solve the organizational side. Without a habit of naming canon versions and assigning ownership, a new tool only creates another home for variants. We have seen teams adopt prompt management, then make ten unnamed versions of every prompt without a deprecation plan. The problem moved, not disappeared.
Tooling is necessary, not sufficient. It must be paired with the practices.
The question ahead
Ask not "how much glue code do we have?" Ask, "if an engineer joined this week, where would they learn how production prompt requests are built?" If they must read several files and ask around, the problem is real. A clear, documented, tested path puts you in a different position.
One point deserves emphasis: working glue code is still working code. Don't remove it for purity. Understand it, keep it stable, and run evals through the same path as production traffic. Together, these remove most risk even when the code remains messy.