We chose to build Monogram in early 2024 after years spent building LLM features inside growing companies. The same issues kept appearing: prompts in code, no pre-shipping tests, and regressions users caught first. A third teammate joined a few months later, and we built what we had wanted ourselves.
This year-end account covers what happened, not a polished retrospective. Some things went as planned. Many did not.
What worked early
We built the eval engine before the prompt editor. It seemed backwards, but proved right. Our first hypothesis was that "run evals before you ship" was the core value. Building eval infrastructure first let us test whether developers wanted it before funding the full workspace.
It worked. Engineers in early design partner discussions immediately understood the eval problem. They had seen prompt changes that seemed safe, shipped without tests, and degraded a subset of outputs. The eval engine resonated, confirming we were working on the right problem.
Building for ourselves also worked. During the first few months, we continued consulting for early-stage companies while developing Monogram, so we used its early tooling on those LLM projects. Real project work reveals product friction in a way interviews cannot replace.
What went wrong
For most of the first half of the year, we treated deployment too abstractly.
We initially framed deployment around infrastructure: staged rollouts, blue-green deployments for prompt versions, and rollback mechanisms. Mature LLM platforms need those capabilities. But most engineers we spoke with were focused on a simpler question: how can I change a prompt safely, and know after shipping whether it broke anything?
We began with where we thought the problem would end, not where teams were. That made deployment more complicated than needed. Weeks went into a model early users did not yet need, wasting time and delaying the product.
The turning point came in August. We demoed staged rollouts to a developer who had spent about six months building an LLM feature. He was polite, not excited. Then he said: "What actually hurts is not knowing whether a prompt change is better or worse. I ship it and watch the support queue."
We refocused then. Deployment became simpler, with eval gates instead of rollout mechanics at its center. Prompt versioning linked to evals became the main story. Listening to the actual problem, rather than pursuing our imagined solution, led us there.
Competition proved noisier than expected
When we began, LLM tooling was crowded in observability, including logging, tracing, and cost monitoring, but sparse in evals. By the time our product worked, evals were crowded too. Several teams, including well-funded ones, shipped eval tooling in 2024.
We handled this by staying focused on our angle: the eval-to-deploy connection, prompt versioning as a first-class concept, and one workspace for building and shipping. We tried not to follow every competitor release. That was difficult for a small team with real competitive anxiety. Each reaction to a competitor cost time that could have gone to user-informed work.
More interest in eval tooling helps us because it validates the problem. Teams starting with eval-only tools eventually want the eval-to-deploy link we built. We prefer a growing market to a niche one.
SDK choices we'd revise
We'll cover this in a dedicated post, but two 2024 SDK decisions aged poorly.
First, the early Python client relied too heavily on chained method calls. It read fluently but hid intermediate steps, making debugging frustrating. We have since moved to an explicit config-object pattern.
Second, we made the client async-first. Most production LLM applications run async, so the default made sense. But most developers exploring the platform used synchronous scripts. Requiring asyncio.run() for a simple eval script added friction. We added synchronous wrappers in Q3, but should have included them initially.
Our view of 2025
The eval engine is solid and deployment works. Our focus now is bringing evals earlier into development. Teams usually add them after a feature works. We want them writing prompts and evals together, iterating on both before anything reaches production.
We are also improving connections to teams' existing CI/CD tools. Eval gates work, but connecting them to GitHub Actions or a standard deploy workflow remains rough. That is a 2025 priority.
The company remains three people who communicate constantly and decide within an afternoon. That will change, so we are protecting what matters most: the feedback loop. We need to keep speaking with product engineers and turning lessons into shipped changes within a week.
Thanks to everyone who tested early versions, shared feedback, filed bugs, or explained what failed. Developer tooling is unusual because users often understand the problem as well as the builders. Explaining what does not work is a meaningful contribution, and we appreciate it.