For a few years, the recipe for a capable language model resembled manufacturing: gather or synthesize large volumes of training rows, run gradient descent, ship the weights. Magpie is a clean example of that philosophy. By feeding an aligned model nothing but its own pre-query template, researchers got it to generate millions of plausible user instructions, then answer them — an assembly line that produced alignment data at scale with no human in the loop. The model was both the factory and the raw material.

That approach worked, and it still matters. But the center of gravity in LLM systems has moved. Capability is no longer something you bake entirely into weights at training time; it is increasingly assembled at runtime, through typed message formats, function schemas, and interoperability standards like the Model Context Protocol. This post traces that shift — from static data production to dynamic composition — and maps out what it means for anyone building software on top of these models. It picks up where my earlier piece on Harmony and training formats left off: if channels are the interface layer between weights and runtime, the question here is how that interface came to carry more of the system’s intelligence than the training corpus itself.

The Data Factory Era and Its Ceiling

Static dataset generation solved a real bottleneck. Human-written instruction data was scarce, expensive, and legally fraught, and techniques like Magpie, Self-Instruct, and Evol-Instruct showed that models could manufacture their own curriculum. The Magpie method — exploiting the autoregressive template so the model completes the “user” side of a conversation it was never given — raises a question worth sitting with: how much conversational structure do post-trained weights already contain, waiting to be extracted?

The ceiling, however, is built into the method. A synthesized corpus is a snapshot: its distribution is frozen the moment training ends. Every new API, coding convention, or enterprise system the model should interact with would require fresh rows and fresh gradient steps. Facts and behaviors also get entangled — teach a model about a specific weather API by memorization, and the knowledge rots the day the API changes. The factory could produce volume; it could not produce currency or context.

The Escape Route, Stage by Stage

The escape route emerged in stages, and the sequence is easier to follow if each stage is read as an answer to the previous one’s open problem.

Stage one: act by prompting. ReAct interleaved reasoning traces with actions expressed as plain text — the model wrote “Action: search[…]” and a wrapper script parsed it, ran the query, and pasted the result back into the prompt. This proved a base model could operate external systems with no special training at all.

The open problem: everything hinged on heuristic parsing of free-form text, which broke whenever the model phrased an action differently.

Stage two: learn the habit from synthesized data. Toolformer had the model annotate its own training text with candidate API calls, kept the insertions that measurably improved next-token prediction, and fine-tuned on the result. It is a transitional artifact — a data-factory technique in service of a dynamic behavior — and it made tool invocation a learned reflex rather than a prompting trick.

The open problem: the tool set was fixed at training time, so every new capability meant another training run.

Stage three: pass the tools in at runtime. Native function calling moved the schemas into the request itself. The model reads JSON descriptions of available functions in context and emits arguments that validate against them. What the weights retain is a structural pattern — read schema, select function, construct arguments — which is why a model fine-tuned on a few thousand examples can call APIs it has never seen.

The open problem: output still arrived as one undifferentiated stream, leaving the runtime to guess whether a given blob of JSON was a call, a draft, or an answer.

Stage four: type the conversation itself. ChatML gave messages explicit roles and boundaries; Harmony extended this with output channels — analysis for internal reasoning, commentary for actions bound to the runtime, final for the user-facing answer. Ambiguity that stage three left to heuristics became a typed contract the serving stack can rely on.

The open problem: every model vendor and every capability provider still wired their integrations one-off.

Stage five: standardize the boundary. MCP defines how a client discovers tools, resources, and prompt templates from independent servers at connection time. Nothing about a specific database, ticketing system, or search index needs to live in the weights; the capability arrives with the connection, described in text the model already knows how to read. Integration stops being a training problem and becomes a configuration problem.

What LSP Taught MCP

Stage five has a precedent in developer tooling. Before the Language Server Protocol, every editor needed a bespoke integration with every programming language — an M×N growth in glue code. LSP collapsed it to M+N with one contract: any editor speaking the protocol gets completions, diagnostics, and refactoring from any compliant language server.

MCP applies the same compression to models and capabilities, and the analogy runs deeper than architecture. LSP succeeded because it let editors and compilers evolve on separate schedules; the protocol boundary is what made independent progress possible. The same decoupling is what lets a tool server built today work with a model released next year.

The Breakdown of Monolithic Training

Seen through this lens, training itself has decomposed into layers with distinct jobs. Pretraining supplies linguistic and world priors. Instruction tuning teaches the conventions of roles and turns. A further fine-tuning pass teaches the grammar of action — when to act, how to construct arguments, how to integrate returned results. Channelized formats teach the model to separate deliberation from invocation from presentation.

None of these layers tries to be the whole system anymore. The monolith — one corpus meant to instill every fact and behavior — has given way to a division of labor: weights learn transferable patterns, while the runtime supplies the specifics of the moment. One way to compress this: training provides the grammar, the environment provides the vocabulary. Synthetic data generation still has a role, but its target changed. You generate examples to sharpen the pattern — schema reading, argument construction, result integration — not to enumerate the world.

A Conceptual Map: How Models Actually Write Code

Code generation makes the dynamic view concrete, because correctness is externally checkable. Picture the system as four concentric layers. At the core sit the weights, trained on mixed text-and-code corpora — they contribute priors about syntax, idiom, and likely program structure. Around them sits the format layer: channels and schemas that let planning, action, and presentation travel as distinct, machine-readable streams. Around that sits the runtime loop: compilers, test runners, linters, static analyzers, and protocol-connected services the model can invoke. The outermost layer is feedback — execution results flowing back into context, reshaping the next generation step.

A coding agent drafts a plan in its reasoning lane, requests a test run through the action lane, receives a traceback, and revises. Functional correctness emerges from this circulation between model and environment, not from a hidden verification oracle inside the parameters. The training corpus made the first draft plausible; the loop makes the final artifact correct. That distinction is the argument of this post in miniature: a model that writes working software is not reciting its dataset — it is participating in a feedback system whose most important components live outside the network.

Actionable Insights for Developers

First, treat schemas as prompts. The descriptions in your function and server definitions are read by the model at inference time; an afternoon spent on precise parameter docs often beats a week of fine-tuning. Second, keep one model on the hot path. Channels exist so that a single inference stream can serve the UI, the executor, and the logger — add model hops only when a validator or router earns its latency.

Third, log per lane. When reasoning, actions, and answers are typed separately, you can evaluate structural validity — did the arguments parse and validate? — independently from answer quality, and you can mask or weight lanes differently if you later train on your own traces. Fourth, aim synthetic generation at patterns rather than facts: Magpie-style pipelines work well for teaching argument construction and result integration, and poorly for anything that changes monthly. Finally, build against open contracts rather than vendor endpoints — a capability exposed through a standard protocol survives model swaps; a bespoke integration does not.

Toward Circular, Modular — and Legible — Systems

The endpoint of this evolution is not the death of the data factory but its relocation. Agent deployments now generate the artifact that factories once had to fabricate: complete, structured traces of plans, invocations, results, and outcomes. Filter those traces by success, and they become training rows; fine-tune on them, and the improved weights produce better traces. The pipeline has bent into a circle, with the production environment as its own curriculum generator. Typed lanes, schema-described capabilities, and protocol boundaries make each component of that circle — weights, runtime, servers, evaluators — replaceable on its own schedule, the way LSP let editors and compilers evolve independently.

There is a newer force reshaping the training side of this loop: research into what happens inside the network, at the level of activations. Sparse autoencoders showed that the hidden states of a transformer can be decomposed into thousands of individually interpretable features — concepts, personas, syntactic roles — as demonstrated in early dictionary-learning work and scaled to production models in Anthropic’s monosemanticity research. In parallel, representation engineering and activation addition showed that behavior can be shifted by adding directions in that feature space at inference time — no gradient steps required. Read against the arc of this post, that is a familiar move: yet another capability migrating from the weights to the runtime.

This line of work is starting to change how training decisions get made. OpenAI’s study of persona features and emergent misalignment traced a fine-tuning failure — narrow training on insecure code producing broadly misaligned behavior — to identifiable directions in the model’s internal representations, then used that diagnosis to design a small corrective fine-tune. Anthropic’s attribution graphs trace multi-step computations through a model, offering a way to check whether a fine-tune changed the mechanism or merely the surface behavior. The practical pattern emerging from both: inspect the feature space before and after training, and let what you find decide what to retrain, what to steer, and what to leave alone.

The field is still early here, and the open questions are the interesting part. Do features stay stable across fine-tunes, so that a monitor built today survives next quarter’s training run? Can feature-level checks run cheaply enough to sit inside the agent loop itself, alongside the compilers and test runners? And if conversation formats gave us typed channels for text, will some future contract expose internal state the same way? The trajectory of the last few years suggests a consistent direction: systems built from parts that are separately trainable, separately swappable, and — increasingly — separately inspectable. The teams that internalize this will spend less time manufacturing static corpora and more time designing the contracts and feedback loops through which their systems teach themselves.