TB: Agentic AI Systems

Overview

Multi-agent AI systems outperform single agents on long, complex, subjective tasks — but not because single agents lack capability. Single agents fail due to context degradation (coherence breaks as context fills) and self-evaluation bias (agents overpraise their own work). The structural fix is a GAN-inspired separation: a generator that produces, an evaluator that independently critiques, and structured context resets that replace summarization. As models improve, harness complexity doesn't shrink — it advances toward more ambitious applications.

Core Tension

More sophisticated harnesses produce dramatically better output but at significant cost — a 6-hour/$200 run versus a 20-minute/$9 run that produces broken output. The right harness complexity is a function of the current model generation: assumptions baked in for one model may be unnecessary scaffolding for the next. Harness design must be adaptive, not fixed. Re-examine every component on each new model release.

Key Insights Across Sources

Single-agent failure modes are structural, not incidental

  • Context management: models lose coherence as context fills. Some exhibit "context anxiety" — prematurely wrapping up work as they sense context nearing its limit. Context resets (fresh agent + structured handoff document) outperform summarization because the new agent starts with high-quality compressed state rather than a degraded long context.
  • Self-evaluation bias: agents consistently overpraise their own work, especially on subjective tasks where no objective verification exists. Separating the generation and evaluation roles breaks this loop — an evaluator that didn't produce the artifact has no ego investment in its quality.
  • These are not model bugs to be patched; they are structural properties of how transformer-based agents use context. The architectural response (separation + reset) is more robust than prompting tricks.

GAN-inspired architecture: generator, evaluator, planner

  • Rajasekaran (Anthropic, 2026): three-agent full-stack harness — Planner (expands brief to spec, deliberately avoids granular implementation details to prevent cascading errors), Generator (implements incrementally, self-evaluates before handoff), Evaluator (uses Playwright to test like a real user; validates against sprint contracts).
  • Sprint contracts are pre-negotiated success criteria per sprint — more reliable than free-form evaluation because the standard is defined before the artifact exists, not after.
  • Evaluator value is asymmetric: it remains useful even as generators improve, because bugs are still bugs. The evaluator is the last line of defense against regressions that look correct at the code level.

Human-in-the-loop placement is a first-order design decision

  • In the JFC HEP framework (Moreno et al. 2026), a mandatory human gate sits before full unblinding — the pipeline halts and presents a structured summary; the human must APPROVE / REQUESTCHANGES / HALT. This is not a concession to caution; it is the correct epistemic design. Scientific judgment at the critical decision point cannot be automated.
  • The general principle: identify the decision where human judgment is genuinely load-bearing (not just conservative), and put the gate there. Everywhere else, let the agents run.

Karpathy's autoresearch: the human programs the process

  • autoresearch (Karpathy): give an agent a single-file training setup and a fixed 5-minute time budget per experiment; it modifies, trains, evaluates, and iterates overnight. ~100 experiments per night.
  • The human's job is writing program.md — the research org specification, not individual experiments. The interesting meta-question becomes: what research org design achieves the fastest progress?
  • This is the asymptotic form of the role shift: the human moves up one level of abstraction per generation of models.

Harness design is adaptive, not permanent

  • Remove non-load-bearing components when models improve. With Claude Opus 4.6, sprint decomposition became unnecessary — the model handled longer uninterrupted stretches. The evaluator remained valuable; the sprint structure did not.
  • Cost scaling from Rajasekaran: solo (no harness) → $9, 20 min, broken output; full harness → $200, 6 hr, polished app; simplified harness with Opus 4.6 → $125, 3h 50m, functional DAW. The quality gap is enormous even after simplification.
  • Iterating harness design based on observed model behavior — not theory — is the empirical approach. The right abstraction for harness design is currently unknown; treat it as an experimental variable.

Steinberger's OpenClaw: personality, self-modification, and open-source agentic engineering

  • Lex Fridman #491 (Steinberger, 2026): "agentic engineering" — intentional, structured agent orchestration — is categorically different from "vibe coding" (undisciplined prompting). Steinberger calls vibe coding "a slur." The practitioner runs 4–10 agents simultaneously, reviews database-touching code carefully, skips boring data transforms.
  • Self-modifying agents are already a practical reality, not a theoretical future: OpenClaw (175k+ GitHub stars) was given full access to its own source code, enabling the agent to autonomously modify itself — "You just prompted it to existence, and then the agent would just modify its own software."
  • The "agentic trap": sophisticated orchestration setups eventually simplify back to short prompts after the practitioner achieves domain mastery. Intermediate scaffolding is a learning crutch, not permanent architecture.
  • Personality and culture are load-bearing in open-source AI tools — not cosmetic. The soul.md file (agent writes its own personality definition, inspired by Anthropic's constitutional AI) differentiated OpenClaw from technically comparable competitors. "It's hard to compete against someone who's just there to have fun."
  • Security caveat: smarter models resist prompt injection better, but capability and attack-surface scale together — a "weird three-dimensional trade-off." Private network deployment and allow-list configs reduce risk significantly.

Karpathy's "decade of agents": what still needs to be solved

  • Source: autoresearch (Karpathy) + Dwarkesh interview (Oct 2025). See ~/ClaudeBrain/papers/2025-karpathy-agi-decade-away.md
  • Karpathy argues current agents are "very early" — impressive, but missing: continual learning, reliable multimodality, robust computer use, and off-manifold generalization. None of these are solved by scale alone.
  • The "off the data manifold" failure is the key diagnostic: agents fail hardest on inputs unlike their training distribution — architecturally novel codebases, custom implementations, non-standard patterns. Three tiers: (1) autocomplete = current sweet spot; (2) vibe coding = good for boilerplate and unfamiliar languages; (3) full agentic = fails on novel architectural problems. "They're not very good at code that has never been written before."
  • Nanochat as proof: models kept trying to use DDP (Distributed Data Parallel) even when Karpathy used a custom gradient sync; they're "over-defensive" — adding try-catch blocks, making production code that wasn't asked for, using deprecated APIs. Autocomplete was the only reliably useful mode.
  • RL for agent training is "sucking supervision through a straw" — an entire trajectory's reward is compressed into a scalar that upweights every action in a successful run regardless of causal contribution. Process supervision is theoretically better but fails in practice because LLM judges are gameable — the "dhdhdhdh" adversarial example: model learns to output nonsense that achieves 100% reward by finding out-of-distribution inputs that fool the reward model.
  • Synthetic training data causes "silent collapse" — model outputs occupy a tiny slice of possibility space; training on them degrades distributional diversity without obvious symptoms. Humans collapse over time too (children say surprising things because they haven't overfit yet). Dreaming as anti-overfitting mechanism: injects entropy via off-distribution scenarios. Extended self-improvement loops require external entropy injection to avoid degradation.
  • Cognitive core theory: ~1B parameter model trained on aggressively curated data could achieve human-level reasoning if separated from memorized knowledge. Current frontier models spend most parameters on memory, not cognition. Pre-training does two things: (1) picks up knowledge (often harmful — causes reliance on memorized patterns); (2) develops intelligence by observing algorithmic patterns. Goal: isolate the cognitive part, strip memory, force the model to look things up.
  • Continual learning gap = missing sleep/distillation equivalent: humans build context windows during the day; sleep distills experience into weights. LLMs have no equivalent. Vision: sparse weight update (LoRA-style) per person from each session's experience.

Two unexploited multi-agent ideas (Karpathy, Oct 2025)

  • Source: Dwarkesh interview; See ~/ClaudeBrain/papers/2025-karpathy-agi-decade-away.md
  • LLM culture: a giant scratchpad that an LLM edits as it reads/works; LLMs writing books for other LLMs; a growing, self-directed repertoire of knowledge accumulating across agents. No equivalent currently exists.
  • Self-play: analogous to AlphaGo — one LLM creates problems of escalating difficulty for another LLM to solve. Neither idea has been convincingly demonstrated at frontier scale.
  • Both require LLMs to be more cognitively capable. Current models are "savant kids" — perfect memory, convincing output, but cognitively kindergarten-level. They can't create culture because "they're still kids."
  • These are the two most promising directions for multi-agent systems beyond the current generator/evaluator paradigm.

The march of nines: why demos are deceiving (Karpathy on self-driving)

  • Source: Dwarkesh interview (Oct 2025). Applicable to any safety-critical agent deployment.
  • Every additional nine of reliability (90% → 99% → 99.9% → …) is a constant amount of work. Karpathy went through ~2–3 nines at Tesla over five years; more remain for self-driving.
  • Self-driving: first demo CMU 1986, Waymo near-perfect demo 2014, still not economical at scale in 2025. "This is not even near done."
  • Waymo has hidden teleoperation centers — "people somewhere out there beaming in from the sky." Human-in-the-loop more extensive than visible.
  • Production software shares the same property: a coding agent making one catastrophic mistake per 7 years sounds safe, but at high token throughput the wall-clock time to failure is much shorter. Surface area of software is larger than self-driving.
  • Implication for agentic deployment: demos are just the first nine. Every nine from there is the same amount of additional work. Don't confuse the demo with the product.

Related Concepts

Cross-Topic Connections

  • AI and Experimental Particle Physics — JFC is a domain-specific instance of these general harness design principles; the multi-agent review system instantiates generator/evaluator separation in the HEP context
  • Attention and Deep Work — the split between generator and evaluator roles mirrors the cognitive principle of separating creation from critique
  • Leadership and Teams — the human-as-orchestrator role mirrors Commander's Intent from Extreme Ownership / Call Sign Chaos: providing context downstream is leadership, regardless of whether the subordinate is human or model

Contributing Articles

  • Technological Optimism and Appropriate Fear (Clark) — Anthropic's Jack Clark on the only intellectually honest stance toward frontier systems: optimism and fear at the same time. "What we are dealing with is a real and mysterious creature, not a simple and predictable machine." The vocabulary of stewardship is doing real work
  • Is Claude Mythos Terrifying or Just Hype? (Newport) — counter-pole to lab self-marketing: independent verification of Mythos's vulnerability-discovery numbers showed modest gains, reproducible by cheaper models. The asymmetric epistemics of frontier-model claims
  • Why Is OpenAI Dropping Video? (Free Press) — the Sora retreat read as a cost-side decision masquerading in critic discourse as a capability-side one; AI-stance taxonomy of denialists / minimizers / adapters
  • Vitalik's Self-Sovereign LLM Setup — local-first as an agentic engineering posture: aggressive sandboxing, supply-chain hygiene, the LLM as dependency reducer rather than dependency adder

Backlinks