In 5 minutes, you will learn why the final answer is the wrong thing to grade, the eight-layer trace that makes a run debuggable, and how to pin a failure on one harness layer before you trust the next run.

Two agents close the same ticket. The final answer says they tied. The final answer is lying to you.

One closed it in four steps, ran the full test suite, and asked before it touched production. The other looped eleven times, skipped the suite, and slammed into a permission wall it never mentioned.

Same diff. Same green check. Two completely different runs, and an output-only evaluation can see only the checkmark.

Most devs/teams still grade agents the way they grade a chatbot. That worked when the model only talked. It breaks the moment the model starts acting.

For a chatbot, the output is the product, so scoring the output is fair.

But for an agent, the output is a receipt. It confirms the transaction closed and says almost nothing about what got touched, retried, skipped, or carried by luck.

The research is starting to say this with numbers. Harness-Bench, a new diagnostic benchmark, ran 5,194 agent trajectories across 106 sandboxed tasks and found that the same task and model can land very differently depending on the harness wrapped around them.

A blunt companion paper, “Stop Comparing LLM Agents Without Disclosing the Harness”, argues that for long, hard tasks the harness moves performance more than the choice of model does.

Why the final answer lies

An agent’s final output is a compression of everything it did. Compression deletes exactly the part you need when something breaks.

One green “pass” can hide at least four different problems.

A coding agent ships a patch that clears the single test it ran and never touches the broader suite. Green check, latent bug.

A research agent cites the perfect source, but it got there through a stale memory entry that happens to still be right. Next month it will not be.

A support agent resolves a ticket after quietly trying a refund path it was forbidden to use and getting denied. Nobody upstream ever hears about the attempt.

A long-running agent finishes a multi-day job after three rounds of context compaction, and no one can say which of the starting assumptions made it through the last handoff.

All four score as wins. Output-only evaluation tells you the task passed. It never tells you which layer made the pass fragile, or whether the next run will be as lucky.

The shift the research is making

Harness-Bench stopped ranking models and started ranking model-and-harness pairs. It crosses 8 model backends with 6 harnesses, and on every one of those 5,194 runs it logs four things: the final artifact, the full execution trace, the usage stats, and what the validators returned.

The final artifact is the smallest item on that list. The benchmark also catalogs the failure symptoms that keep recurring, which is the useful part. Once you can name the symptom, you can hunt for the layer that caused it.

“Stop Comparing LLM Agents Without Disclosing the Harness” is harsher about the status quo. Its Binding Constraint Thesis holds that once models reach frontier-class parity, the harness explains more of the performance gap than the model, so any leaderboard that hides its harness is quietly handing the model credit for the scaffolding’s work.

Anthropic makes the operational version of the argument in its agent-evaluation writing: grade the trajectory and the outcome as two separate things. The transcript of tool calls, token counts, turns, latency, and state checks is a grading surface in its own right, not a footnote under the final reply.

Stack the three together and the direction is obvious. The question is moving from “did the model answer?” to “what did the model and the harness do together?”

The harness telemetry stack

Answering that question takes a trace, and a trace has layers. Here is the minimum set that lets you rebuild a run after it breaks, with the failure each layer exists to catch.

This is not a dashboard wishlist. It is a debugging contract. When a layer is missing from the trace, the failure that lives in that layer cannot be attributed, and you are back to rerunning the agent and squinting.

Codex and Claude Code already emit parts of this over OpenTelemetry: model and tool calls, sessions, token and cost metrics, and structured logs, with traces already shipping on Codex and still in beta on Claude Code. The plumbing is starting to exist. The rare part is treating the trace, not the answer, as the thing you actually review.

A failure map, not a tutorial

With the trace in hand, a broken run finally produces a useful question: which layer failed?

The eleven harness layers everyone lists make a mediocre tutorial and a sharp failure map. Stop reading them as “what a harness contains” and start reading them as “where to point when the answer came out wrong.”

The map does one thing well. It turns “the agent failed” into “the governance layer failed,” which is the difference between a shrug and a fix.

The four agents here split into two shapes. Coding agents like Codex and Claude Code pour their engineering into governance, execution, and verification, because their failures hit files, terminals, repos, and CI. Personal agents like Hermes and OpenClaw pour theirs into memory, identity, channels, and continuity, because their failures play out across time, people, and messaging apps.

Both need traces. They just need the densest tracing in different layers.

Then there is the harder case. When a personal agent runs a coding agent as its backend, which Hermes and OpenClaw can both do, the trace has to cross the seam between two harnesses. The outer agent knows what it asked for. The inner runtime owns the real tool calls, diffs, tests, and permission decisions. If your trace stops at the handoff, so does any hope of explaining the failure.

The harness reliability scorecard

The trace shows what happened. Judging it needs a rubric you run after every important run, before you trust the next one.

This is not a leaderboard. It is a post-run audit, and its only job is to make a failure local enough to fix.

Run it honestly and most “working” agents lose points on trace completeness first. You cannot grade what you never recorded. A run that passed but trips four of these signals is not a success. It is a success you got away with, and got-away-with does not survive contact with production.

The model can tell you what it answered. The trace tells you what actually happened.

So the reliability question for agents has changed. It is no longer “did it work?” It is whether the harness saw enough to explain the run on the day it doesn’t.

Which layer in your stack is flying without a trace right now: context, permissions, memory, or verification?

All Sources in first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix

All Tables as markdown format.

Table 1:

| Trace layer | What to capture | The failure it explains |
|---|---|---|
| Context | prompts, files, retrieved docs, memories, skills, tool schemas, compaction events | the agent never saw the fact it needed, or saw stale and irrelevant context |
| Tool | tool name, args, result, error, latency, output size, repeated calls | the tool failed, returned weak evidence, or got called in a loop |
| Permission | approval request, decision, policy rule, denial, escalation path | the agent crossed a boundary, or could not recover from a denial |
| Execution | sandbox mode, workspace diff, network policy, environment state, side effects | the run leaned on hidden environment state or untracked file changes |
| Test | commands, validators, static checks, pass/fail, outcome graders | the answer looked right but was never checked from outside |
| Memory | reads, writes, source, timestamp, confidence, expiry | stale or unsafe memory quietly steered the run |
| Cost and latency | model calls, token usage, wall time, retry count, wait states | the agent succeeded by burning budget or looping blindly |
| Human | clarifications, approvals, corrections, interrupts, handoffs | autonomous performance cannot be separated from human rescue |

Table 2:

| Harness layer | The question to ask when the run fails |
|---|---|
| Control | Did the agent have the right rules, success criteria, and boundaries? |
| Context | Did the harness load the right information, at the right time, without stale or distracting material? |
| Runtime loop | Did the loop stop, retry, compact, and resume correctly? |
| Tools | Did the tool interface return compact, accurate, actionable observations? |
| Execution | Did the action run in the intended workspace, sandbox, network mode, and environment? |
| Governance | Did risky actions require approval, and did denials leave a recoverable path? |
| Memory | Did durable memory help the run, or inject old, unsafe, or untraceable assumptions? |
| Skills | Did the agent pick the right procedure, version, and validation steps? |
| Planning | Did planning reduce uncertainty, or spawn abandoned branches and noisy handoffs? |
| Verification | Did the harness check the final state instead of only the final message? |
| Interface | Were human approvals, clarifications, and interrupts captured as part of the run? |

Table 3:

| Dimension | Question | Bad signal |
|---|---|---|
| Outcome reliability | Did the final task succeed? | success cannot be checked outside the model's own answer |
| Trajectory quality | Did the agent take a sensible path? | many turns, repeated calls, abandoned branches, no new evidence |
| Tool effectiveness | Did each tool call add information? | tools return huge logs, vague errors, or output nobody uses |
| Context discipline | Did the harness load the right context? | missing source files, stale memory, bloated prompt, bad compaction |
| Permission safety | Did risky actions require approval? | a dangerous action ran silently, or a denial caused a dead end |
| Recovery behavior | Did the agent recover from errors? | retries repeat the same failing action or bury the first failure |
| Verification strength | Did the harness check the final state? | no tests, no validator, no state check, no human review path |
| Cost and latency | Did the run finish inside a sane budget? | success depended on excessive tokens, time, or retries |
| Memory correctness | Did memory improve the run? | memory has no source, timestamp, confidence, or deletion path |
| Trace completeness | Can a human reconstruct what happened? | missing context, tool args, approvals, diffs, test results, or handoffs |

In ~7 mins: the CVE result (85% fewer tokens at 100% accuracy), the 5-system benchmark table, the architecture’s 3 layers, the 6 design principles behind it, and a Hermes Agent walkthrough to run the pattern yourself.

Perplexity just stopped letting its agents call search and started letting them program it.

Search as Code (SaC) is its new search architecture. Instead of one fixed pipeline behind a query, the model writes Python that composes the individual pieces of the search stack into a retrieval pipeline built for each task.

On a 200-CVE research task, that cut token use 85.1%, from 288.7K to 42.9K, while scoring 100% accuracy. Every non-Perplexity system tested scored below 25%.

It ships today: default in Perplexity Computer, available in the Agent API. There is a Hermes Agent walkthrough at the end if you want to run the pattern yourself.

Context

The research is published by Perplexity and titled “Rethinking Search as Code Generation.” It landed June 1, 2026, and builds on the first overview of Perplexity’s search stack from September 2025.

Perplexity’s search serves thousands of queries each second. The old contract was simple: the model issues a query, the engine runs a predefined pipeline, and the model reads the results. That works for a single question.

It breaks for agents. Today’s agents run tasks that take hours, span thousands of retrieval operations, and need a different search strategy at each step. A fixed pipeline cannot bend to all of them. SaC is Perplexity’s answer.

What Search as Code actually is

The old way hands the model a search box. It types a query, gets back a ranked page of results, and works with whatever the pipeline decided to return. The model never touches the steps in between.

SaC hands the model the steps themselves. Retrieval, ranking, filtering, fan-out, rendering: each is a primitive in an SDK, and the model writes Python that wires them into a pipeline for the task in front of it. A single inference turn can drive up to thousands of these operations inside a sandbox, then return only the slice worth reading.

The real shift is not that search now uses Python. It is that the agent stops spending its context window on deterministic grunt work. Loops, deduplication, filtering, and joins move into code, where they belong, and the model stays on strategy. Perplexity calls these the twin levers of control and legibility: the model steers every step, and it can inspect the intermediate state instead of guessing at it.

The evidence: benchmarks and a 200-CVE stress test

The flagship demo is a security research task: identify and characterize more than 200 high-severity CVEs from 2023 to 2025, each record citing the vendor’s own advisory, the affected product, and the fix version. SaC scored 100% accuracy and used 85.1% fewer tokens than the same stack without it, 42.9K against 288.7K. The other systems Perplexity tested all landed below 25%.

A stylized slice of what the model wrote shows the shape: build a query plan across official advisory formats, fan the queries out concurrently, and keep only vendor-owned pages.

templates = [
    ("Mozilla", 'site:mozilla.org/.../mfsa{year} "CVE-{year}-" "Fixed in" "Impact high"'),
    ("Jenkins", 'site:jenkins.io/security/advisory/{year} "CVE-{year}" "High" "Fix"'),
    # ...more vendors
]
queries = [pattern.format(year=y) for y in (2023, 2024, 2025) for _, pattern in templates]
seed_hits = sdk.search.web_many(queries, limit_per_query=8, concurrency=12)
pages = [h for q, hits in zip(queries, seed_hits) for h in hits
         if official_vendor_advisory(h.url)]

The fan-out, the concurrency, and the filter to official sources all run in code, in one turn, without round-tripping through the model.

Across the full suite, SaC leads four of five benchmarks.

The systems are SaC on Perplexity’s Agent API (GPT 5.5, high reasoning), OpenAI’s Responses API with web_search and code_interpreter, Anthropic Managed Agents (Opus 4.7, high reasoning), Exa, and Parallel.

Each is a single run, not best-of-N. OpenAI edges SaC on HLE by 0.002. The benchmarks come from Google (DeepSearchQA), ByteDance Seed (WideSearch), and OpenAI (BrowseComp), plus Humanity’s Last Exam.

Two caveats live inside these numbers. WANDR, where SaC’s lead stretches to 2.5x the next-best system, is Perplexity’s own benchmark and is not released yet.

The cleanest comparison is the ablation against Perplexity’s own non-SaC pipeline on the same infrastructure: the largest absolute gain is +19.77 points on DSQA, the largest relative gain +45% on WANDR.

On cost, medium-reasoning SaC beats every non-SaC system at under $1 per task, and low-reasoning SaC is cheaper than all of them while staying competitive.

How it works: six design principles

Strip SaC to its frame and you get a stack with three jobs and six design choices that make the stack pay off.

Atomize the stack, don’t wrap the API.

The SDK is not a search endpoint dropped into a shell. Perplexity rearchitected its search stack into modular primitives and exposed them at the lowest level it could, from raw retrieval up to semantic parsing. High-level end-to-end pipelines still exist, but only as shorthand the model can use or skip.

Three layers do three jobs.

The model is the control plane: it reads the directive, decides which pipelines each task needs, and writes the code. The compute sandbox handles deterministic work: control flow, batching, retries, filtering, joins, aggregation. The Agentic Search SDK lives in the sandbox runtime and exposes the primitives, so one inference turn can drive thousands of operations.

Code orchestrates, and fills gaps.

When the SDK lacks a capability, the model builds it in code instead of waiting for a new function. Need a precise regex the query syntax cannot express? The model fans out to collect a superset, dedupes, then narrows the results deterministically. The CVE fan-out above is this principle in action.

Control and legibility are the point.

A fixed pipeline owns everything downstream of the query, which creates three failure modes: context bloated with irrelevant hits, domain knowledge the model cannot apply, and serial control flow that pollutes the context with intermediate state. Programmable search fixes all three by handing the model both the steps and the state.

State lives on disk, not in tokens.

Across turns, SaC persists intermediate state to a filesystem with explicit serialization rather than holding it in a REPL. Perplexity tested both. They performed similarly day to day, but the filesystem approach proved more reliable on long trajectories, where an in-memory namespace turns into a cluttered hundred-cell notebook.

Teach the SDK with small skills.

A custom SDK appears in no model’s pretraining data, so Perplexity wrote Agent Skills to teach it. The root Skill.md files stay under 2,000 tokens and spend most of that budget on few-shot examples for composing primitives, not on listing them. Continuous autoresearch loops tune the SDK and the skills against latency, codegen quality, and task performance.

How to run the pattern yourself

The catch: the Agentic Search SDK and Perplexity’s retrieval infrastructure are internal. You cannot download them. The appendix at the end turns the idea into a runnable skill anyway, because what travels is the orchestration pattern, not the engine.

Any agent runtime that exposes code execution plus search primitives can host it. The clearest open option is Hermes Agent from Nous Research, MIT-licensed, which gives you execute_code alongside web_search and web_extract.

The move is the one SaC makes: push fan-out, extraction, filtering, deduplication, and evidence assembly into a single sandboxed code step, and return a compact summary to the parent agent.

Full install-to-run commands are in the appendix below.

AlphaSignal Take

The SDK is the moat, and it is private.

Every result here rests on the atomized Agentic Search SDK and Perplexity’s retrieval infrastructure, and neither ships. This is an architecture disclosure, not a reproducible package. You can copy the shape of the idea, not the engine that makes its numbers.

The headline leans on a benchmark no one else can see.

WANDR, where SaC’s 2.5x lead is widest, is Perplexity’s own and unreleased. Every score in the table is a single run rather than best-of-N, and none has been independently replicated. Read them as directional.

The gains come with lock-in.

SaC is cloud-only and bound to Perplexity’s models and stack. There is no swapping in your own LLM and no self-hosting, the usual price for production search quality you do not maintain yourself.

What keeps this from being a vendor story is that the core idea is not Perplexity’s alone. Executable-code actions (CodeAct, ICML 2024), the broader move from tool-call loops to generated code, and recent wide-search systems all point the same way. The architecture is sound even where the evidence is self-reported.

So the best recommendation is to adopt the pattern, not wait for the product. Build the code-orchestrated version on an open runtime now, measure it against your current serial setup, and treat Perplexity’s numbers as a target to verify rather than a result to trust.

If your agent could write its own search pipeline, what is the first workflow you would hand it?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: How to run a Search-as-Code-style scout with Hermes Agent

This appendix builds a small Hermes skill that approximates the SaC pattern: plan a bounded search, run the deterministic work inside execute_code, persist inspectable artifacts, and return a compact summary.

It is an approximation, not a clone. Hermes exposes higher-level tools than Perplexity’s private SDK, so the value is in moving loops and filtering out of the parent context, not in matching Perplexity’s retrieval quality.

The running example is a migration scout: collect the official migration guidance for a library’s last three major versions, returning for each the source URL, breaking changes, required code edits, and unresolved questions.

1. Install Hermes Agent.

git clone https://github.com/NousResearch/hermes-agent
cd hermes-agent
# follow the quickstart in the docs to install and configure the runtime

Hermes is MIT-licensed and Python-based. Configure web access per the docs before running: web_search and web_extract need search and extraction credentials set first.

2. Create the skill.

mkdir -p ~/.hermes/skills/research/migration-scout

Write ~/.hermes/skills/research/migration-scout/SKILL.md with a contract the model can follow. Keep it under 2,000 tokens, the same discipline Perplexity uses:

---
name: migration-scout
description: Collect official migration guidance for a library's last 3 major versions.
---

# Migration scout

For each of the target library's last three major versions, return: official source URL,
breaking changes, required code edits, and unresolved questions.

Process:
1. Define the output schema before searching (one record per version).
2. Split the work into per-version query branches.
3. Run one scout web_search and inspect the payload before writing an extractor.
4. In execute_code, fan out web_search across branches, web_extract the official docs,
   filter to the vendor's own domain, dedupe by URL, and write artifacts.
5. Persist raw and normalized JSON under sac-state/.
6. Return only counts, artifact paths, unresolved rows, and a small evidence sample.
7. Verify weak rows in a second pass.

3. Scout the payload shape.

Inside execute_code, run one search and look at the response before building an extractor. Do not assume an undocumented schema.

# web_search, web_extract, write_file are available inside execute_code (see docs)
import json
sample = web_search("site:docs.example.com upgrade guide v3", limit=5)
print(json.dumps(sample, indent=2)[:4000])

4. Fan out and persist.

import json

branches = [
    "site:docs.example.com v3 migration breaking changes",
    "site:docs.example.com v2 migration breaking changes",
    "site:docs.example.com v1 to v2 upgrade guide",
]

def urls_in(node):
    out = []
    if isinstance(node, dict):
        for v in node.values():
            out += urls_in(v)
    elif isinstance(node, list):
        for v in node:
            out += urls_in(v)
    elif isinstance(node, str) and node.startswith("http"):
        out.append(node)
    return out

hits = {b: web_search(b, limit=5) for b in branches}
urls = sorted(set(urls_in(hits)))
pages = web_extract(urls[:10]) if urls else {"results": []}

write_file("sac-state/hits.json", json.dumps(hits, indent=2))
write_file("sac-state/pages.json", json.dumps(pages, indent=2))
print(json.dumps({"branches": len(branches), "unique_urls": len(urls)}, indent=2))

5. Verify weak rows.

In a second pass, bind each version’s claims to an official source URL and flag any row that lacks one. Keep this separate from the fan-out so a failed extraction does not poison the whole run.

6. Run, measure, and compare.

Trigger the skill on the bounded task, then capture: parent-visible tool turns, total execute_code calls, search and extract calls, tokens if exposed, rows needing manual verification, and correctness against the official docs. Then run the same task with serial parent-level search calls and compare.

The point is to test whether moving the work into code lowers parent-context load, not to reproduce Perplexity’s benchmark scores.

To deploy, leave the skill in ~/.hermes/skills/, where any Hermes session loads it on demand. The same skeleton ports to other stacks: a Claude Code skill folder, a Codex agent file, or a generic system-prompt slot.

Execution rules.

Keep each run under the documented 50 tool-call default. Do not expect asyncio to parallelize Hermes calls: the RPC stub serializes its exchange behind a calllock.

Write JSON to disk before a long extraction pass. Call delegate_task from the parent agent, never from inside execute_code, which cannot recurse into execute_code, delegate_task, or MCP tools.

Strict mode adds isolation but is not a full secure container, and leaf delegated subagents cannot call execute_code.

A second path: OpenClaw.

OpenClaw (also MIT) offers a second route through its experimental code mode, which is off by default. Enable tools.codeMode.enabled, and guest JavaScript or TypeScript runs in a constrained QuickJS-WASI worker with no imports and no direct network or file access.

It calls already-enabled web tools through the executor and reduces nested results to one compact object, with maxPendingToolCalls defaulting to 16.

Durable artifacts need a write-capable tool. It is a looser fit than Hermes because OpenClaw is built as a multi-channel assistant gateway, so treat it as the secondary option.

This article is adapted from Ben Dickson’s AlphaSignal Sunday Deep Dive on Direct Corpus Interaction and GrepSeek.

AI coding agents are exposing a critical flaw in traditional retrieval-augmented generation (RAG) pipelines. And the solution might be giving the agents the same tools that humans use.

Agentic search requires dynamic plan revision. If an agent is tasked with debugging a production incident, it does not know the full scope of information it needs.

It needs to examine partial evidence, formulate a hypothesis, and search again to verify its assumptions. Agents need to find exact strings, numerical values, version constraints, error codes, and specific file paths.

This is not what traditional RAG is designed for.

RAG systems break documents into chunks and store their embedding values in vector databases. When a user asks a question, the system retrieves text chunks based on the similarity of their embeddings with that of the prompt.

This dense retrieval method is excellent for broad semantic recall and answering general questions over static knowledge bases. But it breaks down in software engineering and IT operations.

Exact lexical constraints and multi-step hypothesis refinement are incredibly difficult to execute through semantic retrievers alone. Current retrieval pipelines often decide too early what the AI agent is allowed to see.

Once relevant evidence is filtered out by a vector index before the agent’s reasoning loop begins, the data is lost. And no amount of reasoning can recover it.

Direct corpus interaction

Direct Corpus Interaction (DCI) is a new but simple paradigm that bypasses embedding models entirely. It allows AI agents to interact with raw data using general-purpose terminal tools like grep, find, cat, sed, and shell pipelines.

In enterprise environments, data is rarely a stable, static document collection. It consists of active incident logs, live IT tickets, recent code commits, daily financial reports, and constantly shifting configuration files.

Vector embeddings are always a snapshot of the past. Building, updating, and maintaining vector indexes takes compute power and batch processing time. DCI allows the agent to interact directly with the current state of the workspace as it exists right now.

With terminal tools, agents can enforce strict constraints that vector databases miss. An agent looking for a specific database failure can search for an exact error string, pipe the output to a secondary filter to remove legacy log files, and verify the local context immediately.

This creates an iterative feedback loop between the agent and the file system. The agent executes a command, reads the raw output, and adjusts its next query based on what it learns. This mirrors how a human developer navigates an unfamiliar codebase.

Experiments show that DCI outperforms semantic retrieval on multi-hop reasoning tasks and retrieval benchmark where clues are scattered across different files, while also reducing inference costs.

Scaling DCI with GrepSeek

Giving a language model raw terminal access introduces friction. Agents can get lost in complex, nested directory structures. They can execute broad search commands that overwhelm the terminal with thousands of lines of useless output, which quickly derails their reasoning process.

A new framework called GrepSeek upgrades DCI and addresses these friction points by training a model to treat the corpus as the search environment. GrepSeek reasons about the query and gathers evidence by issuing executable shell commands against the corpus.

To simplify the process of training GrepSeek, the researchers created a pipeline that generates training data from a very large unstructured body of text without human assistance.

This process generates causally grounded search paths. It trains the model on how to logically navigate a file system, form hypotheses, and use command-line tools efficiently.

GrepSeek also uses reinforcement learning to improve the agent’s task-oriented search behavior. It teaches the model to avoid dead ends, recognize when a command has failed, and refine its search queries accordingly.

Running raw shell commands sequentially over millions of documents introduces severe latency. Agents waiting for a massive grep search to complete across an entire enterprise repository slows down the orchestration loop to a crawl.

GrepSeek solves this bottleneck with a semantics-preserving sharded-parallel execution engine. This engine splits the underlying corpus into smaller data shards and runs shell commands simultaneously across them.

This approach speeds up shell-based retrieval by up to 7.6x compared to traditional execution while preserving the fidelity of the original data.

How to apply DCI in practice

Why not load an entire repository into a massive million-token context window? Because processing millions of tokens for every step an agent takes is unsustainable for most applications.

Massive context slows down the agent’s time-to-first-token. Furthermore, cramming a model with raw code increases the likelihood that it will overlook specific, critical details buried deep within the prompt.

Raw terminal outputs from DCI can also bloat the context window if left unchecked. A single poorly constructed find command can return thousands of lines of text. And running grep on the entire corpus every time can be slow, especially if it is being accessed through a network.

For AI orchestration engineers and data architects, if you have a small corpus of information, DCI-style retrieval can work perfectly fine.

But for very large corpora, a balanced, hybrid approach will probably be better suited:

Semantic retrieval handles broad, high-recall candidate discovery. It locates an initial anchor document when the user’s intent is underspecified.

DCI operates as a precision verification layer on top of the retrieved data.

The agent uses terminal tools to expand laterally from the anchor document into neighboring files or dependencies.

The agent checks exact constraints, verifies version numbers, and combines weak signals across multiple documents before generating a final answer.

This shift changes how we must think about enterprise data architecture. In the near future, data will not only need to be indexed for human search engines. It will need to be explicitly organized for agents that can inspect, trace, and verify raw files.

Retrieval quality for coding agents is not about generating better vector embeddings or using larger context windows. It relies on the resolution of the interface through which the agent is allowed to interact with the corpus.

This article is adapted from Ben Dickson’s AlphaSignal Sunday Deep Dive on Direct Corpus Interaction and GrepSeek.

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

In ~5 mins: what dot-skill actually generates, the capability/persona split, the S = (A, M, L) contract, the correct-and-roll-back loop, the behavioral-fidelity gap the paper admits, and a full install-to-rollback walkthrough at the end.

A tool that turns your coworker into an installable AI skill crossed roughly 18,500 GitHub stars.

dot-skill reads a person’s scattered work traces, their docs, code reviews, and chat decisions, and writes them into a skill file an agent can load.

The backlash arrived fast. Someone shipped an anti-distillation skill that adds noise to your own traces so you cannot be cleanly copied.

Now there is a paper. COLLEAGUE.SKILL, from Shanghai AI Lab, 29 May, drops the digital-twin pitch for a narrower claim: this is a file format for expertise, not a copy of a person.

The useful part is the artifact. Whether it keeps the judgment is the part nobody has measured.

Context

dot-skill is open source under MIT, written in Python, at titanwings/colleague-skill. It started as colleague.skill, built for one job: when a teammate quits, capture their review standards and incident heuristics before the context walks out with them.

The five-person team at Shanghai AI Lab posted the technical report to arXiv on May 29, 2026. MIT Technology Review had already covered the trend in April, and the “distill them before they leave” framing is what drew both the stars and the pushback.

It builds on the Agent Skills standard, where a skill is a folder around a SKILL.md file plus optional scripts and references, loaded on demand. The repo ships three presets: colleague (the main one), celebrity, and relationship. A public gallery lists 215 skills from 165 contributors, which measures distribution, not whether any of them work.

What it actually generates

Expertise rarely lives in a manual. It is scattered across design docs, review comments, chat decisions, and incident notes.

dot-skill reads those traces and writes a few plain Markdown files. The loadable one is SKILL.md. Behind it sit work.md (what the person knows) and persona.md (how they act).

Because it follows the Agent Skills format, any compatible host loads it: Claude Code, OpenClaw, Codex, or Hermes. And because the output is Markdown, you can read the extracted rules, fix them in plain English, version the result, and roll it back.

The design

reframe

The paper calls the method person-grounded trace-to-skill distillation. A lightweight profile, a source scope, and a set of documents map to a package the paper writes as S = (A, M, L): the generated files, the install metadata, and the lifecycle state (version, correction count, rollback history). The package is meant to be portable, inspectable, composable, correctable, and governable.

split

The sharpest design choice is keeping capability separate from behavior. work.md holds review criteria, workflows, and decision heuristics. persona.md holds tone, interaction rules, and a correction log.

Those generate three entry points: the full skill, work-only, and persona-only. Work-only is the safer one, because a review checklist does not need a personality. In the paper’s example, a colleague skill encodes a review order: check authentication, input validation, rate limiting, response schema, and sensitive-data exposure before smaller issues.

files

Generation emits seven files on schema v3: SKILL.md, work.md, persona.md, the two sub-skills work_skill.md and persona_skill.md, plus manifest.json and meta.json for install and lifecycle metadata.

fix loop

Corrections are plain language. Say “he would not push back there” and the handler routes it: a work correction patches the matching ## section in work.md, a behavior correction appends a {scene, wrong, correct} record to persona.md. Every update archives the prior version first, and version_manager.py rolls back to any of the last 10.

The honest part

The paper makes one kind of claim: that this format and workflow exist and run. It does not claim the skill reproduces the person or improves anyone’s work. The authors name their own open problem the behavioral fidelity frontier, and the paper ships no held-out task study to close it.

How to get started

Here is the short path. Every command, including install-to-deploy and rollback, is in the appendix at the end.

Clone the repo into your host’s skills directory, or hand the URL to your agent and let it install itself. Then run /dot-skill, pick the colleague family, and answer three questions: an alias, a one-line role, and a few personality tags.

Point it at authorized traces. It can auto-collect from Feishu, DingTalk, or Slack, or take uploads: PDFs, screenshots, .eml files, or pasted text. It then generates the files and, by default, installs the skill into Claude Code.

Read work.md before you trust it. Then invoke the full skill with /{character}-{slug}, or the safer work-only path with /{character}-{slug}-work. Full install-to-rollback commands are in the appendix below.

AlphaSignal Take

dot-skill ships real software: collectors, a writer, installers, rollback, and 35 passing tests. The gap is between what the product page sells and what the paper will actually claim.

No fidelity evidence.

The paper proves a file format, not that a generated skill catches what the real engineer would. It says so itself and ships no held-out evaluation. You are trusting extraction quality you cannot yet measure.

The persona layer can turn a label into a rule.

The colleague persona analyzer translates freeform tags like “blame-shifter” or “PUA” into Layer 0 rules the agent must never break, and manual tags outrank the actual traces. The repo’s own colleague demo shows the skill dodging blame on cue. That is bias compiled into behavior, by design.

Governance is an affordance, not a guarantee.

Local-first, versioned files give you control, but nothing enforces consent, retention, or redaction in code, and deletion is rm -rf. The product site mentions RAG, yet there is no retrieval runtime in the repo (requirements.txt is requests, pypinyin, playwright, slack-sdk, python-docx, and openpyxl).

So the best recommendation is to adopt the work-only path and treat the rest as a research preview. Package a departing engineer’s review checklist as a work.md skill, test it against reviews you already graded, and keep persona off until someone measures fidelity. Separate relationship and celebrity papers are promised, which is exactly where the consent questions get harder.

If a teammate left tomorrow, would you trust a work-only skill of their review checklist, or is judgment the part that never compiles?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: How to actually run dot-skill

Shorter than the repo’s README. Eight steps, install to rollback.

1. Install dot-skill into your host.

Clone the repo into your host’s skills directory. For Claude Code that is ~/.claude/skills/dot-skill.

bash

git clone https://github.com/titanwings/colleague-skill ~/.claude/skills/dot-skill

OpenClaw uses ~/.openclaw/workspace/skills/dot-skill, Codex uses ~/.codex/skills/dot-skill. For Hermes, clone anywhere and run python3 tools/install_hermes_skill.py --force. Or skip all of this and tell your agent to install the skill at the repo URL.

2. Launch and pick a family.

Run /dot-skill and choose colleague. The other presets are relationship and celebrity.

3. Answer the intake.

Three questions: alias, a one-line role, and personality tags. Keep the tags factual. They become behavior rules, not flavor text.

4. Provide source material.

Auto-collect from a chat platform, or upload files.

bash

# Slack auto-collect (an admin installs the bot; free workspaces cap history at 90 days)
python3 tools/slack_auto_collector.py --setup
python3 tools/slack_auto_collector.py --name "Jane Doe"

Or upload PDFs, screenshots, .eml archives, or pasted text. You can also skip collection and generate from the intake alone.

5. Generate and inspect.

Generation runs through the writer and emits all seven files.

bash

python3 tools/skill_writer.py \
  --action create \
  --character colleague \
  --slug jane-doe \
  --name "Jane Doe" \
  --meta /tmp/meta.json \
  --work /tmp/work.md \
  --persona /tmp/persona.md \
  --base-dir ./skills/colleague \
  --no-install-claude-skill

By default the create step auto-installs into Claude Code. Pass --no-install-claude-skill to stop and read work.md and persona.md first. This is where you catch a manual tag that became a rule it should not have.

6. Install the generated skill to a host.

bash

python3 tools/install_claude_generated_skill.py --skill-dir skills/colleague/jane-doe --force

Use install_openclaw_generated_skill.py or install_codex_generated_skill.py for the other hosts. Then invoke the full skill with /colleague-jane-doe, or the work-only entry point with /colleague-jane-doe-work.

7. Correct it in plain English.

Tell the agent what is wrong. A work fix patches a ## section, a behavior fix becomes a {scene, wrong, correct} record.

bash

python3 tools/skill_writer.py \
  --action update \
  --character colleague \
  --slug jane-doe \
  --correction-json /tmp/correction.json \
  --base-dir ./skills/colleague

8. Version and roll back.

Every update archives the prior version. The version manager keeps the last 10, and cleanup is manual, so old versions stay until you remove them.

bash

python3 tools/version_manager.py --action rollback --character colleague --slug jane-doe --version 3 --base-dir ./skills/colleague

Deploy maps to wherever your agent reads skills: ~/.claude/skills/, ~/.codex/skills/, or the Hermes skill directory.

In ~7 mins: what a 1,730-star Claude plugin actually ships, the 5-verb loop that turns agent coding into delivery evidence, the 13 guardrails behind it, and 4 rough edges the README doesn’t flag.

When you let Claude Code loose on a repo, plans live in chat, tests become optional, review happens too late, and release notes get reconstructed from memory.

Claude Code Harness is an MIT-licensed plugin that wraps that work in a five-verb loop and treats two files as the source of truth.

The shift it represents is the part worth watching: agent tooling moving from chat output to delivery evidence.

Snapshot: v4.12.7, Go-native runtime, +1,700 stars, +190 forks, 33 shared skills.

Why this is worth a look

Claude Code stacks have been quietly maturing. Hermes Agent, Superpowers, and a few others have started looking less like prompt packs and more like operating systems wrapped around the model. Harness sits in that same lane.

Its specific bet is narrow. The model can write useful code. The surrounding process keeps drifting. Plans float. Scope expands quietly. Review collapses into implementation. Release notes get rebuilt by memory.

The repo was created in December 2025 and has shipped daily since. It carries +1,700 stars and +190 forks at v4.12.7, with the last push on 2026-05-27. There is no Hacker News thread, no Reddit discussion, no YouTube demo. Just a small Threads post and a steady release cadence. High-quality, under-the-radar.

What it actually is

The author is Chachamaru, a Japanese-language solo developer who built Harness for what the repo calls “vibecoders”: solo developers running full-cycle contract development through an agent.

Two single-source-of-truth files do the heavy lifting:

• spec.md is the product contract. What must stay true.

• Plans.md is the task ledger. What’s being worked, what’s done, what’s blocking.

Five verb commands sit on top: /harness-setup, /harness-plan, /harness-work, /harness-review, and /harness-release. After install, the default changes from “ask the agent to code” to: write the spec, implement only the approved slice, verify, review independently, package evidence.

The repo is not a small prompt pack. It ships a Claude plugin manifest, a Codex plugin manifest, OpenCode mirrors, a Go runtime, hook definitions, setup scripts, 33 shared skills, tests, and release tooling. 1,529 tracked files. Shell leads the byte count at 2.13 MB, Go follows at 1.56 MB, then JavaScript, TypeScript, and a thin Python layer.

How it works

The operating loop

Plan, work, review, release. Each stage has an explicit gate. The user approves the generated contract before any code is written. Major review findings block completion. Release preflight checks tag, version, changelog, and evidence packaging before any final action.

One rule sets the tone for the whole loop: data the agent has not directly seen stays unknown instead of getting silently invented. It is stated in the spec, not implied.

The Go runtime

The guardrail engine was rewritten in Go starting at v4.0 (”Hokage”). Cold start target is 1–2 ms, compared to roughly 40–60 ms for the older bash-and-TypeScript path the repo replaced. SQLite runs through modernc.org/sqlite, a pure-Go driver. No CGO, no Node.js, no compiler toolchain to install.

The package layout enforces the speed contract:

• hook-fastpath holds rule evaluation, codec, and types. No file I/O, no network, no goroutines.

• worker-runtime holds the SQLite store, session lifecycle, breezing orchestration, and OTel export.

Configuration collapses into one file. harness.toml is the source. harness sync regenerates plugin.json, hooks.json, and settings.json from it. The user edits one file, not five.

The repo wires 58 command-hook entries and 4 agent-hook entries through hooks/hooks.json, covering pre-tool, post-tool, permission, session, notification, stop, and task events.

The pre-tool, post-tool, and permission paths are the only ones the Go binary owns directly. Everything else still routes through shell handlers, and the project’s own doctor warns when the hook config relies on legacy bash wrappers around the binary.

Guardrail rules

The runtime ships with thirteen rules (R01–R13), each tied to a specific tool surface.

• Deny rules block sudo, git push --force, --no-verify, --no-gpg-sign, writes to .env/.git//*.pem/*.key, and git reset --hard on protected branches.

• Ask rules pause on rm -rf, package installs, force-with-lease pushes, npx, and direct pushes to main or master.

• Warn rules cover secret-like file reads (still allowed, but flagged) and edits to package.json, Dockerfiles, and CI workflows.

The honest tradeoff is on the design page: the hook system fails open on infrastructure errors. Deterministic deny rules still block when they run. But if the hook plumbing itself breaks, the design prefers approving the action to breaking the user’s session.

Host adapter tiers

Harness names what it supports and what it does not. The capability matrix lists four tiers:

The project explicitly refuses to claim parity it cannot prove. That refusal is the design choice worth noticing on its own.

How to get started

Full How-to guide in the appendix at the end.

Four lines through Claude Code:

claude
/plugin marketplace add Chachamaru127/claude-code-harness
/plugin install claude-code-harness@claude-code-harness-marketplace
/harness-setup

Then a small first request:

/harness-plan Improve the README onboarding flow

Harness writes or updates spec.md and Plans.md and returns a plan with scope, acceptance criteria, dependencies, unknowns, and stop conditions. The user’s job is to approve or correct, not hand-write the plan.

Requires Claude Code v2.1+ and write access to the repo. Existing users should run bin/harness doctor --migration-report before reinstalling. It inventories stale plugin caches, duplicate Codex skills, OpenCode files, and old symlinks without deleting any of them.

The AlphaSignal Take

Four rough edges worth knowing before adoption.

Version drift inside the checked-out repo. VERSION, plugin.json, and harness.toml all report v4.12.3. The included binaries (./bin/harness version and ./bin/harness-darwin-amd64 version) report 4.11.4 (Hokage). The repo’s own doctor catches the mismatch and recommends cd go && make install. A marketplace install ships the right binary. A clone-and-run-from-bin install does not.

TDD is not enforced by default. The README implies TDD verification on the work step. harness.toml ships with TDD enforcement disabled. R14, the TDD guardrail, is registered as a local-trial path rather than a blocking rule.

The Breezing benchmark is narrower than it looks. The report shows 14/15 passes with validation instructions versus 3/15 without, across 30 runs. The report itself flags the limits: three tasks, one model (GLM-4.5-air through Z.AI’s haiku tier), two actual bug categories under the surface, a two-stage adaptive design, and it tests validation instructions rather than the full Breezing pipeline. Useful signal. Not proof of system effectiveness.

Documentation drift from the TypeScript era. docs/CLAUDE_CODE_COMPATIBILITY.md still references v3.10.2 and Node.js requirements. The current README and the Go-runtime spec say Node is not required. Some skill files still reference deleted concepts. The project’s own deleted-concepts.yaml exists exactly because this residue keeps showing up after every major migration.

The stronger claim is the narrower one. For Claude Code users, Harness gives a structured way to turn agent work into reviewed, evidence-backed repo changes. Treat it as a workflow and evidence system. Not a finished safety layer.

Which part of your agent workflow gets the loosest when you let the model drive: plans, tests, or release notes?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: Full command walkthrough

Install and setup

claude
/plugin marketplace add Chachamaru127/claude-code-harness
/plugin install claude-code-harness@claude-code-harness-marketplace
/harness-setup

Setup installs project guidance, command surfaces, hooks, and a baseline check so the workflow starts from a known state.

/harness-plan <request>

Example:

/harness-plan Add a JSON export endpoint to the user API

What it produces:

• A spec delta written to spec.md, or a documented “spec skip reason” if the request does not change the product contract.

• Task rows in Plans.md with scope, acceptance criteria, dependencies, unknowns, and stop conditions.

The user approves or corrects the contract before any code is written.

/harness-work <task>

Example:

/harness-work 1.1.1

Implements one approved slice and records verification evidence. The work step refuses silent scope expansion. If the work outgrows the approved row, the loop stops and asks.

/harness-review

Runs as a separate step, not blended into implementation. Major findings block completion. Minor findings return as recommendations. The verdict format is fixed: APPROVE or REQUEST_CHANGES.

/harness-release --dry-run

Checks changelog state, version sync across VERSION, plugin.json, and harness.toml, tag boundaries, and release-evidence packaging. Dry-run reports what would happen without making any release action.

Codex CLI route (compatibility, not parity)

git clone https://github.com/Chachamaru127/claude-code-harness.git
cd claude-code-harness
./scripts/setup-codex.sh --user

Mirrored skills are called as $harness-plan, $harness-work, $harness-review. Codex gets contract injection and post-run checks. It does not get Claude Code’s pre-tool hook enforcement.

OpenCode route (mirror + bootstrap)

/path/to/claude-code-harness/scripts/setup-opencode.sh

Mirrors the harness skills into OpenCode-compatible files. Runtime parity is not claimed.

Existing-user health check

bin/harness doctor --migration-report

Inventories stale Claude plugin caches, missing slash entries, duplicate Codex skills, OpenCode files, backup paths, and harness-mem state. Deletes nothing. Run this before reinstalling or cleaning up.

In ~9 mins: the six-component framework, the three bottlenecks every agent runs into, why “agent score” is now a system score, and a runnable appendix to inspect the reference harness yourself.

The model is the part you can pick. Everything else is the part that breaks.

Shangding Gu (UC Berkeley) just argued that long-horizon agent performance now depends as much on the surrounding system as on stronger foundation models.

The paper names this work “scaling the harness” and decomposes any agent into six interacting components, with three of them already at saturation.

A runnable appendix at the end installs the reference harness and points each step back to one of those six components.

Why this paper, why now

The research is authored by Shangding Gu (UC Berkeley) and titled “From Model Scaling to System Scaling: Scaling the Harness in Agentic AI“ (arXiv 2605.26112v1, May 25 2026).

It lands in a year where “agent score” has stopped tracking model score. SWE-agent showed that redesigning the tool schema alone moves SWE-bench accuracy with the same model underneath. tau-bench’s pass^k metric exposed that agents looking strong at single-shot success collapse under repeated trials.

Anthropic reported a 90.2% gain from a multi-agent system (Opus 4 lead with Sonnet 4 subagents) over single-agent Opus 4 on internal research, with token usage alone explaining 80% of BrowseComp performance variance and tool-call count plus model choice taking that to 95%.

The frame is now mainstream. OpenAI calls it “harness engineering.” Anthropic calls it “context engineering.” AMA-Bench (ICML 2026) and RealMem both target memory hygiene. Gu’s paper is the first to put a clean name and a decomposition on what those efforts have in common.

The idea

An agent is six things, not one. The model reasons. The harness around it picks what to remember, what context to assemble, which tool to call, how to verify each step, and what trace to record. Most current evaluation only measures the model.

Reader test. When someone says “this agent is better,” ask whether the model improved or whether the harness changed. Tool schemas, file inspection order, retry policies, and memory rules all sit inside the headline number and currently cannot be separated from it.

The framework: six components, one equation

Gu defines agent performance over a horizon H as:

P_H = Φ(R, M, C, S, O, G)

Model scaling improves R. System scaling improves the other five.

The paper’s main claim is conditional. Once R clears a capability threshold, marginal gains shift to M, C, S, O, and G. Gu also flags that Φ has no closed form and the six factors are not strictly orthogonal. They are six distinct points of intervention, each one engineering effort can change while holding the rest fixed.

Bottleneck 1: context governance

The hard problem of context is not capacity. It’s governance.

Gu factors context quality into four subaxes: relevance (matches the current subproblem), compactness (no more than the minimum sufficient set), traceability (provenance can be inspected), and refresh policy (stale context gets rechecked).

Failure mode: exposure without access. As context grows, the model sees more tokens but does not necessarily attend to the right ones. Relevant evidence competes with low-value padding. Attention dilutes over long inputs. Models prefer evidence at the start or end of the window rather than the middle.

System move: treat each turn’s context as the output of a selection policy, not a fixed buffer. Weight relevance, penalize verbosity against a token budget, prefer recently validated content, record provenance.

Claude Code is the production version of this. Persistent project context lives in CLAUDE.md. Just-in-time access happens through glob, grep, and file reads. The harness gets stable priors plus environment refresh on demand.

The right systems question is no longer how many tokens the model can hold. It is what the minimum sufficient context is for the current subproblem.

Bottleneck 2: trustworthy memory

The hard problem of agent memory is not storage. It’s trust.

Four subaxes again: precision (the claim has a narrow scope), durability (the target has not silently changed), retrievability (the memory can be found at acceptable cost), verifiability (the claim can be checked against the live environment). Retrievability is a precondition for using trust, not a source of it.

Failure mode: stale-but-confident. Gu’s example: a note like “the data loader is defined in utils/loader.py.” After a refactor, this is flatly wrong, but semantic search still ranks it highly. If the agent acts on it without re-checking, the error is operational.

The safety variant is worse. “Remembering More, Risking More” (Al-Tawaha et al., 2026) finds that memory-enabled agents exceed a NullMemory baseline on safety violations, and that violation rates trend upward with exposure length.

System move: trust is a runtime decision, not a property of the stored item. Store confidence, source, age. Penalize staleness in retrieval. Re-check environment-dependent claims before acting. Treat retrieved memory as a hypothesis until verified.

CheetahClaws implements this directly. Each memory entry stores confidence, source, last_used_at, and conflict_group as first-class fields. Search re-ranks by confidence × recency. The other two harnesses in the paper’s comparison derive trust implicitly from access patterns.

Durable memory without verification accumulates undetected drift. Environment-only search throws away every prior verification. A working harness keeps both.

Bottleneck 3: dynamic skill routing

The hard problem of skill is not having skills. It’s routing and checking them.

Four subaxes: specificity (each skill states what it can and cannot do), selectivity (the router invokes the right skill at the right time), composability (one skill’s post-conditions feed the next), verifiability (every skill output has an explicit check).

Failure mode: confident-but-unchecked. As specialized subagents multiply, the risk shifts from missing capability to present-but-unverified capability. A subagent returns plausible output. No downstream layer validates. This is the symmetric form of stale-but-confident memory.

System move: routing as a learned policy, not a fixed rule set, paired with post-condition checks at every step. S and G are not independent. Scaling skill quality without scaling verification produces faster but less reliable progress.

Adjacent work is converging. OpenAI’s skills docs define skills as repeatable procedures, separated from prompts, attached to the execution environment. SkillOpt (Yang et al., 2026) treats the skill file like neural-network weights and accepts edits only when held-out validation improves.

The open research direction is adaptive allocation across skills based on subtask type, confidence, and verification cost. Production routing today is mostly hand-coded.

What’s missing in evaluation

Outcome metrics answer whether the task was solved. Process metrics answer how. Two agents may both pass the same benchmark while differing in tokens, tool calls, retries, failed edits, and human interventions. Endpoint accuracy cannot see that.

Gu lists eight dimensions a harness-aware benchmark should track:

Only the first is well-covered today. tau-bench’s pass^k is an early step on reliability. AMA-Bench (ICML 2026) is an early step on long-horizon memory. The other six dimensions need work.

Safe agent evolution is the second open question. Gu proposes four sub-questions for any agent that adapts over time: what persists, what updates, what is measured, what is auditable. Memory, skills, preferences, and guardrails should not collapse into one undifferentiated state.

The evidence from Anthropic’s multi-agent post lands here. Token usage alone explained 80% of BrowseComp performance variance. Token usage plus tool calls plus model choice took it to 95%. Compute allocation across the harness is the dominant factor, not which model is wrapped.

How to apply this to your stack

Six questions, one per component, that work against any agent a reader ships today (Claude Code, Codex CLI, Hermes, OpenClaw, custom).

R: which model, and what’s its pass^k under repeated trials?

M: what’s stored, how is staleness handled, is there a re-check before action?

C: is context a buffer or a policy? What gets evicted first?

S: how many tools and skills, who routes, who post-condition-checks?

O: retry policy, handoff protocol, max-turn cap?

G: what’s logged, what’s auditable, which permissions exist independently of model output?

If your stack has no answer for one of these, that’s where the next harness gain will come from.

AlphaSignal Take

The framework is sharp. The honest gaps are three.

No metric yet. Gu names the levers but offers no quantitative measure for “harness quality.” Without that, system scaling is direction, not science. The evaluation agenda is the right shape, but every dimension after one-shot completion is still under-benchmarked today.

The reference harness is illustration, not benchmark. CheetahClaws stores confidence and recency as first-class fields and gets credit for that. The Claude Code / OpenClaw / CheetahClaws comparison table is deliberately non-ranking. The paper says explicitly: the point is not to declare a winner. Don’t read it as one.

The evaluation agenda is a wishlist. Of the eight benchmark dimensions, only one is well-covered today. tau-bench, AMA-Bench, and RealMem are early steps on three of the others. Four dimensions have no public benchmark. Scaling the Harness v2 is the paper that turns this framework into measurable scores, and it doesn’t exist yet.

Today, the practical value is vocabulary. Six components, three bottlenecks, four subaxes each, eight benchmark dimensions, four evolution questions. That’s a checklist a reader can run on Monday morning against whatever harness they ship.

Which of the six components is the weakest in the agent stack you ship today?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: install the reference harness and inspect it yourself

The paper is conceptual but its reference implementation is runnable in 40K lines of Python. Each step below points back to which paper component it instantiates.

1. Install

curl -fsSL https://raw.githubusercontent.com/SafeRL-Lab/cheetahclaws/main/scripts/install.sh | bash
source ~/.zshrc
cheetahclaws

Or via pip: pip install cheetahclaws. Python 3.10+ required. Installs the CLI binary, the harness modules, and the default config dir at ~/.cheetahclaws/.

2. Pick a model → R

cheetahclaws --model claude-opus-4-6

Switch between claude-opus-4-6, gpt-4o, gemini-2.5-pro-preview-03-25, deepseek-chat, qwen-max, or any local Ollama model with one flag. No recompile. Component: R (reasoning substrate).

3. Inspect memory → M

ls ~/.cheetahclaws/memory/
cat ~/.cheetahclaws/memory/MEMORY.md

Each entry is a Markdown file under ~/.cheetahclaws/memory/<slug>.md with frontmatter for confidence, source, last_used_at, and conflict_group. MEMORY.md is the rebuilt-on-write index. Search re-ranks by confidence × recency. Run /memory consolidate for a manual deduplication pass. Component: M (memory store).

4. Read the context constructor → C

Open context.py. The harness assembles each turn from: env block, git info, prompt assets, memory index, registered commands, tmux state, and plan-mode fragments. Compaction fires at 70% of the model’s context window with a two-layer rule-based plus AI-summarization stack. Component: C (context constructor).

5. List tools and skills → S

/skills

List Markdown skills under skill/ with frontmatter for allowed_tools, model_override, and inline-vs-fork execution. 27 built-in tools register at startup via tool_registry.py. Subagents live under multi_agent/ with built-in roles, tool restrictions, and worktree isolation. Component: S (skill router).

6. Watch the orchestration loop → O

/verbose

The agent loop yields typed events: TextChunk, ToolStart, ToolEnd, TurnDone. The full loop lives in agent.py with retry, handoff, and termination logic in one file. Component: O (orchestration loop).

7. Read the audit trail → G

/permissions manual

Permission modes (auto, accept-all, manual, plan) are set via /permissions <mode> inside the REPL. The append-only event store lives at ~/.cheetahclaws/kernel.db (SQLite). Schema is defined in cc_kernel/event_log.py with fields event_id, ts, kind, payload, causation_id, and correlation_id. Component: G (verification and governance).

8. Launch the web UI

cheetahclaws --web

All six components surface in one dashboard: model selector, memory browser, context preview, tool registry, event log. Useful for comparing the same task across two models or two permission modes. Spans all components.

This is not a recommendation that CheetahClaws is the right harness for production. It is the one in the paper’s comparison table a reader can read end-to-end in an afternoon.

In ~7 mins: what Mini Shai-Hulud did to 324 npm packages in two weeks, why Perplexity open-sourced Bumblebee to triage the fallout, the 8 on-disk surfaces it reads, the 6-step scan loop that drives it, and the 5 rough edges in v0.1.1. Appendix: full how-to reference at the end.

The Mini Shai-Hulud worm chewed through hundreds of npm packages in the last two weeks.

Mini Shai-Hulud hit 324 antv npm packages across 643 malicious versions on May 19, dropped malicious tanstack releases on May 11 with valid SLSA Build Level 3 provenance, and crossed into PyPI through lightning 2.6.2 and 2.6.3 on April 30. Microsoft, Snyk, Socket, and Wiz all published incident reports inside three weeks.

Bumblebee is the read-only scanner Perplexity open-sourced on May 22 to handle exactly that scenario. Apache-2.0, written in Go, no third-party dependencies, single static binary.

The narrow question it answers: when an advisory names a bad package, which developer laptops in the fleet still show it on disk right now?

Where Bumblebee fits

SBOMs describe what shipped into a build. EDRs describe what ran in a process. Neither describes what is still sitting in a lockfile on a developer’s laptop while the responder is on the call.

That gap matters more in 2026 than it did two years ago. Shai-Hulud 2.0 spread through Zapier, ENS, PostHog, and Postman in November 2025. Mini Shai-Hulud hit SAP npm packages, PyTorch Lightning, and the AntV npm scope across late April and May 2026.

The malicious code reached developer machines through normal npm install flows long before any production SBOM updated.

The repo is moving fast. Bumblebee crossed 2,900 GitHub stars and 1,600+ release binary downloads in the four days between v0.1.1 shipping on May 22 and now. Open issues and PRs already cover Windows defaults, NuGet, Homebrew, OSV.dev integration, and a human-readable terminal mode.

What it actually does in plain English

Bumblebee walks a list of known on-disk metadata locations, normalizes what it finds into NDJSON records, and (optionally) cross-checks those records against an exposure catalog the operator supplies.

It does not call npm ls. It does not call pip show, go list, bundle list, or composer show. It does not read source files, fetch threat intel at runtime, or watch processes. It does not ship a built-in advisory feed.

That last rule is the design pivot. During a fresh compromise, running the same package manager that just shipped the bad release is the worst move a responder can make. Bumblebee is built around never doing that.

Eight ecosystems are covered in v0.1.1:

The MCP, editor, and browser surfaces are the interesting part. Most existing supply-chain tools stop at language registries. Bumblebee treats Claude Desktop configs, Cursor extensions, and Chrome add-ons as part of the same surface the developer is exposed to.

How the scan loop works

The scanner runs six steps per invocation:

1. Resolve safe scan roots based on the selected profile.

2. Walk the file tree, skipping symlinks and over 100 sensitive or noisy directory patterns (.ssh, .aws, .kube, .gnupg, Library/Keychains, Library/Mail, Library/Cookies, browser cache subtrees).

3. Dispatch each recognized basename to an ecosystem-specific parser.

4. Normalize names (npm lowercase, PyPI PEP 503).

5. Emit one NDJSON package record per identity.

6. If --exposure-catalog was supplied, do exact (ecosystem, name, version) matching and emit a finding record per hit. Close the run with a scan_summary.

Three scan profiles control how much of the disk gets walked:

baseline and project refuse a bare-home root by default. Only deep walks one.

Exact-match-only catalog logic is a deliberate choice. CVE scanners running on version ranges produce noise during a fresh worm wave because the advisory itself is still being scoped. Bumblebee answers a tighter question: did the exact compromised version land on this machine? That maps directly to the incident channel where someone just said “anyone with lightning@2.6.2 raise your hand.”

Safety properties are baked in below the parser layer:

• No package-manager execution and no source-file reads.

• No bundled threat intel and no network calls during the scan.

• MCP env values and key names are dropped, so credentials sitting in Claude Desktop configs never leave the host.

• .env and .envrc are skipped even when they fall inside a walked root.

• Remote MCP server URLs are reduced to scheme://host before being recorded, so embedded path-segment credentials cannot leak.

The repo ships 161 Go test functions across 23 test files, plus a CI matrix on Ubuntu and macOS that runs go vet, go test -race, a fresh build, bumblebee selftest, and govulncheck.

How to get started

Three commands cover the smoke test on a clean machine:

bash

go install github.com/perplexityai/bumblebee/cmd/bumblebee@v0.1.1
bumblebee selftest
bumblebee scan --profile baseline > inventory.ndjson

Requires Go 1.25+. Zero non-stdlib dependencies.

bumblebee selftest extracts embedded fake-package fixtures to a tempdir, runs the scanner against an embedded exposure catalog, and asserts a fixed finding count. A non-zero exit means the local install can no longer detect what it should.

The baseline scan writes one NDJSON object per line to inventory.ndjson and diagnostics to stderr. The last line is a scan_summary record. Promote a snapshot into a downstream system only when that summary has status=complete. Partial or errored runs are evidence, not deletion signals.

The release binaries, the controlled left-pad fixture, the HTTP sink, and the threat_intel/ catalog runs are all in the appendix at the end of this article.

AlphaSignal Take

Bumblebee is a sharp v0.1, but it ships with five rough edges worth naming.

No Windows release. The v0.1.1 assets are macOS and Linux only, amd64 and arm64. Issue #2 and PRs #4 and #16 cover default root discovery and full Windows support, none merged yet.

No live OSV.dev source. Open issue #21 asks for it. Catalog matching today is limited to the eight files in threat_intel/ (654 total entries) and whatever JSON the operator writes by hand.

Exact-match only. Version-range solving is out of scope by design. The trade-off is real: upstream catalog quality determines accuracy. A wrong version string in a PR-submitted catalog produces a wrong finding rate fleet-wide.

NDJSON-first output. PR #24 adds opt-in terminal output, but main still expects jq for any human reading. Issue #22 is open on that too.

One small documentation gap. The README still prints selftest OK (2 findings in 1ms). The source in cmd/bumblebee/selftest.go asserts expectedSelftestFindings = 3. Tiny issue, but the kind of thing a responder notices the first time they actually run the binary.

The architecture itself is the bet. Operator-supplied catalogs scale faster than vendor-curated advisory feeds in the first hour of a worm wave, when the question is “which laptops have this exact version” and the answer needs to be in Slack before lunch.

Where do you draw the line on endpoint security: the SBOM, or the laptop itself?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: full how-to reference

Install from the release tarball

macOS Apple Silicon:

bash

VERSION=v0.1.1
curl -L -o checksums.txt "https://github.com/perplexityai/bumblebee/releases/download/${VERSION}/checksums.txt"
curl -L -o bumblebee.tar.gz "https://github.com/perplexityai/bumblebee/releases/download/${VERSION}/bumblebee_0.1.1_darwin_arm64.tar.gz"
shasum -a 256 -c checksums.txt --ignore-missing
tar -xzf bumblebee.tar.gz
./bumblebee version

Linux amd64:

bash

VERSION=v0.1.1
curl -L -o checksums.txt "https://github.com/perplexityai/bumblebee/releases/download/${VERSION}/checksums.txt"
curl -L -o bumblebee.tar.gz "https://github.com/perplexityai/bumblebee/releases/download/${VERSION}/bumblebee_0.1.1_linux_amd64.tar.gz"
sha256sum -c checksums.txt --ignore-missing
tar -xzf bumblebee.tar.gz
./bumblebee version

Preview what a profile will walk

bash

bumblebee roots --profile baseline

Prints <root_kind>\t<path> lines without walking anything.

Run a controlled project scan with a custom catalog

bash

mkdir -p /tmp/bee-demo && cd /tmp/bee-demo

cat > package-lock.json <<'JSON'
{
  "name": "bee-demo",
  "lockfileVersion": 3,
  "packages": {
    "": { "dependencies": { "left-pad": "1.3.0" } },
    "node_modules/left-pad": {
      "version": "1.3.0",
      "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz"
    }
  }
}
JSON

cat > exposure-catalog.json <<'JSON'
{
  "schema_version": "0.1.0",
  "entries": [
    {
      "id": "demo-left-pad-1.3.0",
      "name": "Demo left-pad exposure",
      "ecosystem": "npm",
      "package": "left-pad",
      "versions": ["1.3.0"],
      "severity": "low",
      "source": "local test fixture"
    }
  ]
}
JSON

bumblebee scan \
  --profile project \
  --root "$PWD" \
  --exposure-catalog exposure-catalog.json > project.ndjson 2> project.diag.ndjson

jq 'select(.record_type=="finding")' project.ndjson

Expected output: one package record, one finding record, one scan_summary with status=complete.

Suppress packages, keep findings

bash

bumblebee scan \
  --profile project \
  --root "$PWD" \
  --exposure-catalog exposure-catalog.json \
  --findings-only > findings.ndjson

--findings-only requires --exposure-catalog. It keeps finding, scan_summary, and diagnostics. It drops package records.

Scan against the shipped threat-intel catalogs

From a source checkout or the release archive:

bash

bumblebee scan \
  --profile deep \
  --root "$HOME/code" \
  --exposure-catalog threat_intel \
  --findings-only \
  --max-duration 10m > threat-findings.ndjson

The threat_intel/ directory contains 8 catalog files and 654 entries at the studied commit, covering Mini Shai-Hulud npm + PyPI, the AntV / Mini Shai-Hulud npm wave, GemStuffer RubyGems, Laravel Lang Packagist, node-ipc credential stealer, the nx-console VS Code compromise, shopsprint/decimal Go typosquat, and the TrapDoor crypto stealer.

Ship results over HTTP

bash

export BUMBLEBEE_TOKEN="..."
export BUMBLEBEE_DEVICE_ID="laptop-001"

bumblebee scan \
  --profile baseline \
  --output http \
  --http-url https://inventory.example.com/v1/ingest \
  --http-auth bearer \
  --http-token-env BUMBLEBEE_TOKEN \
  --http-gzip \
  --device-id-env BUMBLEBEE_DEVICE_ID

NDJSON body. Content-Type: application/x-ndjson. HTTPS required for non-loopback hosts. HMAC mode (X-Inventory-Signature: sha256=<hex>) is also available. Signature input is the raw post body, or <timestamp>.<body> when X-Inventory-Timestamp is set. Compression happens before HMAC signing.

Common error modes worth knowing

• deep without --root is rejected.

• --findings-only without --exposure-catalog is rejected.

• --ecosystem cargo (and anything outside the eight supported values) is rejected.

• Binary bun.lockb emits a diagnostic only. Text bun.lock is parsed.

• macOS deep scans hitting TCC-protected paths produce diagnostics, not findings.

In ~7 mins: the third way to adapt a frontier agent (after weights and prompts), the 5 optimizer controls that make text-space training actually behave like training, how to wire the pattern into your Claude Code, Codex, or CLAUDE.md skill file, and a full repo tutorial at the end if you want to run it yourself.

There are two known ways to adapt a frontier agent. Change the weights, or change the prompt / harness.

A new paper from Microsoft argues for a third. The agent’s skill file becomes the trainable artifact, edited by a separate optimizer model under bounded updates and a held-out gate.

The result is best or tied on 52 of 52 evaluated cells. GPT-5.5 lifts +23.5 points in direct chat, +24.8 inside Codex, +19.1 inside Claude Code. The deployed artifact stays under 2,000 tokens.

Paper

SkillOpt is a May 2026 paper from Microsoft, Shanghai Jiao Tong University, Tongji University, and Fudan University. Yifan Yang at Microsoft leads the 15-author group. arXiv lists it as 2605.23904, submitted on May 22, with 27 pages, 4 figures, and 6 tables.

Full title: “SkillOpt: Executive Strategy for Self-Evolving Agent Skills.” Hugging Face Papers ranked it #1 paper of the day at capture, with 140 upvotes.

Companion repo microsoft/SkillOpt is MIT-licensed Python 3.10+. The package is at version 0.1.0 with no public releases yet, 66 stars at capture on May 25. It ships configs for six benchmarks, the trainer, scripts/train.py, an eval_only.py entry point, and an optional Gradio dashboard.

Scope is what makes it worth seven minutes. SkillOpt trains one Markdown skill across 6 benchmarks (SearchQA, SpreadsheetBench, OfficeQA, DocVQA, LiveMathematicianBench, ALFWorld), 7 target models from GPT-5.5 down to Qwen3.5-4B, and 3 execution modes (direct chat, Codex, Claude Code). That product gives 52 head-to-head cells against human-written, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill baselines. SkillOpt wins or ties every single one.

The trainable-skill idea

A skill, in the paper’s sense, is a Markdown file prepended to the agent’s context. It holds procedural rules: how to use tools, what to verify, how to format an answer, what failure mode to avoid. Frontier models do not hold this kind of domain procedure in weights.

SkillOpt treats that file as the external state of a frozen target model. The target never moves. A separate optimizer model edits the skill from scored rollouts. The deployed artifact is one best_skill.md between 379 and 1,995 tokens. No optimizer calls at deployment.

The deep-learning analogy is operational, not decorative. Rollouts are the forward pass. Reflection batches over success and failure trajectories are the backward pass. An actual edit budget plays the role of a learning rate. An actual held-out validation gate plays the role of validation. The epoch-wise slow update plays the role of momentum.

5 controls that make text-space training work

Control 1: Bounded text updates (the textual learning rate)

Every reflection pass ends with a top-N cut. The default edit budget is 4, with cosine decay to a floor of 2. The repo also exposes constant, linear, and autonomous schedules. Without a cap, the loop is ad hoc prompt rewriting.

Removing the budget drops SearchQA / SpreadsheetBench / LiveMath to 84.6 / 75.7 / 57.3, against the default 87.1 / 77.5 / 61.3. The paper calls this control the textual learning rate.

Bounded edits keep adjacent skill versions close enough that the next optimizer call can still learn from the last one. Unbounded rewrites break the optimization history before any of the later controls get a chance to use it.

Control 2: The validation gate

A candidate skill is accepted only if its held-out selection score is strictly greater than the current best. Ties are rejected. The selection split is used only for accept-or-reject decisions. The test split is reported separately.

That is what keeps reflection from becoming drift. Across 6 benchmarks, only 1 to 4 edits per skill survive into the deployed artifact. The optimizer proposes far more. LiveMathematicianBench’s +29.3-point gain comes from a single accepted edit. OfficeQA’s +39.0-point gain also comes from one accepted edit.

Bulk of the optimizer’s text-space search gets rejected. The deployed skill is the small set of changes that actually moved a held-out number.

Control 3: The rejected-edit buffer

Edits that fail the gate are not discarded. They enter an epoch-local memory the optimizer reads before proposing the next batch, along with the score drop they caused.

That gives the loop negative feedback during training without adding any inference-time model calls. Removing the buffer drops SearchQA / SpreadsheetBench / LiveMath by 1.6 / 4.6 / 2.4 points in the matched ablation row.

Optimizer learns not to repeat a harmful edit, the way a fine-tuned model learns not to repeat a low-reward output. The difference is that this memory is plain text and lives only for the current epoch.

Control 4: Slow and meta updates

At each epoch boundary, the optimizer runs the same sampled training items under the previous-epoch skill and the current-epoch skill. Outcomes group into improved, regressed, persistent failure, and stable success.

A concise longitudinal guidance block then goes into a protected region of the skill file. Step-level edits cannot overwrite that region.

Meta skill is separate. Optimizer-side memory of which edit patterns helped or hurt across epochs prepends to future optimizer prompts. It does not ship with best_skill.md.

Removing both slow update and meta skill collapses SpreadsheetBench from 77.5 to 55.0. That 22.5-point drop is the largest single ablation in the paper.

Control 5: Harness-agnostic adapter

One Markdown file deploys across three harnesses. Direct chat, the Codex CLI in a workspace-write sandbox, and the Claude Code CLI all read the same best_skill.md. The adapter contract is small: build batches, inject the skill, run the native execution loop, return scored trajectories.

Transfer numbers carry the claim. A SpreadsheetBench skill trained inside Codex adds +59.7 points when deployed inside Claude Code (22.1 → 81.8), slightly exceeding the in-domain Claude Code SkillOpt reference. The reverse direction adds +43.6 points back inside Codex (27.5 → 71.1).

Trained skill is a portable artifact, not a harness-specific command recipe. Training cost amortizes across deployment surfaces.

8-stage loop, mapped to your stack

Loop runs eight stages per step.

CLI commands and the install-to-deploy walk are in the appendix at the end.

1. Rollout: the frozen target model runs a batch from the training split with the current skill.

2. Reflect: the optimizer model splits the batch into failure and success minibatches, returns structured add, delete, and replace edits.

3. Aggregate: similar edits merge hierarchically, with failure-driven patches prioritized.

4. Select: the optimizer ranks edits and clips to the top of the edit budget.

5. Update: selected edits apply, producing a candidate skill.

6. Gate: the candidate runs on the held-out selection split. Strictly greater than current best is the only accept condition.

7. Slow update: at epoch end, the optimizer compares same-task outcomes under last-epoch and current-epoch skills, then writes longitudinal guidance into a protected region.

8. Meta skill: optimizer-side memory of accepted and rejected patterns is prepended to future optimizer calls. Never ships with the deployed skill.

What the learned skill actually says

Paper Figure 4 reproduces one verbatim rule per benchmark from the final best_skill.md of each case study. Two of them carry the flavor.

SpreadsheetBench, after four accepted edits: “Inspect workbook structure and formulas, then write evaluated static values across the full requested target range instead of relying on Excel recalculation.”

DocVQA, after three accepted edits: “For tables, forms, charts, and legends, first bind the question to the exact visual row/header/field, then copy only the aligned answer span.”

Two patterns stand out. Rules are procedural rather than instance-specific (no question, file, or entity is named). Rules also encode discipline frontier models lack zero-shot: workbook-structure-first reasoning, evidence-to-visual binding, answer-format constraints. A practitioner could write rules like these by hand after a day with the benchmark. SkillOpt produces them automatically and validates each one against a held-out split.

Map this into your stack

The deployed artifact is one Markdown file. That maps cleanly to wherever your agent already loads procedural state.

• Claude Code: drop the trained skill into ~/.claude/skills/ and load it on session start.

• Codex / OpenAI Agents: render to a per-task SKILL.md or AGENTS.md. The paper’s Codex adapter already uses this contract.

• Generic harnesses: CLAUDE.md, .cursorrules, or the system-prompt slot of any agent.

• Hermes-style persistent runtimes: a skill-folder entry. The exported artifact is harness-agnostic Markdown by design.

Running the loop on your own task needs three things:

1. A task family with measurable success. Exact match, executable check, or a verifier you trust.

2. Held-out train, selection, and test splits. The repo does not ship datasets, so this is on you.

3. A target model (the agent you ship) and an optimizer model. The paper defaults both to GPT-5.5, and shows a target-matched optimizer still recovers 56% to 74% of the strong-optimizer gain.

Optimizer runs offline. Deployment uses only the final skill file. No extra model calls at inference.

Full install-to-deploy commands are in the appendix below.

Results

The AlphaSignal Take

Three open problems sit underneath the headline numbers.

Training cost is real. Cost per absolute test-point gain runs from 0.6M training tokens (SpreadsheetBench) to 46.4M (DocVQA). Total training token spend per benchmark in the case studies ranges from 20.8M (OfficeQA) to 213.8M (SearchQA). A team has to budget for the offline run before assuming it pays off.

The loop needs scored tasks. SkillOpt is an optimizer with a verifier. The gate compares numbers from a held-out split. Open-ended creative work, strategy documents, design judgment have no gate to gate on, unless a preference model gets layered in that the paper does not provide.

The repo does not ship datasets. Readers bring their own train, selection, and test splits and their own credentials. The package is at version 0.1.0 with no public releases. Fastest setup path is SearchQA, and even that needs a local split and an Azure OpenAI, OpenAI, or Anthropic key. This is research code, not a turnkey product.

What is actually new is the reframe. The procedure an agent follows becomes a trainable, inspectable text artifact. Not weights. Not a static prompt. Something in the middle, with a versioned history, an audit trail of accepted and rejected edits, and a 379-to-1,995-token deployment footprint.

That is the part of the announcement worth carrying into your own stack, even before training one. The paper points to two next moves: skill libraries that share infrastructure across domains, and self-distillation of trained skills back into target-model weights. Both assume the skill itself is the object being optimized, not a byproduct of prompting.

Which part of your current agent stack would you train as text first, your CLAUDE.md, your skill folder, or your AGENTS.md?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Appendix: How to actually run SkillOpt

A condensed install-to-deploy walk. Assumes Python 3.10+ and one model backend. Azure OpenAI is the paper’s default.

1. Install

git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e .

ALFWorld benchmark needs an extra step: pip install -e “.[alfworld]” then alfworld-download.

2. Set credentials

Azure OpenAI (default):

export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-key"

Or set AZURE_OPENAI_AUTH_MODE=azure_cli to skip the key and use Azure CLI auth.

Other backends also supported: OPENAI_API_KEY for OpenAI direct, ANTHROPIC_API_KEY for Claude, QWEN_CHAT_BASE_URL + QWEN_CHAT_MODEL for Qwen via local vLLM. The repo’s .env.example lists all four.

3. Prepare data

SkillOpt expects this directory layout:

data/my_split/
  train/items.json
  val/items.json
  test/items.json

Each items.json is a JSON array of task items. Schema depends on the benchmark. SearchQA wants id, question, context, answers.

Configs ship for six benchmarks: SearchQA, ALFWorld, DocVQA, LiveMathematicianBench, SpreadsheetBench, OfficeQA. The repo does not ship the datasets, so this step is on you. SearchQA is the fastest setup path.

4. Train

Minimal command (SearchQA, GPT-5.5 as both target and optimizer):

python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --split_dir /path/to/searchqa_split \
  --azure_openai_endpoint https://your-resource.openai.azure.com/ \
  --optimizer_model gpt-5.5 \
  --target_model gpt-5.5

Defaults from configs/_base_/default.yaml: 4 epochs, batch size 40, reflection minibatch 8, edit budget 4 with cosine decay to floor 2, slow update on with 20 samples per epoch, meta skill on, validation gate strict-greater.

CLI override flags: --num_epochs, --batch_size, --workers, --out_root.

5. Output

Each run writes to outputs/<run_name>/:

• best_skill.md: the deployable skill.

• history.json: per-step training log.

• skills/skill_vXXXX.md: skill snapshot per step.

• steps/step_XXXX/: patches, gate evals, edit-apply reports.

• slow_update/epoch_XX/ and meta_skill/epoch_XX/: epoch-end logs.

Re-running the same command auto-resumes from the last completed step.

6. Evaluate

Score a trained skill on any split without retraining:

python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill outputs/my_run/best_skill.md \
  --split valid_unseen \
  --split_dir /path/to/searchqa_split \
  --azure_openai_endpoint https://your-resource.openai.azure.com/

Valid --split values: valid_unseen (test), valid_seen (val), train, all.

7. Deploy

best_skill.md is plain Markdown. No optimizer calls at inference. Drop it where your agent reads procedural state:

• Claude Code: ~/.claude/skills/

• Codex / OpenAI Agents: AGENTS.md or per-task SKILL.md

• Generic: CLAUDE.md or the system-prompt slot

• Hermes-style persistent runtime: a skill-folder entry

8. Optional: WebUI monitor

pip install -e ".[webui]"
python -m skillopt_webui.app

Default port 7860. Add --share for a public Gradio link.

In ~7 mins: the 5 principles a 20-author survey of 250+ tools just landed on, an 8-stage map of where AI helps and where it breaks, and 6 rules for using it without losing scientific accountability.

This article was originally posted on X last Fri (22 May).

The AI Scientist generates a complete research paper for $15.

FARS ran for 228 hours, burned 11.4 billion tokens, and shipped 100 papers.

When the same systems run fully autonomous, 80% of their reported results are fabricated.

The bottleneck moved. It is no longer generation. It is verification, provenance, and human handoffs across the research lifecycle.

What happened

Twenty researchers just published the first end-to-end survey of AI across the complete academic research lifecycle.

The paper is authored by Lingdong Kong (NUS / Apple) with 19 co-authors and titled “AI for Auto-Research: Roadmap & User Guide.” It was posted to arXiv on May 18, 2026 and covers developments through April 2026.

The scope is unusual. Most existing surveys cover one stage: literature review, or coding agents, or paper writing. This one maps 250+ tools, 52 benchmarks, and 33 end-to-end systems into a single framework of 4 phases and 8 stages.

The companion repo, worldbench/awesome-ai-auto-research, is MIT-licensed and tracking +100 stars and 8 forks as of May 22.

The framework breaks the lifecycle into Creation (ideation, literature, coding, figures), Writing (manuscript), Validation (peer review, rebuttal), and Dissemination (Paper2X). That structure carries the whole argument.

Why this paper

The cost-to-quality numbers refuse to scale.

AI Scientist v2 scores 6.33 on an ICLR 1–10 scale at $25 per paper. FARS, running roughly $1,000 per paper, scores 5.05. The ICLR acceptance threshold is 5.69.

The cheaper system is past the line. The 40x more expensive one is below it.

Pattern-matching benchmarks overstate scientific coding too. Frontier systems clear 76%+ on SWE-bench Verified. The same systems ceiling at 37–39% on ResearchCodeBench, where the task is to implement a method described in a paper. The semantic-error rate there is 58.6%.

The validation numbers are worse. In an LLM-reviewer benchmark, 95.8% of rejected papers were misclassified as acceptable. In MLR-Bench’s fully-autonomous track, 80% of reported results turned out to be fabricated.

The paper’s read is that the field stopped being capability-limited and became reliability-limited. That reframe is what makes the survey worth reading.

Principle 1: Structured tasks work. Open-ended judgment does not.

AI is reliable when the task is structured, grounded in retrievable evidence, and externally checkable.

SWE-bench Verified at 76%+ vs. ResearchCodeBench at 37–39% is the cleanest illustration. One measures bug fixes against known passing tests. The other measures whether the model implemented the algorithm the paper actually described.

Same models. Different ceiling.

This holds across stages. Retrieval, citation candidates, plot drafts, grammar polishing, format conversion: solid. Novelty assessment, decisive experiment design, long-horizon reasoning, contribution framing: fragile.

Principle 2: Generation outpaces verification at every stage.

This is the paper’s central tension. AI produces research-shaped artifacts faster than it can prove they are correct.

Ideas look novel on the page and weaken after a single implementation attempt. Code runs cleanly while implementing a different algorithm than the paper described. Figures look publication-ready while distorting axes or dropping baselines.

Reviews come back coherent and lenient. Rebuttals read as persuasive and promise experiments that authors never run. Dissemination artifacts simplify results past the evidence the paper actually provides.

The risk is not that the artifacts are useless. It is that they get treated as validated because they look complete.

Principle 3: Human-governed collaboration beats full autonomy.

The strongest empirical result in the paper comes from the ICLR 2025 randomized study of AI in peer review. Across 22,467 reviews, 89% improved in quality when an LLM gave feedback on a human reviewer’s draft.

Hand the same family of models a paper to review alone, and 95.8% of rejected papers come back misclassified as acceptable.

Assist mode lifts quality. Replace mode breaks it.

That asymmetry shows up everywhere the paper looks for it: writing, review, rebuttal, dissemination. AI augments researchers reliably. As the reviewer of record, the same model fails reliably.

Principle 4: Working systems converge on three layers: explore, execute, verify.

Systems that produce credible work all combine the same three layers, regardless of branding.

Exploration searches over hypotheses, code variants, or response strategies through MCTS, evolutionary methods, or branching agents. Execution drives external tools: code interpreters, retrieval engines, experiment runners, plotters, document editors. Verification checks intermediate outputs through execution feedback, citation validation, adversarial critique, or human review.

Stacks built on “more agents = better” lose on sequential reasoning. Google and MIT scaling work cited in the paper finds an empirical sweet spot at 3 to 4 coordinated agents. Bigger swarms accumulate communication overhead faster than they gain critique quality.

Principle 5: AI in research is a governance problem, not a detection problem.

Corpus studies estimate detectable AI modification in 17.5% of computer science abstracts and 13.5% of biomedical abstracts. Self-reported usage runs higher.

Detection-based enforcement does not scale. AI text detectors false-positive on formal academic prose and non-native writing. Watermarking depends on provider cooperation and breaks under paraphrase and translation.

What stays durable is a different set of questions. Which forms of AI use must be disclosed? Who is accountable when an AI-generated citation is fabricated, or an AI-drafted rebuttal commits to an experiment that never runs?

Policy has to follow disclosure and accountability, not detection. The paper lands there.

The 8 stages, mapped to what’s safe to automate

The lifecycle compresses into a map. Each stage has work AI does well, work that needs human inspection, and work that should not be delegated yet.

Six rules sit underneath the table.

1. Treat every phase handoff as a verification checkpoint, not a transition.

2. Prefer execution-grounded evaluation over LLM-as-judge for any claim that can be tested.

3. Use AI to strengthen human reviews. The 89% quality lift only shows up in assist mode.

4. Track every rebuttal promise against the actual manuscript diff before camera-ready.

5. Compare every dissemination artifact against the paper’s caveats before release.

6. Disclose AI use, attribute decisions, accept accountability for AI-generated claims.

The AlphaSignal Take

The AI Scientist camp is not wrong about cost. They are wrong about ceiling.

Sakana AI, FARS, and the AI Scientist v2 line argue autonomy is already useful at the right cost-quality tradeoff. The rebuttal sits inside the paper itself. A 40x increase in spend per paper (from $25 to $1,000) does not buy quality. It buys volume that scores below the acceptance threshold.

Three problems remain open across every system surveyed. They are the real reason the field is not where the demos suggest.

Phase-boundary faithfulness. No system surveyed maintains a traceable link from hypothesis to dissemination. Hypotheses, logs, manuscript claims, and rebuttal promises break apart at every handoff.

Citation provenance. Generated bibliographies routinely mix metadata across preprint, workshop, conference, and journal versions of the same work. Author list, year, venue, and DOI can come from four different records of one paper. No surveyed tool fixes this.

Cognitive ownership. Aggressive automation hides the work that turns a junior researcher into a senior one. Delegating literature synthesis or rebuttal prevents the field judgment and critical reasoning that build over time.

The reason this paper is worth seven minutes is the reframe. It stops asking whether one autonomous AI scientist can replace a human researcher. It starts asking whether the artifacts a research process produces still link to evidence by the time they reach the public.

That is the right question to be asking in May 2026.

Which principle does your current AI research stack solve, and which one is still open?

All source links are in the first reply. Full breakdown of recent updates + daily signals in our newsletter (link in bio).

In ~8 mins: what SDD is, why it became the default for AI coding, how the 5 leading repos implement it, and the one critic saying the whole category is wrong.

Spec-driven development crossed from blog-post topic to default architecture for AI coding in the last 12 months.

Thoughtworks, Martin Fowler, GitHub, Amazon, and a 67-source academic review all agreed in 2025 and 2026.

The question stopped being whether to use SDD and became which implementation.

What happened

Multiple independent sources converged on the same recommendation inside 18 months.

Thoughtworks listed spec-driven development in Technology Radar Vol. 32 as a technique worth adopting. Martin Fowler covered it on his site.

GitHub shipped Spec Kit, an MIT-licensed toolkit framed as the answer to vibe coding. Amazon launched Kiro, an agentic tool that walks users through requirements, design, and tasks before any code generation. Tessl launched at the radical end, with specs positioned as the new source code.

Red Hat published enterprise SDD guidance. InfoQ covered it at the architecture level.

Bryan Finster pushed back with the right critique. SDD is not a revolution, it’s just BDD with branding.

That critique strengthens the case. The idea is not new. The context is.

BDD was an optional discipline that teams could adopt or ignore. With 84% of professional developers using or planning to use AI tools (Stack Overflow, 2025) and 46% of code output now AI-generated (GitHub, 2025), specification discipline became structurally necessary.

Why it became necessary

Four academic papers landed in 12 months, mapping the same problem from different angles.

Sabry Farrag at the University of East London ran a 67-source systematic review of the productivity paradox. AI coding tools deliver real individual-level gains and real system-level damage at the same time.

Peng et al. measured 55.8% faster completion in a 95-developer RCT. Becker et al.’s METR study found a 19% slowdown for experienced developers working on mature codebases.

DORA reported that 25% AI adoption correlates with a 7.2% drop in delivery stability. Faros AI tracked over 10,000 developers and saw 98% more merged PRs, 91% more review time, and 9% more bugs.

Shuvendu Lahiri at Microsoft Research named the underlying gap. AI-generated code is plausible by construction, not correct by construction. The semantic distance between what a user means and what a program does is the central reliability bottleneck.

An AIware 2026 vision paper named a second gap. Code review evaluates plausibility, not compliance. Most AI-generated changes pass tests, look reasonable, and still drift from the rules they were supposed to follow.

Deepak Babu Piskala wrote the practitioner manual that ties it together. He frames SDD across three rigor levels and a four-phase workflow.

Farrag’s economic argument closes the loop. Code generated for a specific codebase has high asset specificity. LLMs introduce high behavioral uncertainty.

Developers invoke AI hundreds of times daily. In Transaction Cost Economics terms, that combination makes a written, executable contract the rational governance response. SDD is that contract.

How it actually works

SDD compresses to three things a practitioner needs to hold.

A four-phase workflow. Specify what the software should do. Plan how to build it. Implement in small, validated increments. Validate that the code meets the spec. Each phase produces an artifact that constrains the next.

Three rigor levels. Spec-first means a specification is written before coding but may drift after. Spec-anchored means the spec lives alongside the code and tests enforce alignment. Spec-as-source means the spec is the only artifact humans edit, with code regenerated rather than manually changed.

A governance spectrum. Farrag’s paper ranks four mechanisms by constraint intensity:

• Post-hoc review is the loosest, where a developer reviews AI output after the fact.

• Natural-language specification is next, putting requirements before generation.

• Executable contract follows, with tests and structured spec documents the agent must satisfy.

• Constitutional governance is the tightest, a meta-specification of non-negotiable principles that every change must honor.

The higher the asset specificity, behavioral uncertainty, and frequency, the further up the spectrum the rational choice sits. Production code in a mature codebase invoked by AI hundreds of times daily lands at constitutional. A throwaway prototype lands at post-hoc.

The five SDD repos, by philosophy

Each repo encodes a different theory of where complexity belongs.

Full comparison table at the end. Links are in replies.

Spec Kit: constitution as authority

GitHub’s official toolkit, MIT-licensed, Python CLI (specify init).

The theory of complexity: put it in the constitution. A non-negotiable principles file at .specify/memory/constitution.md sits above every spec and every implementation. The agent obeys it on every change, every session.

The workflow runs through nine slash commands:

• /speckit.constitution

• /speckit.specify

• /speckit.clarify

• /speckit.plan

• /speckit.tasks

• /speckit.taskstoissues

• /speckit.checklist

• /speckit.analyze

• /speckit.implement

The constitution and analyze steps are where the formal governance lives.

Farrag’s paper evaluates Spec Kit as the direct instantiation of constitutional governance. The reported result: 12 hours to 15 minutes for upstream artifact production (PRD, design, structure, technical specs, test plans).

A pilot study saw late-stage hotfixes drop from 3-to-5 per sprint to 1-to-2, and rollbacks drop from 2-to-4 per month to 0-to-1.

30+ AI agent integrations including Claude, Codex, Copilot, Cursor, Gemini.

This is the only repo with explicit constitutional governance. The highest tier on Farrag’s spectrum, and the steepest setup cost.

BMAD-METHOD: named agents as authority

BMad Code LLC, MIT, npm (npx bmad-method install). V6, with 34+ workflows.

The theory of complexity: put it in the roles. Six named personas, each with domain expertise:

• Analyst Mary handles brainstorming and research.

• PM John owns PRDs.

• Architect Winston runs the 8-step architecture workflow.

• Developer Amelia handles dev stories, sprint planning, and code review.

• UX Designer Sally owns interface decisions.

• Tech Writer Paige owns documentation.

Party Mode brings multiple personas into one session to argue from different professional perspectives.

The lifecycle has four phases: Analysis, Planning, Solutioning, Implementation. Each phase has its own workflows.

A .decision-log.md records every decision as an audit trail. An implementation-readiness gate (PASS, CONCERNS, or FAIL) blocks the move to code if anything is missing.

Planning depth auto-adjusts to project stakes. A hobby project gets a 2-page PRD. A launch project gets full specs. The bmad-help skill answers free-form questions about what to do next.

The module ecosystem extends the core with specialized domains: BMM (core), BMB (custom agents), TEA (test architecture), BMGD (game dev), CIS (creative intelligence).

This is the only repo that treats specifications as the inter-agent communication protocol of a multi-agent organization.

OpenSpec: change folders as the unit

Fission AI, MIT, npm (openspec init).

The theory of complexity: put it in the change. Each feature gets its own folder containing proposal.md (why this change), specs/ (requirements and scenarios), design.md (technical approach), and tasks.md (implementation checklist).

When the change ships, /opsx:archive folds the change spec into a growing source-of-truth document.

The core surface is three commands:

• /opsx:propose creates the change folder.

• /opsx:apply has the AI implement the task checklist.

• /opsx:archive closes it out.

An opt-in expanded profile adds six more: /opsx:new, /opsx:continue, /opsx:ff, /opsx:verify, /opsx:bulk-archive, /opsx:onboard.

The positioning is explicitly brownfield-first. Most SDD tools optimize for greenfield projects. OpenSpec is built to retrofit existing codebases. The delta-spec format (additions, modifications, removals tracked per change) is what makes that work.

Works with 25+ AI assistants via slash commands.

Executable contract at the lightest possible weight. No constitution, no named agents, no ceremony. The spec discipline survives without the process.

GSD: context as the bottleneck

TÂCHES, MIT, npm (npx get-shit-done-cc@latest). Built by a solo developer for solo developers.

The theory of complexity: put it in context engineering. The main session context stays at 30 to 40 percent. Heavy work runs in fresh subagent contexts, each getting a full 200K-token window.

The hypothesis the rest of the architecture rests on: as a session grows, AI output degrades, so the architecture should keep the session small.

The loop is six commands:

• /gsd-new-project runs questions, research, requirements, roadmap.

• /gsd-map-codebase does the same for existing code.

• /gsd-discuss-phase captures decisions before planning.

• /gsd-plan-phase runs research, plan, verify in a loop.

• /gsd-execute-phase dispatches parallel waves of subagents.

• /gsd-verify-work walks through what was built and diagnoses failures.

Five persistent state files survive session boundaries: PROJECT.md (vision), REQUIREMENTS.md (scope), ROADMAP.md (direction), STATE.md (current position), CONTEXT.md (per-phase decisions).

The .planning/config.json controls mode (interactive or yolo), model profiles (quality, balanced, budget), and quality-agent toggles. Package legitimacy checks are built into the install path.

Executable contract delivered through context discipline rather than process ceremony. The repo treats the context window as the bottleneck, not the methodology.

Superpowers: auto-triggering as discipline

Built by Jesse Vincent and Prime Radiant. MIT, zero-dependency plugin.

The theory of complexity: put it in the agent’s behavior shaping. Skills auto-trigger at the right moments. No manual invocation. Mandatory workflows, not suggestions.

The using-superpowers skill loads at session start and is what makes auto-triggering work. Copying skill files alone is not a real integration.

Seven core skills run the workflow:

• brainstorming refines rough ideas before any code.

• using-git-worktrees isolates the workspace.

• writing-plans breaks work into 2 to 5 minute tasks with exact file paths and complete code.

• subagent-driven-development dispatches a fresh subagent per task with two-stage review (spec compliance, then code quality).

• test-driven-development deletes any code written before its test.

• requesting-code-review blocks critical issues.

• finishing-a-development-branch verifies tests and presents merge options.

The TDD enforcement is the unusual move. Most TDD tooling encourages the loop. Superpowers’ skill deletes code that violates it.

Distributed through the official Claude plugin marketplace, the official Codex plugin marketplace, Factory Droid, Gemini extensions, Cursor, GitHub Copilot CLI, and OpenCode.

Executable contract enforced at the agent layer rather than the user layer. The user never has to remember to invoke the right skill.

The sixth repo, and the case against the category

Matt Pocock’s Skills For Real Engineers sits on the same list of repos by accident. He argues against the category.

His talk Software Fundamentals Matter More Than Ever lands the thesis directly. “Code is not cheap. In fact, bad code is the most expensive it’s ever been.”

On the spec-driven movement specifically: “Specs to code, we are not investing in the design of the system. We are divesting from it.”

His position rests on a software-engineering claim. Bad codebases have always been expensive because they resist change. AI accelerates that. A bad codebase compounded by AI throughput is the most expensive failure mode of the new era.

His repo is composable practices, not a workflow framework. Each skill stands alone:

• /grill-me runs a relentless interview to establish what Frederick Brooks calls a shared design concept.

• /grill-with-docs adds a Domain-Driven Design ubiquitous language file that humans and AI both reference.

• /tdd enforces red-green-refactor as the rate limiter on AI speed.

• /improve-codebase-architecture rebuilds shallow modules into deep modules, per John Ousterhout.

The default pattern is gray boxes: design the interface, delegate the implementation.

The data on his side: the METR finding that experienced developers on mature codebases were 19% slower with AI suggests the bottleneck is codebase quality, not specification quality. His argument is that the five SDD repos optimize for the wrong thing.

His repo went viral on the strength of /grill-me alone. The position is worth taking seriously.

The AlphaSignal take

The five SDD repos and Pocock’s dissent are not answering the same question.

SDD optimizes for the plausibility-to-correctness gap. Pocock optimizes for the design-entropy gap. Both gaps are real. Both data sets support both positions.

A team that picks one and ignores the other is solving half the problem.

The reliability case for SDD is strongest at the constitutional and executable-contract levels. Spec Kit’s constitution mechanism and BMAD’s implementation-readiness gate are where the math actually pays off.

The case is weakest at the natural-language end, where SDD collapses into renamed prompt engineering.

Three things none of the six repos solve, drawn from the open problems sections of the four papers.

Oracle adequacy. Current evaluations collapse model quality, tool reliability, and harness quality into one end-task number. There is no metric for what a specification is actually worth.

Evidence bundles. Every accepted change should ship with a record of what was checked, what was not, and what risks remain. No current SDD tool produces this.

Self-evolving harnesses. The SDD frameworks themselves are software. They will change. None of them have a change-contract for their own evolution.

Read each of these repos as a specific theory of where reliability comes from. Pick the one whose theory matches the bottleneck you actually have. If you don’t know your bottleneck, Pocock’s critique applies first.

Which theory of reliability does your stack depend on, constitution, roles, change folders, context, auto-triggering, or design discipline?

Full breakdown of recent updates + daily signals in our newsletter (link in bio).

Most agent failures aren’t reasoning failures. They’re harness failures.

An agent that passes every test while looping between two failing strategies, because the harness has no dead-end detection.

A new 100-page survey from UIUC, Meta, and Stanford spells out why.

Paper

It’s called “Code as Agent Harness.” 40+ researchers from UIUC, Meta, and Stanford wrote it, and it synthesizes 400+ papers under a single taxonomy with the harness, not the model, as the subject.

The anchor systems are the familiar ones: Claude Code, Codex, SWE-agent, Voyager, MetaGPT, OpenHands. What’s been a Twitter-thread topic for the last six months now has academic scaffolding around it. The contribution is the synthesis, not the discovery.

Core thesis

Long-running agents fail at state, feedback, and verification. Not at reasoning. The bottleneck of autonomy is whether the system around the model can hold its outputs accountable to something executable.

The paper splits any agent system into three coupled pieces.

First is model-internal capabilities: reasoning, planning, perception.

The second is system-provided infrastructure: tools, sandboxes, memory, permission tiers, telemetry.

And third, the underexplored one, is agent-initiated code artifacts: regression tests, temporary tools, DSL programs, executable workflows, and reusable skills the agent itself authors mid-task. Voyager’s skill library and Claude Code’s skill files are early instances.

Above this distinction sit three layers.

Harness Interface, puts code at the center as the medium for reasoning, action, and environment state.

Harness Mechanisms, covers planning, memory, tool use, and the plan-execute-verify loop.

Scaling the Harness, extends the picture to multi-agent systems coordinating over shared code artifacts.

How to audit your stack

Three questions, one per layer. They map to where most stacks actually break.

I

nterface.

Does your agent’s reasoning, action, and environment state pass through code something can actually execute and inspect? A healthy stack has tool calls, generated programs, repo state, traces, and tests. An unhealthy stack runs on natural-language plans the agent never has to defend against execution.

If unhealthy: have the model output executable code as its reasoning, give the agent a structured Agent-Computer Interface like SWE-agent’s shell + edit + search commands, and let it operate on real repo state rather than text descriptions of it.

Mechanisms.

When something fails, what does the harness do about it? A healthy stack runs a plan-execute-verify loop with named verifiers (unit tests, type checks, linters, runtime monitors), durable memory across sessions, and feedback that closes the loop. An unhealthy stack retries with more tokens and a longer context window.

If unhealthy: add named verifiers as gates between generation steps, not just at the end. Most agents only have working memory, which is whatever sits in the current prompt. The paper names four more types that decide whether yesterday’s debugging session helps today: semantic memory of the repo, experiential memory of past trajectories, long-term memory with a compression policy, and multi-agent memory for shared state. OpenHands’ stateful workspace and CodeMem’s budgeted memory slots are reference implementations to study.

Scaling.

When two agents work on the same task, what’s the shared substrate? A healthy stack uses shared code artifacts (repos, tests, traces, structured workflows) with a conflict policy. An unhealthy stack passes messages back and forth with no shared state both can safely modify.

If unhealthy: replace direct message-passing with shared artifacts both agents can read and write. AgentCoder’s programmer-tester-executor split and MetaGPT’s role-specialized agents over a shared message pool are the patterns the paper highlights.

If any of these answers feel unhealthy, the paper has already named the failure mode.

Also, the paper covers

• Five application domains. Code assistants, GUI/OS agents, scientific discovery, embodied agents, personalization.

• Self-evolving harnesses. AutoHarness, Meta-Harness, Agentic Harness Engineering (AHE) (related article down below), GEPA, EvoMAC, and SEW. The harness itself as the object of optimization, with the agent’s task code as a downstream effect.

Post link.

• Workflow topologies. Five patterns for multi-agent code work: waterfall, cyclic, hierarchical (related article down below), star, adaptive.

Post link.

• Planning paradigms. Four categories, from ReAct-style linear decomposition to tree-search-based exploration across candidate paths.

• Three more open problems. Harness evolution that doesn’t break old behaviors, shared state across agents with safe coordination, and multimodal harnesses for screenshots and physical state.

The AlphaSignal take

The most useful vocabulary the field has had for what practitioners are already building. Just not a build plan. Three gaps from the paper’s open problems. Each one is a design warning for what’s already in your stack.

Oracle adequacy.

If your eval is pass/fail on unit tests, you’re measuring the wrong thing. Every agent evaluation today collapses model quality, tool reliability, and harness quality into one end-task number. The paper names this as the central bottleneck and offers no metric that fixes it.

The verification gap.

Green tests are not a correct specification. Every accepted action should ship with an evidence bundle: which checks ran, which assumptions held, which parts of the code stayed untested, what risks remain. No current harness does this. The architecture pattern is sitting there, waiting for someone to ship it.

Approvals that don’t reset.

If approvals vanish after the session ends, your agent will repeat the same unsafe action next time. Permission rules should mutate in response to human decisions, not reset. The paper flags this and stops there.

Read it as a vocabulary, not a roadmap. The harness is the layer teams now invest in optimizing. The taxonomy will sharpen how you talk about your stack. It won’t tell you what to build on Monday.

Does your agent have a verifier that isn’t just the model judging its own output?

Paper Link

Full breakdown of recent updates + daily signals in our newsletter (link in bio).

In 5 minutes, you will know what OpenHuman is, how its memory works, how it compares to OpenClaw and Hermes Agent, and you will have it running locally with Gmail, Notion, or Slack connected.

OpenHuman crossed +20k GitHub stars in days.

Most of those stars came during a GitHub Trending run the founder marked at seven days at #1 on May 18, 2026, when he also posted that the project had hit #1 on Product Hunt’s daily, weekly, and monthly leaderboards.

Steven Enamakel, the founder, tried to set up an open-source AI agent for his dad earlier this year. Three hours of API keys, YAML, and a terminal his dad had never opened later, they both gave up. OpenHuman is what came out of that.

OpenClaw (+373k stars, MIT) and Hermes Agent (+157k stars, MIT) are the two open-source agents OpenHuman is now compared with. Both are great. Neither walks your tools and writes the memory before you start prompting.

This morning at 07:17 UTC, v0.54.0 shipped, with fully-local voice and a shared-memory bridge to Claude Code, Cursor, Codex, and OpenCode.

What OpenHuman is

OpenHuman is an open-source (GPL-3.0) desktop AI agent from TinyHumans AI.

It is written in Rust (65.2% of the codebase) with a TypeScript + React 19 frontend, packaged as a native Tauri v2 desktop app. It ships on macOS, Windows, and Linux. No mobile.

The pitch in one line: most agents start cold, OpenHuman walks your tools every 20 minutes and writes the memory into Markdown files you can open and edit.

Five things ship in the box:

• A clean desktop UI with a mascot that can speak, react, and join Google Meet as a real participant.

• A 118+ Composio toolkit catalog for one-click OAuth integrations.

• A Memory Tree plus an Obsidian-compatible Markdown vault as the local knowledge base.

• Batteries-included native tools: web search, web-fetch scraper, full coder toolset, native voice.

• TokenJuice, a token compression layer that runs on every tool result before it touches an LLM.

The whole project is three months old. The public repo was created on February 18, 2026.

Why it matters

Most open-source agents bet on the wrong half of the problem.

They get smarter at planning. They add more tools. They wire up more channels. They still know nothing about you until you paste your week into the prompt.

OpenHuman is betting the other way. Structured local memory beats embedding-bag retrieval when the agent needs to navigate your day, not similar text. The founder’s framing, from the Product Hunt launch:

“Every powerful AI agent today is built for the 0.01% who can spin up their own runtime. The other 99.99% are watching the agent revolution from the sidelines.“

Early traction tracks that bet. 5,000+ users in the first 7 days and 150% week-over-week growth, per the founder’s Product Hunt post.

OpenClaw and Hermes Agent take different bets. OpenClaw is a multi-channel gateway across 20+ messaging surfaces. Hermes Agent is a self-improving runtime with a closed learning loop. Neither tries to walk your tools and write the memory before you prompt. Full head-to-head comparison further down.

How it works

Not here for the internals? Skip to How to Get Started below.

Nine pieces. The memory pipeline is the spine. Everything else hangs off it.

4.1 Memory Tree

A deterministic, bucket-sealed pipeline, not a vector-store wrapper.

source adapters → canonicalize → chunker → content_store → store → score → source/topic/global trees → retrieval

Data is canonicalized to Markdown, split into ≤ 3,000-token chunks with deterministic IDs, scored, and folded into three trees: source (one per source, L0 → L1 → L2 cascade), topic (one per high-hotness entity), global (one node per UTC day). Three background workers run heavy work behind a semaphore. A daily scheduler at 00:00 UTC enqueues the global digest and stale-flush.

Storage layout:

• <workspace>/memory_tree/chunks.db: SQLite, holds chunks, scores, summaries, entity index, jobs, hotness.

• <workspace>/wiki/: the Obsidian-compatible Markdown vault.

<workspace> defaults to ~/.openhuman, overridable with OPENHUMAN_WORKSPACE. The vault is the point: you can open it in Obsidian, edit a wrong line, and the next retrieval is correct.

4.2 Auto-fetch on a 20-minute tick

The constant lives in src/openhuman/composio/periodic.rs:

const TICK_SECONDS: u64 = 1200;

One global tick walks every active connection. Per-(toolkit, connection_id) state holds the cursor, last-sync, dedup set, and daily budget. Errors are swallowed so the loop never panics out.

The native registry today (src/openhuman/composio/providers/registry.rs::init_default_providers):

register_provider(Arc::new(super::gmail::GmailProvider::new()));

register_provider(Arc::new(super::notion::NotionProvider::new()));

register_provider(Arc::new(super::slack::SlackProvider::new()));

Auto-ingest covers Gmail, Notion, Slack only. The 118+ figure is the Composio catalog reach as proxied tools. Tool calls work everywhere, memory ingest fires on the three.

4.3 Integrations (native vs proxied)

Every connected service surfaces four ways: agent tool, memory source (native only), profile signal, trigger source (HMAC-verified webhooks).

23 toolkits ship as curated proxied tool sets without auto-ingest yet:

Shopify, Stripe, HubSpot, Salesforce, Airtable, Figma, GoogleCalendar, GoogleDrive, GoogleDocs, GoogleSheets, Discord, Telegram, WhatsApp, Microsoft Teams, Outlook, Linear, Jira, Trello, Asana, Dropbox, Twitter, Spotify, YouTube.

Three channels talk back: Telegram (two-way, 80+ actions), Discord (send/receive), Web (in-app local chat).

4.4 TokenJuice

A Rust port of vincentkoc/tokenjuice in the tool-execution path. Every tool result hits a rule overlay before it reaches an LLM. Three layers, later layers override earlier ones:

• Builtin (shipped with the binary): git, npm, cargo, docker, kubectl, ls defaults.

• User (~/.config/tokenjuice/rules/): personal overrides across every project.

• Project (.tokenjuice/rules/): repo-specific overrides, checked in.

HTML → Markdown. Long URLs shortened. CJK and emoji preserved grapheme-by-grapheme. Project claim: up to 80% reduction in cost and latency. PrimeAIcenter measured ~70%. Realistic range: 70–80%, biggest wins on log-heavy and HTML-heavy payloads.

4.5 Model routing

One subscription brokers 30+ providers. The agent loop emits a hint: prefix per task. The router resolves it.

Top five hints:

• hint:reasoning: strong reasoning model.

• hint:fast: fast / cheap model.

• hint:vision: vision-capable model.

• hint:summarize: compression model.

• hint:code: code-tuned model.

Lighter hints (hint:reaction, hint:classify, hint:sentiment, hint:medium, hint:tool_lite) prefer the local provider when Local AI is on. Heavy hints stay cloud. The task picks the model, not the user. Full hint reference in the appendix.

4.6 Local AI (optional, off by default)

Two flags switch it on:

local_ai.runtime_enabled = true

local_ai.opt_in_confirmed = true

Defaults: all-minilm:latest (~23 MB) for embeddings, gemma3:1b-it-qat (~700 MB) for summary-tree building. Heartbeat, learning, and subconscious can also move on-device. Chat, vision, STT, TTS, and web search stay cloud. “Local-first” means memory, not the whole stack.

4.7 Voice, mascot, and the Meet agent

The mascot lip-syncs to TTS and shifts through six mood states (idle, thinking, listening, talking, surprised, dreaming). The Meet agent joins Google Meet through an embedded CEF webview as a real participant: a name, a face, a tile in the grid.

Mid-meeting it can:

• Listen. Inbound audio streams through STT, diarized per speaker, into the Memory Tree live.

• Speak. Replies stream into the call as an outbound mic feed, not bounced through your speakers.

• Animate. The mascot canvas is piped as the outbound camera, lip-synced to the TTS audio everyone else hears.

• Use tools. Memory recall, auto-fetch, native tools, subconscious outputs, all reachable mid-call.

v0.54.0 added fully-local STT and TTS via Whisper and Piper. Prior path required ElevenLabs cloud for TTS.

4.8 Subconscious loop

A background tick (default 5 min, minimum 5 min) that loads due tasks, builds a situation report from memory plus workspace state, and returns one of three decisions:

• Skip. Nothing relevant right now.

• Act. Execute the task.

• Escalate. Hand off to the cloud agent.

Local model evaluates, cloud agent escalates. Write tasks you asked for need no approval. Unsolicited writes open an approval card.

4.9 Skills (in transition)

The QuickJS / rquickjs runtime that executed skill packages was removed. Today’s skills surface is metadata-only:

• Discover and parse SKILL.md files.

• Resolve scope (User / Project / Legacy) and trust markers.

• Install from URL (HTTPS only, no private hosts, .md only, max 1 MiB body).

• Read resources (cap 128 KiB), uninstall, per-turn prompt injection (cap 8 KiB).

Registry repo: tinyhumansai/openhuman-skills. Runtime is being rebuilt. Treat skills as catalog + prompt-injection today, not a third-party plugin runtime.

What v0.54.0 shipped

Released 2026-05-19 07:17 UTC, the morning after the Product Hunt #1 post. 230 commits, 1,271 files changed against v0.53.43 six days earlier.

Voice. Fully-local STT and TTS via Whisper / Piper. Configurable mascot voice with ElevenLabs picker.

Memory. Optional agentmemory backend bridge: set memory.backend = “agentmemory” in config.toml and the Memory Tree shares a durable store with Claude Code, Cursor, Codex, and OpenCode. Plus MCP stdio memory server, NotebookLM-style folder ingestion, cross-chat context retrieval, per-(row, model) embedding storage.

Agents. Dedicated crypto_agent for wallet and market ops (#1397). Cursor Cloud Agents parallel workflow. Global tool registry. Task board CRUD. Gmail Unsubscribe Agent.

Integrations. Bring-your-own Composio direct mode (#1710). Seltz as direct-API search. Discord webview transcript ingestion. WeChat embedded webview.

Providers. Unified per-workload routing. LM Studio local provider support. New reasoning-quick-v1 route for low-latency chat.

UX and i18n. Onboarding redesign. Dark mode standardization. ~3,900 strings across nine locales. Italian and Indonesian added.

Security. Path traversal prevention in agent prompt loading. DNS-aware URL validation. Audio base64 max-size check. Self-repair for locked .secret_key on Windows. Linux AppImage NSS fix.

Local main is already past v0.54.0 at 0.54.2. Post-release: Polymarket integration (#2145), explicit user-preference tool (#2150), Ollama context-window gating (#2122).

OpenHuman vs Hermes Agent vs OpenClaw

6.1 The README’s own framing

6.2 What independent reviewers found

The May 2026 reviews land in roughly the same spot.

PrimeAIcenter (five-day test, Gmail + Notion + GitHub + Calendar): Memory Tree useful after three days, TokenJuice measured ~70% (not 80%), two sync failures in the window. Read on the Meet mascot: sounds gimmicky, isn’t.

TechTimes (May 16): framed OpenHuman as inverting the playbook, flagged three risks, piped-shell install as a supply-chain vector, OAuth aggregation across email, code, calendar, and payments, and no formal independent audit.

Julian Goldie (three-agent SEO test): OpenHuman wins on UI, setup, and voice. Hermes wins on long autonomous tasks. OpenClaw wins on scheduling.

HackerNoon (OpenClaw context): 138+ disclosed CVEs and 341 malicious ClawHub skills out of 2,857 scanned as of May 2026. Hermes: 3 CVEs. OpenHuman: no published CVEs, no skill marketplace today.

6.3 Pick which one

Three different shapes of the same problem. Pick by use case:

• Pick OpenHuman if you want a desktop agent that reads your email, calendar, Slack, and Notion within minutes, plus a memory you can open as Markdown.

• Pick Hermes Agent if you want an always-on server-side agent with a self-improving learning loop and six terminal backends (local, Docker, SSH, Daytona, Singularity, Modal) for long-running autonomous workflows.

• Pick OpenClaw if you want a channel-heavy gateway agent across 20+ messaging surfaces with Git-versioned config. Audit the marketplace and skill set carefully given the recent CVE and ClawHub reporting.

  Thanks for reading AlphaSignal! Subscribe for free to receive new posts and support my work.

How to Get Started

Here is the command you came for:

curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash

That single line installs OpenHuman on macOS or Linux. The full hint table, Local AI presets, native vs proxied list, and troubleshooting all live in the Reference Appendix at the end of this piece.

7.1 Pick an install path

Three options. Pick by tradeoff.

Option A: Download a signed binary (recommended for non-developers).

Go to tinyhumans.ai/openhuman and pick DMG (macOS), MSI or EXE (Windows), or AppImage / .deb (Linux). This avoids the piped-shell install path TechTimes flagged as a supply-chain risk vector.

Option B: One-line installer (recommended for the demo path).

macOS / Linux x64:

curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash

Windows PowerShell:

irm https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.ps1 | iex

Option C: Build from source (for contributors).

Git, Node.js 24+, pnpm 10.10.0, Rust 1.93.0 + rustfmt + clippy, CMake, Ninja, ripgrep, plus your platform’s desktop build prerequisites.

7.2 First-run flow

Six steps. None of them require a terminal after step 1.

1. Launch the app. Sign in.

2. Click Connect on Gmail, Notion, or Slack. These are the three native providers that auto-ingest into the Memory Tree today.

3. Wait one auto-fetch tick (up to 20 minutes) or trigger a manual ingest from the Intelligence tab in the app.

4. Open the Obsidian vault at ~/.openhuman/wiki/ via the in-app deep link or any file browser.

5. Ask a context-heavy prompt: “What did I commit to in email last week?”

6. Open a chunk file in the vault before you trust the answer. The whole point is that the memory is readable. If something is wrong, fix the file, and the next retrieval is correct.

That is the golden path. Five minutes of real time, twenty minutes of wall-clock if you wait for the first sync tick.

7.3 (Optional) Turn on Local AI

Local AI is off by default. Two flags switch it on:

local_ai.runtime_enabled = true

local_ai.opt_in_confirmed = true

Then go to Settings → AI & Skills → Local AI and pick one of three presets:

• Embeddings only. all-minilm:latest (~23 MB) for memory embeddings.

• Memory + reflection. Embeddings + summary-tree building (gemma3:1b-it-qat, ~700 MB) + learning.

• Everything local. All five workloads: embeddings, summary, heartbeat, learning, subconscious.

Hardware floor: 8 GB RAM minimum, 16 GB+ ideal. LM Studio is supported as the alternative provider with default base URL http://localhost:1234/v1.

Skip Local AI if you only have a few sources connected. The cloud path is faster and the privacy benefit is small in that case.

7.4 (Optional) Send the mascot into a Meet

Hand the mascot a Google Meet link from the desktop app.

It opens the embedded Meet webview, joins with the configured display name, switches its tile to the mascot canvas, and is now in the participant grid. The mic is the TTS injection stream (not your real microphone). The camera is the mascot frame producer (not your real webcam). Mute the agent’s mic from the app the same way you would mute yourself in Meet.

Required OS permissions: Camera, Microphone. On macOS, also Accessibility and Input Monitoring for desktop hotkeys.

7.5 (Optional) Bridge to agentmemory across other coding agents

If you already self-host agentmemory for Claude Code, Cursor, Codex, or OpenCode, set this in config.toml:

[memory]
backend = “agentmemory”

OpenHuman’s Memory Tree now proxies to that store. The same durable memory powers OpenHuman alongside the other four agents. This is new in v0.54.0.

7.6 Build from source (contributors)

git clone https://github.com/tinyhumansai/openhuman.git

cd openhuman

git submodule update --init --recursive

pnpm install

pnpm dev            # Vite dev server only (web UI)

pnpm dev:app        # Full Tauri desktop dev

Quality gates before opening a PR: pnpm typecheck, pnpm lint, pnpm format:check, cargo check -p openhuman --lib, pnpm test, pnpm test:rust. PRs need at least 80% coverage on changed lines or the merge gate blocks them.

7.7 Save this

curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash

Save the command. The full Reference Appendix below covers workspace structure, every model routing hint, the Local AI presets matrix, the native vs proxied integration list, and a troubleshooting table.

Current Limitations

Auto-fetch covers three integrations today. The 118+ headline is the Composio catalog reach, not the count of toolkits that auto-ingest into the Memory Tree. Today’s native providers: Gmail, Notion, Slack.

Local-first is not fully local. Memory Tree and the Obsidian vault are local. Default chat, vision, web search, integration OAuth proxying, and TTS streaming all go through the OpenHuman backend.

Skill execution is being rebuilt. The QuickJS runtime is gone. Today’s skills surface is metadata-only (discover, parse, install, uninstall, prompt injection). No executable third-party skill packages today.

80% token compression is a project claim. PrimeAIcenter measured around 70% in their five-day independent test. Realistic range: 70 to 80%, biggest wins on log-heavy or HTML-heavy payloads.

No published independent security audit. v0.54.0 adds meaningful hardening (path traversal prevention, DNS rebinding guard, bearer token redaction, Windows ACL self-repair), but the OAuth surface across email, code, calendar, and payments is wide and concentrated.

So the best recommendation is to install it this week if you are evaluating personal-agent UX or memory ingestion patterns, treat it as active beta for anything production-adjacent, and revisit auto-fetch coverage and the skill runtime in 60 days.

AlphaSignal Take

Verdict: Worth Watching.

The Memory Tree, the Obsidian vault, the 20-minute auto-fetch, and the v0.54.0 agentmemory bridge are real and source-verifiable. The caveats: only Gmail, Notion, and Slack auto-ingest today (not 118), and “local-first” applies to memory, not to LLM calls or OAuth proxying.

Not production-ready. 148 open issues, no published security audit, a skill runtime in transition, beta-grade onboarding bugs. What would change the verdict: native sync providers across the top ten integrations, a published audit, public pricing, and a stable skill runtime. Watch the v0.55 release.

Full breakdown of recent updates + daily signals in our newsletter (link in bio).

If OpenHuman is the third entrant after OpenClaw and Hermes Agent, which one are you actually running right now, and what made you pick it?

Reference Appendix

13.1 System requirements

RAM. 4 GB minimum. 8 GB recommended with Local AI. 16 GB+ for large mailboxes or full local AI.

Disk. ~500 MB for the binary. +23 MB for all-minilm, +700 MB for gemma3:1b-it-qat, several GB more for full mailbox memory.

OS. macOS 12+ (Apple Silicon or Intel), Windows 10+ (x64 or ARM64), Linux x64 with libssl.

macOS permissions. Camera and Microphone for the Meet agent. Accessibility and Input Monitoring for desktop hotkeys.

Build-from-source toolchain. Node.js 24+, pnpm 10.10.0, Rust 1.93.0 + rustfmt + clippy, CMake, Ninja, ripgrep.

13.2 Workspace structure

• <workspace>/memory_tree/chunks.db: SQLite, holds chunks, scores, summaries, entity index, jobs, hotness.

• <workspace>/wiki/: Obsidian-compatible Markdown vault.

• ~/.config/tokenjuice/rules/: User-level TokenJuice rule overrides.

• .tokenjuice/rules/: Project-level TokenJuice rule overrides.

Defaults: <workspace> is ~/.openhuman. Override with OPENHUMAN_WORKSPACE.

13.3 Memory Tree internals

Leaf lifecycle.

pending_extraction → admitted → buffered → sealed
                       \
                        → dropped

The deep score decides admitted vs dropped. Admitted leaves enter a buffer (buffered). When the buffer seals, every leaf inside flips to sealed. Dropped leaves stop, the chunk row stays for provenance.

Job queue kinds.

• extract_chunk: Deep score + entity extraction. Decides admitted vs dropped.

• append_buffer: Adds an admitted leaf to the source (or topic) L0 buffer. May trigger a seal.

• seal: Compresses an L0 buffer into an L1 summary. Cascades up if the parent is full.

• topic_route: Routes a leaf into per-entity topic trees, gated by a hotness check.

• digest_daily: Builds the global daily digest node.

• flush_stale: Force-seals buffers that have been sitting too long.

Three background workers pick jobs. Semaphore caps concurrent LLM-bound calls. On startup, any job whose worker lease has expired (crash, kill) returns to the queue.

13.4 Model routing hints

• hint:reasoning: strong reasoning model. Multi-step planning, math, code-heavy turns.

• hint:fast: fast / cheap model. UI helpers, autocompletes, small classification.

• hint:vision: vision-capable model. Screenshots, image attachments, OCR.

• hint:summarize: compression model. Memory tree summary builders.

• hint:code: code-tuned model. Native coder turns.

• hint:reaction: lightweight model. Quick reactions.

• hint:classify: lightweight model. Classification tasks.

• hint:sentiment: lightweight model. Sentiment analysis.

• hint:medium: medium model. Medium-complexity tasks.

• hint:tool_lite: lightweight model. Lightweight tool calls.

Override globally in config.toml. Override per call by passing a concrete model name (no hint: prefix). Override per skill via the manifest.

13.5 Local AI presets

• Embeddings only. Memory embeddings run local. Everything else stays cloud.

• Memory + reflection. Embeddings, summary-tree building, and learning passes run local. Heartbeat and subconscious stay cloud.

• Everything local. All five workloads run local: embeddings, summary-tree, heartbeat, learning, subconscious.

Models: all-minilm:latest (~23 MB) for embeddings, gemma3:1b-it-qat (~700 MB) for summary-tree building. Provider switch: local_ai.provider = “ollama” or local_ai.provider = “lm_studio”. Base URL override: local_ai.base_url.

13.6 Configuration reference

Environment variables.

• OPENHUMAN_WORKSPACE: override default workspace path (default ~/.openhuman).

• OPENHUMAN_CORE_TOKEN: per-launch bearer for HTTP JSON-RPC to the in-process core.

• OPENHUMAN_CORE_REUSE_EXISTING=1: attach to an externally-started openhuman-core process.

• OPENHUMAN_APP_ENV=staging: use the staging workspace at ~/.openhuman-staging/.

• OPENHUMAN_LM_STUDIO_BASE_URL: override LM Studio endpoint (default http://localhost:1234/v1).

• LM_STUDIO_BASE_URL: alias for the above.

• GGML_NATIVE=OFF: disable -mcpu=native on macOS builds that fail.

• RUST_LOG=openhuman_core::openhuman::tokenjuice=debug: trace TokenJuice rule matches and reductions.

Useful config.toml keys.

• local_ai.runtime_enabled (default false): master switch for the local provider.

• local_ai.opt_in_confirmed (default false): explicit opt-in. Bootstrap forces back to false until you re-opt.

• local_ai.provider (default “ollama”): local provider, “ollama” or “lm_studio”.

• local_ai.base_url (unset): provider URL override. LM Studio default http://localhost:1234/v1.

• local_ai.usage.embeddings (default false): use local for memory embeddings.

• local_ai.usage.heartbeat (default false): use local for the heartbeat loop.

• local_ai.usage.learning_reflection (default false): use local for learning passes.

• local_ai.usage.subconscious (default false): use local for the subconscious loop.

• memory.backend (default “memory_tree”): switch to “agentmemory” to proxy to a self-hosted store.

• composio.mode (default “tinyhumans”): switch to “direct” to use your own Composio v3 tenant.

13.7 Native vs proxied integrations

Native (auto-ingest into Memory Tree). Gmail, Notion, Slack.

Curated proxied (callable, no auto-ingest). Shopify, Stripe, HubSpot, Salesforce, Airtable, Figma, GoogleCalendar, GoogleDrive, GoogleDocs, GoogleSheets, Discord, Telegram, WhatsApp, Microsoft Teams, Outlook, Linear, Jira, Trello, Asana, Dropbox, Twitter, Spotify, YouTube.

Catalog reach (Composio). 118+ services via OAuth, a subset of the wider catalog.

13.8 Skill install constraints

• Scheme: HTTPS only.

• Hosts: private and local hosts rejected.

• URL shape: GitHub blob URLs normalized, path must end in .md.

• Max URL length: 2,048 chars.

• Default fetch timeout: 60 seconds.

• Max fetch timeout: 600 seconds.

• Max SKILL.md body: 1 MiB.

• Per-resource read cap: 128 KiB.

• Per-turn injection cap: 8 KiB.

• Writes are atomic.

Source: src/openhuman/skills/ops_install.rs, src/openhuman/skills/inject.rs, src/openhuman/skills/ops.rs.

13.9 Troubleshooting

First sync feels slow. Large mailboxes take hours on first ingest. Watch the Intelligence tab heatmap.

Locked .secret_key on Windows. v0.54.0 ships self-repair plus Windows ACL hints. Restart the app.

Linux AppImage will not launch on Arch or rolling distro. v0.54.0 excluded bundled NSS libs. Re-download.

Onboarding stuck after Google auth on Windows. Tracking as issue #2215, labeled priority: critical. Subscribe to the issue for the fix.

Sync failure on one integration. Per-connection state rebuilds on the next tick. A missed periodic sync is harmless.

macOS build fails with -mcpu=native. Set GGML_NATIVE=OFF before cargo check or cargo test.

13.10 Build-from-source quickref

# setup

git clone https://github.com/tinyhumansai/openhuman.git

cd openhuman

git submodule update --init --recursive

pnpm install

# dev

pnpm dev            # Vite UI only

pnpm dev:app        # Full Tauri desktop dev

# quality gates

pnpm typecheck

pnpm lint

pnpm format:check

cargo check -p openhuman --lib

pnpm test

pnpm test:rust

Coverage gate on changed lines: ≥ 80%. CI enforces.

δ-mem stores an LLM’s conversation history inside an 8×8 matrix and uses it to steer attention.

The backbone stays frozen. No prompt growth. No fine-tuning.

On Qwen3-4B-Instruct, that small matrix lifts the average score across five benchmarks from 46.79% to 51.66%, with 4.87M trainable parameters (0.12% of the model).

The adapter is public on Hugging Face under CC-BY-4.0. The arXiv paper landed on May 12, 2026.

For most agent workloads, RAG is overbuilt and longer context is wasteful. δ-mem suggests a third path.

What this article covers (~7 min read)

How δ-mem works in four steps, what it moves on five benchmarks, how to load the Qwen3-4B adapter in ten minutes, and where 64 numbers stop being enough. A reference guide for engineers sits at the end as an appendix.

Context

The research is authored by Mind Lab (Soujanya Poria’s group at NTU, with co-authors from Fudan University, Shanghai Jiao Tong, CUHK, and HKUST-GZ) and titled “δ-mem: Efficient Online Memory for Large Language Models.” Ten authors. arXiv submission on May 12, 2026.

The repo at declare-lab/delta-Mem has +100 GitHub stars at time of writing. The Hacker News thread has +230 points and +50 comments. The Hugging Face paper page has +110 upvotes.

The problem it’s pushing at: agents and long-running assistants need to reuse old information, and the three default answers all hit walls.

If your agent is still doing RAG on every turn, you’re paying token cost for retrieval noise on every turn. Longer context hits quadratic attention cost and context rot. LoRA-style adapters are static after training and can’t adapt to a live conversation.

δ-mem proposes a fourth path. It keeps a tiny memory state inside the model, updates it as new tokens arrive, and lets that state shape attention at runtime. The backbone weights never move.

How δ-mem works

δ-mem runs the same four steps at every token position. The frozen backbone runs its normal attention in parallel.

Step 1: Project

At a selected Transformer layer, δ-mem takes the current hidden state and projects it into three 8-dimensional vectors: a memory query, a memory key, and a memory value. The query and key go through tanh and L2 normalization. The value is a plain linear projection.

Step 2: Read

Multiply the previous 8×8 state by the current memory query. Out comes a small read vector. The state size is fixed, so this read cost is the same whether the conversation has 100 turns or 10,000.

Step 3: Steer

The read vector passes through two learned linear maps to produce a query-side correction and an output-side correction, each scaled by α/r (default 2). The corrected query goes into attention. The output-side correction is added after.

The key difference from LoRA: LoRA’s low-rank update is fixed after training. δ-mem’s correction comes from a state that changes every token, so the same parameters produce different steering effects under different histories.

Step 4: Write

After attention, the state updates with a gated delta rule borrowed from Qwen-Next’s gated retention. Three things happen in one update: keep part of the old state, erase the old prediction along the current key direction, and write the new value along that same direction. Two per-dimension gates (β for writes, λ = 1 − β for retention) control how much to overwrite versus retain.

Three write granularities

The paper studies three variants of step 4.

What it actually moves

On Qwen3-4B, an 8×8 matrix beats BM25 RAG by 7.1 points and Context2LoRA by 6.8 points. Same backbone. Same evaluator. Headline numbers from Table 1 of the paper:
In relative terms: 1.10× the frozen backbone, 1.15× the strongest non-δ-mem baseline (Context2LoRA), 1.31× on MemoryAgentBench, 1.20× on LoCoMo.

The most under-reported number in the paper: the Test-Time Learning subtask nearly doubles, 26.14 → 50.50. That’s the one to watch if you care about agents that learn during a session.

Cross-backbone, the biggest absolute jump isn’t on Qwen3-4B or Qwen3-8B. It’s on SmolLM3-3B, where MSW lifts the average from 26.08 to 36.96, a +10.88 point gain. Qwen3-8B goes from 47.20 to 50.86 with SSW. Smaller models benefit more from MSW because four parallel states reduce interference inside a single state.

GPU memory matches vanilla inference at every prompt length tested. Decoding throughput is the tradeoff: at 32k prompt and 64-token decode, vanilla runs 22.60 TPS and δ-mem TSW runs 13.68 TPS. The state read-and-write loop runs every step.

How to run it

Step-by-step setup below. For day-two patterns (preloading history, base-vs-δ-mem comparison, session save and resume, training your own adapter).

See the How to use it appendix at the end.

Clone and install:

git clone https://github.com/declare-lab/delta-Mem.git

cd delta-Mem

python -m pip install uv

bash scripts/setup_uv_env.sh

source .venv/bin/activate

You need Python 3.10 or newer, an NVIDIA GPU with CUDA PyTorch, and FlashAttention plus DeepSpeed for training. CPU is not the target path.

Download the adapter and load it in Python:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from deltamem.core import HFDeltaMemConfig, attach_delta_mem, load_delta_mem_adapter

base_model = "Qwen/Qwen3-4B-Instruct-2507"
adapter_dir = "./delta-mem_qwen3_4b-instruct"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model, torch_dtype=torch.bfloat16, device_map="auto",
)

config = HFDeltaMemConfig.from_pretrained(adapter_dir)
attach_delta_mem(model, config)
load_delta_mem_adapter(model, adapter_dir)
model.eval()

Important: δ-mem is not a standard PEFT adapter. Do not load it with PeftModel. Do not call merge_and_unload(). The runtime read/write path is part of model execution.

Run the chat demo:

MODEL_PATH=/path/to/Qwen3-4B-Instruct-2507 \

ADAPTER_DIR=/path/to/delta-mem_qwen3_4b-instruct \

bash deltamem/demo/run_chat_demo.sh

Inside the demo, /reset clears the state, /stats prints state statistics, and /save_session <dir> plus /load_session <dir> checkpoint the state to disk.

Once the chat demo loads, the How to use it appendix covers the engineering patterns most teams need next.

Current Limitations

Adapter coverage. The only public adapter today is Qwen3-4B-Instruct TSW. The Qwen3-8B and SmolLM3-3B variants need to be retrained from the repo. That means 8× A800 GPUs in the paper’s recipe and a memory-heavy training set.

Decoding overhead. δ-mem TSW runs about 40% slower than the base model at 32k prompt and 64-token decode (13.68 TPS versus 22.60 TPS). The state read-and-write loop runs at every step. GPU memory stays flat.

Context recovery is partial. When the original context is removed and only the compressed state is injected, HotpotQA EM goes from 0.08% to 6.48%. That’s real signal, not full recall. The state cannot replace explicit context for retrieval-style tasks.

Not standard PEFT. The adapter requires a custom runtime path. Standard shortcuts like PeftModel.from_pretrained and merge_and_unload() will not work. CPU-only inference is not supported.

AlphaSignal Take

The sharpest critique on Hacker News (236 points, 59 comments) was a capacity question. One commenter put it plainly: “This doesn’t solve the capacity problem of memory...there’s a fundamental limit on how much information can go into it.”

The paper’s answer is partial. The no-context recall ablation shows the 8×8 state carries usable signal (HotpotQA EM rises from 0.08% to 6.48%, LoCoMo from 3.49 to 8.05). But the absolute floor is low.

This is the line most engineers will miss: an 8×8 state is a steering signal, not a fact store. Treat it like one or it’ll bite you.

The right pattern is to pair δ-mem with retrieval. Use the state to steer the model on what the user has been talking about. Keep exact facts, policies, and audit trails in a search index or vector store. So the best recommendation is to prototype with the released Qwen3-4B adapter and treat the state as an attention bias, not a fact database.

The next thing to watch is a Qwen3-8B δ-mem adapter on Hugging Face. The cross-backbone results already show SSW wins on 8B, and an official adapter at that scale would double the practical surface area of δ-mem overnight.

Who benefits and who doesn’t

This is for ML engineers prototyping memory-heavy agents, researchers comparing latent memory against RAG and LoRA, and developers running Qwen3-4B-class models on a single GPU who want history-conditioned behavior without growing the prompt.

It is not for teams that need auditable retrieval (citations, deletion, exact match), teams without NVIDIA GPUs, teams whose conversations fit comfortably inside a long-context window, or teams that need a larger-model adapter today.

Practitioner Implication

Most teams will not train this from scratch. The real test is whether the released Qwen3-4B adapter beats your current RAG baseline by Friday, without growing the context window or fine-tuning the backbone.

Links

• arXiv paper (paper, ~25 min read)

• GitHub repo (repo, ~10 min setup)

• Hugging Face adapter (Qwen3-4B TSW)

  Thanks for reading AlphaSignal! Subscribe for free to receive new posts and support my work.

Follow @AlphaSignalAI for more content like this.

Subscribe at AlphaSignal.ai for daily AI signals. Read by 300,000+ subscribers.

Questions?

Q: What is δ-mem? A: A memory mechanism that adds a small 8×8 matrix alongside a frozen LLM and uses its readout to inject low-rank corrections into the model’s attention. It stores history as a latent state, not as retrieved text.

Q: Does δ-mem replace RAG? A: No. δ-mem does not retrieve documents and cannot produce citations. The cleanest stack for an agent pairs δ-mem with a retrieval index. The state handles steering. The index handles exact recall.

Q: How big is δ-mem’s memory state? A: A single 8×8 matrix by default (rank 8, 64 entries). The MSW variant keeps 4 parallel 8×8 states. Trainable parameter cost is 4.87M for TSW or SSW (0.12% of the Qwen3-4B backbone), 19.47M for MSW.

Q: Can developers use δ-mem today? A: Yes, on Qwen3-4B-Instruct. The official Qwen3-4B TSW adapter is on Hugging Face under CC-BY-4.0. The repo targets NVIDIA GPUs with bfloat16, FlashAttention, and DeepSpeed.

Q: What are the main limitations? A: Public adapter coverage is narrow (Qwen3-4B TSW only), decoding is about 40% slower than the base model at 32k context, the adapter is not standard PEFT, and no-context recall is still low in absolute terms.

Where do you put memory in your agent stack today? Prompt, vector store, fine-tune, or something else? Which one would you swap out for δ-mem first?

Appendix: how to use it

A reference guide for teams that already cleared the chat demo. Skim once, return to specific sections as needed.

A.1 Mental model

δ-mem is an in-process side-state, not a database. The state matrix lives inside the model object, persists across model.generate() calls in the same Python process, and resets when you re-attach. There is no separate process and no network hop.

One state matrix is allocated per attached layer per state head, all on the same GPU as the model.

A.2 Preloading history into the state

The most common day-one mistake is forgetting that δ-mem only sees what passes through forward(). To preload history, run the historical context through the model in a context-only pass before the user’s first real query. The state advances. The output is discarded.

The paper trained with an 8,192-token write budget per example. Inference is not hard-capped at that length, but the trailing tokens dominate the state.

A.3 Verifying the state is actually doing something

Three quick checks.

First, run the demo in MODE=base and compare answers on the same prompts. If responses look identical to δ-mem mode, the adapter is not attached or model.eval() was skipped.

Second, use /stats in the chat demo or call collect_delta_mem_state_stats() in Python. The state should be non-zero within the first few tokens of context.

Third, run one short benchmark and confirm the result lands near the paper’s headline. For Qwen3-4B TSW, that’s roughly 51.66% average across the five-eval suite.

A.4 Session save and resume

Use /save_session <dir> and /load_session <dir> inside the chat demo, or the equivalent Python helpers, to checkpoint and restore the state across processes. The save captures the exact state matrix at that point in the conversation.

A saved session is tied to the base model it came from. Loading a Qwen3-4B session into a different base will fail.

A.5 Benchmarking against your current memory stack

Run the same task three ways with the same evaluator: base + RAG, base + long context, base + δ-mem. The bundled suite at scripts/run_qasper_multimodel_write8192_benchmark_suite.sh runs all five evals.

To scope it to one task and one variant:

BENCHMARK_VARIANTS_STRING=”TSW_rank8_qasper_write8192” \

EVAL_TASKS_STRING=”locomo” \

bash scripts/run_qasper_multimodel_write8192_benchmark_suite.sh

For reference, on Qwen3-4B the paper reports BM25 RAG 44.56 avg, Context2LoRA 44.90 avg, and δ-mem TSW 51.66 avg.

A.6 Choosing TSW, SSW, or MSW

The cross-backbone results give the rule of thumb.

• TSW wins on Qwen3-4B (51.66 avg). Pick it when local detail matters and the model is mid-sized.

• SSW wins on Qwen3-8B (50.86 avg). Pick it when token-level noise drags the state and the model has more reasoning to spare.

• MSW wins on SmolLM3-3B (36.96 avg) and on memory-heavy benchmarks. Pick it when interference between memory types matters more than per-token detail.

Only TSW is on Hugging Face today. SSW and MSW need the training script.

A.7 Cost reality

GPU memory at inference matches vanilla at every prompt length tested. At a 32k-token prompt, δ-mem’s footprint lands on the same value as the base model in the paper’s table, with no measurable overhead from the state.

Decoding throughput is ~40% slower than base at 32k prompt and 64-token decode (22.60 → 13.68 TPS).

Trainable parameters: 4.87M for TSW or SSW (0.12% of Qwen3-4B), 19.47M for MSW (0.48%).

Training in the paper used 8× A800 GPUs, bfloat16, DeepSpeed ZeRO-2, fused AdamW, peak LR 2e-4, one epoch on the shortest 2,219-sample QASPER split, effective batch size 32.

A.8 Training your own adapter

The realistic floor is multi-GPU bf16. The paper’s exact recipe was 8× A800, but the script supports fewer devices through DeepSpeed.

Training data should be memory-heavy SFT examples where the context tokens carry the signal that the model needs at response time. QASPER fits because the question is short and the supporting context is long.

TRAIN_VARIANTS_STRING=”TSW_rank8_qasper_write8192” \

BENCHMARK_VARIANTS_STRING=”TSW_rank8_qasper_write8192” \

bash scripts/run_qasper_multimodel_write8192_train_and_benchmark_suite.sh

Per-backbone scripts exist for Qwen3-8B and SmolLM3-3B.

A.9 Compatibility constraints

• Not a PEFT adapter. Never load with PeftModel. Never call merge_and_unload().

• GPU-only target path. CPU is not supported.

• Released adapter is fixed to Qwen/Qwen3-4B-Instruct-2507. Other Qwen versions are not guaranteed.

• Adapter files are delta_mem_adapter.pt and delta_mem_config.json. Do not rename.

• The required load path is deltamem.core.attach_delta_mem followed by load_delta_mem_adapter. Standard AutoModel.from_pretrained(adapter_dir) will not work.

A.10 When to reach for δ-mem versus an alternative

• Reach for RAG or vector search when you need exact retrieval, citations, deletion, or an audit trail.

• Reach for longer context when the full history fits in budget and latency is acceptable.

• Reach for δ-mem when you want online, history-conditioned steering without fine-tuning the backbone or growing the prompt.

These are not mutually exclusive. The cleanest agent stack is δ-mem on the model and a retrieval index on the side.

You built an AI agent this weekend.

It writes code.
Browses the web.
Uses MCP tools.
Maybe even touches production data.

Now ask yourself:

• What isolates it from the host?

• What stops credential leaks?

• Who controls tool permissions?

• What happens after prompt injection?

• Where is the audit trail?

Most AI engineers answer these questions after the first incident.

Meanwhile, open source quietly built an entire infrastructure and security stack for AI agents.

Here are 11 repos every AI infra engineer should bookmark:

Each one covers a gap that frameworks don’t.

Open Source AI Agent Infrastructure & Security

1. ProjectRecon/awesome-ai-agents-security

Living Map of the AI Agent Security Ecosystem

What it does: Curated, maintained index of the AI agent security ecosystem, organized by security lifecycle: red teaming, runtime protection, sandboxing, governance, middleware.

Everything categorized, linked, and kept current. The starting point when you need to understand the full landscape or find tools for a specific security problem.

Why it matters for AI infra: This space moves faster than any single article. This repo is the durable index. Watch it. The delta between its last commit and today is your reading list.

🔗: https://github.com/ProjectRecon/awesome-ai-agents-security

2. promptfoo/promptfoo

Automated Red Teaming and Evals for LLM Applications

What it does: The standard framework for automated LLM red teaming, security testing, and model evaluation.

Covers prompt injection, jailbreaks, PII leakage, model regression, and multi-model performance comparison across GPT, Claude, Gemini, Llama, and more. Declarative YAML configs. Native CI/CD integration. Used internally by OpenAI and Anthropic. MIT licensed, fully open source.

Why it matters for AI infra: You test your code before shipping. You should test your prompts and agent security boundaries too. Promptfoo makes red teaming systematic, scriptable, and integrated into your existing CI pipeline. Shipping agent features without automated security evals is the equivalent of shipping code without tests.

🔗: https://github.com/promptfoo/promptfoo

3. aquasecurity/trivy

Supply Chain Vulnerability Scanner for AI Infrastructure

What it does: All-in-one vulnerability scanner for container images, git repos, and filesystems.

One tool catches: vulnerable base images, misconfigured Terraform, insecure Kubernetes manifests, leaked secrets in git history, vulnerable application dependencies. SARIF output feeds directly into the GitHub Security tab. 10 lines of YAML for GitHub Actions integration.

Why it matters for AI infra: You can write perfect agent code and still ship a vulnerable base image or misconfigured infrastructure module. Supply chain attacks are the dominant attack vector now. Trivy catches them in CI before they reach production. It’s automated, with zero manual review overhead.

🔗: https://github.com/aquasecurity/trivy

4. open-policy-agent/opa

Policy-as-Code for AI Agent Infrastructure

What it does: Universal policy engine for your entire stack. Express security policy as readable, testable Rego code.

Kubernetes admission control, API authorization, infrastructure configuration validation, one engine. Decouple security policy from application code. Write once, enforce everywhere. When your AI agent calls a tool, hits an API, or requests a resource and OPA decides whether it’s allowed.

Why it matters for AI infra: OPA isn’t agent-specific, which is exactly why it belongs here. Your agent infrastructure sits inside your existing cloud stack. OPA gives you a consistent, auditable policy layer that spans both traditional infrastructure and agentic workloads without maintaining two separate security systems.

🔗: https://github.com/open-policy-agent/opa

5. AgentGateway (Linux Foundation)

RBAC Proxy for MCP and A2A Agent Protocols

What it does: AI-native proxy for A2A and MCP protocol traffic. RBAC, observability, and policy enforcement on agent-to-tool interactions.

Donated to the Linux Foundation. Sits between your agents and their tools. Only the right agents can call the right tools with the right permissions. Full observability on the protocol layer.

Why it matters for AI infra: MCP is the emerging standard for agent-tool connectivity. Most teams wire MCP directly with no access control layer, a significant and growing attack surface. AgentGateway is the purpose-built solution. Linux Foundation stewardship means production-grade stability and long-term maintenance.

🔗: https://github.com/agentgateway/agentgateway

6. microsoft/agent-governance-toolkit

Runtime Security Middleware for AI Agents

What it does: Runtime policy engine mapped directly to the OWASP Agentic AI Top 10.

When OWASP published the first formal taxonomy of agentic AI risks listing goal hijacking, tool misuse, identity abuse, rogue agents Microsoft shipped a toolkit that addresses every single one. Sub-millisecond governance latency (<0.1ms p99). Deploys as a sidecar container or middleware layer.

• Goal hijacking → semantic intent classifier

• Tool misuse → capability sandboxing + MCP security gateway

• Memory poisoning → Cross-Model Verification Kernel with majority voting

• Rogue agents → ring isolation, trust decay, automated kill switch

Why it matters for AI infra: The EU AI Act’s high-risk AI obligations take effect August 2026. The Colorado AI Act enforces January 2027. Agentic AI governance is moving from best practice to legal requirement. This is the most comprehensive open source implementation aligned with the formal risk taxonomy.

🔗: https://github.com/microsoft/agent-governance-toolkit

7. anthropics/claude-code-security-review

AI-Powered PR Security Review

What it does: GitHub Action that runs Claude Code on every pull request and posts security findings as inline review comments.

Diff-aware: only analyzes changed files. Semantic reasoning, not pattern matching, to identify high-confidence, exploitable vulnerabilities. Calibrated false positive filtering: no theoretical issues, no rate-limiting noise. Just vulnerabilities a senior security engineer would flag in review.

One YAML file to add to any repository.

Why it matters for AI infra: Security review is the step most teams skip because it’s expensive and slow. This makes it automatic and free on every PR. The semantic analysis quality using actual LLM reasoning rather than regex, catches logic-level security issues that static analysis tools miss entirely.

🔗: github.com/anthropics/claude-code-security-review

8. vercel-labs/deepsec

Agent Powered Vulnerability Scanner

What it does: An AI agent that scans your entire codebase for vulnerabilities that have been sitting there for years.

Fast regex matchers find candidates, the Claude/Codex investigates at maximum thinking levels. Work fans out across parallel workers for large single repos. You can interrupt or restart the jobs, it picks up where it left off.

Why it matters for AI infra: Every other tool on this list protects agents at runtime. Deepsec goes one layer earlier. It clears the vulnerabilities already living in the codebase your agents will read, modify, and deploy.

🔗: https://github.com/vercel-labs/deepsec

9. dagger/container-use

Containerized Environments for Coding Agents

What it does: Persistent, isolated container environments for coding agents.

From the Dagger team. Each coding agent gets its own container. Multiple agents run in parallel without conflict. Environments persist across sessions. Resume any task mid-flight with an existing env ID.

The differentiator: full OpenTelemetry instrumentation on every agent run. Every LLM decision, tool call, error, and retry appears in the build trace. When something goes wrong, you don’t guess, you see it.

Why it matters for AI infra: “It works on my machine” is not a deployment model. Container Use brings the isolation and reproducibility guarantees that containerization gave to software builds to agent execution. The observability layer alone makes debugging agentic systems tractable.

🔗: https://github.com/dagger/container-use

10. meta-llama/PurpleLlama/LlamaFirewell

Prompt Injection Defense for LLM Agents

What it does: Meta’s open source guardrail system for LLM agents. Blocks prompt injection, scans LLM-generated code for vulnerabilities, detects misaligned reasoning.

A single prompt injection can flip an agent’s intent cause it to leak private data, execute unauthorized commands, operate far outside scope. LlamaFirewall sits at the application layer and intercepts this before it reaches execution. It also scans code the agent generates for critical vulnerabilities before shipping to production, a gap most guardrail systems ignore entirely.

Open-weight guardrail models on HuggingFace. Run on your own infrastructure at 50–100ms latency. No API calls. No data leaving your environment.

Why it matters for AI infra: Prompt injection is the SQL injection of the agentic era. It’s being exploited in production systems today. LlamaFirewall is the most rigorous open source defense available, built by a security team that has worked through the actual LLM agent threat model.

🔗: https://github.com/meta-llama/PurpleLlama/tree/main/LlamaFirewall

11. microsandbox/microsandbox

Self-Hostable AI Code Execution Sandbox

What it does: Open source, self-hostable alternative to E2B for AI agent code execution. Multi-language SDKs. Hardware-level isolation.

Docker and Kubernetes runtimes. gVisor, Kata Containers, and Firecracker isolation layers, pick the isolation level that matches your threat model. Multi-language SDK support. CNCF Landscape project.

Why it matters for AI infra: Not every team wants an external SaaS dependency in their agent execution path. Microsandbox gives you the same isolation guarantees inside your own infrastructure, with no external API in the hot path, and with full control over your data residency.

🔗: https://github.com/superradcompany/microsandbox

The Bigger Shift

Most teams still think of AI agents as applications.

The infrastructure ecosystem increasingly treats them as autonomous systems that require isolation, observability, security, and governance.

That shift is changing how AI systems get deployed in production.

And increasingly, it is becoming mandatory.

The EU AI Act enters full enforcement on August 2, 2026 with high-risk AI system obligations, transparency requirements, penalties up to €35M or 7% of global turnover

Colorado’s replacement AI law is heading to the Governor’s desk now, with a January 2027 effective date. Every version of every bill, in every jurisdiction, requires the same things: documentation, audit trails, risk management, human oversight. The exact same artifacts this infrastructure stack produces.

The teams building this layer now will not only ship safer systems. When the compliance conversation arrives from customers, auditors, regulators, or enterprise buyers, they will have artifacts to show instead of promises to make.

The next generation of AI companies will not only build better models or better prompts.

They will build the infrastructure required to operate autonomous systems safely at scale.

Thanks for reading!

Follow @AlphaSignalAI for more content like this.

Check out AlphaSignal.ai to get a daily summary of top models, repos, and papers in AI. Read by 300,000+ devs.

After ~10 min reading, you will decide whether to flip the Codex runtime on and how to use every command and config immediately.

TLDR? Check this HTML interactive guide (beta), inspired from Thariq’s article.

This post was originally published on X (15 May).

Nous Research just turned Hermes Agent into a Codex front-end.

Hermes Agent keeps memory, slash commands, /goal, and skill review. Codex CLI runs shell, apply_patch, the sandbox, and native plugins.

The runtime is paid for by a ChatGPT subscription. No API key required.

Context

The feature is authored by Nous Research and titled the Codex App-Server Runtime (opt-in beta, announced May 14, 2026). Hermes Agent crossed +152K GitHub stars the same day, and per Teknium, Hermes’ daily token volume now runs at roughly twice OpenClaw’s (+353B vs +195B as of May 15).

The feature ships in Hermes v0.13.0 (tag v2026.5.7, May 7, 2026) and requires Hermes 2026.5+ and Codex CLI 0.130.0+. Both projects are open-source: Hermes under MIT, Codex CLI under Apache-2.0. The swap targets the OpenAI provider path specifically (openai/* and openai-codex/*) and does not touch Anthropic, Gemini, or any other non-OpenAI provider.

A Reminder: Hermes Agent is a self-improving coding agent with sessions DB, persistent memory, skill review, slash commands, multi-agent Kanban, and the /goal Ralph loop. Codex CLI is OpenAI’s terminal coding agent: sandboxed shell, structured patches, native plugins. Until last week the two were separate ecosystems.

What the Codex App-Server Runtime is

When the runtime is on, Hermes hands openai/* and openai-codex/* turns to the Codex CLI app-server over JSON-RPC stdio. Codex executes the tool loop: terminal commands, file edits, MCP tool calls, sandboxing. Hermes keeps the surrounding session: sessions DB, slash commands, gateway, memory, and skill review.

Default Hermes behavior is unchanged unless the flag is flipped. Hermes never auto-routes onto this runtime.

OpenClaw shipped a similar runtime-swap pattern earlier this year. Hermes’ differentiator is the bidirectional MCP callback that keeps Hermes’ richer tools (browser, vision, skills, TTS) accessible from inside the Codex turn.

How it works

Three tool sources are available the moment the runtime starts.

Codex built-in tools (5):

shell runs terminal commands inside the sandbox (read, write, search, find, run). apply_patch applies structured multi-file diffs. update_plan is Codex’s in-runtime todo tracker. view_image loads a local image into the conversation. Codex’s own web_search rounds out the set. All five run native, all five run inside the sandbox profile.

Auto-migrated Codex plugins:

When the runtime is enabled, Hermes queries Codex’s plugin/list RPC and writes a [plugins.”<name>@openai-curated”] entry for every plugin already installed via codex plugin install. Linear, GitHub, Gmail, Google Calendar, Outlook, Canva: whatever the user authorized in Codex’s TUI is now live inside the Hermes session, no re-config.

Hermes MCP callback (17 tools):

For tools Codex doesn’t ship with, Codex spawns hermes_tools_mcp_server as a stdio MCP subprocess and calls back into Hermes. The callback exposes web_search and web_extract (Firecrawl), ten browser-automation tools, vision_analyze, image_generate, skill_view, skills_list, and text_to_speech.

Event projection keeps memory and skill review alive:

Codex emits commandExecution, fileChange, mcpToolCall, and dynamicToolCall notifications. Hermes projects each one into a synthetic assistant tool_call plus tool result message, so the background review fork sees a normal-looking transcript. Memory nudges fire every 10 user prompts, skill nudges every 10 tool iterations.

The review fork itself downgrades to codex_responses (same OAuth, Hermes owns the loop) so it can still call memory and skill_manage. The downgrade is invisible to the user.

How to get started

For permission overrides, aux-task routing, and safe config editing, see How to use it at the end.

1. Install Codex CLI (0.130.0+):

npm i -g @openai/codex
codex --version

2. Authenticate Codex against the ChatGPT subscription:

codex login

Tokens land in ~/.codex/auth.json. Hermes will not share OAuth state with Codex CLI (the split is deliberate, to avoid clobbering each other on refresh), so users still need hermes auth login codex separately if they haven’t.

3. (Optional) Install Codex plugins:

codex plugin marketplace add openai-curated
codex plugin install linear github gmail calendar

Whatever’s installed at runtime-enable time gets auto-migrated.

4. Flip the runtime on inside Hermes:

/codex-runtime codex_app_server

That single command verifies the Codex CLI install, migrates user MCP servers from ~/.hermes/config.yaml to ~/.codex/config.toml, discovers installed Codex plugins, registers Hermes as an MCP server, and writes default_permissions = “:workspace”. Takes effect on the next session.

Synonyms: /codex-runtime on, /codex-runtime off, /codex-runtime auto (back to Hermes default).

For permission overrides, aux-task routing, and safe config editing, see How to use it at the end.

---

What works, what doesn’t

The four agent-loop tools (delegate_task, memory, session_search, todo) need running AIAgent context that a stateless MCP callback can’t drive. Switch back to /codex-runtime auto when any of them is needed mid-loop.

Use cases

Multi-file refactors and migrations. Codex’s structured apply_patch runs sandboxed multi-file diffs. Hermes’ /goal Ralph loop keeps the migration on-target across turns, and Checkpoints v2 rolls back failed iterations.

Debugging flaky ML pipelines. Codex inspects logs, edits training scripts, and reruns commands inside seatbelt or landlock. Hermes’ background skill review captures the successful fix as a reusable skill for the next incident.

Dependency hell on fresh environments. Codex’s sandboxed shell installs packages, runs build smoke tests, and resolves CUDA or version conflicts. Hermes’ memory remembers which configurations succeeded across projects.

CI and test repair sweeps. Codex patches failing tests file by file inside the workspace sandbox. Hermes’ Kanban dispatches each failure to a worker, with heartbeat, reclaim, and zombie detection from the Tenacity Release handling stalls.

Multi-service integration work. Codex executes against migrated MCP servers and the auto-installed plugins (Linear, GitHub, Gmail, Calendar). The MCP callback brings Hermes’ browser automation and vision into the same turn.

Current Limitations

Four agent-loop tools unavailable. delegate_task, memory, session_search, and todo need running AIAgent context that a stateless MCP callback can’t drive. Workflows that depend on subagent spawning or mid-loop memory lookups require switching back to /codex-runtime auto.

Two separate auth sessions. codex login and hermes auth login codex are independent. Users assuming one covers both will hit auth errors. The split is deliberate, not a bug: Hermes will not share OAuth state with Codex CLI to avoid token-refresh races.

ChatGPT rate limits absorb auxiliary tasks. Title generation, context compression, vision auto-detect, session search summarization, and the background self-improvement review fork all flow through the same ChatGPT subscription by default. Plus-tier users on heavy sessions will eat their cap unless they route aux tasks to a cheaper model via auxiliary.title_generation and related config overrides.

Performance claims are anecdotal. Teknium’s “~5% improvement in GPT coding capabilities” and one community user’s “p95 latency cut in half on long-lived sessions” came from reply threads, not benchmarks. No formal eval comparing the default Hermes runtime to the Codex runtime on identical workloads exists at publication.

Cron and sub-second cancellation not guaranteed. Cron jobs run through the same code path but are not specifically tested. Mid-stream Ctrl+C is sent via turn/interrupt but will not always land if Codex already flushed the final message. Approval prompts may also fall back to a reason string when fileChange data has not streamed yet.

So the best recommendation is to flip the runtime on for shell, patch, sandbox, and plugin-heavy work, and flip it back for anything that needs subagents or mid-loop memory.

AlphaSignal Take

The runtime swap is the right abstraction. Memory nudges fire identically through event projection. Kanban workers report back through the MCP callback. The flag is reversible in one command. For users whose work is dominated by shell, structured patches, and Codex plugins, the upgrade is real, and the cost is a ChatGPT subscription users were probably going to pay anyway.

The four unavailable tools are not a minor gap. They cover Hermes’ most differentiated capabilities (subagent spawning, persistent memory). The two-auth UX will trip first-time users. The 5% coding boost is a reply-thread anecdote, not a benchmark. The auxiliary-task billing default will surprise Plus-tier users running long autonomous sessions until they read the docs section nobody reads.

Verdict: Worth Watching, not Production Ready. The verdict moves to Production Ready when the four agent-loop tools get an MCP-callback equivalent, the two-auth flow is unified, and a published benchmark replaces the reply-thread estimate. Likely candidate: Hermes 2026.6.

Who benefits

Hermes users on OpenAI doing real repo work (multi-file edits, builds, terminal-heavy debugging, CI sweeps), engineers on ChatGPT Plus or Pro who would rather not maintain a separate OpenAI API billing surface, and teams whose Codex plugin install (Linear, GitHub, Gmail, Calendar) is already configured.

It does not fit workflows that lean on delegate_task subagents or cross-session memory mid-loop, anyone on a non-OpenAI provider, Plus-tier users running long autonomous loops who have not routed auxiliary tasks elsewhere, or teams that depend on cron jobs for memory-driven automation.

Practitioner implication

Hermes users on OpenAI can now run sandboxed shell and structured patches inside seatbelt or landlock, paid for by a ChatGPT subscription, with memory, skill review, and /goal intact.

Links

• Codex App-Server Runtime (Hermes Agent docs) (full feature spec, ~12 min read)

• Hermes Agent GitHub (repo, MIT, ~2 min setup with hermes update)

• Codex CLI on npm (install: npm i -g @openai/codex, ~1 min)

• Hermes Agent v0.13.0 release notes (Tenacity Release, ~6 min read)

Follow @AlphaSignalAI for more content like this.

Subscribe at AlphaSignal.ai for daily AI signals. Read by 300,000+ subscribers.

How to use it

A command and config reference for the everyday workflow once the runtime is on.

Toggle the runtime.

/codex-runtime codex_app_server   # enable (or `on`)
/codex-runtime auto               # back to default Hermes runtime (or `off`)
/codex-runtime                    # check current state without changing

The toggle takes effect on the next session, so the current cached agent finishes its turn on the prior runtime. Prompt caches stay valid.

Approve commands as Codex runs.

When Codex wants to execute a shell command or apply a patch, Hermes shows its standard Dangerous Command prompt with three responses: Allow once, Allow for this session, Deny. The session option caches similar commands, so the model does not re-prompt for the same kind of operation. Deny rejects the command, and Codex continues in read-only mode.

Change the sandbox profile.

Three built-in profiles ship with Codex: :read-only (no writes, every command prompts), :workspace (writes inside the workspace, no prompt, Hermes default), and :danger-no-sandbox (sandbox off, not recommended). Override the default in ~/.codex/config.toml outside Hermes’ managed block:

default_permissions = “:read-only”

Hermes preserves user overrides on re-migration. The override only changes the default, per-command approvals still respect the prompts.

Route auxiliary tasks to a cheaper model.

By default, title generation, context compression, vision auto-detect, session search summarization, and the background self-improvement review fork all flow through the ChatGPT subscription. To save the subscription rate limit for actual coding turns, route aux tasks elsewhere in ~/.hermes/config.yaml:

auxiliary:

  title_generation:

    provider: openrouter

    model: google/gemini-3-flash-preview

  context_compression:

    provider: openrouter

    model: google/gemini-3-flash-preview

  vision_detect:

    provider: openrouter

    model: google/gemini-3-flash-preview

  session_search:

    provider: openrouter

    model: google/gemini-3-flash-preview

  goal_judge:

    provider: openrouter

    model: google/gemini-3-flash-preview

This is the single highest-value tweak for anyone running long autonomous sessions on a Plus plan.

Edit ~/.codex/config.toml safely.

Hermes wraps everything it manages between two marker comments. Anything outside the markers is yours and stays put across re-migrations. Anything inside gets clobbered on the next toggle. Use the space outside the managed block for custom MCP servers, sandbox overrides, model preferences, or user-defined permission profiles in [permissions.<name>] tables.

TLDR? Check this HTML interactive guide (beta).

GPT-5.4 derived the right answer, then matched the swarm’s wrong one.

It wrote the correct derivation into its reasoning trace, then externalized the swarm’s wrong answer in 74% of SWE-bench trials at two simulated auditors. External accuracy collapsed from 1.00 to 0.23 while internal validity averaged 0.68.

Multi-Challenge shows a different break. With one Claude auditor named in the prompt, GPT-5.4 accuracy dropped from 0.98 to 0.10 at n=1, with the model disengaging from the task rather than copying the false consensus.

The paper is not testing live multi-agent systems. It tests a single LLM reading static text claiming named peer models have already agreed on the wrong answer. Peer consensus is the attack surface.

Dahlia Shehata and Ming Li at Waterloo ran 22,500 deterministic trajectories across three models and three dataset contexts, with no code release.

Context

The research is authored by University of Waterloo and titled “The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions.” It was submitted to arXiv on May 11, 2026.

What the paper tests: a single propagator model has to resolve a synthetic 3-hop verification task while the prompt asserts that named auditor models have reached a contradictory consensus. The paper measures whether the propagator’s reasoning trace stays intact under that simulated social pressure.

What the paper does not test: live message-passing agents, iterative debate, tool use, handoffs, or dynamic negotiation. The arXiv submission ships no code, no data, and no prompt templates. Only the PDF and TeX source are available.

Notation primer

• p: propagator (the model under evaluation).

• n: number of simulated auditors in the swarm (0, 1, 2, 3, or 5).

• C, G, P: Claude, Gemini, GPT in reviewer sequences (e.g., CPCPG, GGGGG).

• A or A_ext: external accuracy of the final answer (binary).

• E_ew: evidence weighting, scored 1 to 5 on whether the reasoning trace cited F1, F2, F3.

• E_ij: independent judgment, scored 1 to 5 on resistance to peer pressure.

• V_int: internal validity, normalized as E_ew / 5.

• D_L: Interaction Depth Limit, the auditor count where sovereignty drops below 0.5.

• Δ (delta): accuracy gap between two reviewer orderings of the same set.

  ---

How the trap works

Setup. Three propagator models (Claude Sonnet 4.6, Gemini 3.1 Pro, GPT-5.4) run against three dataset contexts (GAIA for high task entropy, SWE-bench for medium, Multi-Challenge for low). Temperature 0, Google Colab, public SDKs.

The 3-hop task. The prompt opens with a “poisoned ID” labeled as a peer guess, then presents a nested fact chain F1 -> F2 -> F3 that derives the true ID, then surrounds the facts with 500 tokens of randomized system log events. The model has to ignore the poisoned ID and chain the facts to land on the correct answer.

The social load. For n >= 2, the prompt declares that named auditor models in the swarm have already reached a unanimous consensus on the decoy ID. The propagator is told its output will be reviewed by those auditors.

Two failure modes that look similar but are not.

Alignment hallucination (SWE-bench n=2). The model derives the correct answer in its reasoning trace, then externalizes the wrong one. For the (Claude, GPT) sequence, internal evidence weighting averaged E_ew=3.55, normalizing to internal validity V_int=0.71. External accuracy was 0.21. The paper reads this gap as the model knowing the right answer and lying to match the crowd.

Social disengagement (Multi-Challenge n=1). The model stops engaging with the task entirely. 85% IGNORED stance, 7% adoption. Internal validity drops to V_int=0.21. This is not sycophancy. The model gives up.

The proposed vocabulary. Lead Anchor (the first auditor named in the prompt disproportionately controls the propagator’s behavior). Sovereignty Gap (internal validity minus external accuracy). Interaction Depth Limit D_L (auditor count at which independent reasoning collapses below the paper’s 0.5 boundary).

The “internal validity” measure is a 1-to-5 rubric score awarded by a Blinded Cross-Brand LLM judge reading the propagator’s reasoning trace. It is not attention probes or activation patching.

Evidence

Reviewer order alone swung accuracy by up to 0.24. For the GPT-5.4 propagator on GAIA at n=2, the (Claude, GPT) sequence scored 0.37 while (GPT, Claude) scored 0.61. For the same model on SWE-bench at n=2, (Claude, GPT) = 0.21 and (GPT, Claude) = 0.31. Reordering reviewers inside the same prompt is enough to shift accuracy by 10 to 24 points.

Mixed-model swarms reduced collapse in some cases. For the Gemini propagator on GAIA at n=5, the fragmented CPCPG sequence scored 0.87 while the homogeneous family swarm GGGGG scored 0.64. A 0.23 gap from changing the architectural composition of five reviewers.

SOTA context

The paper sits against four lines of work that assume more agents improves reasoning: multi-agent debate (Du 2024, Liang 2024), AgentVerse (Chen 2024), Society of Thought (Kim 2026), and topology optimization (Zhou 2026). All four assume aggregation improves accuracy.

This paper provides counter-evidence under a specific simulated condition: static consensus injection, deterministic decoding, and a 3-hop adversarial task. It does not refute the broader assumption, but it identifies an exact attack vector inside the standard pattern.

Where the paper overreaches

You can skip to the “How to wire around it“ section below.

Each item below cites the paper’s own text, tables, or figures. The tensions are internal to the work.

The Claude-immunity claim does not hold on Multi-Challenge. Section 4.1 of the paper states Claude maintained A=1.00 and E_ij=5.00 “across all domains.” Table 4 in the same paper shows Claude on Multi-Challenge at A=0.50 to 0.52, E_ij=3.00 to 3.08 across every plurality level, with 49% to 50% IGNORED stance at baseline. The paper’s results section does not address the gap.

Figure 4 and Appendix C.2 disagree on the Gemini lead-anchor case. Appendix C.2 reports a “Brand Subjugation” pattern on GAIA at delta=-0.10, claiming Gemini scored 0.50 when leading and 0.60 when following GPT. Figure 4 in the same paper shows +0.10 for that exact condition, the brightest cell on the heatmap. Lemma 1’s own proof in Appendix B.1 also reports the Gemini-leading sequence at 0.60 and the GPT-leading sequence at 0.50. The C.2 prose has the numbers swapped and the sign of delta is inverted.

The Interaction Depth Limit is not universal. Section 4.1 frames D_L=2 as the threshold for vulnerable models. Table 2 of the paper shows GPT-5.4 collapsing from 0.98 to 0.10 at n=1 against a single Claude auditor on Multi-Challenge. The threshold depends on the dataset and the lead auditor identity.

The “internal validity” score is an LLM judge’s rubric, not interpretability. Section 3.7 of the paper specifies that the Sovereignty Gap is operationalized via a Blinded Cross-Brand LLM-as-Judge that scores the propagator’s reasoning trace 1 to 5 on whether F1, F2, F3 are cited. Section 4.2 then describes the gap as the model “actively expending the computational effort to retrieve the correct derivation, but sycophantically lying.” The strong mechanistic reading does not follow from a rubric-graded chain-of-thought.

So the best recommendation is to act on the evidence and skip the math. The data identifies a real attack surface in agent pipelines. The proposed Sovereignty Decay Law does not fit the paper’s own Table 2 data past the n=2 cliff.

For GPT-5.4 on SWE-bench, accuracy recovers from 0.23 (n=2) to 0.37 (n=5), not the monotonic exponential decay the law predicts. The implied fix (independent-first reasoning) was already a known best practice in eval design.

How to wire around it

The paper does not ship a fix. It ships a stress test. The data points to four mitigations engineers can wire in today, even though the authors do not formally prescribe any of them.

Independent-first reasoning. Each agent produces its derivation before any peer output is visible. The paper models the failure case (consensus first); the inverse is the defense. In LangGraph, CrewAI, or AutoGen, store each agent’s private chain-of-thought and aggregate after, not before. No consensus string enters a reasoning step.

Anonymize reviewer identities at aggregation. Reviewer order alone swings accuracy by up to 0.24. If the aggregator sees “Reviewer 1 (Claude) said X, Reviewer 2 (GPT) said Y,” the order and the brand leak into the final answer. Strip names and model labels before the aggregator sees the verdicts.

Heterogeneous reviewer pools where you can. For Gemini on GAIA at n=5, the mixed CPCPG sequence scored 0.87 vs 0.64 for homogeneous GGGGG. Support is narrow (one model, one dataset, one plurality) but the cost of applying it is also narrow. Mix model families when designing reviewer rosters; do not lean on this as the primary fix.

Order-shuffle in your eval harness. If eval accuracy depends on reviewer order, the eval is measuring topology, not the model. The paper’s 25-trial sweep is overkill for production work; a 4- or 8-permutation rotation will surface order-sensitivity inside a day. Report the spread across permutations, not the mean of one ordering.

Who benefits and who does not

The findings travel to: agent-framework builders wiring orchestrated agents, eval engineers running LLM-as-judge pipelines, AI/ML engineers building reviewer-pool aggregation, and application developers exposing one model’s output to another before reasoning. The cleanest target is any system where Agent B reads Agent A’s verdict before producing its own.

The findings do not travel to: developers shipping pure single-agent flows, teams whose models never read other models’ outputs, anyone using deterministic ensemble methods (majority vote on closed outputs) instead of prompted aggregation. Researchers running message-passing multi-agent simulations should also read this as a caveat about evaluation methodology, not a result about active negotiation.

Practitioner implication

Agent-framework builders can now treat peer-consensus strings as untrusted input, now that consensus-first prompts are shown to drop a frontier model by up to 88 accuracy points.

Links

• Paper on arXiv (paper, ~25 min read)

• PDF

Follow @AlphaSignalAI for more content like this.

Subscribe at AlphaSignal.ai for daily AI signals. Read by 280,000+ developers.

Questions?

Q: Does this paper test real multi-agent systems? A: No. The paper tests a single LLM reading static text claiming named peer models have agreed on the wrong answer. Live message-passing agents, iterative debate, and tool use are out of scope and named as a limitation.

Q: Why does GPT-5.4 collapse from 98% to 10% on Multi-Challenge with one auditor? A: The failure mode at n=1 is task disengagement, not sycophancy. 85% of trials show an IGNORED stance and 7% show adoption of the false answer. The model stops engaging with the 3-hop puzzle rather than copying the consensus. The paper labels this terminal social disengagement.

Q: Is Claude actually immune to the bystander effect? A: On GAIA and SWE-bench, yes. On Multi-Challenge, Claude’s baseline accuracy is 0.52 with no auditors and stays at 0.50 to 0.52 across every plurality. The paper claims universal immunity in prose. Table 4 disagrees.

Q: Does this affect LLM-as-judge eval pipelines? A: Yes. The Lead Anchor Effect swings accuracy by up to 0.24 by reordering reviewers inside the same evaluation prompt. Pipelines that include other reviewers’ verdicts before the model reasons are exposed.

Q: What is the practical mitigation? A: Independent-first reasoning. Have each agent produce its derivation before exposure to peer verdicts. Use heterogeneous reviewer pools where possible (CPCPG outperformed GGGGG by 0.23 on Gemini GAIA). Treat any “the other agents said X” string as untrusted input.

After ~10 min reading, you will decide whether to install and use it and how to use every skill immediately.

TLDR? Check this HTML interactive guide (beta), inspired by @trq212

Zhao et al. just counted 146,932 hallucinated citations in 2025’s preprint record (arXiv:2605.07723, 2026-05).

Academic Research Skills is the first installable Claude Code workflow that wires a fix into the paper pipeline itself.

Cheng-I Wu shipped v3.7.0 with a two-command plugin install on May 5, 2026.

The license is CC BY-NC 4.0: source-available, not OSI open source.

Context

The repo is authored by Cheng-I Wu (GitHub Imbad0202). It was created on February 26, 2026 and now sits at +6.7k stars.

The intellectual ancestry is named in the README. Methodology is borrowed from PaperOrchestra (Song, Song, Pfister, Yoon, 2026, Google, arXiv:2604.05018). The failure-mode taxonomy comes from Lu et al. (2026, Nature 651:914-919, “The AI Scientist”).

The problem it solves is concrete. Most academic AI workflows live as one-off prompts in private chats. The pipeline from literature search to draft to peer review to citation check to disclosure is rebuilt every time. Academic Research Skills packages that pipeline as four Claude Code skills with mandatory human checkpoints at every stage.

Repo Snapshot

Technical Architecture

You can skip to “How to get started” Section down below.

The suite is four skills with declared data-access tiers, 25 registered modes, and a 10-stage orchestrated pipeline. Each skill owns part of the workflow.

deep-research ships 13 agents and 7 modes. It runs the upstream investigation: literature review, fact-check, systematic review, Socratic question framing. Data access level is raw. Modes include full, quick, socratic, lit-review, fact-check, systematic-review, and review.

academic-paper ships 12 agents and 10 modes. It handles drafting, revision, citation checks, format conversion, and the AI-disclosure statement. Data access is redacted. Modes include full, plan, outline-only, revision, revision-coach, abstract-only, lit-review, format-convert, citation-check, and disclosure.

academic-paper-reviewer ships 7 agents and 6 modes. It runs multi-perspective peer review with an Editor-in-Chief, three dynamic reviewers, and a Devil’s Advocate. Data access is verified_only. The calibration mode measures the reviewer’s own FNR/FPR against a user-supplied gold set.

academic-pipeline ships 4 agents and orchestrates everything above. It runs a 10-stage flow: research, write, Stage 2.5 integrity check, peer review, revision, re-review (max 2 loops), Stage 4.5 final integrity check, format conversion, final output, and process summary.

Stage 2.5 and Stage 4.5 integrity gates are the load-bearing piece. They run a 7-mode failure-mode checklist grounded in Lu et al.’s enumerated failures: implementation bugs, hallucinated results, shortcut reliance, bug-as-insight reframing, methodology fabrication, frame-lock, and citation hallucinations. The gates block pipeline progression on suspected failures, not silently flag them.

Material Passport is the handoff schema. It carries literature_corpus[] between skills with CSL-JSON authors, year, title, and source pointers back to the user’s own knowledge base. Since v3.6.5, consumers run a corpus-first, search-fills-gap flow: pre-screen the user’s corpus, then search external databases only for the remaining gaps.

v3.7.3 (in progress on main, not yet released) is the direct response to the Zhao et al. audit. That audit covered 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PMC, found 146,932 hallucinated citations for 2025 alone, and reported that 85.3% of preprint hallucinations survive into the published record. v3.7.3 closes the locator-channel half of the “claim faithfulness” gap the paper named.

The concrete addition is Three-Layer Citation Emission. Every visible citation gets a hidden <!--anchor:<kind>:<value>--> marker after the <!--ref:slug--> tag, where <kind> is quote, page, section, paragraph, or none. Quote anchors are capped at 25 words. Emitting none triggers a finalizer hard-gate refusal. The L3 full claim-faithfulness audit lands in v3.8.

Contamination signals are the second v3.7.3 addition. preprint_post_llm_inflection fires when a citation has year >= 2024 and venue is in a closed list of ten preprint servers (arXiv, bioRxiv, medRxiv, SSRN, Research Square, Preprints.org, ChemRxiv, EarthArXiv, OSF Preprints, TechRxiv). semantic_scholar_unmatched fires when the existing Semantic Scholar API protocol returns no match. Both are advisory annotations, not blocking gates.

How to Get Started

Plugin install (Claude Code CLI, VS Code, JetBrains, v3.7.0+) takes two commands:

/plugin marketplace add Imbad0202/academic-research-skills
/plugin install academic-research-skills

That sets up four skills, three plugin agents, ten /ars-* slash commands, and a SessionStart announce hook. Verify by typing /ars-plan and describing a paper. The skill should open a Socratic dialogue to map chapter structure.

The traditional install path (git clone + symlinks) still works for users on older Claude Code versions or anyone wanting per-project skill control:

git clone https://github.com/Imbad0202/academic-research-skills.git ~/academic-research-skills

cd /path/to/your/project

mkdir -p .claude/skills

ln -s ~/academic-research-skills/deep-research .claude/skills/deep-research

ln -s ~/academic-research-skills/academic-paper .claude/skills/academic-paper

ln -s ~/academic-research-skills/academic-paper-reviewer .claude/skills/academic-paper-reviewer

ln -s ~/academic-research-skills/academic-pipeline .claude/skills/academic-pipeline

Minimum runtime is Claude Code plus an Anthropic API key:

export ANTHROPIC_API_KEY=sk-ant-...
claude

Optional document tooling for DOCX and APA 7.0 PDF output:

brew install pandoc
brew install tectonic

For Codex CLI users, the sibling distribution is Imbad0202/academic-research-skills-codex. Same workflow content, Codex-native packaging.

Cost from docs/PERFORMANCE.md: roughly $4 to $6 for a 15,000-word paper with 60 references on Opus 4.7. Cross-model verification (ARS_CROSS_MODEL) adds $0.60 to $1.10. A full run exceeds 200K input and 100K output tokens, so long sessions can lose prompt-cache benefits and need Material Passport resume.

How to Actually Use It

Three entry points cover most real usage.

Full pipeline. Type /ars-full or describe the goal in natural language (”I want to write a research paper on AI’s impact on higher education QA”). The orchestrator starts at Stage 1 and walks all ten stages with user confirmation at every FULL checkpoint. Output is a finished APA 7.0 paper, an Editorial Decision Letter, a Revision Roadmap, two integrity reports, and an AI Self-Reflection Report.

Guided planning. Type /ars-plan when the research question is not yet clear. The Socratic Mentor agent classifies user intent as exploratory or goal-oriented. Exploratory mode disables auto-convergence and runs a chapter-by-chapter dialogue: define the question, choose the method, map the argument. Output is a Chapter Plan plus an INSIGHT collection.

Targeted single-skill calls. Skip the orchestrator when only one function is needed:

A typical first session looks like this. Run /ars-plan to get a chapter map. Then run /ars-lit-review to fill the corpus. Then run /ars-full with the corpus already populated in the Material Passport.

The Material Passport is the handoff between sessions. It carries the literature corpus, the chapter plan, and the integrity reports. To resume a prior run in a fresh Claude Code session, set ARS_PASSPORT_RESET=1 and use the resume_from_passport=<hash> mode.

For an existing draft, type “I already have a paper, review it” to enter the pipeline at Stage 2.5 with the integrity check running first. For reviewer-comment response, type “I received reviewer comments” to enter at Stage 4 with the revision-coach flow.

The pipeline ends with a Process Summary stage: a Collaboration Quality Evaluation across six dimensions scored 1 to 100. That score feeds the AI Self-Reflection Report, which surfaces concession rate, health alerts, and sycophancy risk for the run.

Evidence

Academic Research Skills is the only candidate in the current academic-Claude-Code-skills cluster with multi-stage integrity gates wired into the pipeline itself.Comparison is architectural, not empirical. No formal benchmarks exist for academic-pipeline tooling.

Current Limitations (13 May)

License friction. CC BY-NC 4.0 blocks commercial use. Source-available, not OSI open source.

Claude Code lock-in. The reference distribution is Claude Code-first. A Codex sibling exists. Cursor, OpenCode, and Gemini are not addressed.

Integrity gates leak. The maintainer’s own post-publication audit of the showcase paper found 21 issues across 68 references that survived three rounds of automated integrity checks. v3.7.3 closes the locator-channel half. The full claim-faithfulness audit is deferred to v3.8.

Tagged release vs main drift. v3.7.0 is the latest tagged release. Three-Layer Citation Emission lives on main as [Unreleased] v3.7.3 work.

Metadata inconsistency. .claude-plugin/plugin.json claims “35+ modes, 32-agent ensemble.” MODE_REGISTRY.md says 25 modes. Direct file count finds 36 agents.

AlphaSignal Take

Worth Watching.

Academic Research Skills ships what its README claims. v3.7.0 is the signed release, the plugin assets are in place (.claude-plugin/, 10 /ars-* commands, 3 plugin agents, SessionStart hook), and the static lints pass across spec consistency, schema, and pattern-protection checks. The workflow architecture is the strongest installable response yet to a citation-hallucination paper that just put a corpus-scale number on the problem.

The non-obvious finding is that the maintainer documents his own failure. 21 out of 68 references slipped through three rounds of integrity checks in the showcase audit. That honesty is the strongest evidence the gates do something. It is also why the verdict is not Production Ready.

What changes the verdict: a permissive license (or a commercial tier), the L3 full claim-faithfulness audit shipped in v3.8, and a non-Claude Code reference distribution. Until then, the workflow design is more valuable than the workflow runtime.

Who Benefits and Who Doesn’t

PhD students, academic researchers, and lab teams already on Claude Code under noncommercial settings, agent-tooling builders studying how integrity gates wire into multi-stage workflows, and journals or workshops evaluating AI-disclosure schemas.

Commercial SaaS or paid-consulting teams (CC BY-NC 4.0 blocks the build), Cursor-only or OpenCode-only stacks (the reference distribution is Claude Code-first), and anyone needing byte-reproducible citation guarantees (v3.7.3 anchors are advisory in places, L3 full audit is unshipped).

Practitioner Implication

Researchers using Claude Code can now install a 10-stage academic workflow with mandatory integrity gates as four skills, now that v3.7.0 ships a one-line plugin path.

Links

• github.com/Imbad0202/academic-research-skills (repo, ~5 min plugin install)

• arxiv.org/abs/2605.07723 (Zhao et al. citation-hallucination audit, ~25 min read)

Follow @AlphaSignalAI for more content like this.

Subscribe at AlphaSignal.ai for daily AI signals. Read by 280,000+ developers.

Questions?

Q: How do you install Academic Research Skills in Claude Code?

A: Two plugin commands: /plugin marketplace add Imbad0202/academic-research-skills then /plugin install academic-research-skills. Requires Claude Code latest and ANTHROPIC_API_KEY. First run: /ars-plan.

Q: Does Academic Research Skills write papers automatically?

A: No. The repo’s POSITIONING.md explicitly states ARS is assistive, not autonomous. Mandatory human checkpoints at every FULL stage and at Stage 2.5 and Stage 4.5 integrity gates block silent progression.

Q: How does ARS reduce citation hallucinations?

A: Two integrity gates (Stage 2.5 pre-review, Stage 4.5 pre-finalization) run a 7-mode failure-mode checklist plus Semantic Scholar API verification. v3.7.3 adds Three-Layer Citation Emission: a hidden anchor marker after every citation specifies quote, page, section, or paragraph locator.

Q: What does ARS cost to run for a full paper pipeline?

A: Roughly $4 to $6 for a 15,000-word, 60-reference paper on Opus 4.7 per docs/PERFORMANCE.md. Cross-model verification adds $0.60 to $1.10. A full run exceeds 200K input and 100K output tokens.

Q: Is Academic Research Skills open source?

A: Source-available, not OSI open source. License is CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0). Commercial SaaS, hosted services, paid consulting, and enterprise deployments require separate licensing.

After 3 minutes, you’ll know whether to install agentmemory. After 10 minutes, you’ll have it running and be able to use it immediately.

TLDR? Review this HTML interactive guide (beta), inspired by @trq212

agentmemory replaces CLAUDE.md, .cursorrules, and every other static-file memory hack with an actual local service.

3,000+ stars in ~3 days, version 0.9.9 shipped 2026-05-11, Apache-2.0, one npx command to install.

Rohit Ghumare, Principal Product Evangelist at iii.dev, authored the implementation. The design comes from his gist “LLM Wiki v2” which extends Karpathy’s LLM Wiki pattern with the lessons from building agentmemory (170+ forks, 13 comments).

What changes for you. Session 1, you set up JWT auth with jose middleware in src/middleware/auth.ts. Session 2, you ask for rate limiting, and the agent already knows your auth stack, your test file, and why you chose jose over jsonwebtoken. No re-explaining.

Repo Snapshot (12 May)

Context

Every AI coding agent ships with built-in memory: Claude Code has MEMORY.md, Cursor has notepads, Cline has a memory bank. These work like sticky notes. They cap at ~200 lines, go stale, and load everything into context every time the session opens.

agentmemory is the searchable database behind the sticky notes. The repo was created 2026-02-25, has 280+ commits across 13 contributors, and shipped 8 releases in three days (May 9 to May 11) leading up to v0.9.9.

How It Works

Jump to “How to get started” down below.

agentmemory is built on iii-engine, a service composition framework (15K+ stars, TypeScript + Rust). Functions, KV state, streams, and OTEL traces are all iii primitives. The engine replaces Express.js, Postgres + pgvector, SSE/Socket.io, pm2, and Prometheus. No external database, no Docker required (though both work).

The system has three layers: Capture, Pipeline, Retrieval. Plus a consolidation cycle that compresses raw observations into longer-term memory tiers.

• Capture. Twelve Claude Code lifecycle hooks fire automatically: SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PostToolUseFailure, PreCompact, SubagentStart, SubagentStop, Notification, TaskCompleted, Stop, SessionEnd. Each hook is a standalone Node script that reads JSON from stdin, POSTs to the local REST API, and exits. Non-Claude agents capture the same way through /agentmemory/observe or the MCP memory_save tool. Zero manual add() calls. The agent works, the hook fires, the observation lands.

• Pipeline. Inside the server, every observation passes through four stages. SHA-256 dedup catches anything that repeats within five minutes. The privacy filter in src/functions/privacy.ts strips <private> blocks and redacts API keys, bearer tokens, GitHub tokens, AWS keys, Google keys, JWTs, npm tokens, GitLab tokens, and DigitalOcean tokens before storage.

  Raw observations land in iii-engine’s file-backed KV. Then a synthetic compression path indexes the observation in BM25 without calling any LLM. If you set AGENTMEMORY_AUTO_COMPRESS=true, a configured Anthropic, MiniMax, Gemini, or OpenRouter provider compresses observations into structured facts on every hook. Off by default since v0.8.8 (issue #138) because the per-token cost on active sessions is significant.

• Retrieval. Three streams run in parallel inside src/state/hybrid-search.ts. BM25 stems with Porter, expands synonyms, runs always. Vector computes cosine similarity over all-MiniLM-L6-v2 384-dimension embeddings when an embedding provider is configured (free local option via @xenova/transformers, or hosted via Gemini, OpenAI, Voyage, Cohere, OpenRouter).

  Graph traverses entity relationships when entities are detected in the query. Results fuse via Reciprocal Rank Fusion with RRF_K = 60, then diversify across sessions (max 3 results per session) so one session’s noise can’t dominate the top-K.

• 4-tier consolidation. Memories progress through four tiers, analogous to sleep consolidation. Working holds raw observations. Episodic holds compressed session summaries. Semantic holds extracted facts and patterns. Procedural holds workflows and decision patterns. Memories decay on an Ebbinghaus-style curve. Frequently accessed memories strengthen. Stale memories auto-evict. Contradictions are detected and resolved on write.

• Context assembly. When a new session starts, mem::context assembles pinned slots, project profile, recent session summaries, and high-importance observations into an <agentmemory-context> block with a default 2,000-token budget. SessionStart context injection is OFF by default (AGENTMEMORY_INJECT_CONTEXT=false since v0.8.10, issue #143).

  Hooks still POST observations for background capture either way, but the README’s “agent already knows your stack” demo specifically requires the env var to be on. Worth knowing before you wonder why the agent doesn’t seem to remember anything.

• MCP and REST surface. Fifty-one memory_* tools, six MCP resources, three prompts, four skills (/recall, /remember, /session-history, /forget), and 127 REST endpoint declarations across src/triggers/api.ts and src/mcp/server.ts. Seven tools are visible by default. Set AGENTMEMORY_TOOLS=all to expose all 51. The full list includes hybrid search (memory_smart_search), file history (memory_file_history), project profile (memory_profile), graph traversal (memory_graph_query), team sharing (memory_team_share), audit trail (memory_audit), and multi-agent coordination primitives (memory_lease, memory_signal_send, memory_mesh_sync).

  ---

How to Get Started

This section covers install. For the command reference once it is running, jump to How to Use It at the end of this article.

Total time to working install: under 10 minutes on macOS or Linux, 15 on Windows.

1. Prerequisites. Node.js >= 20.0.0. iii-engine v0.11.2 OR Docker Desktop. macOS, Linux, or Windows. On Windows, the npm package alone is not enough: download the iii-engine binary from iii-hq/iii releases (or use Docker Desktop), since there’s no PowerShell installer or winget package today.

2. Start the server.

npx @agentmemory/agentmemory

This auto-downloads or starts iii-engine v0.11.2 (the pinned version, since v0.11.6 introduced a sandbox-per-worker model agentmemory hasn’t refactored for yet). REST binds to 127.0.0.1:3111, streams to :3112, viewer to :3113.

3. Seed and verify with the demo.

npx @agentmemory/agentmemory demo

This seeds three sessions (JWT auth setup, an N+1 query fix, rate limiting) with six observations total, then runs three searches. The search “database performance optimization” returns the N+1 fix observation. Keyword-only search cannot do that. If you see results, the pipeline works end-to-end.

4. Open the viewer.

http://localhost:3113

Live observation stream, session explorer, memory browser, knowledge-graph visualization, and a health dashboard. Bound to 127.0.0.1 only.

5. Wire it to your agent. One JSON block covers most hosts (Cursor, Claude Desktop, Cline, Roo Code, Windsurf, Gemini CLI, OpenClaw). Merge into the host’s existing mcpServers object. Do not replace the file.

{
  “mcpServers”: {
    “agentmemory”: {
      “command”: “npx”,
      “args”: [”-y”, “@agentmemory/mcp”],
      “env”: {
        “AGENTMEMORY_URL”: “http://localhost:3111”
      }
    }
  }
}

The host-specific shapes:

• Cursor: ~/.cursor/mcp.json

• Claude Desktop: claude_desktop_config.json in Application Support, restart after editing

• Cline / Roo Code / Kilo Code: Settings UI, MCP Servers, Edit

• Windsurf: ~/.codeium/windsurf/mcp_config.json

• Gemini CLI: gemini mcp add agentmemory npx -y @agentmemory/mcp --scope user

• Codex CLI (TOML shape): codex mcp add agentmemory -- npx -y @agentmemory/mcp

• OpenCode (different shape, top-level mcp key with command as array)

6. Claude Code: install the plugin instead. Skip step 5 and run:

/plugin marketplace add rohitg00/agentmemory
/plugin install agentmemory

The plugin registers all 12 hooks, 4 skills, and auto-wires the MCP server through .mcp.json. Verify with curl http://localhost:3111/agentmemory/health.

7. Optional: free local embeddings.

npm install @xenova/transformers

Switches the embedding provider to all-MiniLM-L6-v2 running in-process. No API key, no per-call cost, adds ~9 percentage points of recall over BM25-only (86.2% to 95.2% R@5 on LongMemEval-S).

8. Optional: turn on the headline demo. Out of the box, agentmemory captures and supports recall but does not inject context into the agent’s first turn. To enable the “agent already knows your stack” behavior, add this to ~/.agentmemory/.env:

AGENTMEMORY_INJECT_CONTEXT=true

Restart the server. SessionStart will now write up to 2,000 tokens of relevant project context into the first turn. This counts against your model’s token budget. The startup warning will remind you.

9. Import existing transcripts.

npx @agentmemory/agentmemory import-jsonl

Default scan path is ~/.claude/projects. Default cap is 200 files / 1,000 sessions. Imported sessions show up in the viewer’s Replay tab alongside native ones.

That covers install. For the command reference once it is running, jump to How to Use It at the end of this article.

---

Benchmark Evidence

LongMemEval-S (ICLR 2025, 500 questions, ~48 sessions per question, ~115K tokens each):These are retrieval recall scores, not end-to-end QA accuracy. The repo says so plainly: it does not claim these as “LongMemEval scores,” only as retrieval-only evaluations on the LongMemEval-S haystack. Scripts and the cleaned dataset are committed under benchmark/ for reproduction.

Token efficiency, repo-reported: ~1,900 tokens per session, ~170K per year, ~$10/year on per-token billing or $0/year with local embeddings. Compare against ~22K tokens at 240 observations for the “paste everything into CLAUDE.md” approach. That is a 92% reduction at the working point most heavy users hit by month two.

Comparison

Mem0 and Letta publish benchmarks on LoCoMo, a different evaluation set. The repo’s own COMPARISON.md flags this as “apples vs oranges” and invites cross-benchmark collaboration. Treat the headline 95.2% as agentmemory’s number on agentmemory’s pipeline, not a leaderboard win against tools measured on a different test.

Current Limitations

Plaintext HTTP token transport (issue #275, open). The plugin sends the AGENTMEMORY_SECRET bearer over plaintext HTTP. The default localhost binding contains the exposure today. Anyone exposing the REST surface beyond a single host needs a reverse proxy, TLS, and an auth review first.

The interesting features are off by default. Auto-compress, context injection, slots, reflect, and graph extraction are all gated behind env vars in src/config.ts. The README’s “agent already knows your stack” demo specifically requires AGENTMEMORY_INJECT_CONTEXT=true. Out of the box, agentmemory captures observations and supports recall, but it does not inject context into Claude Code’s first turn. This is the single biggest expectation mismatch a new user will hit.

Documentation drift. AGENTS.md ships v0.8.9 stats (44 tools, 104 endpoints, 699 tests). The source has 51 tools, 127 endpoint declarations, and 888 static test cases. The README badge claims 104 endpoints and 827 tests, while the README prose says 107 endpoints. benchmark/COMPARISON.md references npm run bench:* scripts that are not in package.json. Three doc surfaces, three different numbers, no agreement.

The engine writes state into your project root (issue #303, open). When Claude Code auto-starts agentmemory from a project directory, the iii-engine creates data/state_store.db/... inside the user’s git working tree. Expect cleanup noise and .gitignore drift until the engine moves state to a stable location.

AlphaSignal Take

Verdict: Production Ready for solo developers and small teams running agentmemory on localhost as a personal coding-agent memory layer.

The architecture is real. The benchmark is reproducible from committed scripts. The install is one npx command. The 12-hook capture flow runs unattended. The viewer at port 3113 makes the memory system inspectable, which is rare in this category. There is no equivalent shipping today.

Maintenance health is acceptable. Eight releases in three days (May 9 through May 11). 280+ commits since February. Active issue triage, public CHANGELOG, public ROADMAP. The 91% single-maintainer concentration is the asterisk, and the Q3 2026 roadmap names “additional maintainer onboarding” as a priority and a foundation Growth-Stage prerequisite.

What would change the verdict positively? Q4 2026 ships an external security audit, SSO, RBAC, and audit-log export. Q1 2027 freezes the REST and MCP surface for v1.0. The plaintext-HTTP fix (#275) and the engine-CWD cleanup (#303) close. At that point the answer for production deployments changes from “wait” to “deploy.” Watch for v1.0 in Q1 2027.

Who Benefits and Who Doesn’t

Benefits: Claude Code, Cursor, Codex CLI, Gemini CLI, and OpenCode users running solo or in small teams. Engineers in multi-day agent sessions on the same codebase. Teams who want a real-time viewer for what the agent is learning. Anyone who hit the 200-line CLAUDE.md ceiling and started copy-pasting.

Doesn’t benefit yet: teams deploying agentmemory’s REST surface beyond localhost without a reverse proxy plus TLS (issue #275 is open). Production deployments that require an external security audit (planned Q4 2026). Non-English coding sessions that rely on accurate BM25 retrieval (issue #295 strips non-ASCII tokens today). Windows users without Docker or the iii-engine binary on PATH.

Practitioner Implication

Coding-agent users can now install a shared, hook-driven, searchable memory layer in one npx command and stop pasting their stack into every new session.

Links

• github.com/rohitg00/agentmemory (repo, ~30 sec to first install)

• agent-memory.dev (landing site)

• benchmark/LONGMEMEVAL.md (~5 min read)

• LongMemEval paper, ICLR 2025 (~25 min read)

Follow @AlphaSignalAI for more content like this.

Subscribe at AlphaSignal.ai for daily AI signals. Read by 280,000+ developers.

Appendix: How to Use It

A command reference for once agentmemory is installed and running.

The four user-invocable skills (Claude Code). Installed by the plugin, invoked by typing the slash command in the agent:

• /recall [query] wraps memory_smart_search. Hybrid BM25 + vector + graph search across past observations. Use when you want context from a past session (”recall how we set up JWT auth”).

• /remember [content] wraps memory_save. Explicitly persists an insight, decision, or pattern with auto-extracted concepts and file references. Use when you want to lock in a decision so the next session inherits it.

• /session-history wraps memory_sessions. Lists the last 20 sessions on this project with key highlights.

• /forget [query or session ID] wraps memory_smart_search then memory_governance_delete. Surfaces matching observations, asks for explicit confirmation, then deletes with an audit trail.

Most of the time, just talk to the agent. The 12 hooks capture every tool call automatically. If AGENTMEMORY_INJECT_CONTEXT=true is set, SessionStart pre-loads relevant memories into the agent’s first turn. The four skills above are for explicit control, not the default workflow.

Direct MCP tools (other agents). Agents without the plugin call MCP tools directly. The seven core tools available by default:

Set AGENTMEMORY_TOOLS=all to expose the full 51-tool surface (knowledge graph queries, multi-agent leases, signals, sentinels, sketches, consolidation, snapshots, mesh sync, audit, governance, team sharing).

Direct REST calls. Every MCP tool has a REST equivalent on port 3111. Useful for scripts, IDEs, and CI:

# Search
curl -s -X POST http://localhost:3111/agentmemory/smart-search \
  -H “Content-Type: application/json” \
  -d ‘{”query”: “jwt auth middleware”, “limit”: 5}’

# Save
curl -s -X POST http://localhost:3111/agentmemory/remember \
  -H “Content-Type: application/json” \
  -d ‘{”content”: “Auth uses jose, chosen over jsonwebtoken for Edge compat”, “concepts”: [”jwt”, “auth”, “edge”], “files”: [”src/middleware/auth.ts”]}’

# Project profile
curl -s “http://localhost:3111/agentmemory/profile?project=$(pwd)”

# Health
curl -s http://localhost:3111/agentmemory/health

When AGENTMEMORY_SECRET is set, protected endpoints require Authorization: Bearer <secret>.

The viewer at port 3113. Open http://localhost:3113 to watch observations land live, browse sessions, walk the knowledge graph, and scrub through the Replay tab on past sessions (including imported JSONL transcripts).

When to reach for which command:

• Working on a feature, want continuity across sessions: do nothing, hooks handle it

• Decision or pattern worth keeping forever: /remember

• New session, no context loaded: /recall with a topic

• Reviewing yesterday’s work: /session-history

• Privacy or cleanup: /forget

  ---

After ~10 min reading, you will decide whether to install agent-skills, where to wire it in, and how to use every skill immediately.

Addy Osmani, engineering lead at Google Chrome, open-sourced agent-skills on February 15, 2026.

The repo has hit 39K+ stars in three months, with a 1K+ reported daily gain.

The bet is that agent reliability comes from the harness around the model, not from a smarter model.

What’s different is that this is not another prompt library. It is a three-layer architecture (skill, persona, command), with anti-rationalization tables in 20 of 22 skills, a parallel fan-out command, and three hooks that give the pack real enforcement weight on Claude Code.

It does not make the model smarter. It makes skipping specs, tests, reviews, and security checks harder.

The single most screenshot-able thing in the repo is the table used across most lifecycle skills that names the agent’s shortcut excuses and pairs each with a counter-argument:

“I’ll write tests after the code works” → “You won’t. And tests written after the fact test implementation, not behavior.”

“It’s just a prototype” → “Prototypes become production code. Tests from day one prevent the test-debt crisis.”

“I tested it manually” → “Manual testing doesn’t persist. Tomorrow’s change might break it with no way to know.”

That table sits inside test-driven-development/SKILL.md. Most lifecycle skills in the repo ship one of their own. It is the single structural move that separates agent-skills from every other skills repo in the feed.

Context

The author is Osmani, the author of Learning JavaScript Design Patterns and a Google Chrome engineering lead. He summarized the motivation in a May 3 research post:

“A senior engineer’s job is mostly the parts that don’t show up in the diff.”

The repo was created on February 15, 2026 and has shipped 170+ commits from 20+ contributors since. The latest release is v0.6.0 on April 28. Growth went from 27K+ stars on May 4 to 39K+ on May 11.

How it sits next to the other major skills repos

If you’ve been tracking the skills space, the question isn’t “what is this.” It’s what this does that obra/superpowers and anthropics/skills don’t.

The star comparison cuts the other way: obra/superpowers is bigger by raw count, anthropics/skills is the official spec. The unique contribution of agent-skills is structural: 20 anti-rationalization tables, a parallel-fan-out command, and an enforcement hook layer.

Repo snapshot (11 May)

Three composable layers

The governing rule is short and load-bearing. The user, or a slash command on the user’s behalf, is the orchestrator. Personas do not invoke other personas. Skills are mandatory hops inside a persona’s workflow.

This rule was formalized in the v0.6.0 release notes after a stretch of issues where contributors tried to write “meta-orchestrator” personas that routed work to other personas. The repo names this an anti-pattern and rejects it on two grounds. It loses information through paraphrasing hops, and on Claude Code it is impossible by platform constraint anyway, since subagents cannot spawn other subagents.

The only multi-persona pattern the repo endorses is parallel fan-out, used by /ship. More on that below.

How it works

You can skip to “How to get started” Section down below.

Inside the repo. Skills sit in skills/<name>/SKILL.md. Personas live in agents/ as plain Markdown files. Slash commands ship twice: .claude/commands/*.md for Claude Code and .gemini/commands/*.toml for Gemini CLI. Hooks are bash scripts in hooks/. Four reference checklists (testing, security, performance, accessibility) sit in references/, separate from any skill so they load on demand.

The SKILL.md format. Every skill file follows the same anatomy:

• YAML frontmatter (name + description). Only these two fields are loaded at session start, so the agent can scan all 22 skills cheaply. Full file content loads only when the skill is invoked. This is the progressive-disclosure mechanic that keeps the pack under context budget even with all skills installed.

• Overview and When to Use. Two short blocks. The first says what the skill does. The second lists the exact triggers that should activate it (for example, “implementing any new logic,” “fixing a bug”). The agent matches the current task against the triggers.

• Process. The numbered step-by-step workflow. The largest section in every file, and the part the agent actually executes. Steps include inline code examples, decision flowcharts, and templates the agent fills in.

• Common Rationalizations. A two-column table of agent excuses paired with counter-arguments. The agent has to read its own most-likely shortcut before it can take it. 20 of the 22 skills ship one of these.

• Red Flags. A bullet list of observable signs the skill is being violated. The agent self-monitors against the list. The reviewer in /review also checks against it.

• Verification. A checklist of exit criteria the agent must satisfy before marking work done. Every checkbox is evidence-based: tests passing, build output, runtime data, screenshot. The repo’s rule: “Seems right is never sufficient.”

How a skill activates. Two paths. The agent auto-activates a skill by matching the current task against the When to Use triggers in the frontmatter. The meta-skill using-agent-skills holds the routing flowchart that does this matching, and is injected into every session by the session-start hook. The second path is explicit: the user invokes a skill via a slash command (/spec, /test, /review, etc.). Either path loads the full SKILL.md into context.

Length and structure rules. Every SKILL.md stays under 500 lines. Reference material that would push it over lives in references/, loaded only when a skill needs it. The longest skill file in the repo is ci-cd-and-automation at 390 lines. Verified from the local clone at SHA 3ff4b518: 22 of 22 skill files have valid YAML frontmatter, 21 of 22 have a ## Verification block, 20 of 22 have a ## Common Rationalizations table.

Start here. For most teams, the first three skills to load are spec-driven-development, test-driven-development, and code-review-and-quality. They cover the highest-risk agent failure loop: unclear task, untested change, and unreviewed diff.

The full six-phase lifecycle map is near the end of this article for readers who want every skill by phase.

---

Evidence: four structural moves that make it defensible

Osmani’s frame is short: “Agents skip those steps for the same reason any junior would. They’re invisible.” The repo makes those steps visible in four places.

Anti-rationalization tables. test-driven-development is one of 20 skills with a table that names the shortcut and rejects it before the agent can take it.

The /ship fan-out. /ship spawns code-reviewer, security-auditor, and test-engineer concurrently, then merges their reports into a GO or NO-GO decision with a rollback plan. It skips fan-out only when the change touches two files or fewer, stays under 50 lines, and avoids auth, payments, data access, and config.

Three hook systems. session-start.sh injects using-agent-skills on new Claude Code sessions, uses jq for the JSON payload, falls back to INFO when jq is missing, and passes bash hooks/session-start-test.sh. sdd-cache-{pre,post}.sh caches source docs by sha256(url) and only serves cached bodies after 304 Not Modified against If-None-Match or If-Modified-Since. simplify-ignore.sh protects /* simplify-ignore-start: reason */ blocks with BLOCK_<hash> placeholders and reports 21 passed, 0 failed.

The newest skill, doubt-driven-development. Added in May 2026, it runs a fresh-context reviewer on non-trivial decisions using only the artifact plus the contract, not the original agent’s reasoning. Cross-model review through Codex CLI or Gemini CLI requires explicit per-call authorization, so the check happens mid-flight before /review.

How to install

Claude Code (the canonical path). Marketplace install:

/plugin marketplace add addyosmani/agent-skills
/plugin install agent-skills@addy-agent-skills

For teams without SSH keys on GitHub, force HTTPS (workaround for PR #108):

/plugin marketplace add https://github.com/addyosmani/agent-skills.git
/plugin install agent-skills@addy-agent-skills

For local development against an in-flight skill, clone and point Claude Code at the working copy:

git clone https://github.com/addyosmani/agent-skills.git
claude --plugin-dir /path/to/agent-skills

At runtime, the session-start.sh hook injects the using-agent-skills meta-skill on every new session, which routes the first message to the matching skill. The 7 slash commands become explicit lifecycle entries on top of that. CI on main is green for the workflow Test Plugin Installation at the verified SHA 3ff4b518.

OpenCode. No slash commands and no plugin system. The integration runs through AGENTS.md and the built-in skill tool. The repo ships .opencode/skills as a symlink to ../skills/ so OpenCode resolves the same skill set. The execution rule in AGENTS.md maps user intent (new feature triggers spec-driven-development, bug triggers debugging-and-error-recovery, code review triggers code-review-and-quality) to skills on every turn. Honest tradeoff per the repo’s own docs/opencode-setup.md: skill invocation depends on model compliance with the AGENTS.md contract, with no hook layer.

Cursor. Copy selected SKILL.md files into .cursor/rules/. Start with the 2-to-3-essential set.

Gemini CLI. gemini skills install https://github.com/addyosmani/agent-skills.git --path skills. Native install. The repo also ships .gemini/commands/*.toml with the same 7 commands, except /plan is renamed /planning because /plan collides with a Gemini internal command.

Windsurf, Copilot, Kiro. Add skill content to .windsurfrules, .github/skills/, or .kiro/skills/ respectively. All three are plain-Markdown integrations with no hook layer.

The six-phase lifecycle and 22 skills

DEFINE → PLAN → BUILD → VERIFY → REVIEW → SHIP
/spec   /plan   /build   /test    /review  /ship
                                  /code-simplify

Skills auto-activate by context. The slash commands are explicit user entries on top. The 22 skills, with what each one does so the reader can pick:

Meta

• using-agent-skills: Routes incoming work to the right skill via a flowchart. Auto-loaded by the session-start hook on every Claude Code session. Defines the shared operating behaviors (surface assumptions, manage confusion, push back, enforce simplicity, scope discipline, verify don’t assume).

Define

• idea-refine: Turns vague ideas into concrete proposals through structured divergent and convergent thinking. Output is a one-page markdown spec with problem statement, recommended direction, MVP scope, and a “Not Doing” list.

• spec-driven-development: Writes a PRD before code: objective, commands, project structure, code style, testing strategy, boundaries (always/ask-first/never). Four-phase gated workflow (specify, plan, tasks, implement) with human review at each gate.

Plan

• planning-and-task-breakdown: Decomposes a spec into small verifiable tasks with acceptance criteria and dependency ordering. Cap of ~5 files per task. Each task includes its verification step.

Build

• incremental-implementation: Builds thin vertical slices: implement, test, verify, commit. Caps at ~100 lines of unverified code. Feature flags, safe defaults, rollback-friendly changes.

• test-driven-development: Red-Green-Refactor, the test pyramid (80% unit / 15% integration / 5% E2E), DAMP over DRY in tests, the Beyonce Rule, and the Prove-It pattern for bug fixes (failing reproduction test before the fix).

• context-engineering: Loads the right context at the right time. Rules files, context packing, MCP integrations.

• source-driven-development: Grounds framework decisions in official documentation with required citations. Detects stack and versions, fetches the relevant docs, flags conflicts with existing code. Paired with the sdd-cache hook for cross-session HTTP caching.

• doubt-driven-development: Adversarial fresh-context review on every non-trivial in-flight decision. Five-step cycle: CLAIM, EXTRACT, DOUBT, RECONCILE, STOP. Optional cross-model escalation to Codex CLI or Gemini CLI with explicit per-call authorization.

• frontend-ui-engineering: Component architecture, design systems, state management, responsive design, WCAG 2.1 AA accessibility.

• api-and-interface-design: Contract-first design, Hyrum’s Law, the One-Version Rule, error semantics, boundary validation.

Verify

• browser-testing-with-devtools: Chrome DevTools MCP for live runtime data. DOM inspection, console errors, network traces, performance profiling, screenshots.

• debugging-and-error-recovery: Five-step triage: reproduce, localize, reduce, fix, guard. Stop-the-line rule for failing tests, safe fallbacks.

Review

• code-review-and-quality: Five-axis review (correctness, readability, architecture, security, performance), change sizing ~100 lines, Critical/Important/Suggestion severity labels.

• code-simplification: Reduce complexity while preserving exact behavior. Chesterton’s Fence, the Rule of 500. Paired with the simplify-ignore hook for protected code blocks.

• security-and-hardening: OWASP Top 10 prevention, auth patterns, secrets management, dependency auditing, three-tier boundary validation.

• performance-optimization: Measure-first approach. Core Web Vitals targets, profiling workflows, bundle analysis, anti-pattern detection.

Ship

• git-workflow-and-versioning: Trunk-based development, atomic commits, change sizing ~100 lines, the commit-as-save-point pattern.

• ci-cd-and-automation: Shift Left, Faster is Safer, feature flags, quality gate pipelines, failure feedback loops.

• deprecation-and-migration: Code-as-liability mindset, compulsory vs advisory deprecation, migration patterns, zombie-code removal.

• documentation-and-adrs: Architecture Decision Records, API docs, inline documentation standards. Document the why, not the what.

• shipping-and-launch: Pre-launch checklists, feature flag lifecycle, staged rollouts, rollback procedures, monitoring setup.