Hoja-Solutions

advanced-tool-use

Give an agent access to many tools without flooding its context with every tool definition. Covers the three advanced-tool-use patterns: a search tool the agent calls to discover tools on demand, programmatic tool calling where the model writes code that invokes tools and filters results before they return, and tool-use examples that teach correct invocation. Use this when an agent's context fills with tool schemas, an agent picks the wrong tool out of dozens, someone connects many MCP servers at once, or a single task chains several tool calls. Trigger on "too many tools," "tool definitions eat my context," "tool search," "programmatic tool calling," "agent calls the wrong tool," and similar. This is scaling access to a large tool catalog; writing a single tool's description and response shape is tool design (covered by community packs), and gating which commands run is action gating.

auditing-agent-behavior

Audit an agent for risky behavior before it ships, the way you would red-team it, but automated. An auditor agent runs many multi-turn scenarios against your target agent from seed instructions you write, and a judge model scores the transcripts for deception, sycophancy, oversight subversion, power-seeking, and cooperation with misuse. Covers writing seed instructions that probe your agent's real risks, adapting the scoring rubric to your domain, and reading flagged transcripts. Use this when someone wants to red-team or safety-test an agent, asks how to find deceptive or manipulable behavior, audits a model or agent before deployment, or compares behavior across model versions. Trigger on "red-team my agent," "audit agent behavior," "test for deception," "alignment testing," and similar. This is auditing behavior for safety; measuring task success is build-agent-evals, and stopping objective-gaming is reward-hacking-and-inoculation.

build-agent-evals

Build automated evaluations for an AI agent from scratch: collecting tasks from real failures, choosing code/model/human graders, picking pass@k vs pass^k, building an isolated harness, and keeping the suite honest over time. Use this whenever someone wants to measure, benchmark, or regression-test an agent, write an eval harness for an LLM agent, decide how to grade non-deterministic output, set up an LLM-as-judge, or asks any version of "how do I know if my agent is actually getting better." Trigger even when they say "tests for my agent," "eval set," or "agent benchmark" rather than the word "evals," or when they ask about benchmark contamination or a model recognizing the eval. Not for container or resource limits making scores flaky across runs; that's calibrate-eval-infrastructure.

DevOps & Infrastructure Listed

calibrate-eval-infrastructure

Stop the machine from deciding your benchmark. Configure and validate the container and runtime resources for an agentic coding eval so infrastructure noise stays inside statistical bounds instead of swinging scores more than the models do. Use this whenever someone runs SWE-bench or any agentic coding benchmark in containers, sees scores jump between runs for no code reason, suspects OOM kills or flaky infra are skewing results, sets container memory or CPU limits for an eval harness, or wants to trust a leaderboard delta. Trigger on "my benchmark scores are inconsistent," "OOM during eval," "how much memory should the eval container get," and similar. Not for designing the eval tasks or graders themselves; that's build-agent-evals.

coding-agent-scaffold

Design the tool interface for a coding agent so the model stops misusing it. Covers the minimal two-tool scaffold (a bash tool plus a file editor), exact single-match string replacement, absolute-path rules, and error-proofing the tool descriptions so common model mistakes become impossible. Use this whenever someone is building a coding agent or SWE-bench-style harness, designing a bash or file-edit tool for an agent, deciding how much scaffolding to impose, or debugging an agent that keeps editing the wrong place, fumbling multi-line edits, or escaping shell commands wrong. Trigger on "build a coding agent," "str_replace tool," "agent keeps breaking the file," and similar. Not for general MCP or service tool design; this is the bash plus file-editor interface specifically.

constitutional-classifiers

Screen an agent's input and output against a policy you write, so it refuses the content classes you disallow without over-refusing the ones you allow. Covers writing a constitution that lists allowed and disallowed content for your app, screening user input before the model and model output before delivery, generating synthetic examples from the constitution to test and harden the screen, setting a stricter policy for an autonomous agent than for a chat assistant, and tracking the over-block rate. Use this when someone needs content guardrails on an agent, wants to block a category of request or output, hardens an agent against jailbreaks, or deploys an agent in an abuse-prone or regulated domain. Trigger on "content policy for my agent," "block disallowed output," "jailbreak defense," "input and output filtering," and similar. This is policy screening of agent I/O; keeping the agent from obeying instructions hidden in the content it reads is defending-against-prompt-injection.

defending-against-prompt-injection

Keep an agent from obeying instructions hidden in the content it reads. Covers placing fetched pages, emails, and API responses in tool_result blocks instead of the system prompt; wrapping that content as JSON with explicit source fields; labeling its provenance; screening tool output with a fast classifier before the agent acts; and stating an untrusted-content policy in the system prompt. Use this when someone builds an agent that reads the web, email, shared files, or any third-party text, asks how to stop indirect or cross-content prompt injection, or finds an agent following instructions buried in a fetched page. Trigger on "indirect prompt injection," "agent followed instructions in a web page," "untrusted tool output," and similar. This is content handling inside the context window; limiting the damage once an attack lands is environment containment (sandboxing-agentic-systems), and gating outbound commands is action gating.

durable-agent-architecture

Structure a long-lived agent service so any part can crash and resume. Decompose it into brain (model plus harness), hands (ephemeral sandbox and tools), and session (a durable event log), each replaceable on its own, with wake/resume semantics and credentials kept out of the execution environment. Use this whenever someone designs a production or long-running agent backend, asks how to make agents crash-recoverable or resumable, worries about losing session state when a container dies, needs to scale agents as a service, or asks where to keep credentials for an agent that runs code. Trigger on "agent infrastructure," "resume an agent after a crash," "agent runs for hours," "where do tokens live," and similar. Not for parallelizing work across agents or coordinating a shared repo; see multi-agent-orchestration and parallel-autonomous-agents.

multi-agent-orchestration

Run an orchestrator-worker system for breadth-first research: a lead agent plans, spawns three to five subagents with their own context windows, and synthesizes their findings. Covers when multi-agent actually beats a single agent and when it just burns tokens, how to delegate so subagents do not overlap, broad-to-narrow search, writing findings to a filesystem, and how to evaluate the system. Use this whenever someone wants to parallelize research or exploration across agents, asks how to coordinate a lead and subagents, considers a multi-agent setup, or asks whether multi-agent is worth it for their task. Trigger on "orchestrator and workers," "parallel research agents," "lead agent spawns subagents," "should this be multi-agent," and similar.

parallel-autonomous-agents

Coordinate several unsupervised agents working on one shared git repo without collisions. Covers the autonomy loop that lets each agent pick the next task and respawn without a human, file-based lock files that claim work, machine- readable test output so a test suite steers the agents instead of a person, and context hygiene for long unattended runs. Use this whenever someone wants multiple agents grinding on one codebase in parallel, asks how to stop agents from duplicating work or clobbering each other, sets up an unattended or overnight agent run, or asks how agents claim and release tasks. Trigger on "parallel agents on one repo," "agents keep doing the same task," "autonomy loop," "unsupervised agents," and similar. Not for breadth-first research across subagents (that's multi-agent-orchestration) or crash-recovery architecture (that's durable-agent-architecture).

reward-hacking-and-inoculation

Keep an agent from gaming a checkable objective instead of doing the work. An agent told to make tests pass may delete the failing test, hard-code the expected output, or special-case the grader; the task looks done and its intent is defeated. Covers spotting reward-hacking in transcripts and evals, inoculation prompting that names when a shortcut is allowed so the agent does not learn that cheating generalizes, designing graders and tasks that do not pay out for shortcuts, and checking downstream behavior. Use this when an agent passes evals but ships wrong work, edits or skips tests to go green, satisfies the letter of a goal against its intent, or when someone designs rewards or graders for an agent. Trigger on "reward hacking," "agent cheats the test," "agent gamed the grader," "specification gaming," and similar. This is keeping the objective honest; measuring an agent in general is build-agent-evals.

sandboxing-agentic-systems

Contain an agent that runs code or reads untrusted content, layer by layer. Covers OS-level filesystem and network isolation that also catches spawned subprocesses, an egress proxy that checks request provenance, treating tool outputs and fetched pages as prompt-injection vectors, and keeping credentials outside the sandbox behind a proxy. Use this whenever someone designs a sandbox for a coding or computer-use agent, asks how to safely let an agent run shell commands or browse, worries about prompt injection from tool results, needs to limit what an autonomous agent can reach or delete, or asks where an agent's credentials should live. Trigger on "sandbox the agent," "agent runs untrusted code," "prompt injection," "restrict network access," and similar. This is environment containment; deciding which actions need approval before they run is a separate concern (action gating).

using-the-think-step

Decide where an agent should reason during a task and how to prompt for it. Covers three places a model can think and how to choose between them: extended thinking before the turn, interleaved thinking between tool calls, and the no-op "think" tool that logs reasoning mid-chain at a point you pick. Also covers when a mid-task step helps, when it only adds latency, the contract interleaved thinking imposes on the request, and why the think tool's gains come from system-prompt guidance rather than the tool itself. Use this whenever someone builds an agent that follows policies or makes long sequential decisions, asks about a think or scratchpad tool or interleaved thinking, finds an agent skipping rules or mishandling tool output mid-task, or wants more deliberate tool use. Trigger on "think tool," "interleaved thinking," "reasoning between tool calls," "let the agent reason before acting," "agent ignores the policy," and similar. Not for general prompt engineering or one-shot chain-of-thought.