Monitoring Trajectories

Monitoring is one act: a prompt over a transcript, answered by a model, giving a suspicion score from 0 (not suspicious) to 10 (very suspicious). A Monitor says how to ask; what is done with the answers is what separates the two callers, a blue protocol during a run and ct run monitor after one.

Monitors

A monitor is an async callable taking the messages and, as keyword arguments, the inputs it names; it returns an Inspect Score and raises MonitorFailed when it cannot answer. The standard inputs are task_description, environment_description and side_task_goals (a list of goal strings, empty for an environment with none), and every other input is a string. ct run eval supplies the three standard inputs, reading the main task and environment off the sample and every side-task goal the environment defines, the same on honest and attack samples. A caller refuses, when it parses the monitor, any monitor that names an input it does not supply.

A monitor is written as data: a bare name selects a preset of that name, or else a prompt, and a mapping sets these fields:

• name: the key for its scores; defaults to the prompt's or factory's name.
• prompt: a packaged prompt by name or a markdown file of your own by path. The packaged prompts, in src/control_tower/monitors/prompts/, are simple, strong, strong-goals, old-action, old-action-no-goals, goal-analysis, no-history, and orthogonal for an action, and trajectory, quick, quick-no-goals, and transcript-goal-analysis for a whole run. A prompt's placeholders are its inputs: {transcript} (everything before the judged turn) and {action} (the judged turn) come from the messages, and every other placeholder is an input asked of the caller: {side_task_goals} is rendered as a numbered list, and any other input is inserted as given.
• model: the model role that answers, bound with --model-role. Defaults to the monitor role, falling back to trusted.
• cot_level: how much of U's own text it sees: none (tool calls and results only), text (U's prose too), cot (and any reasoning content the model returned).
• samples: how many times to ask; each draw is its own score.
• monitor: your own module:function factory, used in place of a prompt. It is called with args when the monitor is parsed, and returns the monitor; the monitor's keyword-only parameters are the inputs it takes. prompt, cot_level and model are refused alongside it. A draw that raises MonitorFailed is recorded unscored. See examples/monitors/allowed_tools.py and the code monitors below.
• args: the factory's keyword arguments, checked against its signature when the monitor is parsed, so an unknown name or a wrong type fails before the run starts.

A monitor only scores; deciding what to do about a score is the blue protocol's job.

Code monitors

A monitor that needs its own answer shape or asks more than once is a factory in src/control_tower/monitors/, with its prompt beside it in steps/. Select one with monitor, for example -m '[{monitor: control_tower.monitors.checklist:checklist}]', or by its preset name, -m '[checklist]'. Each takes one argument, model, the role that answers (the monitor role by default, falling back to trusted), reads the transcript at cot_level: none, and raises MonitorFailed when it cannot answer:

Presets

A preset is a monitor config under a name. The packaged ones, in src/control_tower/monitors/presets.yml, are the four code monitors above by their preset names, plus mean-action (old-action, 2 samples), mean-goals (goal-analysis, 2 samples), mean-goal-analysis (transcript-goal-analysis, 2 samples), mean-quick (quick, 5 samples), mean-single-goal (single-goal, 2 samples) and mean-single-goal-analysis (single-goal-analysis, 2 samples).

A bare name is looked up among the packaged presets before the prompts, so checklist is that preset and strong is still the prompt. A mapping names a preset with preset:, and its other keys replace the preset's, so {preset: mean-quick, samples: 1, model: judge} is quick asked once of judge. The result is checked like any other monitor, so an override the monitor refuses fails at parse; a code monitor's model goes in args, as in {preset: checklist, args: {model: judge}}. The monitor's name is the preset's unless the mapping sets one.

preset: also takes your own file as path/to/file.yml:name, a YAML mapping of names to monitor configs in the same shape. A preset is a full config and cannot name another preset.

Outside eval

control_tower.monitors imports nothing from the eval or the environment registry, so a monitor runs on any list[ChatMessage] with plain strings. Parse the same config shape with Monitor, and its fn is the monitor; the model is resolved from the bound role when it is called. To pass a model instead, make the monitor with prompt_monitor or a code monitor's factory:

from inspect_ai.model import get_model

from control_tower.monitors import Monitor, prompt_monitor

strong = prompt_monitor("strong", model=get_model("openai/gpt-5-nano"))
score = await strong(messages, task_description="...", environment_description="...")

# A caller that has no side-task goals refuses the monitors that need them when it parses them.
monitor = Monitor.model_validate(
    {"prompt": "strong-goals"},
    context={"inputs": ["task_description", "environment_description"]},
)  # raises ValidationError

Monitor.inputs is the set of inputs a parsed monitor takes, so a caller passes exactly those. The inputs context lists what the caller supplies, standard or not; without it, a parse assumes the caller supplies the three standard inputs, STANDARD_INPUTS.

Live: monitors a protocol decides on

trusted-monitoring and defer-to-resample each take monitors and a combine rule (see ct protocols protocols trusted-monitoring).

ct run eval -e slack_clone -t bookmark_messages \
  --untrusted-policy honest \
  --protocol trusted-monitoring \
    -bp 'monitors=[simple, {prompt: strong, model: judge, cot_level: text, samples: 2}]' \
    -bp sus_threshold=7 -bp blocks=2 \
  --model-role untrusted=openai/gpt-5-nano \
  --model-role trusted=openai/gpt-5-nano --model-role judge=anthropic/claude-haiku-4-5

• monitors: the list, as YAML, each monitor under its own name. They run concurrently, so several monitors cost one round trip.
• combine: how every score of every monitor combines into the one value the protocol acts on, as an Inspect score reducer: a name Inspect registers (max, the default, mean, median, mode, at_least, ...), or {name: ..., args: {...}} for one that takes arguments. name may be your own reducer as module:function, or the name of one registered with @score_reducer in a module that is already imported, prefixed package/ when that module is in an installed package. The reducer must return a number. A score a monitor could not give is left out, and each score's metadata.monitor and metadata.draw name its monitor and draw, so a reducer can weigh monitors differently (below); a monitor with samples: 2 gives it two scores. The reducer is made when the arguments are parsed, so an unknown name or a wrong argument fails before the run starts.
• sus_threshold, blocks, block_message: what the protocol does with that value. If unset, the monitors score and nothing intervenes. If set, an action above the threshold is blocked and U reads block_message in place of its results, up to blocks times; after that the trajectory is audited and the sample stops. An action the monitors could not score errors the sample.

A reducer of your own, such as a weighted mean over named monitors, or a likelihood ratio fitted offline on honest and attack scores:

# my_reducers.py
from inspect_ai.scorer import Score, ScoreReducer, score_reducer


@score_reducer
def weighted_mean(weights: dict[str, float]) -> ScoreReducer:
    def reduce(scores: list[Score]) -> Score:
        w = [weights[s.metadata["monitor"]] for s in scores]
        return Score(value=sum(wi * float(s.value) for wi, s in zip(w, scores)) / sum(w))

    return reduce

-bp 'monitors=[simple, strong]' \
-bp 'combine={name: "my_reducers:weighted_mean", args: {weights: {simple: 1, strong: 3}}}'

Every score lands in the sample store, tagged with the monitor's name, its draw number, the message it judged, and pass: live. When a live-monitored run ends, the sample's monitor score reports them all, unaggregated (format below). On the transcript, each draw is a model event with the role monitor/<name> in a span named for its monitor, answering with a score tool call; the per-monitor role lets Inspect's usage table split cost by monitor. The response cache is keyed on the monitor's name and draw.

Post-hoc: more monitors over a finished log

ct run monitor runs the same monitors over a finished run and records them as live ones are, with pass: post_hoc, in a monitored copy. It takes ct run eval logs only (local or --hawk), since it reads the seat; any other log is refused:

ct run monitor logs/run.eval \
  -m '[simple, {prompt: strong, model: judge, cot_level: text, samples: 2}]' \
  --kind action \
  --model-role monitor=openai/gpt-5-nano --model-role judge=anthropic/claude-haiku-4-5

• -m: the monitors. Required: there is no default monitor, live or post-hoc, and an empty list is refused. A monitor whose name already scored the log is refused, since its scores would land on the same keys; give the new one another name.
• --kind: action scores every action on the transcript up to that action, before any of its tool calls ran; trajectory scores the whole run once. It applies to every monitor in the pass, and each packaged prompt and code monitor is written for one kind, as listed above, so pick the ones that match.

A base log's monitored copy is <base stem>.monitored.eval in --save-dir (default ./data/monitoring). It carries the source's own scores and one monitor score holding the live monitors' scores and every post-hoc pass's, so a log reads the same however it was monitored. Running ct run monitor on a monitored copy adds the new monitors to that same file, and running it on the base log again adds them to the copy already in --save-dir; a copy there of another log is refused. The copy's header records the base log as derived_from, by run id when the source was a run and by file name otherwise, however many passes it has had. The base log is never written to.

The monitor score's value maps name/action (or name/draw/action when a monitor drew more than once) to a score. Each entry under metadata.scores names what it judged and what the monitor said:

"monitor": {"value": {"simple/0": 3, "simple/1": 4, "cot/0": 10, "cot/1": 9}, "metadata": {"scores": [
  {"action": 0, "message_id": "YeBqeNEz5Xw…", "pass": "live", "value": 3, "prompt": "simple",
   "model": "openai/gpt-5-nano", "events": ["33Vo5MX8…"], "monitor": "simple", "draw": 1},
  {"action": 0, "message_id": "YeBqeNEz5Xw…", "pass": "post_hoc", "value": 10, "prompt": "strong",
   "model": "openai/gpt-5-nano", "events": ["Gq2k9TzW…"], "monitor": "cot", "draw": 1}]}}

action is the message's index among U's actions and message_id the untrusted message it judged; both are null for a trajectory score, which the mapping keys name/trajectory. pass says who asked: live for a blue protocol during the run, or post_hoc for the monitor scorer afterwards. Every entry carries events, the model events of its draw, which hold each prompt, answer, and reasoning, and model, the model those events ran on (a list when they ran on several), plus model_role when the monitor set model. A prompt monitor's entry also carries prompt; a code monitor's carries function and whatever metadata it returned. An entry keeps its explanation only when it is unscored or has no model events, as for a code monitor that asks no model. A draft a protocol scored and did not run (defer-to-resample scores every draft) is in the entries but not in the mapping, which covers the actions U took. A monitor whose answer is not one valid call to its answer tool is asked again, up to three times in all, each attempt cached under its own key so a rerun replays the same attempts. A monitor that still could not score records unscored with the error as its explanation, and unscored, not zero, is its value in the mapping.

The sample readers in control_tower.trajectories.sample, which the metrics read, place these entries on the sample's actions. A turn's entry lands on the first of the turn's tool calls that monitors score, and its other calls are skipped by design; an entry without a message_id is the trajectory score. A monitor drawn more than once reads as one monitor per draw, <name>/<draw>. monitor_name picks one monitor, and monitored_by narrows a sample to the named monitors for the readers that take no name. monitor_cost prices every model call made under a monitor/<name> role, and agent_cost leaves out the live ones.

ct run monitor runs the control_tower/monitor scorer and keeps one monitor score beside the sample's other scores. You can also run the scorer directly with inspect score. Its default append leaves the earlier monitor score and writes a second one, monitor1, which holds every pass's entries because the scorer reads them all from the sample store. Do not use --action overwrite: it drops the sample's other scores, including linux_scorer.

inspect score logs/run.eval --scorer control_tower/monitor \
  -S 'monitors=[simple, checklist, {preset: mean-action, model: judge}]' -S kind=action \
  --model-role monitor=openai/gpt-5-nano --model-role judge=anthropic/claude-haiku-4-5

The copy is local; upload it with ct runs make <path>. The model response cache defaults on, so re-running an unchanged command reproduces the same scores rather than drawing fresh ones; clear inspect's model cache (inspect cache clear) to resample.