Human-in-the-loop theater
The wrong debate teams keep having: full automation versus full manual control.
High-performing teams do neither. They automate deterministic execution and reserve human decisions for risk boundaries where judgment materially changes the outcome.
The question isn't "should humans be involved?" It's "where does a human actually improve the decision?"
Human review adds real value when:
- There's an ambiguous risk tradeoff
- A policy exception is being handled
- A cross-team priority conflict needs resolution
- A customer-impacting incident is in context
It does not add value for repetitive deterministic checks that already have explicit pass/fail criteria. Asking for approvals there doesn't increase safety - inceases queue time.
Define approval boundaries as architecture decisions, not habits. Production promotion, security control overrides, temporary policy exceptions, emergency rollback bypasses - these are the real boundaries. Everything else should be workflow-driven.
In 2025 and 2026, another boundary belongs on that list: upstream trust changes.
If a workflow introduces a new third-party action, changes an action from commit SHA pinning back to a mutable tag, expands runner token scope, or opens a new secret-bearing execution path, that is not a routine plumbing edit. That is a governance event. Treat it like one.
Three governance states:
- Pass: all required automated gates succeeded
- Block: required gate failed or hard policy violation
- Escalate: insufficient automated evidence; human decision required
When escalation is needed, the handoff must be compact: what failed, which gates are impacted, evidence links, operational risk, recommended options. No log dumps. Reviewers need decision support, not archaeology.
The warning sign that something is wrong: shadow governance. Engineers saying "we got verbal approval" with no artifact. Manual rerun decisions not reflected in workflow records. Decision criteria differ across teams. If this is happening, your governance model isn't encoded strongly enough.
Don't jump from manual operations to full autonomy. Build trust through evidence:
- Observe mode - workflow recommends, humans decide.
- Assisted mode - workflow auto-executes low-risk gates, humans approve at boundaries.
- Governed autonomy - workflow handles most paths, escalates defined exceptions.
The recent supply-chain incidents make this progression even more important. Teams that jump straight to "the automation knows what to do" usually discover too late that the automation also happily executes poisoned dependencies, attacker-controlled tags, or compromised maintainer releases if nobody encoded the trust boundary in source.
What we built at Kolsetu: governance is compiled-in, not enforced by convention. allow_default_branch=false hard-fails the run if someone triggers orchestration from main or master - no exceptions, no verbal overrides. Every gate resolves to one of three explicit states: passed, skipped-by-input, or failed. strict_full_plan=true hard-fails the run if any defined gate doesn’t resolve to an explicit state - no silent passes. Escalation always includes a structured packet: what failed, which gates are impacted, evidence links, recommended options. The packet is non-negotiable, because without it escalation becomes a fire alarm with no address.
Observability built for a single job
Agentic workflows are often introduced as "smarter automation."
Operationally, they behave like distributed systems - multiple stages, async dependencies, partial failures, retries, timeouts, human handoffs.
If your observability model assumes a single linear job, incident handling will be slow. And if you've built gates 8–11 from Part 2 (language coverage, system metrics, record integrity, log hygiene), you've already created exactly this kind of distributed-state dependency. Language coverage evals, for instance, span multiple calls, multiple languages, session-level context. There's no single log line that tells you it passed.
Observe four layers:
- Orchestration layer - dispatch events, gate transitions, timing
- Worker execution layer - individual gate runs, inputs used, outputs produced
- Output/mutation layer - what actually changed and when
- Governance decision layer - pass/block/escalate decisions and their evidence
For every run, capture: run identity and correlation IDs, gate transitions with timestamps, dispatch and completion events, artifact references and hashes, output operations performed, final decision state and reason.
And for any workflow that can touch secrets, builds, packages, or deploy paths, capture the trust evidence too: which action SHAs actually ran, which package versions resolved, what outbound hosts were contacted, which token scopes were used, and whether the run crossed an approval boundary.
Without this, retries are blind and audits are expensive.
Build for replay, not just retry.
Retry is useful when transient failure is likely. Replay is essential when you need confidence in causal analysis - rerunning a segment with the same inputs, constraints, and policy context to understand why something failed, not just that it did.
"It works now, unknown why" is not an acceptable recovery state.
Classify failures consistently: input failure, policy failure, dependency failure, logic failure, output failure. This classification belongs in your summaries and alerts.
That dependency failure bucket is not theoretical anymore. It now covers exactly the scenarios teams keep getting burned by: a compromised action tag, a poisoned release artifact, suspicious runner memory access, or unexpected outbound traffic during a build step that "usually just installs tools."
Alerts should be actionable in under one minute: gate name, failure class, correlation ID, direct run link, suggested first action, and any abnormal trust signal (new network destination, unpinned dependency, token scope change, or unexpected execution path). If an alert doesn't help an engineer take the first step in under 60 seconds, it's incomplete.
After every meaningful incident, ask: was this preventable by source rules? Was failure surfaced at the right stage? What single instrumentation field would have shortened diagnosis? Codify the answer in source and tests. Not in memory.
What we built at Kolsetu: every gate in the pre-merge summary includes workflow name, run ID, run URL, and status. Gate fallback inputs are logged so reviewers see what the orchestration actually used, not just what was passed. When replay is needed, exact inputs, constraints, and gate execution order are fixed at compile time in the source .md - not reconstructed from memory. Failure mode is explicit: "Stop immediately on first hard failure and produce a concise failure report with run URLs and next action."
Operationally, the lesson from the last wave of supply-chain compromises is simple: if you cannot quickly answer "what exact upstream code executed, with what permissions, and what did it try to talk to?" then your observability is not ready for production automation.
Measuring activity instead of outcomes
Most teams measure the wrong things:
- Number of workflows created
- Number of agent runs
- Reduction in manual comments
These are activity metrics. The real question is whether delivery speed improved while quality and control held.
Three-layer metric model:
Layer 1 - Delivery performance:
- Merge lead time
- Time from PR ready to final decision
- Queue time at governance boundaries
Layer 2 - Quality and reliability:
- Escaped defects linked to merge process failures
- Reopen/revert rate for merged changes
- Failed gate rerun frequency
Layer 3 - Risk and governance:
- Policy violation rate
- Threat detection block rate
- Human intervention rate by category
- Exception approval count
Use both leading and lagging indicators. Leading (gate pass stability, prompt ambiguity defects, alert-to-action latency) to steer quickly. Lagging (incident rate, mean time to recover, stakeholder trust in automated merge readiness) to validate direction.
Set a baseline before rollout: prior 30-day merge decision timing, existing manual touchpoint count, current release incident profile. Measure at 30/60/90 days post-adoption. This gives a fair comparison and prevents narrative-driven reporting.
When performance improves, ask what drove it. Workflow design? Team behavior change? Reduced volume? Environment stabilization? Don't attribute everything to the agentic workflow.
After 90 days, "good" looks like:
- Lower merge decision latency
- Fewer manual handoff steps
- Stable or improved defect/revert profile
- Clear boundaries for human escalation
- Better auditability of release decisions
The only problem? If speed improves but risk signals degrade, you haven't solved the problem. You've made it faster to create it.
And that is exactly what these recent incidents exposed. Plenty of teams had fast pipelines. They did not have trustworthy pipelines.
What we built at Kolsetu: the pre-merge gate ends with a coverage report - 11 gates, each with explicit status (passed / skipped / failed). After 90 days, we know exactly which gates fail most, which are slowest, and where retries cluster. The signal isn’t “how many workflows ran” - it’s “how many branches cleared all 11 gates clean without a retry.” That’s what correlates with production stability.
The full series checklist - all 8 foundations:
- Explicit gate graph
- Source/compile managed workflow logic
- Security controls native, not bolted on - including immutable refs, tight permissions, and explicit trust boundaries
- Orchestration and worker responsibilities separated
- Prompts written as protocols
- Human approvals boundary-based, not habitual
- Replayable observability
- Measuring outcomes, not activity
Boom. If you can answer yes to all eight, your agentic workflow program is operating as an engineering system, not a demo.
Further reading:
