Measuring human-in-the-loop: time to decision, rule coverage and fatigue
Time to decision at the median and 90th percentile, the share settled by rules, expired requests and signs of approval fatigue, with a script to compute them.
· 8 min read
An approval step can fail in two quiet ways. People take so long that agents give up and work stalls, or people approve so fast that the step is a formality. Neither shows up as an error. Four numbers show both, and all four come from the record you already keep.
1. Time to decision: the median and the 90th percentile
Measure from when the request was made to when a person decided it. Leave out what rules decided (that takes no time) and what expired or was withdrawn (nobody decided).
- The median is the usual wait. It tells you whether approvals fit the agent’s work at all.
- The 90th percentile is the slow tail, and it is what breaks agents. If it is near the time limit agents give their requests (15 minutes unless they say otherwise), the slow ones expire.
Look at both per action. A median of two minutes for refunds and two hours for bulk sends may be exactly right. A long tail for one action usually means its requests reach the wrong people, or arrive when nobody is watching.
2. The share answered by standing rules
Of all requests, how many did a rule decide? This is the measure of whether people are kept for judgement calls. Read it per action:
- Near 0%
- Every request reaches a person. Fine for rare, serious actions. For a frequent one, it is where fatigue starts.
- Rising slowly
- Allow rules are being added as the record earns them. The healthy direction for routine actions.
- Near 100%
- People no longer see this action. Check that its rule still has a limit and a budget you would sign today.
3. Expired requests
An expired request is safe: the agent did not act. It is still a failure, of the work. Causes are few and fixable:
- Nobody heard about it. Turn on email, Slack or browser notifications.
- It went to people who were away. Route the action to named people with escalation, so anyone may decide after a wait.
- The time limit is shorter than the work allows. A payment that two people must approve can be given a day.
4. Signs of approval fatigue
Fatigue has no single number, but it leaves marks in the record. Watch three, per action:
- Approval rate near 100%
- Over many requests, nothing is ever denied or edited. Either it should be a rule, or nobody is reading.
- Very fast approvals
- Approvals within a few seconds of arriving, for requests with details worth reading.
- Bursts
- Many approvals by one person within a minute: clearing the inbox, not deciding.
The cure for fatigue is fewer questions, not faster clicks. Move the action you always approve to a rule with a limit, and the remaining questions get the attention they need.
Computing them from the record
Download the record as CSV from the Record tab, then run this with Python 3.9 or newer, no packages needed:
"""Approval metrics per action from a VariantGate record export (CSV)."""
import csv
import math
import re
import sys
from collections import defaultdict
from datetime import datetime
from statistics import median
FAST = 5 # seconds: an approval this quick was barely read
FRACTION = re.compile(r"\.(\d+)")
def when(text):
text = FRACTION.sub(lambda m: "." + m.group(1)[:6].ljust(6, "0"), text)
return datetime.fromisoformat(text.replace("Z", "+00:00"))
def p90(values):
ordered = sorted(values)
return ordered[math.ceil(0.9 * len(ordered)) - 1]
rows_by_action = defaultdict(list)
with open(sys.argv[1], newline="", encoding="utf-8") as file:
for row in csv.DictReader(file):
rows_by_action[row["action"]].append(row)
print("action,requests,by_rule,expired,median_s,p90_s,approved,fast")
for action, rows in sorted(rows_by_action.items(), key=lambda kv: -len(kv[1])):
people = [r for r in rows if r["decided_by_rule"] == "no"
and r["status"] in ("approved", "denied")]
waits = [(when(r["settled_at"]) - when(r["created_at"])).total_seconds()
for r in people]
approved = [w for r, w in zip(people, waits) if r["status"] == "approved"]
by_rule = sum(r["decided_by_rule"] == "yes" for r in rows)
expired = sum(r["status"] == "expired" for r in rows)
print(",".join(str(x) for x in [
action, len(rows),
f"{by_rule / len(rows):.0%}", f"{expired / len(rows):.0%}",
round(median(waits)) if waits else "", round(p90(waits)) if waits else "",
f"{len(approved) / len(people):.0%}" if people else "",
f"{sum(w < FAST for w in approved) / len(approved):.0%}" if approved else "",
]))It uses the export’s action, status, decided_by_rule, created_at and settled_at columns. For a request that needs two people, settled_at is the last approval. One export holds the 10,000 most recent requests that were decided, expired or withdrawn; group by the key column instead of action to see the same numbers per agent.
For a quick look without a script, the Report per agent panel under Agents shows, for each agent key over 30 days, 90 days or 12 months, the median time a person took to decide and the share a rule decided.
How often to look
Weekly, while you are still adding rules; monthly once they settle. Each look should end in a change or a deliberate decision not to make one: a new allow rule, a lower limit, a routing rule, a longer time limit. Why rules are the answer to most of what you find is in approval fatigue and standing rules; what each column of the record means is on the record.