Effectiveness & the operating manual

Who this is for: Owner and Manager — requires protocols.engagement.read. Where: Protocols → Effectiveness

A play makes a claim. This is where the claim is checked.

Measured per version, not per play#

Effectiveness is scored against the version an engagement ran under, never against the play as a whole.

That matters more than it sounds. Suppose a trigger was firing far too widely, you noticed, and you tightened it. Score the play as one thing and the fix gets averaged into the problem it fixed, and the play looks mediocre forever. Score per version and you can see the exact moment it started working.

The same applies to a partial rollout: a version running at 40% is compared against the targets that did not receive it, and because the split is deterministic on a cohort hash, the two groups are stable rather than reshuffled on every evaluation.

The dual-lens report#

One report deserves calling out, because it answers a question most systems quietly destroy the evidence for.

When a guest books, they are sold as something — a segment, at a rate, through a channel. That is a fact about the sale, captured at sell time and never rewritten. Separately, the classification loop forms a behavioural hypothesis about the same guest from how they actually behave.

The dual-lens report puts those side by side. Where they disagree, you are usually looking at either a distribution problem or a segmentation assumption that stopped being true.

The sold segment is never overwritten when a guest is reclassified. Rewriting it would be the easy implementation and it would destroy the only report that can show the gap.

Where new plays come from#

You do not have to think of every play yourself.

Mining runs deterministically over the debrief corpus — the accumulated (what we did, what we said, what happened) tuples your team has been producing every time they answer a debrief — and looks for theme-to-outcome correlations. The strongest of those are drafted into candidates: real plays in the real schema, with their evidence stapled on.

A reviewer then promotes, edits, or rejects a candidate. Three things about that are worth being explicit about, because they are what makes running a drafter safe at all:

1. A candidate has no write path. The drafter can write candidates and nothing else. It cannot reach a running play, a guest profile or a customer record. A candidate does nothing whatsoever until a person promotes it.
2. A draft cannot make itself live. Whatever comes back is forced to draft status before it is stored, and a generated script that claims to be running is refused. There is no automatic promotion path anywhere in the system.
3. It is on the record. The model, token counts, cost and a one-way digest of the ask are recorded on the candidate and audited. A background model call is on the same trail as a human action.

Rejecting a candidate is not a dead end either — the reason feeds the next mining round.

The corpus that feeds all of this is read through a PII-minimised seam and carries its own retention window. See Governance.

The operating manual#

A promoted play is written in two registers at once: the machine schema the runtime executes, and a human SOP entry someone can read, teach to a new hire, and quote in a policy review.

Each register fails on its own, in a way most hotels will recognise:

• A schema nobody can read becomes a black box, and staff stop trusting it.
• A wiki page nobody executes drifts from what the system actually does, usually within a month.

Written together and pinned to the same version, neither can drift from the other without somebody noticing.

The prose is generated deterministically from the play and its evidence — no model call. A "why" sentence is only worth reading if the numbers in it came from the measurement rather than from a generator.

And it is human-owned: an entry is authored at promotion, and anyone may rewrite any part of it afterwards. Once edited, it is never silently regenerated over the top of your words.

What's next#

• Authoring plays — versions, envelopes and rollout.
• Governance — consent, minimisation and retention.
• Analytics — the property-level numbers a play is ultimately trying to move.