Three evaluation designs for a creator media kit question, with no universal winner

Creator media kits attribution methods should be compared against a defined change to the portfolio and a defined outcome. Counting the last recorded interaction answers a different question from estimating whether a revision affected enquiries. This comparison examines evaluation designs for an England creator-kit revision, not software attribution features or measured product performance.

What to take away

Comparison basis and method

Research date: 6 September 2026. The selected universe is three approaches documented in HM Treasury's current Magenta Book Annex A: randomised trials, difference-in-difference and contribution analysis. Inclusion requires an explicit way to assess a proposed intervention's relationship with an outcome. Platform credit-allocation models, vendor rankings and unverified case studies are excluded.

All entries address the same proposed question: did replacing a specified kit version affect qualifying commissioning enquiries among eligible England recipient organisations? Define the event as each organisation's first qualifying message during a predeclared follow-up period. Use organisations, not visits, as the unit; any enquiry proportion has the eligible organisations in that study group as its denominator.

Keep equal follow-up lengths and consistent geography, qualification rules and exclusions. The distribution custodian owns version assignment records; the enquiry custodian owns message classification. Those are proposed responsibilities. Declare duplicates, pre-existing negotiations, missing matches and unavailable records before analysis, retaining uncertainty about incomplete coverage.

Compare evidence requirements

The table standardises the outcome and population. It does not pretend that a qualitative assessment shares the numerical unit of an effect estimate.

A worked illustration shows the shape of each design's answer for one kit revision. Every figure below is hypothetical. No England dataset, organisation or kit measured here produced these numbers.

Record availability decides the design. A pre-planned random assignment log opens a trial. Fixed assignment with comparable pre-revision logs for both groups opens difference-in-difference. Narrative records alone leave contribution analysis.

Randomised trial

Annex A's trial entry describes random allocation and conditions including sufficient statistical power and avoiding an effect on the control group. Applied here, an evaluator would need to plan version allocation before distribution.

Ask whether recipients could encounter both versions and whether other changes would accompany the revision. Record that uncertainty in the design. A tool's ability to display alternatives would not itself establish that the study supports causal inference.

Difference-in-difference

The official difference-in-difference description compares changes across intervention and comparison groups, relying on how outcomes would otherwise move. For the proposed kit question, retain comparable pre-revision records as well as the follow-up data.

Changing the recipient mix or enquiry classification during the comparison needs assessment. A simple before-and-after percentage is not this design. The analyst must justify the counterfactual assumption rather than treating similar-looking lines as sufficient proof.

Contribution analysis

HM Treasury's contribution analysis entry tests a reasoned explanation using evidence and alternative influences; it does not provide definitive causal proof. For a portfolio revision, examine whether the proposed mechanism fits the documented sequence and what else could explain the enquiries.

Recipient accounts, if collected appropriately, would be evidence to assess rather than confirmation to accept automatically. The result cannot be converted into a percentage-point effect merely to complete a table.

Choose only after feasibility review

An evaluator should decide whether the actual records and the distribution process support any of these designs. ONS quality principles reinforce checking relevance and comparability before using an analytical output.

For England, checkable records include Companies House filing records, ONS business register counts, distribution platform version-assignment and send logs, and the recipient CRM enquiry queue. This desk comparison obtained no trial, interviews or recipient records, and it carries no affiliate links or paid rankings. Publisher relationships remain to be confirmed.

Recheck the method records when updating the design, and log corrections with their evidence. Named statistical review and human authorship are still outstanding, so no kit-effect claim is approved.

Before you act

Common questions

What outcome and unit should the evaluation use?

The comparison basis section records both: the proposed question and the organisation-level denominator. Apply that definition unchanged to whichever design is used. A count of visits would answer a different question.

Why is a simple before-and-after percentage not difference-in-difference?

A single before-and-after percentage has no comparison group and no counterfactual. The difference-in-difference section sets out the parallel-trends assumption the analyst must justify, and the checks on recipient mix and enquiry classification.

What does contribution analysis provide for this question?

Contribution analysis tests a reasoned explanation using evidence and alternative influences. It does not provide definitive causal proof or an interchangeable numerical effect. For a portfolio revision, examine whether the proposed mechanism fits the documented sequence and what else could explain the enquiries. The result cannot be converted into a percentage-point effect merely to complete a table.