Holdout / pilot

Before you convert everything, design a small live trial with a comparison arm, plus a deliberate run in degraded mode. Name in advance what result would count as the whole system improving, not just one task feeling faster.

Before you convert everything, design a small live trial with a comparison arm, plus a deliberate run in degraded mode. Name in advance what result would count as the whole system improving, not just one task feeling faster.

Example

Run unit three of the licensing material in the app and unit four on paper index cards for two weeks. Then take a timed 40-question quiz on both. Also try one session with the app's summarizer off, to see what it was actually carrying.