Back to library
NetflixNetflix

A/B Test Analysis — Ship or Kill?

Data & Analytics

What it tests

Statistical reasoning, business judgment, and ability to make a clear recommendation when data is ambiguous

Format

  1. 1Candidate receives real A/B test results with deliberately conflicting signals: conversion up but revenue down, mobile vs desktop split
  2. 2Must write structured analysis with clear ship/kill recommendation
  3. 3Must respond to conflicting stakeholder opinions (PM vs revenue analyst) with their actual position

What to look for

  • Do they catch the Simpson's Paradox in the mobile/desktop split?
  • Ship/kill recommendation: is it clear or fence-sitting?
  • Response to stakeholders: do they take a position or try to please everyone?
  • What they'd test next: is it informed by what they learned, or generic?

Adaptation guide

Replace the sample data with a real A/B test from your company. The more ambiguous the result, the more signal you get from the candidate's reasoning.

Full description

Format:

  1. Candidate receives A/B test results with conflicting signals
  2. Writes structured analysis with clear recommendation
  3. Must respond to conflicting PM vs analyst positions

Time: 90 minutes

What to look for:

  • Statistical literacy (Simpson's Paradox detection)
  • Clear recommendation, not fence-sitting
  • Stakeholder communication under disagreement

Adaptation: Use a real A/B test from your company with ambiguous results.

Use this template in NouSpark

Start with this assessment template and customize it for your hiring process.

Create Assessment