A big problem with conventional experimentation or A/B testing is that the treatment effect can vary over time and it can depend on the entire history of the treatment. Usual comparisons miss this...

Andrea Bronzini points to this article and writes:

I’m working on experimentation tooling and wrote a brief note about a failure mode I keep seeing: “winning” A/B tests often lose uplift after rollout. My claim is that the typical setup mixes novelty response with steady-state effect because assignment happens after first exposure, and the outcome is time-dependent.

My reply: I think he is identifying a real problem. The problem he identifies is a special case of the issue that the treatment effect can vary over time and it can depend on the entire history of the treatment, and I guess that ultimately the best solution is to model these sources of variation.

I also like this bit from his webpage:

He saw how the industry’s obsession with certainty slowed businesses down… so he decided to fix it. . . . Confidence ≠ Certainty.
Data is never complete.
And that’s why we need to embrace taking decisions with risk.
Progress belongs to those who act while others wait.