How the Biggest Names in Tech Test Product Changes

• A workshop held before 2025 CODE@MIT with academics and companies including Netflix, Meta, Pinterest, Airbnb and Uber weighed how well short-term experiments predict long-term results.
• Short tests usually point in the right direction, but they can mislead when content quality shifts, gaming platforms push monetization, or pricing tests skew toward early bookers.
• Industry teams favor simple, explainable proxies over complex models, and say one well-designed long-term experiment can teach more than a stack of historical data.

Every company wants to know how an important change will play out months or years down the road. But few can afford to wait that long to find out.

That challenge is the focus of a new working paper by a team including , an MIT Sloan professor and leader of the Data and Analytics research group at the MIT Initiative on the Digital Economy (IDE). Their paper, Evaluating for the Long Term: Learnings from Industry, is based on a daylong workshop conducted the day before last year’s (CODE@MIT). The workshop aimed at uncovering how accurate short-term experiments are in predicting long-term results.

The working paper’s authors drew on the insights of 26 experts from four universities and 15 companies that operate online platforms, including Netflix, Meta, Pinterest, Airbnb and Uber. Insights from these organizations offer a practical look at how the best experimentation teams in tech operate. They also reveal how short-term test results hold up over time, the traps that can cause unreliable data, and how tests should be structured for the best results.

How do companies test product changes?

Testing product changes is essential to understanding their potential for success, and doing it at scale can be relentless. The workshop revealed that large companies with at least 10 million monthly active users—they include Microsoft, Amazon, Google and Meta—run anywhere from 10,000 to hundreds of thousands of experiments a year.

Most of these tests are short:

• A typical experiment runs anywhere from a few days to a few months.
• The median test length from companies using third-party experimentation platforms is just one to three weeks.
• Yet for most companies, testing wraps up in under two weeks.
• Long tests—lasting three months or more—are reserved for major features. At least seven companies in the group of 15 say they’ve run tests this long.

Participants agreed that while they prioritize long-term business outcomes, it’s infeasible to run experiments long enough to observe those outcomes directly. Instead, the working paper says, the dominant industry practice is to run short-run experiments, then extrapolate the observed treatment effects to the long term.

Are short-term signals reliable?

Companies are always concerned that short-term test results won’t hold up. However, most participants said the signs presented in short-term tests do often align with long-term results. For example:

• Netflix compared launch decisions made at two weeks against decisions made at two months across 200 A/B tests. The decisions agreed 95% of the time.
• After Microsoft reviewed its experiment history, the company couldn’t find a single case where a statistically significant result in one direction later flipped to the other.
• When Pinterest tested a badging treatment, it sharply increased daily active users soon after the experiment. While the magnitude of the treatment effect decreased over time (from 7% to 2.5%), the long-run results were still positive.

Where short-term data misleads

That said, there are times when short- and long-term decisions point in opposite directions. As the working paper explains, participants flagged three specific situations where this is likely to happen:

• Confusing signals from changing content quality or relevance. A previous industry white paper found that content-quality signals fail to correlate with predicted engagement in the short term, even though retention improved in the long term. In other words, a change that initially looks like a step backward can over time build trust. For example, when Facebook filtered notifications by relevance for certain users, the time spent by those users on the social media site initially declined. But after a year, the time those users spent on Facebook rose when compared with a control group.
• Hyper-monetization on gaming platforms. When gaming platforms make product changes that extract more fees from existing players, the companies increase their revenue in the short run. But over the long run, these same changes can instead erode revenue due to lost user retention.
• Pricing experiments affected by lead-day bias. On travel and hospitality sites, how far in advance a booking is made matters. That’s because travelers who book early behave differently from those who book at the last minute. For this reason, a short test tends to overrepresent whichever group happens to book during the early window, and sometimes it flips the long-term effect entirely. For example, Airbnb—which typically runs pricing algorithm tests for about four months—found that lead-day bias can cause the short-run effect to differ entirely from that of the long-run.

Two related nuances came up repeatedly. One is novelty effects, where a treatment’s impact shifts noticeably in just the first few days of a test. While this is common, participants differed on how often it leads to a changed launch decision. One platform discards the first few days of data by default, while others felt novelty effects rarely move the outcome.

A second nuance is retention-based metrics, such as daily active users. These are especially prone to delayed reveals. On social media platforms, for example, the effect of a content ranking change was described as having a “half-life” of roughly one month. In other words, it took about one month for a short-term effect to reach half its eventual long-term effect.

Overall, these examples reveal that short-term numbers, while helpful, can fail to provide the full story.

Platform leaders share best practices

You might think the fix for short-term uncertainty is a more sophisticated model that weighs dozens of signals to predict the long-term outcome more precisely. But participants reported that the opposite approach works even better.

In theory, complex, multivariable models can outperform simple ones. But if the team cannot explain why a metric moved, then no one will trust the model. Instead, participants consistently favor plain, explainable proxies. These offer metrics that people both believe in and use.

Continual testing is important, too. Correlations in past data are prone to bias; often, they don’t hold up in a new context. Even a single well-designed long-term experiment can teach a team more than a stack of historical data that’s been cleverly crunched.

Digital experimentation and what’s still unsolved?

As the new working paper explains, the workshop couldn’t provide all the answers. Indeed, participants identified a few challenges that remain open:

• Products keep evolving mid-experiment. A change being tested often isn’t fixed. Instead, it’s a moving target that’s continuously refined, even while an experiment runs.
• Early users aren’t always representative. Early adopters may engage differently than most users. For example, the people who respond to your two-week test may look nothing like your full user base six months later.

Fortunately, the discussion continues. Practitioners, academics and platform leaders will pick up the conversation when they gather in Cambridge, Mass., for the 2026 CODE@MIT conference on digital experimentation at MIT on Nov. 13-14, 2026. Join us: Registration is open now.

Meanwhile, you can read the full working paper – Evaluating for the Long Term: Learnings from the Industry, by Dean Eckles et al.

The post appeared first on MIT Initiative on the Digital Economy.