F3 Lightning: How Meta Accelerates Ads ML Features to Production at Scale

• Facebook Feature Framework (F3) is Meta’s framework for authoring, compiling, and serving the machine learning features underlying ads ranking models, where a single request can require more than 20,000 features. We first presented F3 at AI @ Scale 2020.
• F3 reduced execution computational costs by compiling all features into a single merged graph, but that graph also served as the release unit, so a single failure or invalid definition blocked the entire release pipeline.
• This post introduces F3 Lightning, an infrastructure layer built on top of F3 to resolve this challenge by decoupling feature releases from shared execution, so each feature is compiled, validated, deployed, and rolled back independently.
• The hardest part to solve was preserving F3’s efficiency improvements. Early per-feature release prototypes increased CPU usage because independent features no longer shared computation. We aimed to close this gap through four optimizations: stable computation identifiers, a single shared execution plan, one shared memory buffer across all features, and dense index mapping for direct array access.
• With F3 Lightning, comparable extraction features now reach production ~3.6x faster than the prior F3, with no increase in infrastructure compute cost. This shorter release cycle also enables more model iterations in the same period.

A Primer on F3’s Shared Execution Architecture

Production machine learning systems rarely consume raw data directly. Before a model can make a prediction, its feature-serving system may need to read stored signals, filter events to a time window, count or sum values, normalize them, and combine intermediate results.

The simplest design to complete this work treats every feature as an independent function. That is easy to understand, but it becomes wasteful at scale, because features are not computationally independent. A seven-day click count and a seven-day impression count may read the same event source and apply the same time-window filter. A click-through rate feature may then reuse both results. Running each feature separately would repeat that filter, and every other shared step, on every request.

F3 approached this differently, by representing feature extraction as a directed acyclic graph (DAG). Each node performs one unit of work, such as reading an input, applying a filter, or calculating an aggregation. An edge means one computation depends on the result of another. The graph’s outputs are the feature values a model consumes.

Engineers author each feature independently in a high-level language. The F3 compiler turns each definition into a small graph, then merges the graphs for every feature a service uses into one merged graph (Figure 1). Where features share an operation, such as the seven-day filter above, the compiler collapses the duplicates into a single node. That shared work then runs once per request, not once for every feature that needs it. At runtime, feature extraction is a traversal of this merged graph.

The Shared Graph Made Every Release All-or-Nothing

F3’s merged graph shared computation efficiently, but it came with a major catch: The graph was also the release, and one malformed definition could hold back every other feature in the merged graph.

Figure 2 illustrates this challenge. It shows seven independently authored features, A through G, where five definitions are valid, but D and G contain unrelated errors. The “Attempt 1” build fails on D, so none of the seven features ships. After D is corrected, the “Attempt 2” build reaches G and fails again. Only after both definitions are fixed in “Attempt 3” does the complete merged graph make it successfully to production.

Failures became more frequent as F3 scaled. F3 originally served a few hundred features, and at that size merges succeeded and releases were largely smooth. Today that number is more than 20,000, and different feature types need their own checks depending on how they compute, so new features and modified features are held to different testing bars. A feature that a model already consumes needs model-related tests before it can be released. Those tests are computationally expensive enough that they run only in a later release stage, and, because the merged graph ships as one unit, every other feature in it waits for them too.

F3 Lightning: Making Each Feature Its Own Release and Sharing Computation

After F3 launched, we made several rounds of improvements to the merged graph. Each one brought back release speed for a while, but the gains eroded as more features landed, because all features still shipped together in the merged graph. So F3 Lightning was built to make every feature releasable on its own while preserving F3’s efficiency.

As illustrated in Figure 3, F3 Lightning introduced two changes to help achieve this:

• Deployment granularity: Each feature is an independently releasable configuration. It is validated, shipped, and rolled back without affecting any other feature.
• Efficiency parity: F3’s efficiency came from the merged graph, so giving every feature its own release removes the source of it. Matching that efficiency while keeping per-feature releases required several key optimizations in how F3 Lightning executes.

Deployment Granularity: How F3 Lightning Isolates Feature Releases

Isolation starts in the compiler, where each feature is compiled on its own into a single configuration. That feature configuration carries everything needed to extract that feature — every node it needs, in order, with nothing borrowed from a neighbor. If feature C has a problem, feature A still ships, because the two configurations have no relationship to each other.

Because features are no longer merged at release time, each one can be selected and released on its own schedule. The release path becomes an ordered queue instead of a batch that ships all at once or not at all. Changes are prioritized by what they unblock. A new feature that no model consumes yet can go out quickly on standard validation, while a change to a feature a live model already reads gets extra testing first.

Efficiency Parity: How F3 Lightning Preserved F3’s Serving Efficiency

Our first prototypes separated deployment only. Features shipped independently, but execution was no longer shared, so the prototypes used substantially more CPU than traditional F3. Each independently released configuration arrived at the serving process as its own graph and was executed separately, repeating every read, filter, and aggregation that F3’s merged graph had eliminated.

The fix was to bring shared execution back without bringing the merge back. At startup, F3 Lightning does not rebuild a merged graph. It takes the active feature configurations, consolidates the work they share, and arranges it into a single execution order. Getting back to parity took four steps: a compiler prerequisite followed by three startup optimizations.

Step 1: Stable Identifiers Make Shared Work Recognizable

To deduplicate execution, the runtime must recognize when two independently compiled configurations contain the same computation. Node numbers can’t do this, because numbering is local to each feature’s graph. The same seven-day filter might be node 2 in one configuration and node 7 in another, and inserting a single node renumbers everything after it. Traditional F3 never had this problem, because one compiler run saw every feature and could assign identifiers globally.

F3 Lightning instead has the compiler derive each identifier from what the work does, not where it sits: the operation, its parameters, and the identifiers of its inputs (see Figure 6). Because each input’s identifier already reflects everything upstream of it, the identifier captures the whole chain of work that produced the result. Where input order doesn’t affect the result, inputs are put in a standard order. Aggregations also include the properties that define their behavior, such as the time window and grouping keys. As a result, two features that define the same operation over the same inputs receive the same identifier, even if they were compiled and released weeks apart. If a computation’s meaning changes, its identifier changes with it.

Step 2: One Execution Plan Across Every Feature

At service startup, F3 Lightning builds a single execution plan across all active features, using a topological sort to place every node after the nodes it depends on. When several features contain a node with the same identifier, the plan keeps one copy and connects every consumer to it, so that computation runs once no matter how many features use it.

Merging guarantees each computation runs once, but the plan is still a graph that the executor walks pointer by pointer on every request. The same startup pass therefore flattens it into an ordered list, so each request runs a single forward pass down the list instead of traversing a graph.

Sharing one plan raises a risk: A single failing computation could affect every feature. To prevent this, if a computation fails or its input is unavailable, F3 Lightning marks everything downstream of it as not ready, and the executor skips that work. A failure affects only the nodes that depend on it, and independent branches keep running.

Step 3: A Shared Memory Buffer, Because Deduplication Was Not Enough

After steps 1 and 2 were complete, no computation ran twice, yet CPU use was still above traditional F3. The remaining compute cost came from memory layout. Each configuration carried its own request-time data container, so intermediate results were scattered across separate memory allocations. Walking the plan in order meant jumping between them, and each jump cost a cache miss.

F3 Lightning replaced those per-feature containers with one buffer per request. At startup, each feature’s references are rebound to slots in that buffer.

Three improvements follow from the single buffer:

Locality: Consecutive plan steps now touch adjacent memory, so the values a step reads are usually already in cache. This restores the sequential access pattern the merged graph always had. It also removes the per-feature allocation on every request.

One readiness vector: A single vector, stored apart from the values, replaces per-feature tracking, so the executor can check whether a value is ready without reading the value itself. The marker has to be separate because an empty list is a valid result, so “empty” can’t also mean “not computed yet.”

No duplicate copies: Most feature values are lists and maps, and consumers no longer get their own copy. The shared slot holds a reference to the producer’s result, so a computation used by 20 features is computed once and stored once. Single values (scalars) are still copied, because copying a small value is cheaper than following a pointer. This is safe because each buffer lasts only for one request, and the execution order guarantees each result outlives its last reader.

Step 4: Dense Index Mapping Removes the Last Per-Access Cost

Values now lived together, but finding one was still computationally expensive. Every access had to resolve an identifier, which meant hashing a key, probing buckets, and following a pointer. Lookups weren’t computationally efficient at Meta’s serving volume.

The final fix applied the same principle as steps 2 and 3, doing the work once at startup. F3 Lightning assigns every unique identifier a consecutive integer slot and rewrites every reference in every configuration to point at those slots, so identifiers are used only during initialization. Reading a result becomes a direct array index into the shared buffer, with no hashing and no lookup table on the request path.

With F3 Lightning, comparable extraction features now reach production ~3.6x faster than on traditional F3, with no increase in infrastructure computational cost. That shorter release cycle means models can be improved and retrained more often in the same period. Building F3 Lightning also taught us three lessons:

Deployment and execution should have separate boundaries: If one artifact determines both how components run and when they ship, a failure in any one of them blocks all the others. Splitting the two jobs removed a trade-off between efficiency and release speed.

Work that never changes between requests belongs at startup, not on the request path: Three tasks were identical for every request: spotting shared work, deciding the order to run it in, and finding where each result is stored. F3 Lightning now does them once, when the service starts, so independent releases add no computational cost to individual requests.

Two designs can execute the same operations in the same order and still differ substantially in CPU computational cost: In F3 Lightning, we removed every repeated computation, yet CPU use stayed above traditional F3. Each feature still kept its own copy of the shared results, spread across memory. Consolidating them into one copy per result closed the gap. This is the least obvious result in the project and the one that generalizes furthest. On modern CPUs, where results live and how many copies exist can matter as much as how many computations run.

Acknowledgments

We would like to thank Sourabh Bansal, Ping Chen, Weitao Chen, Benjamin Dummitt, Junhua Gu, Henry Hu, Jing Huo, Joffrey Liu, Rocky Liu, Weixi Ma, Shuby Mao, Sandeep Pandey, Mihir Rastogi, Tim Sombuntham, Matt Steiner, Vivienne Sung, Ritwik Tewari, Vivek Trehan, Arnaud Venet, Ziheng Wang, Gautam Wilkins, Fan Yang, Qiao Yang, Jie Zheng and the entire team behind the development and productionization of F3, Meta’s Ads ML feature computation and serving platform, and F3 Lightning, the capability built on top of it.

The post appeared first on Engineering at Meta.