unconf.md

Starred Gists，GitHub Gist，即时分享代码、笔记与代码片段。关注

Skycache: Remote Analysis & Execution Caching in Bazel

This document is a Markdown export of the Skycache Bazel Unconference presentation, formatted for GitHub Discussions and Gists.

Agenda and Objectives

  • Gather requirements and use cases from the Bazel community.
  • Share details on Google's internal Bazel Skycache reference implementation and architecture.
  • Discuss whether a standardized Remote Analysis Caching API (RAC-API) is the right alignment and how to get there.
  • Gauge interest in forming a community-driven and community-owned RAC-API working group, where the Bazel team can share insights from our internal implementation and core Bazel serialization machinery and serve as a point of contact for design reviews.

Skycache at 30,000 Feet

Even with high remote execution cache hit rates, many teams working in large repositories with complex dependency graphs still experience multi-minute cold analysis times. Before any action can run or check for a remote execution cache hit, Bazel must load BUILD files, apply build configurations, evaluate Starlark rules to generate actions in memory, and pass providers up the dependency graph.

Skycache performs remote Skyframe caching of both analysis and execution SkyKey/SkyValue entries. By serving top-down cache hits directly in Skyframe, Skycache prunes transitive subgraphs and skips analysis and execution work before any action runs or queries the remote execution cache.

In production at Google, Skycache reduced average and P95 build wall times by more than 70% and lowered Bazel heap memory usage by more than 30% for onboarded user groups. Because Skycache is specifically suited for large monorepos and expensive analysis graphs, we want to ensure community efforts are driven by real-world use cases.

Already in Bazel

  • Serialization codecs with zstd compression and size-based chunking
  • SkyValueRetriever and FrontierSerializer for reading and writing Skyframe state
  • RemoteAnalysisCacheClient service interface
  • Partial versioning and invalidation mechanics for SkyValue entries
  • Builds without the Bytes (host-independent, shareable remote output metadata)

Not in Bazel

  • Backend invalidation service, providers, and invalidation/versioning semantics
  • Metadata table for tracking snapshots, configurations, and versions
  • Cache priming pipeline
  • Low-latency co-located key-value storage
  • External dependency management for Skycache (such as external dependency root paths)
  • Standardized wire protocol

Skyframe 101

Consider a simple C++ binary target in Bazel:

# BUILD
cc_binary(
    name = "cc",
    srcs = ["main.cc"],
)
$ bazel query 'deps(//:cc)' --nohost_deps --noimplicit_deps --output=graph
# //:cc -> //:main.cc

$ bazel build //:cc
INFO: Analyzed target //:cc (71 packages loaded, 379 targets configured).
INFO: Found 1 target...
Target //:cc up-to-date:
  bazel-bin/cc

While the BUILD file exposes a simple declarative interface (//:cc depends on //:main.cc), Bazel translates that declaration into an imperative evaluation graph in Skyframe covering file states, package loading, build options, target configuration, and action execution:

flowchart TD
    TC["TargetCompletion (//:cc)"] --> CppLink["CppLink (//:cc)"]
    TC --> CCBin["cc_binary (//:cc)"]
    TC --> Pkg["Package (PACKAGE:)"]
    TC --> BuildConfig["BuildConfiguration (...dab8f700df816ad8f8998bb302a45cc805cb966)"]

    CppLink --> CppCompile["CppCompile (//:cc)"]
    CppLink --> CCBin

    CppCompile --> Artifact["Artifact (main.cc)"]
    CppCompile --> CCBin

    Artifact --> FileMain["File (.../workspace/main.cc)"]
    CCBin --> Pkg
    CCBin --> BuildConfig

    FileMain --> FileStateMain["FileState (.../workspace/main.cc)"]
    FileMain --> FileDir["File (.../workspace/)"]

    Pkg --> FileBuild["File (.../workspace/BUILD)"]
    FileBuild --> FileStateBuild["FileState (.../workspace/BUILD)"]
    FileBuild --> FileDir

    FileDir --> FileStateDir["FileState (.../workspace/)"]

    BuildConfig --> BaselineOptions["BaselineOptions (rExecTransition=false, newPlatform=null)"]
    BuildConfig --> PrecomputedStarlark["Precomputed (starlark_semantics)"]
    BaselineOptions --> PrecomputedBaseline["Precomputed (baseline_configuration)"]

Loading

Incremental Builds in Skyframe

Skyframe handles incremental changes after a full build in two phases: upward invalidation and change pruning.

1. Upward Transitive Closure Invalidation

When a BUILD file is modified, DiffAwareness detects that the underlying FileState for BUILD has CHANGED. Skyframe then marks the upward transitive closure of nodes that depend on it as DIRTY (File -> Package -> cc_binary -> CppCompile -> CppLink -> TargetCompletion).

flowchart BT
    Diff["DiffAwareness"] -->|Detects edit| FSB["FileState: BUILD (CHANGED)"]
    FSB --> FB["File: BUILD (DIRTY)"]
    FB --> Pkg["Package (DIRTY)"]
    Pkg --> CCBin["cc_binary //:cc (DIRTY)"]
    CCBin --> CppCompile["CppCompile //:cc (DIRTY)"]
    CppCompile --> CppLink["CppLink //:cc (DIRTY)"]
    CppLink --> TC["TargetCompletion //:cc (DIRTY)"]

Loading

2. Evaluation and Change Pruning

During the next build, Skyframe re-evaluates bottom-up starting from the dirtied leaf nodes (FileState -> File -> Package). If re-evaluating Package produces the exact same PackageValue as the previous build (for example, if only a comment was edited in BUILD), Skyframe prunes the dirtied upward transitive closure (cc_binary, CppCompile, CppLink, TargetCompletion) and skips re-evaluating them.

Skycache in Skyframe: Top-Down Frontier Pruning

Whereas local incremental builds prune bottom-up, Skycache prunes top-down by caching both analysis values (ConfiguredTarget / cc_binary) and execution values (ActionExecutionValue / CppLink, CppCompile) in a remote cache.

When a user modifies a top-level file (such as //pkg:app.cc) and builds //pkg:app, Skyframe evaluates the active modified zone locally and queries Skycache at the unmodified dependency frontier via SkyValueRetriever:

  • Cache hit #1 (analysis) resolves ConfiguredTargetKey (//pkg:lib as DATA_TYPE_ANALYSIS_NODE).
  • Cache hit #2 (execution) resolves ActionLookupData (Link //pkg:lib.a as DATA_TYPE_EXECUTION_NODE).

Everything below the Skycache frontier is pruned—meaning transitive ConfiguredTarget nodes, Package and BzlLoad evaluations, FileState checks, and transitive compile actions are never evaluated in memory and never queried against the Remote Execution Action Cache (RBE AC).

flowchart TD
    subgraph Active["Active / Modified Zone (Evaluated Locally)"]
        BDK["Top-Level BuildDriverKey: //pkg:app"]
        CTK_App["ConfiguredTargetKey: //pkg:app (Active)"]
        ALD_App["ActionLookupData: Compile //pkg:app.cc (Local Edit)"]
        BDK --> CTK_App --> ALD_App
    end

    subgraph Frontier["Skycache Frontier (Top-Down Cache Hit via SkyValueRetriever)"]
        ALD_Lib["ActionLookupData (DATA_TYPE_EXECUTION_NODE): Link //pkg:lib.a — CACHE HIT"]
        CTK_Lib["ConfiguredTargetKey (DATA_TYPE_ANALYSIS_NODE): //pkg:lib — CACHE HIT"]
    end

    ALD_App --> ALD_Lib
    CTK_App --> CTK_Lib

    subgraph Pruned["Pruned Transitive Subgraph (Skipped — Never Evaluated in Memory or RBE)"]
        T_CTK["Transitive ConfiguredTargetKey: //dep:target_a (Skipped)"]
        T_Pkg["PackageKey & BzlLoadKey: //dep:BUILD, //rules:defs.bzl (Skipped)"]
        T_ALD["Transitive ActionLookupData: Compile //dep:target_a.cc (Skipped — No RBE AC Lookup)"]
        T_File["FileStateKey & DirectoryListingKey: dep/target_a.cc, dep/ (Skipped)"]

        T_CTK -.-> T_Pkg
        T_Pkg -.-> T_File
        T_ALD -.-> T_File
    end

    CTK_Lib -.->|Pruned by Skycache Hit| T_CTK
    ALD_Lib -.->|Pruned by Skycache Hit| T_ALD

Loading

Skycache Architecture at Google

1. Skycache Priming (Write Path)

A continuous priming pipeline populates Skycache from clean repository snapshots:

  1. The Primer service schedules builds of target sets at repository change snapshots roughly every 5 minutes using Bazel in Cloud.
  2. Bazel in Cloud invokes Bazel in write mode.
  3. Bazel serializes SkyValue nodes along with invalidation and versioning data and streams them to the Write Proxy.
  4. The Write Proxy records metadata (snapshot version, change number, build configuration) in Spanner and writes the serialized SkyValue and shared object chunks into RAM key-value stores. 
flowchart LR
    Primer["Primer"] -->|Build targets at change snapshots every 5m| BIC["Bazel in Cloud"]
    BIC -->|Invokes Bazel in write mode| Bazel["Bazel (Writer)"]
    Bazel -->|Serialize SkyValues with invalidation and versioning data| WP["Write Proxy"]
    WP --> Spanner[("Spanner for Metadata (version, change, config)")]
    WP --> KV[("RAM Key-Value Stores")]

Loading

2. Skycache Reader (Read Path)

When a developer or CI invocation runs a build in read mode:

  1. Bazel sends lookup requests for SkyValue keys to the AnalysisCacheService / Storage Proxy.
  2. AnalysisCacheService checks cache metadata in Metadata Spanner and validates the cached entry's file and directory dependencies against the Version Control / Source Service at the client's workspace state.
  3. On a valid entry, AnalysisCacheService fetches the root SkyValue bytes from the RAM key-value stores (cache hit of parent bytes).
  4. AnalysisCacheService returns the root SkyValue bytes to Bazel.
  5. During deserialization, Bazel reconstructs nested objects (RuleConfiguredTarget, providers, NestedSet of Artifact) by pointer-chasing 16-byte content fingerprints via the Storage Proxy.
  6. The Storage Proxy resolves those child chunk fingerprints directly from the RAM key-value stores. 
sequenceDiagram
    participant Bazel as Bazel (Reader)
    participant ACS as AnalysisCacheService / Storage Proxy
    participant Spanner as Metadata Spanner
    participant VCS as Version Control / Source Service
    participant KV as RAM Key-Value Stores

    Bazel->>ACS: 1. Request for SkyValues
    ACS->>Spanner: 2a. Check Skycache metadata
    ACS->>VCS: 2b. Validate against source state
    ACS->>KV: 3. Lookup parent SkyValue bytes
    KV-->>ACS: Cache hit of parent bytes
    ACS-->>Bazel: 4. Respond with root SkyValue bytes
    Bazel->>ACS: 5. Pointer-chase child bytes (getSharedValue)
    ACS->>KV: 6. Lookup child content chunks
    KV-->>ACS: Cache hits of child bytes
    ACS-->>Bazel: Return child bytes to complete SkyValue reconstruction

Loading

Cache Schema and Object DAG

Each root SkyValue record is compact (~380 bytes) and keyed by Fingerprint(FrontierNodeVersion, SkyKey) (for example, DataType = DATA_TYPE_ANALYSIS_NODE (4)). Its value stores a 16-byte Invalidation Key alongside the Root SkyValue Payload Bytes, splitting into two separate DAGs.

Server-Side Invalidation DAG (Walked by AnalysisCacheService)

  • The root nested invalidation node is keyed by a 16-byte PackedFingerprint and holds references to child_nodes[], file_keys[], and listing_keys[].
  • Child nested invalidation nodes (also keyed by 16-byte PackedFingerprint values) are shared across targets.
  • Leaf entries represent FileInvalidationData (keyed by paths such as :pkg/BUILD, :rules/defs.bzl, or Base64(MTSV):path) and DirectoryListingInvalidationData (keyed by directory paths such as ;pkg/include), allowing the server to validate source state without client round-trips.

Client-Side Analysis Object DAG (Pointer-Chased via getSharedValue)

  • The root payload deserializes into a RuleConfiguredTarget containing inline 16-byte PackedFingerprint references to shared content chunks.
  • In Round-Trip 1 (Layer 1), the client fetches shared content chunks (0x01 ZSTD) such as CcInfo / TransitiveInfoProviderMap and ImmutableList.
  • In Round-Trip 2 (Layer 2), the client fetches shared content chunks (0x00 RAW) such as the NestedSet root Object[] array.
  • In Round-Trip 3 (Layer 3), the client fetches child NestedSet Object[] arrays (such as //dep:target_a headers and //dep:target_b headers), deduplicating transitive data across targets. 
flowchart TD
    Root["SkyValue Record (~380 B) — Key: Fingerprint(FrontierNodeVersion, SkyKey) [DataType = DATA_TYPE_ANALYSIS_NODE (4)] — [16B Invalidation Key] + [Root SkyValue Payload Bytes]"]

    subgraph ServerDAG["Server-Side Invalidation DAG (Walked by ACS)"]
        InvRoot["Root Nested Invalidation Node — Key: 16B PackedFingerprint (child_nodes[], file_keys[], listing_keys[])"]
        InvChild["Child Nested Invalidation Node — Key: 16B PackedFingerprint (Shared across targets)"]
        InvFile["FileInvalidationData — Key: ':pkg/BUILD', ':rules/defs.bzl' (or Base64(MTSV):path)"]
        InvDir["DirectoryListingInvalidationData — Key: ';pkg/include'"]

        InvRoot -->|16B child_node| InvChild
        InvRoot -->|file_keys| InvFile
        InvChild -->|listing_keys| InvDir
    end

    subgraph ClientDAG["Client-Side Analysis Object DAG (Pointer-Chasing via getSharedValue)"]
        RCT["RuleConfiguredTarget (Deserialized from Root Payload) — Inline 16B PackedFingerprint refs"]
        ChunkL1_A["Shared Content Chunk (Layer 1) — Key: 16B PackedFingerprint (0x01 ZSTD) — CcInfo / TransitiveInfoProviderMap"]
        ChunkL1_B["Shared Content Chunk (Layer 1) — Key: 16B PackedFingerprint (0x01 ZSTD) — ImmutableList of ActionAnalysisMetadata"]
        ChunkL2["Shared Content Chunk (Layer 2) — Key: 16B PackedFingerprint (0x00 RAW) — NestedSet of Artifact Root Array (Object[])"]
        ChunkL3_A["Shared Content Chunk (Layer 3) — Key: 16B PackedFingerprint — Child NestedSet Object[] (//dep:target_a hdrs)"]
        ChunkL3_B["Shared Content Chunk (Layer 3) — Key: 16B PackedFingerprint — Child NestedSet Object[] (//dep:target_b hdrs)"]

        RCT -->|Round-Trip 1: 16B Fingerprint| ChunkL1_A
        RCT -->|Round-Trip 1: 16B Fingerprint| ChunkL1_B
        ChunkL1_A -->|Round-Trip 2: 16B Fingerprint| ChunkL2
        ChunkL2 -->|Round-Trip 3: 16B Fingerprint| ChunkL3_A
        ChunkL2 -->|Round-Trip 3: 16B Fingerprint| ChunkL3_B
    end

    Root -->|16B Invalidation Key| InvRoot
    Root -->|Root Payload Bytes| RCT

Loading

Current Protocol Buffers

AnalysisCacheService (Root SkyValue Lookup & Validation)

service AnalysisCacheService {
  rpc Lookup(LookupRequest) returns (LookupResponse);
}

message LookupRequest {
  string workspace_id = 1;
  int32 snapshot_version = 2;
  repeated bytes keys = 3;
}

message LookupResponse {
  repeated bytes values = 1;
  repeated MissReason miss_reasons = 2;
  repeated bytes invalidation_fingerprints = 3;
}

SkycacheStorageProxyService (Streaming Shared Chunk Pointer-Chasing)

service SkycacheStorageProxyService {
  rpc Lookup(stream StorageLookupRequest) returns (stream StorageLookupResponse);
}

message StorageLookupRequest {
  int32 batch_id = 1;
  repeated bytes keys = 2;
  string build_id = 3;
}

message StorageLookupResponse {
  int32 batch_id = 1;
  repeated int32 index = 2;
  repeated bytes value = 3;
  repeated bool found = 4;
}
添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论