Fifty years of the data stack

Our work at Columnar is rooted in the history of analytic data systems. Today we’re sharing Data: The First 50 Years, an interactive history of the data stack from 1976 to 2026. It traces how the systems that run analytical queries over tabular data evolved from monoliths into three distinct layers, and how the technologies in each layer gained and lost ground.

From top to bottom, the layers are access (how applications connect and retrieve data), compute (the engines running queries), and storage (how data is persisted). Box widths show our estimates of how that year’s analytical work was divided among technologies. Each layer totals 100%, and the site explains the research and methodology behind the estimates.

Press Play and watch how change in one layer puts pressure on the others. The timeline starts slowly: legacy reporting over flat files and hierarchical databases gradually gives way to relational databases, built on Edgar F. Codd’s relational model, and SQL. The 1980s brought more databases, each coupling its own interfaces, engine, and storage. As those products matured in the 1990s, portability became urgent. ODBC, JDBC, and language-specific APIs gave applications common ways to connect.

Around the millennium, compute and storage began changing rapidly again. Hadoop moved more analytics onto shared files, and S3 and other object stores helped that separation spread through the 2010s. Teams gained flexibility and inherited the job of making all the pieces work.

Managed cloud warehouses offered relief from that operational burden, growing rapidly from the mid-2010s into the 2020s. In turn, renewed concerns about costs and lock-in helped fuel interest in lakehouse and embedded engines querying shared tables. Now managed open tables, including Amazon S3 Tables and offerings inside platforms like Snowflake and Databricks, let vendors handle compaction and maintenance while preserving engine choice, subject to interoperability limits.

After decades dominated by ODBC, JDBC, and DB-API, access is changing again. Coding agents are making it easier for teams to adopt Rust, Go, and other languages. JavaScript and TypeScript, once peripheral to this stack, are becoming more prominent as the lines between analytics, operational systems, and consumer-facing applications blur. The older interfaces leave developers navigating uneven language support and extra data conversions. Arrow and ADBC give them a common columnar format and drivers they can share across languages. Arrow-based database access remains a small slice of the chart, but adoption is accelerating.

Our work on these connections began long before Columnar. At Cloudera, I worked with Wes McKinney on fast, portable connections from Hive and Impala to Python and R. My co-founder David Li and I both contributed to Arrow early on, and David went on to create ADBC. My co-founder Matt Topol built the Snowflake driver, the first ADBC driver for a cloud warehouse. All three of us serve on Arrow’s project management committee, and Matt also serves on Iceberg’s.

The chart’s focus on access, compute, and storage is deliberately reductive. It leaves out other parts of the stack where interoperability matters, including catalogs and semantic layers. We’re especially interested in the work happening there through Apache Polaris and Apache Ossie (incubating). We also leave out the rapid evolution of ingestion and replication tools, real-time processing, and multimodal data systems. The applications above these layers are also changing; we expect agentic inference to become a dominant source of demand.

We see MCP as an overlay above the database access layer. It gives agents a common way to discover and invoke tools, from databases to calendars. That helps tools work across agent applications, but wrapping a database driver in JSON-RPC doesn’t by itself make data retrieval faster. We see more promise in MCP coordinating work while ADBC delivers Arrow results separately, out of band.

Another pattern stands out in the chart: the databases and interfaces that dominated the early 2000s still account for much of today’s stack. They coexist with newer technologies, unlike the pre-relational systems that largely faded from use. We expect that coexistence to continue. More engines, formats, and applications mean more connections to manage. We believe the most valuable new infrastructure will make these choices manageable. That’s what we’re building at Columnar.

Explore the timeline at history.columnar.tech.

Footnotes

1. The title is a playful nod to David Graeber’s Debt: The First 5,000 Years.
2. See the methodology. Many of the databases shown are also widely used in transactional systems, but only their use in analytics is counted here. A chart of transactional workloads would look very different. Despite extensive research, the timeline likely still contains errors and omissions. If a date, grouping, or percentage looks wrong, or a technology is missing, email hello@columnar.tech with supporting sources.