Why We're Building Stategraph: Terraform State as a Distributed Systems Problem

The Pathology of File-Based State

Terraform state, at its core, is a coordination problem. Multiple actors (engineers, CI systems, drift detection) need to read and modify overlapping subsets of infrastructure state concurrently. This is a well-studied problem in distributed systems, with established solutions around fine-grained locking, multi-version concurrency control, and transaction isolation.

Instead, Terraform implements the simplest possible solution: a global mutex on a JSON file.

Consider the actual data access patterns in a typical Terraform operation:

This mismatch between granularity of operation and granularity of locking is the root cause of every Terraform scaling problem. It violates the fundamental principle of isolation in concurrent systems: non-overlapping operations should not block each other.

The standard response, splitting state files, doesn't solve the problem. It redistributes it. Now you have N coordination problems instead of one, plus the additional complexity of managing cross-state dependencies. You've traded false contention for distributed transaction coordination, which is arguably worse.

State as a Graph: The Natural Representation

Infrastructure state is inherently a directed graph. Resources have dependencies, which form edges. Changes propagate along these edges. Terraform already knows this: the internal representation is a graph, and the planner performs graph traversal. But at the storage layer, we flatten this rich structure into a blob.

This is akin to storing a B-tree in a CSV file. You can do it, but you're destroying the very properties that make the data structure useful.

When state is properly normalized into a graph database, several properties emerge naturally:

Subgraph isolation: Operations on disjoint subgraphs are inherently parallelizable. If Team A is modifying RDS instances and Team B is updating CloudFront distributions, there's no shared state to coordinate.

Precise locking: We can implement row-level locking on resources and edge-level locking on dependencies. Lock acquisition follows the dependency graph, preventing deadlocks through consistent ordering.

Incremental refresh: Given a change set, we can compute the minimal refresh set by traversing the dependency graph. Most changes affect a small cone of resources, not the entire state space.

Concurrency Control Through Proper Abstractions

The distributed systems community solved these problems decades ago. Multi-version concurrency control (MVCC) allows readers to proceed without blocking writers. Write-ahead logging provides durability without sacrificing performance. Transaction isolation levels let operators choose their consistency guarantees.

Stategraph implements these patterns at the Terraform state layer:

Each operation acquires locks only on its subgraph. The lock manager uses the dependency graph to ensure consistent ordering, preventing deadlocks. Readers use MVCC to access consistent snapshots without blocking writers.

The result is dramatic improvement in concurrent throughput:

Three teams, three transactions, zero contention. This isn't possible with file-based state, regardless of how you split it.

The Refresh Problem

Terraform refresh is O(n) in the number of resources, regardless of change scope. Change one security group rule and you still walk the entire state. That's an algorithmic bottleneck, not just an implementation detail.

With a graph representation, refresh work can be scoped to the affected subgraph instead of the entire state. Most changes touch only a small fraction of resources, not everything.

Why We Built This

At Terrateam, we've watched hundreds of teams struggle with the same fundamental problems. They start with a single state file, hit scaling limits, split their state, discover coordination complexity, build orchestration layers, and eventually resign themselves to living with the pain.

This is a solvable problem. The computer science is well-understood. The implementation is straightforward once you acknowledge that state management is a distributed systems problem, not a file storage problem.

Stategraph isn't revolutionary. It's the application of established distributed systems principles to a problem that's been mischaracterized since its inception. We're not inventing new algorithms; we're applying the right ones.

The infrastructure industry has accepted file-based state as an immutable constraint for too long. It's not. It's a choice, and it's the wrong one for systems at scale.

Technical Implementation

Stategraph is implemented as a PostgreSQL schema with a backend that speaks the Terraform/OpenTofu remote backend protocol. We chose PostgreSQL for its robust MVCC, proven scalability, and operational familiarity. The schema normalizes state into three primary relations:

resources: one row per resource, with type, provider, and attribute columns.
dependencies: edge table representing the resource dependency graph.
transactions: append-only log of all state mutations with full attribution.

The backend extends Terraform's protocol with graph-aware operations. Lock acquisition and state queries operate directly on the database representation of the graph, enabling precision and concurrency that file-based backends can't provide.

This isn't a wrapper or an orchestrator. It's a replacement for the storage layer that preserves Terraform's execution model while fixing its coordination problems.

Adoption Path

Stategraph reads existing tfstate files and constructs the graph representation automatically. No changes to Terraform configurations are required. The backend protocol is unchanged. From Terraform's perspective, Stategraph is just another backend, like S3 or GCS.

But from an operational perspective, everything changes. Lock contention disappears. Refresh times drop by orders of magnitude. Teams stop blocking each other. State becomes queryable, auditable, and comprehensible.

We're not asking teams to rewrite their infrastructure. We're asking them to store it properly.

The question isn't whether Terraform state should be a graph. It already is. The question is whether we'll continue pretending it's a file.