Skip to contents

The datom Model: Code in Git, Data in Cloud

Source: vignettes/design-datom-model.Rmd
design-datom-model.Rmd

Companion to: Getting Started. Read this when you want to understand why datom split storage in two.

The single design decision that shapes everything else in datom is this: metadata and data live in different places, and that’s intentional.

This article explains the split, the consequences that fall out of it, and the properties you get for free when you accept it.

The split 

+------------------------+         +-------------------------+
|    Git repository      |         |    Object store         |
|                        |         |                         |
|  manifest.json         |         |  dm/                    |
|  metadata.json         |         |    a8ee7a31.parquet     |
|  version_history.json  |  ---->  |    5c1a3f7b.parquet     |
|  project.yaml          |  refers |  ex/                    |
|                        |   to    |    f44910b5.parquet     |
|  (everything text,     |         |                         |
|   diffable, reviewable)|         |  (parquet bytes only,   |
|                        |         |   keyed by SHA)         |
+------------------------+         +-------------------------+

Metadata – the catalog of what tables exist, what their current and historical versions are, and where the bytes live – goes in a git repository. It’s small, text-based, diffable, reviewable, and protected by all the access controls and audit machinery your organization already has around git.

Data – the parquet bytes themselves – goes in an object store. Each parquet file is named by a SHA of its content. The git repo never holds data; it only holds pointers to data.

This is not a storage optimization. It’s an information-architecture choice with downstream consequences.

Property 1: Immutability is automatic

Parquet files are addressed by SHA of their content. Two consequences fall out:

  • Writing the same data twice produces the same filename. The second upload is either a no-op or a harmless overwrite of identical bytes.
  • Different data produces a different filename. A new version never overwrites an old one; both files coexist in the store.

You don’t have to remember to copy old files before overwriting them. The file system makes overwriting impossible by construction. The history your datom_history() call shows is real – those parquet files are still in the store.

Property 2: The metadata is reviewable like code

Because metadata lives in git, every change to a project’s catalog is a commit. That means:

  • git log shows the history of every table.
  • git blame tells you which datom_write() introduced a specific version.
  • Pull requests, branch protection, and code review apply to data changes the same way they apply to code changes.
  • Restoring a project to “the way it was on March 15” is git checkout, no special tooling.

If you’ve ever debugged a missing column by reading commit messages, you already understand why this matters.

Property 3: Readers don’t need write access

The split lets datom support two roles cleanly:

  • Data developers have a local git clone of the project, push to the remote, and write to the data store.
  • Data readers connect to the data store directly – no git clone, no write credentials required. datom_get_conn(store = ..., project_name = "...") (reader form, no path) resolves everything it needs from the metadata mirrored to the data store at write time: the manifest, per-table metadata.json, and versioned snapshots.

The data store can be an object store (S3) or a local filesystem store (datom_store_local()); the reader path is the same either way. What readers never need is access to the git repository (the code store) or write credentials for the data store.

A statistician with read-only access to the data store can read any table and reproduce any historical analysis – the metadata they need is already there, mirrored when the developer wrote.

Property 4: Storage is swappable

Because datom only knows the address of data through the metadata, not the bytes themselves, the storage backend is a substitution point. datom_store_local() puts parquet on a directory; datom_store_s3() puts parquet on S3; future backends (GCS, Azure Blob, etc.) drop in without touching how versions are computed or how history is recorded.

What does not swap is git. The metadata still goes to a git remote, always.

What stays in the metadata

A datom project’s git repository holds, per table:

  • metadata.json – the current state (current data SHA, table type, size, parents).
  • version_history.json – an append-only log of every version ever written.
  • A manifest.json at the project root summarizing all tables.

It does not hold the data, and it never will. The .gitignore in a new project explicitly excludes parquet, csv, and other data formats so nobody can commit them by accident.

What this isn’t

datom is not a content-addressed blob store like git-LFS or DVC. The key difference: datom’s metadata is a structured, queryable catalog – not opaque pointer files in your repo. You ask datom_history(conn, "dm") and you get a data frame, not a git log of .dvc files. The metadata is designed to be read by code, not just by humans.

datom is also not a database. It has no query engine, no ACID transactions across tables, no joins. It is a content-addressed catalog of immutable parquet files, and it’s deliberately not more than that.

Where this leads

Once you accept “metadata in git, data in data store,” several other datom design choices stop looking arbitrary:

  • The two-repo split (governance vs. project) – the organization-scope governance repo is managed by the governance companion package.
  • The ref.json indirection layer for portable, migration-ready storage.
  • The two-flavor SHA scheme (data SHA + metadata SHA) – see Version SHAs: Data SHA vs. Metadata SHA.

Each is a direct consequence of the split, applied to a specific problem. None of them stand on their own.