Getting Started

Goal: Stand up a versioned datom project using a local filesystem store and onboard data using datom_sync() – the primary workflow for bringing files into datom. By the end of this article the project holds multiple versioned tables, and you have seen: syncing one file, updating it, the idempotent no-op, and syncing a batch. No AWS account needed. The same workflow extends unchanged to S3 (see Starting on S3).

Already done the two-minute tour in the README? The tour and this article cover the same ground. If your project initialized cleanly and datom_read() returned TRUE, you can jump straight to Starting on S3 or Tracing Data Lineage.

You are the data engineer for STUDY-001, a Phase II clinical trial. EDC extracts land as files in a staging folder; your job is to onboard them into a shared, versioned data space – multiple engineers writing new extracts, multiple analysts reading any version, all coordinated through a single git history. Every sync is a git commit; every read resolves to an exact version SHA. No one can silently overwrite history, and anyone with access to the repo can reproduce any past analysis by pinning to a SHA.

This first article walks the local-only path. The same workflow – same functions, same commands – works for a shared S3 space: you build an S3 store instead of a local one (see Starting on S3).

Two locations, two roles:

Input folder (mutable, disposable) – a landing zone for data not yet onboarded. Files here can be overwritten or deleted freely. Re-syncing already-onboarded content is a no-op. Nothing here is the source of truth.
Storage (immutable, permanent) – once datom_sync() onboards a file, storage holds the versioned, content-addressed copy. The original input file is no longer needed; the onboarded data stands on its own.

Requirements

datom keeps metadata in git (diff-able, auditable) and data wherever you tell it to live (S3 or a local directory). Even when data lives on a local filesystem, metadata still goes to a git remote – that is how version history stays reproducible across machines.

You need two things, both one-time:

A GitHub account with a personal access token (PAT) scoped to repo. Store it in your OS keychain once with keyring::key_set("GITHUB_PAT"); every article after this picks it up automatically. Using keyring keeps the PAT out of your code and command history.
The gh CLI is not required – datom creates GitHub repos through the GitHub REST API directly using your PAT.

No AWS, no cloud account, no governance repo for this article.

Verify your keyring setup before continuing:

nzchar(keyring::key_get("GITHUB_PAT"))   # should return TRUE

If it errors, set the PAT with keyring::key_set("GITHUB_PAT") first.

Configure your environment

Everything machine-specific lives in one block. Set these values once for your environment; every code chunk after this is copy/paste as-is.

library(datom)
library(fs)

# --- Settings you control --------------------------------------------------
project_name <- "STUDY_001"          # logical project name (recorded in metadata)
repo_name    <- "study-001-data"     # GitHub repo name for the metadata repo

# dev_dir  -- your local clone of the metadata git repo (stays on your machine)
# data_dir -- where parquet bytes are written; point at a shared location
#             (network mount, or an S3 store) for a real team space. Temp dirs
#             are used here so the article leaves nothing behind.
dev_dir  <- path(tempdir(), "study_001_dev")
data_dir <- path(tempdir(), "study_001_data")

# Your GitHub personal access token (PAT), scoped to `repo`. Stored once in your
# OS keychain with keyring::key_set("GITHUB_PAT"); read here by name so the
# token never appears in your code or command history.
github_pat <- keyring::key_get("GITHUB_PAT")
# ---------------------------------------------------------------------------

dir_create(data_dir)

Build a store

A store bundles the addresses datom needs: where parquet bytes go and the GitHub PAT that lets datom push metadata. Governance is not attached (governance = NULL); it is opt-in and added later through the governance companion package when sharing across a portfolio matters.

store <- datom_store(
  governance = NULL,
  data       = datom_store_local(path = data_dir),
  github_pat = github_pat
)

Initialize the data repository

datom_init_repo(
  path         = dev_dir,
  project_name = project_name,
  store        = store,
  create_repo  = TRUE,
  repo_name    = repo_name
)

This creates a GitHub repo, clones it into dev_dir, and commits a project.yaml that records the project’s data store address. The git repo is now live on GitHub. No parquet data is pushed to GitHub – only the metadata commits travel over the wire; the parquet bytes stay in data_dir.

datom_init_repo() also creates an input_files/ directory inside the clone. It is gitignored – files placed there are never committed. It is the inbox for datom_sync().

datom_sync() onboards flat tabular files: csv, tsv, txt, psv, parquet, sas7bdat, xpt, sav, zsav, por, dta, xls, xlsx. Anything else in the inbox (an .rds, a .json) is reported as unsupported_format and skipped without blocking its neighbours – see the ingestion allowlist for why, and for the one-line escape hatch.

Take a moment to inspect the repo structure before moving on:

# Metadata layout in the git clone (note input_files/ -- your sync inbox)
fs::dir_tree(dev_dir)

# Storage layout (empty until first sync)
fs::dir_tree(data_dir)

Connect

conn <- datom_get_conn(path = dev_dir, store = store)
print(conn)
#> -- datom connection
#> * Project: "STUDY_001"
#> * Role: "developer"
#> * Backend: "local"
#> * Root: "/tmp/.../study_001_data"
#> * Path: "/tmp/.../study_001_dev"
#> * Governance: not attached

Step 1: Sync one file

The month-1 extract has just landed – a single demographics CSV. Drop it in the input folder:

# The input folder lives inside the git clone but is gitignored.
# Files placed here are the raw material for datom_sync().
input_dir <- path(dev_dir, "input_files")

write.csv(
  datom_example_data("dm", cutoff_date = "2026-01-28"),
  path(input_dir, "dm.csv"),
  row.names = FALSE
)

Scan the folder to build a sync manifest – datom detects what is new, changed, or unchanged relative to what has already been onboarded:

manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 1 new, 0 changed, 0 unchanged.
manifest
#>   name  file         format original_file_sha status
#> 1   dm  .../dm.csv   csv    7a3b...           new

Now onboard it:

datom_sync(conn, manifest)
#> i Syncing 1 table...
#> v dm synced (new).
#> i Sync complete: 1 succeeded, 0 failed, 0 skipped.

Three things just happened, in this order:

The CSV was read and serialized to parquet in data_dir. No data was pushed to the GitHub repo – parquet bytes never leave your local store.
metadata.json and version_history.json were updated in the git clone and committed.
The metadata commit was pushed to GitHub. The version is now auditable from any machine with repo access, but the raw data stays where you put it.

Confirm with datom_list(), datom_history(), and a round-trip read:

datom_list(conn)
#>   name current_version current_data_sha last_updated
#> 1   dm        a8ee7a31         4b6d0a7e 2026-01-28T...

datom_history(conn, "dm")
#>    version  data_sha timestamp            message
#> 1 a8ee7a31 4b6d0a7e 2026-01-28T09:02:11Z dm synced from dm.csv

datom_read(conn, "dm")

Key point: the input file is now disposable. Delete it and the data is still accessible from storage – storage is the source of truth, not the input folder:

file_delete(path(input_dir, "dm.csv"))
datom_read(conn, "dm")   # still works -- storage is the permanent record

Step 2: Update one file

A month passes. The month-2 extract arrives with new subjects. Overwrite the input file and sync again:

write.csv(
  datom_example_data("dm", cutoff_date = "2026-02-28"),
  path(input_dir, "dm.csv"),
  row.names = FALSE
)

manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 0 new, 1 changed, 0 unchanged.

datom_sync(conn, manifest)
#> i Syncing 1 table...
#> v dm synced (changed).
#> i Sync complete: 1 succeeded, 0 failed, 0 skipped.

The table now has two versions. The previous version is still intact – nothing was overwritten:

datom_history(conn, "dm")
#>    version  data_sha timestamp            message
#> 1 5c1a3f7b 9e8f1c2d 2026-02-28T10:14:02Z dm synced from dm.csv
#> 2 a8ee7a31 4b6d0a7e 2026-01-28T09:02:11Z dm synced from dm.csv

# Read the current version (month 2)
nrow(datom_read(conn, "dm"))
#> [1] 16

# Read the prior version (month 1) by its SHA
hist   <- datom_history(conn, "dm")
m1_ver <- hist$version[nrow(hist)]   # oldest row is the month-1 version
nrow(datom_read(conn, "dm", version = m1_ver))
#> [1] 4

Both versions coexist; any historical snapshot is retrievable by its SHA. This is the property that downstream statisticians, regulators, and auditors rely on.

Step 3: No-op repeat

Run datom_sync() again with nothing changed:

manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 0 new, 0 changed, 1 unchanged.

datom_sync(conn, manifest)
#> i No new or changed files. Nothing to sync.

Re-syncing identical content is a no-op. This is what makes sync safe to run in a scheduled job or pipeline – accidental re-runs cost nothing and pollute no history. The same idempotency applies to datom_write(): writing an identical data frame to the same table name detects the duplicate and skips.

Idempotency here is stronger than “the file’s bytes are unchanged”. If a source system re-exports the same data – new export timestamp in the header, same rows – the file’s bytes change but its content does not. datom records the new file provenance as a version and does not store the data a second time. Version SHAs explains the three hashes that make that distinction.

Step 4: A batch of files

It is now month 3. The data management team has switched from emailing single files to dropping a folder of monthly extracts. Today’s drop contains four domains:

cutoff <- "2026-03-28"

write.csv(datom_example_data("dm", cutoff_date = cutoff),
          path(input_dir, "dm.csv"), row.names = FALSE)
write.csv(datom_example_data("ex", cutoff_date = cutoff),
          path(input_dir, "ex.csv"), row.names = FALSE)
write.csv(datom_example_data("lb", cutoff_date = cutoff),
          path(input_dir, "lb.csv"), row.names = FALSE)
write.csv(datom_example_data("ae", cutoff_date = cutoff),
          path(input_dir, "ae.csv"), row.names = FALSE)

Scan and sync in one pass:

manifest <- datom_sync_manifest(conn)
#> i Scanned 4 files: 3 new, 1 changed, 0 unchanged.
manifest
#>   name  file         format original_file_sha status
#> 1   dm  .../dm.csv   csv    c41a...           changed
#> 2   ex  .../ex.csv   csv    9b08...           new
#> 3   lb  .../lb.csv   csv    72d3...           new
#> 4   ae  .../ae.csv   csv    1e4a...           new

datom_sync(conn, manifest)
#> i Syncing 4 tables...
#> v dm synced (changed).
#> v ex synced (new).
#> v lb synced (new).
#> v ae synced (new).
#> i Sync complete: 4 succeeded, 0 failed, 0 skipped.

dm is changed (month-3 update); ex, lb, ae are brand new. All four are now versioned:

datom_list(conn)
#>   name current_version current_data_sha last_updated
#> 1   ae        3a17b8e2         e91d04ff 2026-03-28T...
#> 2   dm        d0922fc7         c2e80a14 2026-03-28T...
#> 3   ex        f44910b5         88a73e02 2026-03-28T...
#> 4   lb        718e02ca         4c3812dd 2026-03-28T...

Project health checks

datom_status() summarizes the project state at a glance:

datom_status(conn)
#> -- datom status: STUDY_001
#> v Git: clean, in sync with origin
#> i Tables on local: 4
#> i Last commit: <sha> "Update ae"

datom_validate() cross-checks that every table in the manifest has its parquet file in the data store and its metadata in git history:

datom_validate(conn)
#> v 4 tables validated.
#> v Manifest <-> data store: consistent.
#> v Manifest <-> git history: consistent.

If you ever see a discrepancy from datom_validate(), that is the moment to stop and investigate before doing more work – it means the project’s state has drifted from one of git or storage.

Reading as a reader

Writing is a developer action. Reading is what analysts and downstream pipelines do, and they use the reader role – a connection built from the data store alone, with no GitHub PAT and no local clone. You build a reader store by leaving github_pat unset, then connect by project name:

reader_store <- datom_store(
  governance = NULL,
  data       = datom_store_local(path = data_dir)   # same data location
)                                                    # no PAT -> reader role

reader_conn <- datom_get_conn(store = reader_store, project_name = project_name)
print(reader_conn)
#> -- datom connection
#> * Project: "STUDY_001"
#> * Role: "reader"
#> * Backend: "local"

Read any table through the reader connection:

datom_read(reader_conn, "lb")   # labs, current version from storage

datom stores data in Apache Parquet format and reads it back as a tibble. The read does not go through GitHub – the reader resolves the parquet file directly from the data store. A teammate on another machine takes the same path: they need access to the data store, not to the git repo.

Writing a computed table with `datom_write()`

datom_sync() is the primary workflow for onboarding files from a staging folder. For tables derived in code – where the data frame is computed rather than imported from a file – use datom_write() directly:

# Derive a summary table in code
lb <- datom_read(conn, "lb")
lb_summary <- dplyr::summarise(
  dplyr::group_by(lb, LBTESTCD),
  n = dplyr::n(),
  .groups = "drop"
)

# Write it as a versioned table, declaring its parent for lineage.
# datom_parent() resolves the parent's data_sha and source_lineage from
# the stored metadata snapshot -- always use it instead of a raw list.
datom_write(
  conn,
  data    = lb_summary,
  name    = "lb_summary",
  message = "Lab test counts from month-3 LB",
  parents = list(
    datom_parent(conn, table = "lb", version = datom_history(conn, "lb")$version[1])
  )
)
#> v Wrote "lb_summary" (full): "b9c4e21a"

datom_write() takes the same connection and produces the same versioned, content-addressed result. The parents argument declares lineage – this table was derived from lb – making the provenance chain auditable (see Tracing Data Lineage).

The datom table contract

A table version is identified by a canonical hash of its values (data_sha), which means every column has to be a type datom knows how to encode: logical, integer, double, character, factor, Date, POSIXct, difftime/hms, data.table::ITime/IDate, bit64::integer64, or a labelled vector over one of those.

Files onboarded through datom_sync() land inside that set by construction. It is derived tables – the datom_write() path above – where a column can fall outside it: a tidyr::nest() result carries a list column, strptime() returns POSIXlt, an sf table carries geometry, a database round-trip can bring back a blob.

datom_check_hashable() tells you before you write. It needs no connection and no store, so it is safe to run anywhere:

library(datom)

lb_summary <- data.frame(LBTESTCD = c("ALT", "AST"), n = c(12L, 12L))
datom_check_hashable(lb_summary)
#> ✔ All 2 columns are hashable. This table is ready for `datom_write()`.

When a column is outside the contract, the report names it and says what to do about it:

messy <- data.frame(id = 1:2)
messy$measurements <- list(c(1, 2), c(3, 4))

report <- datom_check_hashable(messy)
#> ✖ 1 of 2 columns are not hashable.
#> ✖ Column measurements (<list>): List and blob columns are not hashable. Flatten
#>   to one value per row with tidyr::unnest(), or serialize each element to
#>   character (for example with jsonlite::toJSON() per element), before writing.
#> ℹ Fix these before `datom_write()`, which refuses the whole table until they are resolved.
report$recourse[report$status == "unsupported"]
#> [1] "List and blob columns are not hashable. Flatten to one value per row with tidyr::unnest(), or serialize each element to character (for example with jsonlite::toJSON() per element), before writing."

datom_write() runs the same check itself and refuses the whole table until every column is inside the contract, reporting all offenders at once and leaving no partial state behind. The full type-by-type recourse table and the reasoning live in Version SHAs.

Where you are

A full sync-based onboarding workflow: one file, update, no-op, batch.
Four raw tables versioned; one derived table written directly.
Input files are disposable; storage is the permanent record.
datom_status() and datom_validate() provide consistency checks.
Readers access data from storage alone – no git, no PAT.
datom_check_hashable() is the pre-flight check for the table contract.

From here, Starting on S3 shows the same workflow with data in object storage, Tracing Data Lineage shows how derived tables record where they came from, and Version SHAs explains the three hashes behind version, data_sha, and stored-object integrity.

Teardown

When done, remove the project. Pick the one option that fits, and copy only that chunk:

# Option A -- full scripted teardown (deletes local files AND the GitHub repo).
# Do this BEFORE any manual unlink().
datom_repo_delete(conn, confirm = "STUDY_001")

# Option B -- local only (the GitHub repo stays; delete it from the UI later).
unlink(c(dev_dir, data_dir), recursive = TRUE)

Do not call unlink() before datom_repo_delete() – removing the local clone first strips the GitHub remote reference and the remote repo will not be deleted.