Skip to contents

Preview What a Sync Will Change

Source: R/sync.R
datom_sync_manifest.Rd

Looks at the files in the project's input_files/ folder and returns one row per file, saying whether it is new, changed, unchanged since it was last synced, or in a format datom cannot read. Nothing is written. Review the result, drop any rows you do not want, then pass it to datom_sync().

Usage

datom_sync_manifest(conn, path = NULL, pattern = "*", sources = NULL)

Arguments

conn

A datom_conn object from datom_get_conn().

path

Optional path to input files directory. Defaults to input_files/ inside the repo. Not accepted on a product repo, which reads no files.

pattern

Glob pattern for file matching. Default "*". On a product repo it filters source artifact names instead.

sources

On a product repo only, and required there: one datom_conn, or a list of them, for the projects the set's inputs come from. Each connection's project name is what members are matched on. Refused on an ordinary repo.

Value

On an ordinary repo, a data frame with columns: name, file, format, original_file_sha, status (one of "new", "changed", "unchanged", "unsupported_format").

On a product repo, a data frame with columns project, name, kind, version_from (NA for a new artifact), version_to (NA for a member that was not compared) and status (one of "new", "changed", "unchanged", "ambiguous", "not_checked", "excluded"). Versions are full 64-character strings.

Details

A file counts as changed when its bytes differ from the file last synced under that name.

Files whose format is outside datom's ingestion allowlist (flat tabular formats only) are flagged "unsupported_format" up front, without blocking their allowlisted siblings.

On a product repo (mode: product) it maps the repo's set against source projects instead – see "On a product repo" below.

On a product repo

A product repo owns one set (named in .datom/project.yaml), and this call compares that set, as stored, with what each source project holds now. It reads one manifest per source and the stored set; it writes nothing.

Tables and sets are treated alike: a set a source holds gets a row, and a member that is a set is compared exactly as a table member is, so a set built from other sets syncs the same way.

One row per artifact in the sources whose name matches pattern:

  • new – no member points at it (every row, when the set has no version yet);

  • changed / unchanged – one member points at it, at an older / the current version;

  • ambiguous – two or more members point at it (a live table beside a frozen baseline, say), so neither will move. Move one with datom_update_members(), narrowing by member and tags.

Plus one row for each member the call did not compare:

  • excluded – its artifact is in a source but does not match pattern;

  • not_checked – its project was not passed in sources.

The preview never proposes removing a member. A member whose artifact is no longer listed in its source is named in the messages and left pinned. Members in the repo's own project are outputs and get no row: re-derive them, then move them with datom_update_members().

It stops when sources includes the repo's own project, and when a source connection's project name differs from the name that project's own manifest records.

Tables and sets are saved at different points. On an ordinary repo, datom_sync() writes each table as it syncs. On a product repo it hands the edited set back, and the set is saved only by datom_write_set() – one version for the whole edit, which you can look at or add to first.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Drop a source file into the repo's input_files/ directory.
  file.copy(
    system.file("extdata", "dm.csv", package = "datom"),
    file.path(tmp, "repo", "input_files", "dm.csv")
  )

  manifest <- datom_sync_manifest(conn)
  print(manifest[, c("name", "format", "status")])

  unlink(tmp, recursive = TRUE)
}
#> ℹ Created store directory /tmp/Rtmpvh48Dj/datom-example-1a767b7f2523/storage.
#> ✔ Initialized datom repository "example_project" at /tmp/Rtmpvh48Dj/datom-example-1a767b7f2523/repo
#> ℹ Scanned 1 file: 1 new, 0 changed, 0 unchanged.
#>   name format status
#> 1   dm    csv    new