The repository shall have mechanisms in place to ensure any/multiple copies of digital objects are synchronized.

This is necessary in order to ensure that multiple copies of a digital object remain identical, within a time established as acceptable by the repository, and that a copy can be used to replace a corrupted copy of the object.

Synchronization workflows; system analysis of how long it takes for copies to synchronize; procedures/documentation of synchronization processes.

The disaster recovery plan should address what to do should a disaster and an update coincide. For example, if one copy of an object is altered and a disaster occurs while the second is being updated, there needs to be a mechanism to assure that the copy will be updated at the first available opportunity. The mechanisms to synchronize copies of digital objects should be able to detect bit corruption and validate fixity checks before synchronization is attempted.

The ingest process assures that digital objects are correctly stored S3, Glacier, and Wasabi. As part of the process, there is a series of data validation for items being ingested. The ingest validation process and the recording process take steps to be sure data is copied where it should be,as completed processes. The events of each step are all recorded as a PREMIS events within Registry, where they can be viewed by end users and storage location can be tracked. The data is saved in the Registry database permanently. 

“That all files mentioned in the manifests appear in the payload directory, and that all of their checksums match……That tag manifest checksums are correct” ingest validator

“The preservation verifier checks that all of the files copied by the preservation uploader are actually present in the preservation storage buckets. It issues a HEAD request for each file and ensures that the size matches. When possible, it also ensures that the etag matches the file’s md5 checksum. (This is only possible for smaller files, not for large, mutli-part uploads.)

If any check fails, the verifier marks the ingest as failed, sets a note about the missing/incorrect file in the note field of the WorkItem, and sets the WorkItem’s NeedsAdminReview flag to true.” ingest recorder

Fixity checks are responsible for making sure that the data exists and has not been corrupted in S3 and Wasabi. They do not happen on Glacier as that is not currently possible. 

“Preserv checks fixity on all files in S3 and Wasabi storage every 90 days. We do not run fixity checks on files in Glacier or Glacier Deep archives.”

See our technical documentation for both preservation-services and registry for detailed information on how all these processes work and how data is managed.

Well documented in the systems documentation Andrew created.