Manifest CSV

Manifest CSV

The manifest is a CSV file with file locations and metadata used to bulk upload and download files in Synapse. It is the standard manifest format used by synSyncFromSynapse(), synSyncToSynapse(), and synGenerateSyncManifest() (on both a Project and a Folder), the Synapse UI download cart, and synDownloadListFiles().

Note: This CSV manifest replaces the legacy TSV manifest produced by the legacy syncFromSynapse utility function. The legacy syncFromSynapse and syncToSynapse utility functions are deprecated. Use synSyncFromSynapse() and synSyncToSynapse() on a Project or Folder instead.

Manifest file format

The format of the manifest file is a comma-separated value (CSV) file with one row per file and columns describing the file. The minimum required columns for uploading are path and parentId, where path is the local file path and parentId is the Synapse ID of the project or folder where the file is uploaded to. Values that contain commas are automatically quoted (e.g., "hello, world").

Required fields for upload

Field Meaning Example
path local file path or URL /path/to/local/file.txt
parentId Synapse ID of parent syn1235

Note: The legacy TSV manifest used the columns parent and id, while the CSV manifest uses parentId and ID to align with Synapse REST API field names. If you’re migrating a TSV manifest to CSV, you’ll need to rename parent to parentId and id to ID.

Standard fields

These columns are recognized by synSyncToSynapse() and have specific meaning. Any of these columns may be present in the manifest but only path and parentId are required for upload. Each of these are individual examples and is what you would find in a row in each of these columns. To clarify, “syn1235;/path/to_local/file.txt” below states that you would like both “syn1235” and “/path/to_local/file.txt” added as items used to generate a file. You can also specify one item by specifying “syn1234”

Field Meaning Example
path local file path or URL /path/to/local/file.txt
parentId Synapse ID of parent container syn1235
ID Synapse entity ID syn2345
name name of file in Synapse Example_file
synapseStore whether to upload the file True
contentType content type of file to overwrite defaults text/html
forceVersion whether to update version False
activityName name of activity in provenance Ran normalization
activityDescription text description of what was done Ran algorithm xyz with parameters…
used list of items used to generate file syn1235;/path/to_local/file.txt
executed list of items executed https://github.org/;/path/to_local/code.py

Metadata fields (ignored during upload)

These columns are present in manifests generated by the Synapse UI download cart and synDownloadListFiles(). They are ignored by synSyncToSynapse() and are not treated as annotations.

Field Meaning
error any error in downloading file
versionNumber version of the file
dataFileSizeBytes size of the file in bytes
createdBy user who created the file
createdOn date the file was created
modifiedBy user who last modified
modifiedOn date last modified
synapseURL URL to the file in Synapse
dataFileMD5Hex MD5 hash of the file

Annotations

Any columns that are not in the standard or metadata fields described above will be interpreted as annotations of the file.

Adding annotations to each row:

path parentId annot1 annot2 annot3 annot4 annot5 annot6
/path/file1.txt syn1243 bar 3.1415 “aaaa, bbbb” “[14,27,30]” “Annotation, with a comma” “True”
/path/file2.txt syn12433 baz 2.71 value_1 “[1,2,3]” string without commas “[True,False]”
/path/file3.txt syn12455 zzz 3.52 value_3 “[42,56,77]” a_single_string

Multiple values of annotations per key

Using multiple values for a single annotation should be used sparingly as it makes it more difficult for you to manage the data. However, it is supported.

Annotations can be comma , separated lists surrounded by brackets [].

Because the manifest is a CSV file, multi-value annotations that contain commas are automatically quoted. For example, [a,b,c] will appear in the CSV as "[a,b,c]".

This is an annotation with 3 values:

path parentId annot1
/path/file1.txt syn1243 “[a,b,c]”

Dates in the manifest file

Dates within the manifest file will always be written as ISO 8601 format in UTC without milliseconds. For example: 2023-12-20T16:55:08Z.

Dates can be written in other formats specified in ISO 8601 and they will be recognized. However, synSyncFromSynapse() will always write dates in the UTC format specified above. For example, you may want to specify a datetime at a specific timezone like 2023-12-20 23:55:08-07:00 and this will be recognized as a valid datetime.

Manifest sources

The CSV manifest format is shared across multiple tools:

Source Filename
synSyncFromSynapse() on a Project or Folder manifest.csv
synGenerateSyncManifest() on a Project or Folder user-specified manifest_path
Synapse UI download cart manifest.csv
synDownloadListFiles() manifest_<timestamp>.csv

A manifest generated by any of these sources can be used as input to synSyncToSynapse(), provided the path column is present with valid local file paths. Manifests from the Synapse UI do not include a path column by default, so users must add it before uploading.

Example manifest file

path parentId ID name annot1 annot2 collection_date used executed
/path/file1.txt syn1243 syn5001 file1.txt bar 3.1415 2023-12-04T07:00:00Z syn124;/path/file2.txt https://github.org/foo/bar
/path/file2.txt syn12433 syn5002 file2.txt baz 2.71 2001-01-01T08:00:00Z https://github.org/foo/baz
/path/file3.txt syn12455 syn5003 file3.txt zzz 3.52 2023-12-04T07:00:00Z https://github.org/foo/zzz