--- title: "Manifest CSV" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Manifest CSV} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ## Manifest CSV The manifest is a CSV file with file locations and metadata used to bulk upload and download files in Synapse. It is the standard manifest format used by `synSyncFromSynapse()`, `synSyncToSynapse()`, and `synGenerateSyncManifest()` (on both a `Project` and a `Folder`), the Synapse UI download cart, and `synDownloadListFiles()`. > **Note:** This CSV manifest replaces the legacy TSV manifest produced by the legacy `syncFromSynapse` utility function. The legacy `syncFromSynapse` and `syncToSynapse` utility functions are deprecated. Use `synSyncFromSynapse()` and `synSyncToSynapse()` on a `Project` or `Folder` instead. ## Manifest file format The format of the manifest file is a comma-separated value (CSV) file with one row per file and columns describing the file. The minimum required columns for uploading are **path** and **parentId**, where `path` is the local file path and `parentId` is the Synapse ID of the project or folder where the file is uploaded to. Values that contain commas are automatically quoted (e.g., `"hello, world"`). ### Required fields for upload | Field | Meaning | Example | |----------|----------------------------|-------------------------| | path | local file path or URL | /path/to/local/file.txt | | parentId | Synapse ID of parent | syn1235 | > **Note:** The legacy TSV manifest used the columns `parent` and `id`, while the CSV manifest uses `parentId` and `ID` to align with Synapse REST API field names. If you’re migrating a TSV manifest to CSV, you’ll need to rename `parent` to `parentId` and `id` to `ID`. ### Standard fields These columns are recognized by `synSyncToSynapse()` and have specific meaning. Any of these columns may be present in the manifest but only `path` and `parentId` are required for upload. Each of these are individual examples and is what you would find in a row in each of these columns. To clarify, "syn1235;/path/to_local/file.txt" below states that you would like both "syn1235" and "/path/to_local/file.txt" added as items used to generate a file. You can also specify one item by specifying "syn1234" | Field | Meaning | Example | |---------------------|--------------------------------------------|----------------------------------------------| | path | local file path or URL | /path/to/local/file.txt | | parentId | Synapse ID of parent container | syn1235 | | ID | Synapse entity ID | syn2345 | | name | name of file in Synapse | Example_file | | synapseStore | whether to upload the file | True | | contentType | content type of file to overwrite defaults | text/html | | forceVersion | whether to update version | False | | activityName | name of activity in provenance | Ran normalization | | activityDescription | text description of what was done | Ran algorithm xyz with parameters... | | used | list of items used to generate file | syn1235;/path/to_local/file.txt | | executed | list of items executed | https://github.org/;/path/to_local/code.py | ### Metadata fields (ignored during upload) These columns are present in manifests generated by the Synapse UI download cart and `synDownloadListFiles()`. They are ignored by `synSyncToSynapse()` and are **not** treated as annotations. | Field | Meaning | |-------------------|-------------------------------| | error | any error in downloading file | | versionNumber | version of the file | | dataFileSizeBytes | size of the file in bytes | | createdBy | user who created the file | | createdOn | date the file was created | | modifiedBy | user who last modified | | modifiedOn | date last modified | | synapseURL | URL to the file in Synapse | | dataFileMD5Hex | MD5 hash of the file | ### Annotations Any columns that are not in the standard or metadata fields described above will be interpreted as annotations of the file. Adding annotations to each row: | path | parentId | annot1 | annot2 | annot3 | annot4 | annot5 | annot6 | | --- | --- | --- | --- | --- | --- | --- | --- | | /path/file1.txt | syn1243 | bar | 3.1415 | "aaaa, bbbb" | "[14,27,30]" | "Annotation, with a comma" | "True" | | /path/file2.txt | syn12433 | baz | 2.71 | value_1 | "[1,2,3]" | string without commas | "[True,False]" | | /path/file3.txt | syn12455 | zzz | 3.52 | value_3 | "[42,56,77]" | a_single_string | | #### Multiple values of annotations per key Using multiple values for a single annotation should be used sparingly as it makes it more difficult for you to manage the data. However, it is supported. **Annotations can be comma `,` separated lists surrounded by brackets `[]`.** Because the manifest is a CSV file, multi-value annotations that contain commas are automatically quoted. For example, `[a,b,c]` will appear in the CSV as `"[a,b,c]"`. This is an annotation with 3 values: | path | parentId | annot1 | |-----------------|----------|--------------| | /path/file1.txt | syn1243 | "[a,b,c]" | ### Dates in the manifest file Dates within the manifest file will always be written as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format in UTC without milliseconds. For example: `2023-12-20T16:55:08Z`. Dates can be written in other formats specified in ISO 8601 and they will be recognized. However, `synSyncFromSynapse()` will always write dates in the UTC format specified above. For example, you may want to specify a datetime at a specific timezone like `2023-12-20 23:55:08-07:00` and this will be recognized as a valid datetime. ## Manifest sources The CSV manifest format is shared across multiple tools: | Source | Filename | |----------------------------------------------------------------------|---------------------------------| | `synSyncFromSynapse()` on a `Project` or `Folder` | manifest.csv | | `synGenerateSyncManifest()` on a `Project` or `Folder` | user-specified `manifest_path` | | Synapse UI download cart | manifest.csv | | `synDownloadListFiles()` | `manifest_.csv` | A manifest generated by any of these sources can be used as input to `synSyncToSynapse()`, provided the `path` column is present with valid local file paths. Manifests from the Synapse UI do not include a `path` column by default, so users must add it before uploading. ### Example manifest file | path | parentId | ID | name | annot1 | annot2 | collection_date | used | executed | |-----------------|----------|---------|-----------|--------|--------|---------------------------|--------------------------|------------------------------| | /path/file1.txt | syn1243 | syn5001 | file1.txt | bar | 3.1415 | 2023-12-04T07:00:00Z | syn124;/path/file2.txt | https://github.org/foo/bar | | /path/file2.txt | syn12433 | syn5002 | file2.txt | baz | 2.71 | 2001-01-01T08:00:00Z | | https://github.org/foo/baz | | /path/file3.txt | syn12455 | syn5003 | file3.txt | zzz | 3.52 | 2023-12-04T07:00:00Z | | https://github.org/foo/zzz | ## References - [`synSyncFromSynapse`](../reference/Project_SyncFromSynapse.html) - [`synSyncToSynapse`](../reference/Project_SyncToSynapse.html) - [`synGenerateSyncManifest`](../reference/Project_GenerateSyncManifest.html) - [`synDownloadListFiles`](../reference/synDownloadListFiles.html) - [Managing custom metadata at scale](https://help.synapse.org/docs/Managing-Custom-Metadata-at-Scale.2004254976.html#ManagingCustomMetadataatScale-BatchUploadFileswithAnnotations)