Overview
The File storage service is where clients can store raw files in their original form. The files are still registered as a dataset in the metadata service, as any other type of dataset. It is up to the client to use the files appropriately as the format or internal structure of the files are not known. Clients can interact with the files through the web api and by using the transfer pipeline.
Endpoints¶
Internal endpoints for interaction with the transfer/conversion pipeline
- GET
/api/raw/prepare-download/{datasetId}
Prepare download of data for dataset. - GET
/api/raw/prepare-upload/{datasetName}
Prepare upload of data for dataset.
File synchronization - for integration primarily with desktop clients
- GET
/api/filesync/{id}/blocks
Get file blocks with checksums. - GET
/api/filesync/checksums
Get checksums for selected or all files in a folder. - GET
/api/filesync/prepare-download
Get download info for selected or all files in a folder. - PUT
/api/filesync/upload
Upload file datasets from the staging area or another storage. Files in the staging area moved, all other files are copied. Check the copy operations with POST/api/filesync/upload/status
. - POST
/api/filesync/upload/status
Get upload copy operations status, copy operations started in "PUT /api/filesync/upload".
Main endpoint for accessing an individual file
- GET
/api/raw/dataset/{id}
Prepare download of data for dataset. When you call this service directly you need to provide a SAS token for the given dataset with read privilege to download data for a dataset.
Endpoints for staging files during transfer or conversion
- POST
/api/raw/move-staged-url
Move data from staging storage to dataset. - GET
/api/raw/staging-url
Prepare staging blob storage for data upload. - POST
/api/raw/staging-url
Check if given url is a staging url - GET
/api/raw/staging-urls
Prepare staging blob storages for data upload.