Data conversion

Conversion is the key mechanism for moving data in, out, and around the platform. Users want to be able to convert data to variety of formats as described in Feature #10002.

If you are looking only for upload and download of plain files, you may want to refer directly to the File storage section, especially to the File storage quickstart section.

API and examples

All conversion end points accept the following common request structure:

{
  "readerName": "<reader name>",
  "writerName": "<writer name>",
  "readerParameters": [
    {
      "name": "<name of the parameter>",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "<name of the parameter>",
      "value": {}
    }
  ],
  "transformations": [
    {}
  ]
}

Reader and writer parameters may not be always required. However, particular readers or writers may require parameters and the transfer/conversion will fail if the parameters are not specified. If the type of parameter is not primitive (e.g. int, string, GUID) its value must be serialized in JSON format, otherwise raw value can be filled. Transformations parameter is also optional. Currently we can specify following transformations:

• CrsTransformation - transform coordinate system of the input model to the coordinate system of the output model.

{
  inputSrid: "<optional spatial reference id of input model>",
  outputSrid: "<required spatial reference id of output model>",
  type: "crsTransformation"
}

Clients list valid spatial reference ids with corresponding spatial reference definitions in WKT and proj4 string from GET/api/crs/coordinate-system/list on the gis service.

Convert existing dataset

Reads a given dataset with specified reader and imports it as a new dataset with specified writer into the storage in given project.

POST​/api​/metadata​/dataset​/{id}​/convert

Minimal request body must include:

{
  "outputDatasetData": {
    "name": "<name of the created dataset>"
  },
  "writerName": "<writer name>"
}

Reader doesn't have to be specified, because is infered by stored dataset format, but we can specify exact reader, additional reader and writer parameters together with detailed dataset data and output project ID (by default new dataset is created under the same project):

{
  "outputProjectId": "<guid>",
  "outputDatasetData": {
    "name": "<string>",
    "description": "<string">,
    "metadata": {
      "additionalProp1": {},
      "additionalProp2": {}
    },
    "properties": {
      "additionalProp1": {},
      "additionalProp2": {}
    }
  },
  "readerParameters": [
    {
      "name": "<string>",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "<string>",
      "value": {}
    }
  ],
  "readerName": "<string>",
  "writerName": "<string>"
}

Click to show example shell script to convert existing dataset

HTTP REST APISDK

datasetId="<replacewithdatasetid>"
openapikey="<replacewithopenapikey>"

# convert existing dataset
curl -L -X POST "https://api.mike-cloud-test.com/api/metadata/dataset/$datasetId/convert" \
  -H 'api-version: 3' \
  -H 'dhi-service-id: metadata' \
  -H "dhi-open-api-key: $openapikey" \
  -H 'Content-Type: application/json' \
  --data-raw '{
      "outputProjectId": "<guid>",
      "outputDatasetData": {
        "name": "<string>",
        "description": "<string">,
        "metadata": {
          "additionalProp1": {},
          "additionalProp2": {}
        },
        "properties": {
          "additionalProp1": {},
          "additionalProp2": {}
        }
      },
      "readerParameters": [
        {
          "name": "<string>",
          "value": {}
        }
      ],
      "writerParameters": [
        {
          "name": "<string>",
          "value": {}
        }
      ],
      "readerName": "<string>",
      "writerName": "<string>"
  }'

DatasetOutput output = await TransferClient.ConvertAsync(
    Guid projectId,
    Guid datasetId,
    string datasetName,
    string writerName,
    string readerName = null,
    Guid? outputProjectId = null,
    ICollection<ParameterInput> readerParameters = null,
    ICollection<ParameterInput> writerParameters = null,
    ICollection<Transformation> transformations = null,
    CancellationToken cancellationToken = default);


var filePath = "./Data/EqD2.dfs2";
var fileName = "EqD2.dfs2";
var projectId = _fixture.ProjectId;

var dataset = await _transferClient.UploadAsync(projectId, filePath, "Dfs2Reader", "VtuWriter");

var newDataset = await _transferClient.ConvertAsync(projectId, dataset.Id, fileName, "VtuWriter");

Import dataset with conversion

Reads a given file with specified reader and imports it into the storage with specified writer.

POST/api/conversion/upload-convert

Minimal request body must include:

{
  "originalFileName": "<name of file being imported - required since V2>",
  "uploadUrl": "<blob storage url of file being imported without any query part>",
  "outputDatasetData": {
    "name": "<string>"
  },
  "readerName": "<reader name>",
  "writerName": "<writer name>"
}

How to get uploadUrl see: Dataset import

Many optional parameters can be included, for example to amount to:

{
  "originalFileName": "<name of file being imported - required since V2>",
  "uploadUrl": "<blob storage url of file being imported without any query part>",
  "outputDatasetData": {
    "name": "<string>",
    "description": "<string">,
    "metadata": {
      "additionalProp1": {},
      "additionalProp2": {}
    },
    "properties": {
      "additionalProp1": {},
      "additionalProp2": {}
    }
  },
  "readerParameters": [
    {
      "name": "<string>",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "<string>",
      "value": {}
    }
  ],
  "readerName": "<string>",
  "writerName": "<string>"
}

Below is an overview of the steps where each section is used, using an example import of a dfs2 file into the multidimensional storage.

1 - Request2 - Reader3 - Model4 - Transformations5 - Writer6 - Result

Import file from ftp as is:

POST/api/conversion/upload-convert The upload url must be url encoded including the user name and password.

REST apiSDK

{
  "uploadUrl": "ftp://user:password@nrt.cmems-du.eu/Core/ARCTIC_ANALYSIS_FORECAST_BIO_002_004/dataset-topaz4-bio-arc-myoceanv2-be/20160101_dm-metno-MODEL-topaz4_norwecom2-ARC-b20160101-fv02.0.nc",
  "originalFileName": "20160101_dm-metno-MODEL-topaz4_norwecom2-ARC-b20160101-fv02.0.nc",
  "projectId": "<project id>",
  "outputDatasetData": {"name" : "file_from_ftp.nc" },
  "readerName": "FileReader",
  "writerName": "FileWriter"
}

var transferResult = await transferClient
    .CreateUrlImport(projectId, "ftp://user:password@nrt.cmems-du.eu/Core/ARCTIC_ANALYSIS_FORECAST_BIO_002_004/dataset-topaz4-bio-arc-myoceanv2-be/20160101_dm-metno-MODEL-topaz4_norwecom2-ARC-b20160101-fv02.0.nc", "20160101_dm-metno-MODEL-topaz4_norwecom2-ARC-b20160101-fv02.0.nc","file_from_ftp.nc")
    .ExecuteAsync();

Export dataset with conversion

Reads a given dataset from the storage with specified reader and exports it with specified writer into the blob storage.

POST/api/metadata/dataset/{id}/download-convert

Minimal request body must include:

{
  "writerName": "string"
}

Reader doesn't have to be specified, because is infered by stored dataset format, but we can specify exact reader and additional reader and writer parameters:

{
  "readerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "readerName": "string",
  "writerName": "string"
}

After the transfer is completed, we can download the exported file from blob storage. We get the url of exported file from response of /api/conversion/transfer/{id}.

Click to show example shell script to export dataset with conversion

HTTP REST APISDK

datasetId="<replacewithdatasetid>"
openapikey="<replacewithopenapikey>"

# export dataset with conversion
curl -L -X POST "https://api.mike-cloud-test.com/api/metadata/dataset/$datasetId/download-convert" \
  -H 'api-version: 3' \
  -H 'dhi-service-id: metadata' \
  -H "dhi-open-api-key: $openapikey" \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "readerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "writerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "readerName": "string",
    "writerName": "string"
  }'

await TransferClient.DownloadAsync
    ( Guid projectId,
      Guid datasetId,
      string targetPath,
      string readerName,
      string writerName,
      ICollection<ParameterInput> readerParameters = null,
      ICollection<ParameterInput> writerParameters = null,
      ICollection<Transformation> transformations = null,
      CancellationToken cancellationToken = default);


var filePath = "./Data/EqD2.dfs2";
var fileName = "EqD2.dfs2";
var outputFilePath = Path.Join(_tempDir, Guid.NewGuid() + fileName);
var projectId = _fixture.ProjectId;

var dataset = await _transferClient.UploadAsync(projectId, filePath, "Dfs2Reader", "VtuWriter");

await _transferClient.DownloadAsync(projectId, dataset.Id, outputFilePath, "VtuReader", "VtuWriter");

Update dataset with conversion

Reads a given file with specified reader and updates existing dataset in the storage with specified writer (updater).

POST/api/metadata/dataset/{id}/update

Minimal request body must include:

{
  "originalFileName": "<name of updating file>",
  "uploadUrl": "<blob storage url of updating file without any query part>",
  "readerName": "<reader name>"
}

How to get uploadUrl see: Dataset import

Writer doesn't have to be specified, because is infered by stored dataset format, but we can specify exact writer and additional reader and writer parameters:

{
  "originalFileName": "<name of updating file>",
  "uploadUrl": "<blob storage url of updating file without any query part>",
  "readerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "readerName": "string",
  "writerName": "string"
}

Below is an overview of the steps where each section is used, using an example update of an existing dataset in the multidimensional storage. The dataset is based on NOAA data, and it is updated with a set of timesteps from a new file.

1 - Request2 - Reader3 - Transformations4 - Validation5 - Update6 - Result

Click to show example shell script to update dataset with conversion

HTTP REST APISDK

datasetId="<replacewithdatasetid>"
openapikey="<replacewithopenapikey>"

# Update dataset with conversion
curl -L -X POST "https://api.mike-cloud-test.com/api/metadata/dataset/$datasetId/update" \
  -H 'api-version: 3' \
  -H 'dhi-service-id: metadata' \
  -H "dhi-open-api-key: $openapikey" \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "originalFileName": "<name of updating file>",
    "uploadUrl": "<blob storage url of updating file without any query part>",
    "readerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "writerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "readerName": "string",
    "writerName": "string"
  }'

    DatasetOutput output = await TransferClient.UpdateAsync
    (   Guid projectId,
        Guid datasetId,
        string filePath,
        string readerName,
        string writerName = null,
        ICollection<ParameterInput> readerParameters = null,
        ICollection<ParameterInput> writerParameters = null,
        ICollection<Transformation> transformations = null,
        CancellationToken cancellationToken = default);


    var filePath = "./Data/EqD2.dfs2";
    var projectId = _fixture.ProjectId;

    var dataset = await _transferClient.UploadAsync(projectId, filePath, "Dfs2Reader", "MDWriter");

    await _transferClient.UpdateAsync(projectId, dataset.Id, filePath, "Dfs2Reader");

Append dataset with conversion

Reads a given file with specified reader and appends existing dataset in the storage with specified writer (appender).

POST/api/metadata/dataset/{id}/append

Minimal request body must include:

{
  "originalFileName": "<name of appending file - required since V2>",
  "uploadUrl": "<blob storage url of appending file without any query part>",
  "readerName": "<reader name>"
}

How to get uploadUrl see: Dataset import

Writer doesn't have to be specified, because is infered by stored dataset format, but we can specify exact writer and additional reader and writer parameters:

{
  "originalFileName": "<name of appending file - required since V2>",
  "uploadUrl": "<blob storage url of appending file without any query part>",
  "readerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "writerParameters": [
    {
      "name": "string",
      "value": {}
    }
  ],
  "readerName": "string",
  "writerName": "string"
}

Click to show example shell script to append dataset with conversion

HTTP REST APISDK

datasetId="<replacewithdatasetid>"
openapikey="<replacewithopenapikey>"

# Append dataset with conversion
curl -L -X POST "https://api.mike-cloud-test.com/api/metadata/dataset/$datasetId/append" \
  -H 'api-version: 3' \
  -H 'dhi-service-id: metadata' \
  -H "dhi-open-api-key: $openapikey" \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "originalFileName": "<name of appending file - required since V2>",
    "uploadUrl": "<blob storage url of appending file without any query part>",
    "readerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "writerParameters": [
      {
        "name": "string",
        "value": {}
      }
    ],
    "readerName": "string",
    "writerName": "string"
  }'

    DatasetOutput output = await TransferClient.AppendAsync
    (   Guid projectId,
        Guid datasetId,
        string filePath,
        string readerName,
        string writerName = null,
        ICollection<ParameterInput> readerParameters = null,
        ICollection<ParameterInput> writerParameters = null,
        ICollection<Transformation> transformations = null,
        CancellationToken cancellationToken = default);


    var filePath = "./Data/EqD2.dfs2";
    var projectId = _fixture.ProjectId;

    var dataset = await _transferClient.UploadAsync(projectId, filePath, "Dfs2Reader", "MDWriter");

    await _transferClient.AppendAsync(projectId, dataset.Id, "./Data/EqD2-append.dfs2", "Dfs2Reader");

Readers/Writers

You must specify valid read name and writer name. List of available readers and writers with specific parameters they require can be obtained using end points - GET/api/conversion/reader/list - GET/api/conversion/writer/list

Response of the end point includes transfer id so that the progress of each conversions can be monitored as any other transfer.

As of July 2019 DEV environment can convert: - ASCII Grid <=> GeoTIFF - Shapefile <=> GeoJson - DFS0 => VTU - DFS1 => VTU - DFS2 => VTU - DFS3 => VTU - DFSU => VTU - XYZ => VTU

For more details see list of supported readers and list of supported writers or subpages of this page.