Preface

Transfer client

The ITransferClient is the entry point for operations related to files or existing datasets. You can construct it manually, but typically you will use a DI container to inject an instance of the class. In the following examples we will assume that the variable transferClient holds a reference to an instance of ITransferClient.

Transfer process

Each operation that imports or downloads data is represented by a transfer. Transfer is a process that runs somewhere in the cloud, and may potentially run for a long time. Therefore its lifecycle is initiated at some point by one request to the cloud platform, and then its status has to be tracked by polling or other mechanisms. At the end of its run, the transfer will provide its result - either a new dataset in case of an import, or a url where the requested file can be downloaded

How to download plain files

The most basic operation is downloading a simple file. The operation starts with an existing dataset in the cloud platform (identified by a project id and a dataset id), and ends with a url that can be used to download the file (for a limited timeframe).

var transferResult = await transferClient
                .CreateDatasetDownload(projectId, datasetId)
                .ExecuteAndWaitAsync();

// this is just a sample code on how to consume the url
using var httpClient = new HttpClient();
using var response = await httpClient.GetAsync(transferResult.Url);
using var stream = new FileStream("C:\\Temp\\sdktest\\output.file", FileMode.Create);
await response.Content.CopyToAsync(stream);

How to convert and download plain files

Files can also be converted before they are downloaded (without modifying the source dataset). The conversion is done by selecting the appropriate Reader and Writer for the file. If we have a file in the platform that we know is a Dfs2 file, we can use the Dfs2Reader and a VtuWriter to convert thisfile into a VTU file that we can download.

var transferResult = await transferClient
                .CreateDatasetDownload(projectId, datasetId)
                .WithReader(new Dfs2Reader())
                .WithWriter(new GenericWriter("VtuWriter"))
                .ExecuteAndWaitAsync();

Transformations can also be applied during this operation, as described in the How to upload files guide.

How to download data from queryable storages (Multidimensional, TimeSeries, ...)

In case the source dataset is not a physical file, but rather a queryable object residing in one of the data services (Multidimensional, Timeseries), the download needs to be more involved. The output format has to be selected by specifying a Writer (the Reader in this case is determined by the type of the dataset, so it does not have to be explicitly specified).

In this code the dataset in the multidimensional storage is exported into a Dfs2 file.

var transferResult = await transferClient
                .CreateDatasetDownload(projectId, datasetId)
                .WithWriter(new Dfs2Writer())
                .ExecuteAndWaitAsync();

Keep in mind that not all storages (or Readers in the context of a download/export) will support all writers. For example, the TimeSeries storage does not support exporting data into a Dfs2 file, because such combination simply does not make much sense. The allowed combinations of readers/writers as prescribed by the platform API still have to be observed.

Filtering data

With data coming from dedicated storages like the Multidimensional storage it is advised to filter the data so that the output file only contains the data you need. The main objective is to limit the size of the output, as well as processing time.

The data filtering can be achieved with extensions WithItemFilteringTransformation, WithTimeFilteringTransformation and WithSpatialFilteringTransformation. In this example the multidimensional dataset is filtered with a geometry, item index, and a temporal range.

var transferResult = await transferClient
                .CreateDatasetDownload(projectId, datasetId)
                .WithWriter(new Dfs2Writer())
                .WithItemFilteringTransformation(0) // include only the first item in the output
                .WithSpatialFilteringTransformation("POLYGON((0 0, 0 2, 2 2, 2 0, 0 0))") // filter spatial domain by polygon
                .WithTimeFilteringTransformation(from: new DateTime(2002, 1, 1), to: new DateTime(2002, 1, 1, 0, 10, 0)) // filter temporal domain
                .ExecuteAndWaitAsync();