Monitoring running transfers
When a new Transfer is created, regardless of the transfer type, it goes through several stages and clients may want to monitor the progress and find out when the transfer has completed or failed.
Polling¶
Clients can call the GET/api/conversion/transfer/{id} endpoint at regular intervals. The endpoint returns all details about a specific Transfer, including its status.
{
"type": "Export",
"status": "Completed",
"format": "gdalvector",
"projectId": "6b4c70c4-ef2f-4b39-acb0-1e4b24313e8e",
"exportParameters": {
"datasetId": "04d0992a-056b-44b2-a8d8-93c93ba54f3a",
"outputFileName": "test"
},
"downloadPath": "http://127.0.0.1:10000/devstoreaccount1/importexport/..."
"id": "ee5e0f90-c292-487c-abde-0dbcaed6eb09",
"createdAt": "2019-01-24T08:17:02.4355712",
"createdBy": "862281a7-2e66-4c76-af8d-29c82c723b4b",
"updatedAt": "2019-01-24T08:17:05.8255626",
"updatedBy": "862281a7-2e66-4c76-af8d-29c82c723b4b"
}
The status property can have the following values:
- None - the request has not been queued.
- Pending - Waiting in the queue
- InProgress
- Completed
- Error
When status is Completed, the Transfer details contain also a results section with additional information.
Calling GET/api/conversion/transfer/list-summaries returns information about multiple transfers in one request.
Platform eventing service¶
The Platform eventing service can be used to subscribe to the events emmitted by transfer pipelines. For a guide on how to subscribe to platform events, see Subscribe to events. All transfer pipelines provide transfer started/finished and progress event messages. Example of transfer started event:
{
"specversion": "1.0",
"customerid": "980552ba-58a1-4742-9248-cbcf3b3a4ece",
"projectid": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"userid": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
"resourceid": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"type": "dhi.waterdata.events.transferstartedevent",
"source": "/dhi/platform/transfer",
"id": "ad899ad7-28fd-43f2-aa94-6f51b73921dc",
"time": "2021-08-16T09:42:25.8079022Z",
"subject": "Transfer started",
"dataschema": "https://schemas.mike-cloud.com/platform/transfer/v1/transferstartedevent",
"datacontenttype": "application/json",
"data": {
"TransferId": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"CustomerGuid": "10000000-0000-0000-0000-000000000000",
"ProjectId": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"UserId": "00000000-0000-0000-0000-000000000001"
},
"offset": "90209790600"
}
Example of transfer progress event:
{
"specversion": "1.0",
"customerid": "980552ba-58a1-4742-9248-cbcf3b3a4ece",
"projectid": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"userid": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
"resourceid": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"type": "dhi.waterdata.events.transferprogressevent",
"source": "/dhi/platform/transfer",
"id": "f0a06897-4830-4190-95e9-edf7ef6cf25c",
"time": "2021-08-16T09:42:32.354819Z",
"subject": "Transfer progress",
"dataschema": "https://schemas.mike-cloud.com/platform/transfer/v1/transferprogressevent",
"datacontenttype": "application/json",
"data": {
"TransferId": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"Message": "Transfer Id - '3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23': Uploading outputs",
"LastUpdated": null,
"ProgressPosition": null,
"ProgressPositionMax": null,
"CustomerGuid": "10000000-0000-0000-0000-000000000000",
"ProjectId": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"UserId": "00000000-0000-0000-0000-000000000001"
},
"offset": "90209798400"
}
Example of transfer finished event:
{
"specversion": "1.0",
"customerid": "980552ba-58a1-4742-9248-cbcf3b3a4ece",
"projectid": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"userid": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
"resourceid": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"type": "dhi.waterdata.events.transferfinishedevent",
"source": "/dhi/platform/transfer",
"id": "c0336932-648c-41d7-9e10-d621551ba841",
"time": "2021-08-16T09:42:42.8995453Z",
"subject": "Transfer finished",
"dataschema": "https://schemas.mike-cloud.com/platform/transfer/v1/transferfinishedevent",
"datacontenttype": "application/json",
"data": {
"TransferId": "3e97a3ea-c35f-4d0b-96be-15ed8ed8aa23",
"ErrorMessage": null,
"OutputFileAbsoluteUri": "http://127.0.0.1:10000/devstoreaccount1/importexport/b5e9d645-c555-4ef8-83da-a2d9a865290d?sv=2020-04-08&spr=https,http&se=2021-08-16T21%3A42%3A34Z&sr=b&sp=r&sig=gqjwtUBDd6I7H2AnAwWvPLVsRJOzqvjU6UQ65io5JMc%3D",
"TransferResultType": "Import",
"ResultSizeKB": 2,
"AffectedDatasetIds": [
"da9c62a6-5da4-4fc1-ab15-1a0a8a37bee0"
],
"CustomerGuid": "10000000-0000-0000-0000-000000000000",
"ProjectId": "b3ace22a-d66c-44f7-9421-c8513506ff6b",
"UserId": "00000000-0000-0000-0000-000000000001"
},
"offset": "90209814000"
}
Multidimentional and tiling transfer pipelines will send progress event message every minute and it will include current step text message, step position if relevant and when the progress step message was last updated:
{
"specversion": "1.0",
"customerid": "980552ba-58a1-4742-9248-cbcf3b3a4ece",
"projectid": "5fa6db8e-0d1c-4bd8-8d32-fbf460687425",
"userid": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
"resourceid": "a053e83e-a2f8-42d8-b047-64ddc4fa446c",
"type": "dhi.waterdata.events.transferprogressevent",
"source": "/dhi/platform/transfer",
"id": "99909f4e-ceae-4121-bdce-e0065090c42c",
"time": "2021-08-16T12:25:54.3893824Z",
"subject": "Transfer progress",
"dataschema": "https://schemas.mike-cloud.com/platform/transfer/v1/transferprogressevent",
"datacontenttype": "application/json",
"data": {
"TransferId": "a053e83e-a2f8-42d8-b047-64ddc4fa446c",
"Message": "Transfer Id - 'a053e83e-a2f8-42d8-b047-64ddc4fa446c': MD writer done processing.",
"LastUpdated": "2021-08-16T12:25:53.3893824Z",
"ProgressPosition": "1",
"ProgressPositionMax": "2",
"CustomerGuid": "10000000-0000-0000-0000-000000000000",
"ProjectId": "5fa6db8e-0d1c-4bd8-8d32-fbf460687425",
"UserId": "00000000-0000-0000-0000-000000000001"
},
"offset": "90211734688"
}
SignalR subscription¶
SignalR subscription has been discontinued in March 2019 when we changed the application registration and authentication for the metadata service
SignalR client are available for multiple technologies, here is a sample C# code (provide action func to process progress data):
var connection = new HubConnectionBuilder().WithUrl(_baseUrl + @"/transfer-progress").Build();
connection.On<TransferMonitorData>("UpdateProgress", (data) => action(data));
await _connection.StartAsync();
await _connection.InvokeAsync("SubscribeToProgress", id);
{
"status": "InProgress",
"message": "Some text describing the current task",
}