Do not download if you can query
Environmental data such as spatio-temporal coverage of rainfall, sea surface temperature, or many other simulation inputs and outputs are often presented as large individual files or large collections of small or large files in formats like NetCDF or DFS. These files need to be stored in a way that makes it easy to extract parts of the data in code. Typical file system storage that we know from our laptops and PCs is expensive and difficult to manage for terabytes of data that need to be accessible from multiple service hosts. A cloud blob storage comes to mind. However, some file format libraries, such as NetCDF or DFS don't allow partial reading over the blob storage protocol or over the SMB protocol version that blob storages provide. This often leads to the following solution
var localFilePath = "/mypoject/data/temperature.dfs2"
await _blobStorageClient.DownloadAsync(remoteBlob, localFilePath);
var ncFile = nc.Open(localFilePath);
var variables = ncFile.Variables
///...
The download operation can take a long time and become prohibitively slow when working with many large files. The Platform's dedicated storage are designed to solve exactly this issue.
Consider storing a NetCDF file or DFS file in the Platform's Multidimensional service. The service API allows you to extract only parts of the data using queries, so you don't need to transfer more data than you need.