Skip to content

Storage container structure

To support multitenancy, each tenant (customer) has its own storage container. Each storage container contains individual datsets. Each dataset is defined by unique and static id

image.png

Dataset structure

  • Dataset root folder contains files with dataset definition and optional spatial indexes image.png
  • Subfolders contains data in different formats
  • Raw contains the data in raw format , reorganized by spatial index
  • Other subfolders can contain for example generated tiles, ect.
  • Each subfolder is further organized by item and layer
  • The data blobs contain data organized by the selected strategy

image.png

Dataset metadata

Each dataset folder contains dataset.json blob with full metadata description of the dataset including - dataset identification and version - spatial domain with optional vertical domain definition - temporal domain definition - items definitions - item storage options

Dataset.json sample: 

{
  "Id": "efae3d23-7d0f-4073-ae4b-8e6678ba9d08",
  "Name": "2018010100F000",
  "SchemaVersion": "2019-09-30",
  "SpatialDomain": {
    "Type": "EquidistantGridD2",
    "ElementCount": 8817120,
    "Dy": 0.083335876,
    "Ny": 2041,
    "Y0": -80.0416667,
    "Dx": 0.083328247,
    "Nx": 4320,
    "X0": -180.0416667,
    "SRID": 4326,
    "VerticalDomain": {
      "Name": "VerticalDomain",
      "UnitId": "eumUmeter",
      "Datum": { "Origin": 0.49402499 },
      "Orientation": "Down",
      "MaxZ": [],
      "Transformation": {
        "Depths": [ 0.49402499, 1.541375, 2.645669, 3.819495, 5.0782242, 6.4406142, 7.9295602, 9.5729971, 11.405, 13.46714, 15.81007, 18.49556, 21.59882, 25.211411, 29.444731, 34.434151, 40.344051, 47.373692, 55.76429, 65.807266, 77.853851, 92.326073, 109.7293, 130.666, 155.85069, 186.1256, 222.4752, 266.04031, 318.12741, 380.21301, 453.93771, 541.08893, 643.56677, 763.33313, 902.33929, 1062.4399, 1245.291, 1452.251, 1684.2841, 1941.8929, 2225.0779, 2533.3359, 2865.7029, 3220.8201, 3597.032, 3992.4839, 4405.2241, 4833.291, 5274.7842, 5727.917 ],
        "Name": "NonEquidistantTransformation"
      },
      "LayerCount": 50,
      "HasTimeVariableValues": false
    }
  },
  "TemporalDomain": {
    "Stamps": [ "2018-01-01T12:00:00" ],
    "StartTime": "2018-01-01T12:00:00",
    "EndTime": "2018-01-01T12:00:00",
    "Count": 1
  },
  "Items": [
    {
      "StoredType": "Int16",
      "StoredNoDataValue": -32767,
      "ValueConverter": {
        "ScaleFactor": 0.00061037018895149231,
        "ScaleOffset": 0.0
      },
      "NoDataValue": -19.999999981373549,
      "Name": "Eastward velocity",
      "Quantity": {
        "ItemId": "eumIBedThickness",
        "UnitId": "eumUmeter"
      },
      "Type": "Double",
      "ItemType": "ConvertibleItem`2",
      "ConverterType": "Scaled"
    }
  ],
  "ItemStorage": {
    "UseSpatialIndex": true,
    "TimeGroupSize": 0,
    "StorageProvider": "SingleTimestep"
  }
}

Grid elements storage

Grid is fully defined by its defintition in metadata.

Grid index is based on the QuadTree algorithm. The index is built in memory from grid definition.

image.png

Mesh domain storage

Mesh domains are defined as a collection of Mesh elements.

We are using R-Tree spatial Index (https://en.wikipedia.org/wiki/R-tree) in Sort Tile Recursive implementation from NetTopologySuite.

For more details see mesh storage description