Reader - GFS/CFS NOAA GRIB

ReaderName: NoaaGribReader

This reader is used to import files from GFS or CFS NOAA analysis product to the multidimensional storage. To see which data is available and where to get it, please go here.

The resulting dataset will have the following properties:

• Spatial domain is an equidistant grid
• The CRS is taken from the - it will most likely end up as SRID 700101 - it is a projection to a sphere with lon/lat coordinates.
• Extracted items are named using a long name of the variable and a reference domain, e.g. v-component of wind @ Isobaric surface. If the values are defined with a vertical domain, the layers will be split into separate items and the number of the layer will be appended to the name (e.g. v-component of wind @ Isobaric surface 0 for the first layer). This is done because there is no common vertical domain defined within the file, and the multidimensional storage model only supports one vertical domain that is shared for all items.
• Due to the high number of items contained within each file it is advisable to filter the items so that only relevant data is stored. This is done by using AllowedItemNames reader parameter. The request payload for such filtered conversion might look like this:

      {
        projectId: <projectId>,
        uploadUrl: <fileUrl>,
        outputDatasetData: {
          name: <dataset-name>
        },
        readerName: 'GfsGribReader',
        writerName: 'MDWriter',
        readerParameters: [
          {
            name: 'AllowedItemNames',
            value: '["v-component of wind @ Isobaric surface 0", "v-component of wind @ Isobaric surface 1"]'
          }
        ]
      }
  

• Mapping to EUM enumerations is done in part by the reader itself, but due to high number of different items in these files it is likely that some items will not be mapped by default. In that case please use reader property QuantityMapItems. It is an array of objects that look like this:

  {
    itemName: "Temperature", 
    itemId: "eumITemperature", 
    unitId: "eumUdegreeCelsius"
  }
  

  and allow the user to specify how to map the items in the file to EUM items and units.

Optional reader parameter is "CoordinateInterpretationType", which gives a hint to the reader as to what representation to assign to coordinates found in the file. Values are "CenterOfCell", which means the coordinates represent centers of cells (default), or "BottomLeftBorder", which indicates that the coordinates are given with respect to the bottom left border of the cell.

ReaderName: NoaaGribReaderV2

Similarly to the previous reader, this one is used to read GFS or CFS NOAA data from GRIB files. The difference between the two is the method of extracting and separating the items (variables) from the file. The NoaaGribReader has some deficiencies that may result in the items being mismatched - this reader attempts to remedy that.

Each GRIB file has a number of bands inside. Each band represents a raster (which holds data) with some metadata (which describe what kind of variable the data represents). It is important to note that the metadata description can be quite complicated. The item definition can be comprised of the variable/element name, but also the definition of a reference surface (or surfaces), generating process, interval within which it is valid and so on. This extra information (apart from the element name) is contained within a descriptor called "Product template definition" - see link. It is a set of numbers with known meaning that specify what the band data represents. It is based on this description that the reader tries to identify individual items, and combine bands from the file under each item (in case there are multiple time steps within the file).

Since this process can be a bit opaque to the user, the resulting items have the indication about the original band indices in its name. For example, item that is described by this metadata (output from gdalinfo utility):

Band 2 Block=192x1 Type=Float64, ColorInterp=Undefined
  Description = 1[-] HYBL="Hybrid level"
  Metadata:
    GRIB_COMMENT=Temperature [C]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=TMP
    GRIB_FORECAST_SECONDS=0 sec
    GRIB_IDS=CENTER=7(US-NCEP) SUBCENTER=0 MASTER_TABLE=2 LOCAL_TABLE=1 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2020-02-02T12:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)
    GRIB_PDS_PDTN=0
    GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES=0 0 2 0 197 0 0 1 0 105 0 1 255 0 0
    GRIB_PDS_TEMPLATE_NUMBERS=0 0 2 0 197 0 0 0 1 0 0 0 0 105 0 0 0 0 1 255 0 0 0 0 0
    GRIB_REF_TIME=  1580644800 sec UTC
    GRIB_SHORT_NAME=1-HYBL
    GRIB_UNIT=[C]
    GRIB_VALID_TIME=  1580644800 sec UTC

will produce an item with name Temperature [C] (Band 2) - the human-readable identifier from GRIB_COMMENT section is used as a base, but because there can be multiple bands with the same GRIB_COMMENT representing different temperature measurements, the name is made unique by appending the original band index. In case the item is constructed from multiple bands (because of the presence of multiple time steps within the file) all the indices will be listed there.

It is recommended to use these names to precisely locate the data that you want, and once that is clear, you can use an item-redefining transformation to change the name of the items to something more meaningful.

Parameters

• SRID - (optional) allows the user to override the SRID associated with the dataset.

• CoordinateInterpretationType - (optional) gives a hint to the reader as to what representation to assign to coordinates found in the file. Values are "CenterOfCell", which means the coordinates represent centers of cells, or "BottomLeftBorder" (default), which indicates that the coordinates are given with respect to the bottom left border of the cell.

• DoubleAsFloat - (optional) boolean flag indicating whether to store variables originally represented as double (64 bit float) with single precision (32 bit). The reason why this option is here is the fact that even though the item in the NOAA file may indicate a 64 bit precision, in reality the amount of information can be much smaller, so it might make sense to lower the precision in the resulting model as well to save on storage space.

• BandItemMap - (optional) in case the item-extracting mechanism fails (maybe because the necessary metadata is not present in the file) you can use this reader parameter to specify how to construct individual items. The parameter value is an array of objects, where each object has a property itemName indicating the desired name of the item, and then an array of integers bands, indicating which band indices to include in the item (note that nads are indexed starting with 1). Example:

HTTP REST APISDK

    {
      projectId: <projectId>,
      uploadUrl: <fileUrl>,
      outputDatasetData: {
        name: <dataset-name>
      },
      readerName: 'NoaaGribReaderV2',
      writerName: 'MDWriter',
      readerParameters: [
        {
          bandItemMap: [
            { 'itemName': 'my-temperature', 'bands': [1, 2] },
            { 'itemName': 'my-ice-coverage', 'bands': [3, 4] },
          ]
        }
      ]
    }

    var result = await transferClient
        .CreateFileImport(projectId, "noaa.grib2")
        .WithReader(new GenericReader("NoaaGribReaderV2", new List<ParameterInput>
        {
            new ParameterInput
            {
                Name = "BandItemMap", 
                Value = "[ { \"itemName\": \"my-temperature\", \"bands\": [1, 2] }, { \"itemName\": \"my-ice-coverage\", \"bands\": [3, 4] }, ]"
            }
        }))
        .WithWriter(new MDWriter())
        .ExecuteAndWaitAsync();

ReaderName: NoaaGribReaderV3

The latest version of reader for NOAA grib files. This one is based on NGrib library - https://www.nuget.org/packages/NGrib/

The reader will group data slices by their parameter name (e.g. "Temperature") and it will then create an item for each of these groups. All items must have the same number of timesteps associated with it (so that it complies with the multidimensional model definition).

Parameters

• SRID - (optional) allows the user to override the SRID associated with the dataset.

• CoordinateInterpretationType - (optional) gives a hint to the reader as to what representation to assign to coordinates found in the file. Values are "CenterOfCell", which means the coordinates represent centers of cells, or "BottomLeftBorder" (default), which indicates that the coordinates are given with respect to the bottom left border of the cell.

• DoubleAsFloat - (optional) boolean flag indicating whether to store variables originally represented as double (64 bit float) with single precision (32 bit). The reason why this option is here is the fact that even though the item in the NOAA file may indicate a 64 bit precision, in reality the amount of information can be much smaller, so it might make sense to lower the precision in the resulting model as well to save on storage space.

• AddBandsToItemNames - (optional) boolean flag that can be used to enhance item names with the band indexes upon which the item is based on. This can be useful if there are many items with different timesteps - the item name will the indicate which bands were used for which item.