Skip to content

Reader - CSV Zip

ReaderName: CsvZipReader

This reader can read data from a zip file with many time series data in it.

Transfer parameters

Transfer parameters are used to describe data archive and data files in the archive - file names, fields delimiter and field names.

transfer parameter name value (default) description
Datafile ^(data(?<NUM>.*)\\.txt)$ pattern used to find time series data files in ZIP archive, it is not a full regular expression, just the data and txt parts can be changed.
Metafile metadata$(NUM).txt pattern for time series metadata file, it is not a full regular expression, just the metadata and txt parts can be changed.
FlagsDefFile $(ROOT)flagsdef.txt filename pattern for flags definition file, is not a full regular expression, mainly just the flagsdef and txt parts can be changed, $(ROOT) is used when data files are in sub-folders, but flags definition file is in the root when all time series have the same definition
DatasetMetafile datasetmetafile.txt metadata file for the whole dataset
DataFieldDelimiter ; data fields delimiter in CSV file
TimeDataField REFDATETIMEREF time field name
ValueDataField DATA data value field name
QualityDataField QUALITYID quality value field name, empty if flags are not in the data file or if flags should not be imported, archive must contain flags definitions (FlagsDefFile) for quality values to import quality values from data file

Blob data - csv-in-zip format

Blob data is ZIP archive (application/zip) with two or more files: CSV data file and time series metadata file for each exported time series (mandatory), flags definition file (optional) and dataset metadata file (optional). File names are specified in the transfer parameters (Datafile, Metafile, FlagsDefFile, DatasetMetafile). Files structure in the archive can be flat (all files are in the root folder) or data and metadata files can be in sub-folders.

Simple ZIP archive with one time series (no flags definition, no dataset metadata):

dataarchive.zip
├── data.txt             # time series data
└── metadata.txt         # time series metadata

ZIP archive with two time series, flat file structure:

dataarchive.zip
├── data1.txt            # time series 1 - data
├── metadata1.txt        # time series 1 - metadata
├── data2.txt            # time series 2 - data
├── metadata2.txt        # time series 2 - metadata
├── datasetmetafile.txt  # dataset metadata
└── flagsdef.txt         # flags definition

ZIP archive with two time series, files in subfolders:

dataarchive.zip
├── 0001   ├── data.txt         # time series 1 - data   └── metadata.txt     # time series 2 - metadata
├── 0002   ├── data.txt         # time series 2 - data   └── metadata.txt     # time series 2 - metadata
├── datasetmetafile.txt  # dataset metadata
└── flagsdef.txt         # flags definition

ZIP file example with two time series and flags definitions is attached (flat structure). sample_csv_zip_data.zip

Metadata file

Metadata file is file in JSON format (UTF8) that contains additional information about time series: name, dimension, datatype, description, location, locationlatitude, locationlongitude, item, itemEumIdent, itemMzId, unit, unitEumIdent, unitMzId.

Example

{
"name":"TRN.LT-TB702-Z_KOTE",
"dimension":0,
"datatype":"Instantaneous",
"description":"B747 - Kote Trøjborg bassin",
"locationname":"BAS 747 Trøjborg Bassin, Dr. Margrethesvej 16",
"locationlatitude":56.1516215254962,
"locationlongitude":10.1752495765686,
"location":{
 "type":"Point",
 "coordinates":[10.1752495765686,56.1516215254962]
},
"item":"L",
"itemEumIdent":"eumIWaterLevel",
"itemMzId":100000,
"unit":"m",
"unitEumIdent":"eumUmeter",
"unitMzId":1000
}

CSV data file

Fields are separated with a character specified in the transfer parameter DataFieldDelimiter (;). Numeric data values use decimal dot. Time values are in yyyy-mm-dd hh:nn:ss format. Fields with time, data and quality values are specified in the transfer parameters TimeDataField (REFDATETIMEREF), ValueDataField (DATA) and QualityDataField (QUALITYTXT). Additional fields in CSV file are skipped.

Example

REFDATETIMEREF;DATA;QUALITYID;QUALITYTXT
2016-05-01 00:00:00;5.856770859;2;ok
2016-05-01 00:01:00;5.856770859;2;ok
2016-05-01 00:02:00;5.861111145;2;ok
2016-05-01 00:03:00;5.861111145;2;ok

Flags definition file

Flags definition file is JSON file (UTF8) that contains information about flags values: id, name, description, level.

Example

{
"data": [
{
"id": 1,
"name": "",
"description": "Undefined data quality",
"level": -1000
},
{
"id": 2,
"name": "ok",
"description": "Default data quality",
"level": 0
},
{
"id": 3,
"name": "error",
"description": "Error data quality",
"level": -15
},
{
"id": 4,
"name": "critical",
"description": "Critical data quality",
"level": -10
},
{
"id": 5,
"name": "warning",
"description": "Warning data quality",
"level": -5
},
{
"id": 6,
"name": "estimated",
"description": "Estimated data quality",
"level": -12
}
]
}

Dataset metadata file

Metadata file is file in JSON format (UTF8) that contains additional information about the dataset.

Example

{
"locationname":"BAS 747 Trøjborg Bassin, Dr. Margrethesvej 16",
"projectnumber":"#1235678"
}