Reader - CSV Zip
ReaderName: CsvZipReader
This reader can read data from a zip file with many time series data in it.
Transfer parameters¶
Transfer parameters are used to describe data archive and data files in the archive - file names, fields delimiter and field names.
transfer parameter name | value (default) | description |
---|---|---|
Datafile | ^(data(?<NUM>.*)\\.txt)$ |
pattern used to find time series data files in ZIP archive, it is not a full regular expression, just the data and txt parts can be changed. |
Metafile | metadata$(NUM).txt |
pattern for time series metadata file, it is not a full regular expression, just the metadata and txt parts can be changed. |
FlagsDefFile | $(ROOT)flagsdef.txt |
filename pattern for flags definition file, is not a full regular expression, mainly just the flagsdef and txt parts can be changed, $(ROOT) is used when data files are in sub-folders, but flags definition file is in the root when all time series have the same definition |
DatasetMetafile | datasetmetafile.txt |
metadata file for the whole dataset |
DataFieldDelimiter | ; |
data fields delimiter in CSV file |
TimeDataField | REFDATETIMEREF |
time field name |
ValueDataField | DATA |
data value field name |
QualityDataField | QUALITYID |
quality value field name, empty if flags are not in the data file or if flags should not be imported, archive must contain flags definitions (FlagsDefFile ) for quality values to import quality values from data file |
Blob data - csv-in-zip format¶
Blob data is ZIP archive (application/zip) with two or more files: CSV data file and time series metadata file for each exported time series (mandatory), flags definition file (optional) and dataset metadata file (optional). File names are specified in the transfer parameters (Datafile
, Metafile
, FlagsDefFile
, DatasetMetafile
). Files structure in the archive can be flat (all files are in the root folder) or data and metadata files can be in sub-folders.
Simple ZIP archive with one time series (no flags definition, no dataset metadata):
dataarchive.zip
├── data.txt # time series data
└── metadata.txt # time series metadata
ZIP archive with two time series, flat file structure:
dataarchive.zip
├── data1.txt # time series 1 - data
├── metadata1.txt # time series 1 - metadata
├── data2.txt # time series 2 - data
├── metadata2.txt # time series 2 - metadata
├── datasetmetafile.txt # dataset metadata
└── flagsdef.txt # flags definition
ZIP archive with two time series, files in subfolders:
dataarchive.zip
├── 0001
│ ├── data.txt # time series 1 - data
│ └── metadata.txt # time series 2 - metadata
├── 0002
│ ├── data.txt # time series 2 - data
│ └── metadata.txt # time series 2 - metadata
├── datasetmetafile.txt # dataset metadata
└── flagsdef.txt # flags definition
ZIP file example with two time series and flags definitions is attached (flat structure). sample_csv_zip_data.zip
Metadata file¶
Metadata file is file in JSON format (UTF8) that contains additional information about time series: name
, dimension
, datatype
, description
, location
, locationlatitude
, locationlongitude
, item
, itemEumIdent
, itemMzId
, unit
, unitEumIdent
, unitMzId
.
Example¶
{
"name":"TRN.LT-TB702-Z_KOTE",
"dimension":0,
"datatype":"Instantaneous",
"description":"B747 - Kote Trøjborg bassin",
"locationname":"BAS 747 Trøjborg Bassin, Dr. Margrethesvej 16",
"locationlatitude":56.1516215254962,
"locationlongitude":10.1752495765686,
"location":{
"type":"Point",
"coordinates":[10.1752495765686,56.1516215254962]
},
"item":"L",
"itemEumIdent":"eumIWaterLevel",
"itemMzId":100000,
"unit":"m",
"unitEumIdent":"eumUmeter",
"unitMzId":1000
}
CSV data file¶
Fields are separated with a character specified in the transfer parameter DataFieldDelimiter
(;
).
Numeric data values use decimal dot.
Time values are in yyyy-mm-dd hh:nn:ss
format.
Fields with time, data and quality values are specified in the transfer parameters TimeDataField
(REFDATETIMEREF
), ValueDataField
(DATA
) and QualityDataField
(QUALITYTXT
). Additional fields in CSV file are skipped.
Example¶
REFDATETIMEREF;DATA;QUALITYID;QUALITYTXT
2016-05-01 00:00:00;5.856770859;2;ok
2016-05-01 00:01:00;5.856770859;2;ok
2016-05-01 00:02:00;5.861111145;2;ok
2016-05-01 00:03:00;5.861111145;2;ok
Flags definition file¶
Flags definition file is JSON file (UTF8) that contains information about flags values: id
, name
, description
, level
.
Example¶
{
"data": [
{
"id": 1,
"name": "",
"description": "Undefined data quality",
"level": -1000
},
{
"id": 2,
"name": "ok",
"description": "Default data quality",
"level": 0
},
{
"id": 3,
"name": "error",
"description": "Error data quality",
"level": -15
},
{
"id": 4,
"name": "critical",
"description": "Critical data quality",
"level": -10
},
{
"id": 5,
"name": "warning",
"description": "Warning data quality",
"level": -5
},
{
"id": 6,
"name": "estimated",
"description": "Estimated data quality",
"level": -12
}
]
}
Dataset metadata file¶
Metadata file is file in JSON format (UTF8) that contains additional information about the dataset.
Example¶
{
"locationname":"BAS 747 Trøjborg Bassin, Dr. Margrethesvej 16",
"projectnumber":"#1235678"
}