Binary protocol¶

This section describes the structure of Multidimensional binary query result.

Top level structure:

Section	Content	Occurrence
Header	See header	1
SRID	VarUInt32	1
Data sections	Various data blocks	0..N
Closing tag	Byte	1

Read pseudo code :¶

header = read 4 bytes 
srid = read VarUInt32 
tag = read byte // 1. section tag 
while tag <> closing tag
   read DataSection based on tag
   tag = read byte // next section tag

The header has 4 bytes and the content is fixed.

Position	Length	Content
0	1	Major protocol version
1	1	Minor protocol version
2	1	Reserved
3	1	Reserved

Data sections¶

Each data section consist of

Section identifier tag : 1 byte
Section data : number of bytes differs per section

Section	Tag	Occurrence
SPATIAL	10	0..1
TEMPORAL	20	0..1
ITEM	30	NOT SUPPORTED YET, Use dataset metadata
LAYER	40	NOT SUPPORTED YET, Use dataset metadata
DATABLOCK	99	0..Many
END	255	1

End tag¶

End tag occurs as the last section identifier and indicates that the client should stop reading data. The section tag value is 255.

DataBlock section¶

Datablock section can appear multiple times in the result. Datablock contains : - 3 indexes. The indexes can vary based on the query type. - Array of data : float only in verion 1.1

Data block part	Data type	Occurence
Block indexes	See Block index below	3
Data length	VarUInt32	1
Data array	array of bytes	Data Length

Block index is a struct - Index type : byte - Index Value: VarSInt32

Block index type	Tag
SPATIAL	10
TEMPORAL	20
ITEM	30
LAYER	40

For example, if Block index type indicates ITEM (30), then Index Value can be 7 as item 7. If Block index type indicates TEMPORAL (20), then Index Value can be 10 to indicate time step 10.

Spatial section¶

Only one spatial section can occur.

Spatial domain type tag: 1 byte
Spatial data: size varies and is defined per section

Spatial domain type	Tag value
GRID INDEXES	10
MESH ELEMENTS	20
MESH PAGES	30

Grid indexes¶

Applies to Gridded spatial domain. Grid index is integer position within full dataset grid definition.

Byte array length: VarUInt32
byte array: array of bytes (with length)

Cast byte array to signed int array.

Mesh elements¶

Applies to Mesh spatial domain. Not implemented yet.

Mesh pages¶

Applies to Mesh spatial domain. Mesh page consists of at most 256 elements optimized for minimal serialization size.

Pseudo code:

pageCount = read VarUInt32
for i = 0 to pageCount -1
    read MeshPage

For more details on how to read mesh page see mesh page reader.

Temporal domain¶

Contains an array of ordered Datetimes. Very efficient for mostly equidistant (with a few gaps) temporal domains.

Pseudo code:

ticks = read VarUint64
timeSteps.Add(convert ticks to DateTime);

delta = read VarUint32
while (delta > 0)
    count = read VarUint32
    for i = 0 to count-1
        ticks = ticks + delta
        timeSteps.Add(convert ticks to DateTime)

    delta = read VarUint32

Timesteps are encoded as UInt64 values (ticks): Ticks represent the number of seconds elapsed time since 12:00:00 midnight, January 1, 0001 in the Coordinated Universal Time.

Data types¶

Used data types:

byte : unsigned byte (0-255)
VarSInt32 : Variable signed integer
VarUInt32 : Variable unsigned integer
VarUInt64 : Variable unsigned long integer
double : Little Endian encoded double value

Variable integers are defined as Google protobuf primitive types. It is recommended to use Google protobuf language specific implementation for value decoding.

Non-byte arrays¶

Google protobuf does not handle array buffers well compared to another Google serialization library Google flatt buffers. Arrays are handled as continuous blocks and specific CAST operator is used.

See the appropriate language specific implementation in ![Google flatbuffers]: https://github.com/google/flatbuffers/tree/master/net/FlatBuffers

Encoding details¶

This section is added only for completeness. Use Google protobuf libs to read primitive types if possible.

ZigZag Encode¶

The varint scheme for indicating a value larger than one byte involves setting the "high bit", which is the same thing that standard integers use for storing negative numbers. Negative numbers need to first be encoded using a "zig zag" encoding, which keep absolute values small, but do not set the high bit as part of encoding negative numbers.

The effect is to convert small negative numbers into small positive numbers, which is exactly what we want, since delta values will tend to be small negative and positive numbers:

-1 => 1
 1 => 2
-2 => 3
 2 => 4

A computationally fast formula to generate the zig-zag value is

/* for 32-bit signed integer n */
unsigned int zz = (n << 1) ^ (n >> 31)

VarInt Encode¶

Variable length integers are a clever way of only using as many bytes as necessary to store a value. The method is described here:

https://developers.google.com/protocol-buffers/docs/encoding#varints

The varint scheme sets the high bit of a byte as a flag to indicate when more bytes are needed to fully represent a number. Decoding a varint accumulates the information in each flagged byte until an un-flagged byte is found and the integer is complete and ready to return.