05 December 2018 - Version 0.18.1

Bug

  • [ODB-392] - test_encode_odb fails with Cray compiler 8.6

Improvement

  • [ODB-434] - Fix DynamicParametisation following eckit API change


June 2018 - Version 0.18.0

This version introduces changes required for compatibility with a C++11-only software stack.

Bug

  • [ODB-422] - Error message on duplicate column is nonsensical

Improvement

  • [ODB-362] - Use new lest-compatible unit testing from eckit for ODB tests.
  • [ODB-421] - C++11 as minimum requirement

October 2017 - Version 0.17.6

Bug

  • [ODB-390] - Fix FileMapper TIME for odbServer (correct broken test)

Improvement

  • [ODB-393] - Analysis of ODB meta data coming from marsfs should buffer data

October 2017 - Version 0.17.4

Bug

  • [ODB-388] - odb mdset sometime produces incomplete files
  • [ODB-391] - test_decode_odb has macro with wrong number of arguments

Improvement

  • [ODB-387] - odb compare should accept a list of columns not to test
  • [ODB-390] - Fix FileMapper TIME for odbServer

Specific note

ODB-390 (changes to FileMapper) changes the default handling of TIME in the archiving operations. This has been done to align the versions of ODB API used in operations at ECMWF with those that are externally released. If the older behaviour is desired, this can be restored by setting the environment variable:

ODB_PATCH_TIME_ENCODE_RELATIVE=true

September 2017 - Version 0.17.3

Bug

  • [ODB-356] - Tests fail when dependency order is not obeyed
  • [ODB-379] - ODB-2 distance function out by a factor of 10
  • [ODB-376] - Garbled data can be returned from odb::Reader from malformed ODB file (rather than error)

Improvement

  • [ODB-159] - ODB API 1.0: Use codec which fixes ODB-15 for encoding by default
  • [ODB-362] - Use new lest-compatible unit testing from eckit for ODB tests.

July 2017 - Version 0.17.2 (Internal only)

Bug

  • [ODB-365] - Build failure on FreeBSD
  • [ODB-367] - The "chars" codec stores all of its data twice
  • [ODB-369] - Values can be promoted into missingValue silently, without being registered in statistics
  • [ODB-372] - Writer will happily create output with columns with repeated names
  • [ODB-374] - Failure during writing header of string-based types in odb mdset
  • [ODB-375] - Internal segfault if columns are not correctly initialised

Improvement

  • [ODB-362] - Use new lest-compatible unit testing from eckit for some of the ODB tests.
  • [ODB-363] - Added tests for data encoding/decoding and codecs.

June 2017 - Version 0.17.1

Primary change is the removal of ODB1 dependencies from ODB API (and extraction of those components into odb_tools).

Bug

  • [ODB-351] - Legacy Fortran API syntax error if SQL not terminated with semicolon
  • [ODB-353] - Default paths in oda2request are incorrect, causing failures
  • [ODB-354] - Cmake is checking for FortranLibs when just checking if the Fortran compiler works should be sufficient

23 May 2017 - Version 0.17.0

Bug

  • [ODB-52] - Crash when trying to open files that don't exist
  • [ODB-246] - Problem compiling odbql_wrappers.f90
  • [ODB-324] - ODBServer project should be declared as Fortran?
  • [ODB-331] - Python interface: files not closed after executing SELECT.
  • [ODB-334] - Inconsistent NetCDF variable naming
  • [ODB-342] - Data Governance tools - Creation of new groups through update_odb_obsgroups.sh is broken
  • [ODB-347] - odb sql command line fails with syntax error if terminating semicolon is missing
  • [ODB-349] - A race condition in ODB API SQL (Bitfields' types registration)

New Feature

  • [ODB-83] - ODB API Python interface to accept multiple files as SQL input

Improvements

  • [ODB-127] - $ variables in ODB2
  • [ODB-317] - Make OpenMP optional
  • [ODB-319] - Allow ODB-API to be built without OpenMP support

March 2017 - Version 0.16.2

Fixes

  • [ODB-169] - Add an alias 'mean' to 'avg' for ODB-1 compatibility
  • [ODB-313] - odb_migrator was crashing in verification stage for some ODB databases

New Features

  • [ODB-300] - 'odb count' can accept more then one file name as parameter and return total sum of row counts.
  • [ODB-303] - Incorporate MetOffice patch for ODB Server so time is formatted as 4 digits (optional compile-time feature)
  • [ODB-306] - Make the verification of split optional (option -no_verification)
  • [ODB-312] - Make verification optional in odb_migrator (option -no_verification)

0.16.1

  • [ODB-166] - Portability issues affecting clang compilers builds, including Mac OS
  • [ODB-286] - Improve SQL syntax error messages
  • [ODB-284] - Less verbose logging for functionality called by MARS client and server

0.16.0

  • [ODB-269] - Python API: files were not closed properly. Python programs opening many files could reach operating systems limit on number of open files.
  • [ODB-281] - New SQL functions: (aggregate) first and last.
  • [ODB-271] - Python API: Pass file name to function connect instead of DDL, see updated examples.

0.15.2

  • Fixed issue that caused Metview to crash on some systems (ODB-273)

0.15.1

  • Fixed issue ODB-270 caused by a regression in ecKit (issue ECKIT-169)

0.15.0

NOTE: This version is deprecated due to a regression in ecKit (ECKIT-169) Use version 0.15.1

  • Improvements to Fortran API and examples (ODB-261)
  • Improvements to Python API, examples and tests (ODB-94, ODB-250, ODB-255)
  • Changes necessary after MarsKit library was renamed to metkit (ODB-262)
  • 'odb header' tool can print metadata of a file in DDL syntax (CREATE TABLE statement) when invoked with option -ddl (ODB-242)
  • Improvements to SQL parser implementation, including thread safety (ODB-257, ODB-263, ODB-264)
  • Stability improvements (ODB-256)
  • Internal changes to accommodate changes in ecKit library ( ECKIT-154 (logging), ECKIT-164 (configuration files syntax))

0.14.0

  • First beta version of new Python API, see examples in subdirectory src/api/  The intention is to provide an API conforming with PEP 249 Python Database API Specification v2.0 https://www.python.org/dev/peps/pep-0249/  The new API, unlike the old one, allows for writing files (using INSERT, see examples).

0.13.0

  • Preliminary, beta version of new API for C/C++ and Fortran, see api subdirectory for examples. All comments, suggestions and ideas for improvement will be much appreciated!

  • New SQL syntaxes:
    • CREATE TABLE for associating SQL tables with files and setting metadata of new files
    • 'SELECT ALL *' for reading heterogeneous files (different columns on different parts of a file), available via the new API (ODBQL)
    • INSERT for writing files, available via ODBQL
    • MATCH '(' expression_list ')' IN QUERY '(' select_statement ')'
  • ECML, the embedded scripting language based on MARS language syntax, has been merged into ODB API
  • ODB API has been integrated with MarsKit, which allows direct retrieval of data from MARS or ODB Server
  • ODB Server's functionality for local files available as part of the library
  • A tarball with ODB API and all its dependencies is available, see file odb_api_bundle-0.13.0-Source.tar.gz. To build MarsKit it is necessary to pass -DENABLE_GRIBPP=OFF to cmake

0.10.5

  • Added file group.txt to source code's etc directory. The file is needed for unit testing. An up to date version of the file is in http://apps.ecmwf.int/odbgov/group/csv/ , it is also installed in /usr/local/apps/odb_api/codes on ECMWF systems, and is used by MARS client when archiving ODB data. ODB_API_CODES environment variable can be used to change default location.

  • Added cmake variable EXTRA_ODB_MIGRATOR_LIBS. It can be used to pass additional libraries needed for building odb_migrator, like gomp when building with gfortran.
  • eckit is not distributed with this version. It is available for download here: ecKit Releases

0.10.4

  • Use /usr/local/apps/odb_api/codes/group.txt ( http://apps.ecmwf.int/odbgov/group/csv/ ) instead of deprecated obsgroups.txt (MARS client support code),

  • Fix odb_migrator for cases when SQL SELECT defining data to be converted contained  functions or expressions other than simple column names on the select list,
  • When writing output files create subdirectories if present in path name,
  • Put back changes reverted by accident in change 6389d281bfb (this regression was released with 0.10.3): ODB-75 (LIKE/RLIKE, atoi), ODB-85 (hex bitfields formatting), ODB-108 (WHERE 0).

0.10.3

  • Improve memory consumption of odb split tool (ODB-121)

0.10.2

  • Fix odb_migrator, it was crashing when invoked with no parameters (ODB-91)

  • Fix AIX build issues.

0.10.1

  • Updated to use eckit 0.6.0 and ecbuild 1.3.2
  • Table in the SELECT's INTO clause can be defined with a CREATE TABLE statement (ODB-67)

  • odb sql tool has new flags (ODB-85):

    [--binary]            Print bitfields in binary notation

    [--no_alignment]      Do not align columns

  • Python API for legacy ODB added (documentation to be written)
  • ...

0.9.32

  • "ORDER BY 1" was not working correctly (ODB-49)
  • oda2request tool broken in ODB API 0.9.31 (wrong default path to configuration) (ODB-70)
  • CSV import tool was not working correctly for some bitfield columns (ODB-76)
  • ODB_DOUBLE constant added to Fortran bindings (ODB-77)
  • Python API for accessing old ODB databases (ODB-74)
  • Added odb_migrator option to specify missing value indicators different than default ones (only for REAL and INTEGER currently, DOUBLE always the same as REAL, BITFIELD always 0) (ODB-68)

            Syntax: 
               -mdi <type1:MDI1,type2:MDI2,...>
            e.g.:
              -mdi REAL:2147483647,INTEGER:2147483647

0.9.31

  • Fixed a bug introduced in 0.9.30, where operator >= was behaving like =.
  • Now ORDER BY clause accepts also column numbers, for example: SELECT int(lat/10), int(lon/10), count( * ) ORDER BY 3 DESC
  • Support for #include directive

0.9.30

  • Public header oda.h renamed to odb_api.h
  • Added functions for reading bitfields' metadata: odb_read_iterator_get_bitfield  and odb_select_iterator_get_bitfield to Fortran API
  • Disabled again fix for codec short_real, (ODB-15). The new codec short_real2 will be used by default when all ODB API applications at ECMWF are upgraded to use an ODB API version >= 0.9.24
  • Added syntax for range expressions in SQL WHERE clause, e.g.: <value1> <=  <expression> <=  <value2>

0.9.29

  • Fixed encoding of files containing columns of type DOUBLE (no packing) only.

0.9.28

  • Improvements to the CSV import tool, now it can import bitfields
  • Fixed queries with shift operator (#) appearing as arguments of functions
  • Fixed option -T (do not print header with column names) of the sql tool

0.9.27

  • Upgraded to eclib 7.3.0 - now wrapped in namespace eclib
  • Fixed ODB-26: some expressions with mixed aggregated and non-aggregated functions, e.g.: sum(obsvalue)+rms(obsvalue), were calculated incorrectly

0.9.26

  • Introduced in-memory data tables

0.9.25

  •  Introduced row shift operator (#) (ODB-2)

0.9.24

  • Fix an issue in SELECT DISTINCT ... INTO (some rows contained NULLs instead of correct values)
  • Fix odb header tool so it works properly when string columns contain empty strings (ODB-19)
  • Asynchronous IO can be used optionally when writing files by setting ODB_API_USE_AIO
  • Fix the split tool (ODB-17)

  • Added new codec short_real2 which should replace codec short_real in the future. NOTE: files created with this codec cannot be decoded with versions 0.9.23 and earlier. The new codec short_real2 is not used for encoding by default (ODB-15)

  • CodecOptimizer can be configured by setting ODB_DEFAULT_CODEC

0.9.23

  • Fixed a memory leak

  •  Add options -offset and -length to the sql tool:

            [-offset <offset>]    Start processing file at a given offset

            [-length <length>]    Process only given number of bytes of data

  • Add option -offsets to odb header tool, for printing basic information about blocks of data on a file:

            offset length numberOfRows

  • Fix for ODB-8: Missing values were not handled correctly by SQL formulae if results of SELECT saved on file
  • Semantics of SQL multiplication was different than in the old ODB: now 0 * NULL == 0, not NULL (ODBSUP-13)
  • Result of most SQL functions is now double precision (ODB-11)
  • Improved odb compare tool
  • Added verification to the split tool