What is the compression offered to commercial customers for dissemination products?
ECMWF offers bzip2 compression of disseminated data.
The system pre-compresses the data flagged for automatic dissemination (i.e.: not the data in standby) and sends them according to the normal schedule. If there is delay and there is not enough time to compress the data before the file is scheduled for dissemination, the system will compress the file on the fly. Standby files that you re-queue will be compressed on the same principle (e.g.: compressed on the file or if time allow pre-compressed).
The system attempts to compress the data files for the whole "destination": this means that we cannot explicitly configure it to send a mix of compressed and uncompressed files to a single destination. However, if a compressed file is the same size or bigger than the original file, then the file will be sent uncompressed (i.e.: the system will not send files bigger than the original files). This means that your system must be able to handle both compressed and uncompressed files.
The files appear on your server with the extension .bz2 and will need unpacking before processing. Once the data is uncompressed there should be no difference between these and the normal data.
Compression rate
The compression rate is variable, however we normally see a reduction in the range of 30% to 60%. If you are not already using Second Order packing we can estimate the compression before setting the test. This is not possible if you are using Second Order packing.
Users receiving real-time products via dissemination and using the bzip2 compression, should consider using the parallel bzip2 compression utility which can be found here:
http://lbzip2.org
It is fully compatible with bzip2 (file format and command line level) and allow compressing/decompressing at fast speed.
ECMWF currently testing this utility but already got positive feedback from users about this tool.
Ask to set up a test for a short time and we will organise duplicate delivery on a separate directory on the same server. Once you are satisfied that it works well for you, we will make the change operational.
