You have a URL which contains GRIB/BUFR message(s) and you want to read them directly from the web without first saving to disk.

Step-by-step guide

Let's say you have a stream of in-memory bytes rather than a file which contains one or more GRIB or BUFR messages. You can of course save those bytes to a file and decode as usual but what if you have a slow filesystem, your disk is full or you just want to avoid the disk access overhead?

The following python script shows you how you can decode those messages:

import requests
from eccodes import codes_new_from_message, codes_get, codes_release

url='https://some.gov/pub/data/temperature.grib2'
print('Started downloading...')
r = requests.get(url)
data = r.content # raw bytes

# Convert each field to a handle in turn
offset = 0
while offset < len(data):
    h = codes_new_from_message(data[offset:])
    print(offset, codes_get(h, 'step'), codes_get(h, 'shortName'))
    # Advance through the bytes by the length of the message
    offset += codes_get(h, 'totalLength')
    codes_release(h)

In line 4 we retrieve the data from the specified resource. In this example it's a GRIB file.
Then in line 7 we get the raw bytes from the request which can contain several messages.
In the main loop we start off at the beginning of the stream (offset = 0) and get the first message using codes_new_from_message() which gives us a handle 'h' on the first message.
We can now use the standard codes_get() function to access values of various keys from this message.
In line 15 we advance through the byte data by the length of the first message (by slicing the byte array) so next time when we call codes_new_from_message() we will come across the next message and so on. Note we must call the codes_release() on each handle to free the memory.



1 Comment

  1. Shahram Najm

    Thanks to our colleague Luke Jones for this contribution