In this test case we add the WIGOS station information to a synop message. The program is tested with two SYNOP messages.
This test case differs from the Brazilian one as the input and the output files are already BUFR files but
with different BUFR sequences ( the output file must have the WIGOS sequence 301150).
As a test to assess the problem, we have created the following code
#!/usr/bin/env python
from eccodes import *
import argparse
import pandas as pd
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
This is a test program to encode Wigos Synop
requires
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
To run the program
./wigosMultisubset.py -i synop_multi_subset.bufr -o out_synop_multisubset.bufr -w WIGOS_IDENTIFIERS.csv
Uses BUFR version 4 template and adds the WIGOS Identifier 301150
REQUIRES TablesVersionNumber above 28
Author : Roberto Ribas Garcia ECMWF 09/09/2019
'''
def read_cmdline():
'''
reads the command line to get the input ascii filename and the output bufr file
usage
prog -i <input_bufr_file> -o <output_bufr_file> -w <wigos_csv_info>
'''
p = argparse.ArgumentParser()
p.add_argument('-i', '--input', help = 'input BUFR filename')
p.add_argument('-o', '--output', help = 'output BUFR filename')
p.add_argument("-w", "--wigoscodes",help="csv with the station codes")
args = p.parse_args()
return args
def read_wigosInfo(wigosCSVFile):
'''
dtype={"wigosLocalIdentifierCharacter":object} forces the column to be string compatible
with the wigosLocalIdentifierCharacter key in bufr. This column in the CSV must be created as a TEXT
'''
df=pd.read_csv(wigosCSVFile,sep=",",dtype={"wigosLocalIdentifierCharacter":object})
return df
def main():
'''
reads the arguments from the command line
-i input bufr file
-o output bufr file
-w wigos information ( csv containing the station Name and wigos information ( wigosLocalIdentifierCharacter etc)
'''
args=read_cmdline()
inputFileName=args.input
outputFilename=args.output
wigosFile=args.wigoscodes
'''
reads the wigos information into a pandas dataframe that is queried
for each station to retrieve the station's wigos information
'''
dfWigosInfo=read_wigosInfo(wigosFile)
fin=open(inputFileName,"rb")
ibid=codes_bufr_new_from_file(fin)
codes_set(ibid,"unpack",1)
inUE=codes_get_array(ibid,"unexpandedDescriptors")
nsubsets=codes_get(ibid,"numberOfSubsets")
masterTablesVN=codes_get(ibid,"masterTablesVersionNumber")
# change the masterTablesVersionNumber if is below 29 ( otherwise the WIGOS sequence is not present)
if masterTablesVN<28:
masterTablesVN=28
outUE=inUE.tolist()
# update the unexpandedDescriptors ( BUFR sequence) to add the WIGOS data
outUE.insert(0,301150)
fout=open(outputFilename,"wb")
obid=codes_bufr_new_from_samples("BUFR4")
### important, use master tables version number above 28 as they contain WIGOS keys
# otherwise it won't work
codes_set(obid, 'masterTablesVersionNumber', masterTablesVN)
# set the unexpandedDescriptors of the output file with the new sequence 301150 (WIGOS) + synop sequence
#from Input message
# IMPORTANT, read the number of subsets
codes_set(obid,"numberOfSubsets",nsubsets)
codes_set_array(obid,"unexpandedDescriptors",outUE)
# here wigos information is added, the stationName is used
# to query the dfWigosInfo dataframe and retrieve the station Wigos information (wigosLocalIdentifierCharacter etc)
for i in range(0,nsubsets):
stationKey="#{0}#stationOrSiteName".format(i+1)
stationName=codes_get(ibid,stationKey)
dfo=dfWigosInfo.query("station=='{0}'".format(stationName))
key="#{0}#wigosIdentifierSeries".format(i+1)
codes_set(obid, key,int(dfo["wigosIdentifierSeries"].values[0]) )
key="#{0}#wigosIssuerOfIdentifier".format(i+1)
codes_set(obid, key, int(dfo["wigosIssuerOfIdentifier"].values[0]))
key="#{0}#wigosIssueNumber".format(i+1)
value=dfo["wigosIssueNumber"].values[0]
codes_set(obid, key, value)
key="#{0}#wigosLocalIdentifierCharacter".format(i+1)
value=dfo["wigosLocalIdentifierCharacter"].values[0]
codes_set(obid,key,str(value))
# copies the data from the input message ( ibid) to the output message obid
codes_bufr_copy_data(ibid,obid)
# write to output file ( packing is not needed here as copy_data does it implicitly.)
codes_write(obid,fout)
# release the obid and ibid bufr handles
codes_release(obid)
codes_release(ibid)
fout.close()
fin.close()
if __name__=="__main__":
main()
To run this program ecCodes ( above version 2.8) is required. This program was run with ecCodes version 2.12.5.
A test WIGOS_IDENTIFIERS.csv is created to test that the WIGOS keys are properly populated.
./wigosMultisubset.py -i synop_multi_subset.bufr -o aa.b -w WIGOS_IDENTIFIERS.csv
The program workflow is the following
1) read the input BUFR message and retrieve different keys
unexpandedDescriptors that contains the input BUFR sequence ( synop). The list of unexpandedDescriptors ( BUFR sequence) is updated to add the WIGOS sequence 301150 in front of the list
numberOfSubsets needed after to allocate space for the data in the output message
masterTablesVersionNumber this key is needed, if the masterTablesVersionNumber is below 28 ( does not contain the WIGOS sequence) then is set to 28 to make sure ecCodes finds the WIGOS sequence.
2) Once the unexpandedDescriptors sequence is updated with the WIGOS sequence, we open an output file, create a BUFR handle from a BUFR4 sample and set the information ( masterTablesVersionNumber, numberOfSubets) and ( WIGOS information that comes from the WIGOS_IDENTIFIERS.csv see below Important Notes ).
This WIGOS information is read at the beginning from a CSV file (WIGOS_IDENTIFIERS.csv) and stored in a pandas dataframe dfWigosInfo, then for each station read from the input file, we query the dfWigosInfo for that particular
station and retrieve the different WIGOS fields needed for the output BUFR.
The rest of the information is copied from the input BUFR to the output BUFR by using codes_bufr_copy_data(ibid,obid) that copies the common keys from the ibid to obid.
3) once the common keys are copied and the new WIGOS keys populated properly, we just write the obid handle to the output file and release the handles ibid,obid to avoid exhausting the system's memory.
Important notes
1) The masterTablesVersionNumber must be above 28 otherwise no WIGOS sequence is available.
2) The file WIGOS_IDENTIFIERS.csv contains individual keys for the WIGOS , as mentioned before the WigosLocalIdentifierCharacter has to be encoded as TEXT when creating the CSV file, any other
keys that contain 0 leading values, should be encoded as TEXT as well to prevent the non leading 0 being removed as they would be treated as numeric fields.
The files used are attached here, the out_mutisubsets.bufr contains the resulting output ( with WIGOS identifiers) for the synop_multi_subset.bufr.
The file out_singlesubset.bufr contains the resulting output for the synop.bufr file.
test files and WIGOS csv information.
WIGOS_IDENTIFIERS.csvsynop.bufrsynop_multi_subset.bufr
Output files
The output of out_synop.bufr contains the WIGOS information
"key" : "wigosIdentifierSeries",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosIssuerOfIdentifier",
"value" : 20000,
"units" : "Numeric"
},
[
{
"key" : "wigosIssueNumber",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosLocalIdentifierCharacter",
"value" : "10015",
"units" : "CCITT IA5"
},
[
{
"key" : "blockNumber",
"value" : 10,
"units" : "Numeric"
And the output of the out_synop_multisubset.bufr contains the WIGOS keys for each of the subsets
{
"key" : "wigosIdentifierSeries",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosIssuerOfIdentifier",
"value" : 20000,
"units" : "Numeric"
},
[
{
"key" : "wigosIssueNumber",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosLocalIdentifierCharacter",
"value" : "01027",
"units" : "CCITT IA5"
},
[
DISCLAIMER: This software is intended for testing purposes only. Feedback would be appreciated but technical support can only be given if our workload permits.
Add Comment