Complete documentation for Metview's Python interface is now available on readthedocs!
Metview's Python interface provides access to all of Metview's Macro language functions, with automatic translations between data types. Here's an example that shows how to retrieve model and observation data and compute their difference, and the weighted mean value of that difference, both in Macro and in Python.
Macro | Python |
---|---|
# Metview Macro t2m_fc48 = retrieve( type : "fc", levtype : "sfc", param : "2t", date : -5, step : 48, grid : "o1280") synop = retrieve( type : "ob", repres : "bu", date : -3) synop_t2m = obsfilter( output : "geopoints", parameter : 012004, data : synop) diff = t2m_fc48 - synop_t2m print(integrate(diff)) | import metview as mv t2m_fc48 = mv.retrieve( type = "fc", levtype = "sfc", param = "2t", date = -5, step = 48, grid = "o1280") synop = mv.retrieve( type = "ob", repres = "bu", date = -3) synop_t2m = mv.obsfilter( output = "geopoints", parameter = "012004", data = synop) diff = t2m_fc48 - synop_t2m print(mv.integrate(diff)) |
You can see that the Macro functions are all available through the namespace defined in the import
command.
More examples are available in the Gallery.
Variable types
When calling Macro functions from Python, variables passed as input to or output from those functions undergo a conversion as described in this table:
Macro | Python | Notes |
---|---|---|
number | number | |
string | string | |
list | list | Can also pass a tuple to Macro, and it will be converted to a Macro list |
fieldset | Fieldset | Lightweight wrapper class in Meview-Python |
geopoints | Geopoints | Lightweight wrapper class in Meview-Python |
observations | Bufr | Lightweight wrapper class in Meview-Python |
netcdf | NetCDF | Lightweight wrapper class in Meview-Python |
odb | Odb | Lightweight wrapper class in Meview-Python |
table | Table | Lightweight wrapper class in Meview-Python |
vector | numPy array | |
date | datetime | Can also pass a date or a datetime64 to Macro |
definition | dictionary | |
nil | None |
Working with Fieldsets
Fieldsets work much the same as they do in the Macro language, but watch out for these things:
- length of a fieldset can be found with the
len
function:num_fields = len(my_fieldset)
- indexing starts at 0:
first_field = my_fieldset[0]
- slicing works:
my_fields = fs[0:6:2]
- you can pass a numpy array of indexes:
my_fields = fs[np.array([1.0, 2.0, 0.0, 5.0])]
- comparison operators work the same as in Macro, i.e. they return a fieldset of 1s and 0s:
smaller = fs1 < fs2
- equality and non-equality operators are
==
and!=
- Fieldsets can be directly constructed either as empty, or with a path to a GRIB file:
f = mv.Fieldset()
f = mv.Fieldset(path='test.grib')
- concatenation can be done like this:
my_fieldset.append(my_other_fieldset)
- iteration works:
for f in my_fieldset: #do something
Working with Geopoints
Geopoints also work much the same as they do in Macro, but be aware of these points:
- in Macro, we use
geo_missing_value
to denote missing data values; in Python, we usenumpy.nan
Working with dates
There are several differences between the usage of the date object in Macro and the datetime object in Python. You will find a few examples below to compare the various date manipulation techniques used in Macro and Python, respectively.
Macro | Python |
---|---|
# Metview Macro # creating d = 2000-01-04 09:50:24 today = date(0) yesterday = date(-1) # arithmetic d = d + hour(9) + minute(50) + second(24) # using an increment of 2 days for d = 2018-11-01 to 2018-11-10 by 2 do (...) end for # using an increment of 6 hours for d = 2018-11-01 to 2018-11-10 by hour(6) do x = retrieve( date : yymmdd(d), time : hhmm(d), ...) (...) end for | import metview as mv import numpy as np from datetime import datetime, timedelta # creating d = datetime(2000, 1, 4, 9, 50, 24) today = datetime.today() yesterday = datetime.today() - timedelta(1) # arithmetic d = d + timedelta(hours=9, minutes=50, seconds=24) # using an increment of 2 days d0 = datetime(2018,11,1) d1 = datetime(2018,11,10) dt = timedelta(days = 2) for d in np.arange(d0, d1, dt): (...) # using an increment of 6 hours d0 = datetime(2018, 11, 1) d1 = datetime(2018, 11, 10) dt = timedelta(hours = 6) for d = np.arange(d0, d1, dt): x = mv.retrieve(date =d, time = mv.hour(d), ...) (...) |
Additional data export features
NumPy arrays
Any Metview function that normally returns a vector will return a numPy array when called from Python. dtypes of float32 and float64 are supported. For example, the following fieldset functions return numPy arrays:
a = mv.read('my_data.grib') # returns a Fieldset lats = mv.latitudes(a) # returns a numPy array lons = mv.longitudes(a) # returns a numPy array vals = mv.values(a) # returns a numPy array
Pandas Dataframes
The Geopoints data type has an additional function, to_dataframe()
, which can be used to produce a Pandas Dataframe object as shown:
import metview as mv import pandas as pd gpt = mv.read("gpts.gpt") # returns a Geopoints df = gpt.to_dataframe() # returns a Pandas Dataframe print(df.head())
Output:
date latitude level longitude value 0 2018-01-14 12:00:00 30.0 1000.0 -24.0 288.736 1 2018-01-14 12:00:00 30.0 1000.0 -18.0 288.736 2 2018-01-14 12:00:00 30.0 1000.0 -12.0 286.736 3 2018-01-14 12:00:00 30.0 1000.0 -6.0 NaN 4 2018-01-14 12:00:00 30.0 1000.0 0.0 NaN
Xarray Datasets
The Fieldset object has an additional method, to_dataset()
, which produces an xarray Dataset object from the given fieldset. This is an N-dimensional data array based on the Common Data Model used in netCDF. For example:
import metview as mv t2m_fc = mv.retrieve( type = 'fc', levtype = 'sfc', param = ['2t', '2d'], date = -5, step = list(range(0, 48+1, 6)), grid = [1,1] ) xa = t2m_fc.to_dataset() print(xa)
will produce the following output:
<xarray.Dataset> Dimensions: (latitude: 181, longitude: 360, step: 9, time: 1) Coordinates: * time (time) datetime64[ns] 2018-05-10T12:00:00 * step (step) timedelta64[ns] 0 days 00:00:00 0 days 06:00:00 ... * latitude (latitude) float64 90.0 89.0 88.0 87.0 86.0 85.0 84.0 83.0 ... * longitude (longitude) float64 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ... Data variables: 2t (time, step, latitude, longitude) float32 ... 2d (time, step, latitude, longitude) float32 ... Attributes: Conventions: CF-1.7 comment: GRIB to CF translation performed by xarray-grib
This is based on the Open Source cfgrib package which is planned to be ultimately integrated into xarray.
From version 0.8.6 of Metview's Python interface, it is also possible to convert an xarray dataset to a Metview Fieldset, and also to pass one directly to Metview functions. Note that this uses an experimental feature of cfgrib and will only work for a small subsset of xarray datasets. Examples:
grib = mv.read('temperature_on_pl.grib') x = grib.to_dataset() # convert from Fieldset to dataset f = mv.dataset_to_fieldset(x) # convert back to Fieldset
grib = mv.read('temperature_on_pl.grib') x = grib.to_dataset() # x is now an xarray dataset fs = mv.mean(x*2) # *2 is done by xarray, mean() is done by Metview
Icon functions
Macro functions which correspond to icons, such as retrieve()
, which corresponds to the Mars Retrieval icon, can take their arguments in a number of ways:
Method | Example |
---|---|
Named arguments | a = mv.retrieve(type = 'fc', date = -5) |
Dictionary |
|
Combination |
|
Object-oriented calling
Metview's Python interface also provides a more object-oriented way of using the wrapper classes such as Fieldset. The following two snippets of code are equivalent:
Function | Object method |
---|---|
|
|
Calling user-defined Macro functions from Python
Any user-defined Macro function can be called from Python as long as it satisfies both of these criteria:
- the function must reside in a file of the same name as the function
- the file must be in a directory listed in the
METVIEW_MACRO_PATH
environment variable; this variable can be modified by the user, but by default it will include$HOME/metview/System/Macros
.
If these are satisfied, then the function may be invoked using the call()
command as shown here:
import metview as mv mv.call('my_func', 4, 1, 'my_string_arg'))
The first argument is the name of the function, the subsequent arguments will be passed to the function.
Things to watch out for
Although it's easy to convert an existing macro into a Python script, there are some things to be aware of.
- Macro indexing starts at 1, but Python indexing starts at 0. Aside from the standard Python structures such as lists, this is also true for Metview Python classes such as Fieldset. Given a fieldset
fs
, the first field is obtained in Macro byfs[1]
, but in Python it is obtained byfs[0]
. - Some MARS requests require a class parameter. As class is a restricted keyword in Python, please use class_ in its place.
- In order to support the more interactive coding environments provided by Python, any call to the
plot()
command will immediately produce a plot. This is different from Macro, where plots are delayed until the end. The result of this is that multipleplot()
commands in Python will result in multiple plot windows. Fortunately, a singleplot()
command can be given any number of items, including multiple pages. To see how to produce a multi-plot layout, please see the Layoutx3 Example gallery example. - To produce multiple physical pages in a PostScript file, add calls to
newpage()
within theplot()
command, e.g.plot(dw, data1, visdef1, newpage(), data2, visdef2)
.