Loading Conda

Conda is now available to all users on the new Atos HPC. To use Conda, simply load the module using the following command:

module load conda

WARNING: loading conda will unload all the modules already loaded and will prevent you from loading new ones. This is done to avoid potential conflicts between packages installed in your conda environments and those from the module system (eccodes, ecflow, compilers, etc.).

To go back to the module system, use the reset command:

module reset

Simply unloading the Conda module won't be enough to go back to the defaults state, many modules won't be visible. It's best to reset the module system.

Loading Conda will give you access to the base environment, with only limited packages available. To access more complex environments, you can either load one of the shared environments, with the list available through the command:

conda env list

or create your own environment, following the approaches listed in Getting started with Conda.

Channels

By default, Conda will use the two channels below as a source to create environments. We strongly recommend users to stick with those two channels, especially for operational environments. Users are free to use other channels (typically conda-forge) to create environments, but ECMWF won't provide supports for these environments. Note that, amongst other features, direct access to MARS will only be available through the packages installed using the ecmwf channels, using other channels could give you a working but not optimal environment.

Channel based on conda-forge

ECMWF specific channel

https://nexus.ecmwf.int/repository/conda-forge

https://nexus.ecmwf.int/repository/conda-ecmwf

This channel is based on conda-forge (https://conda-forge.org), one of the most popular Conda channels for scientists, but only supports a restricted number of Python versions (starting from 3.8). WE DO NOT SUPPORT Python 2.7.

Apart from that, most of the packages from conda-forge are available in this channel. The only exception is for the ECMWF packages, which have been intentionally left out, but are available in the conda-ecmwf channels. The conda-forge channel will be updated once a year.

This channel contains ECMWF packages, built specifically for the HPC. This channel will be continuously updated, mostly following the synchronised release workflow. It contains the main ecmwf packages, such as:

  • eccodes

  • ecflow

  • magics

  • metview

  • fdb

  • ...

Defining your Conda configuration file

Advanced users may be interested in fine tuning their configuration file. This can be done through the conda interface: https://docs.conda.io/projects/conda/en/latest/commands/config.html

The following default conda configuration is provided by the module:

channels:
  - https://nexus.ecmwf.int/repository/conda-ecmwf
  - https://nexus.ecmwf.int/repository/conda-forge
pkgs_dirs: # where to download packages, this avoids using $TMPDIR, which can be quickly full 
  - $PERM/conda/pkgs
envs_dirs:
  - $PERM/conda/envs

You can change your default configuration in the .condarc file located in your home folder. For instance:

pkgs_dirs:
  - $SCRATCH/conda/pkgs
conda-build: # where to build recipes, best to avoid $TMPDIR
  root-dir: /scratch/$USER/conda/build

More information on configuration:

https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html