ecFlow's documentation is now on readthedocs!
What is ecFlow



ecFlow is a client/server workflow package that enables users to run a large number of programs (with dependencies on each other and on time) in a controlled environment. It provides reasonable tolerance for hardware and software failures, combined with restart capabilities. It is used at ECMWF to run all our operational suites across a range of platforms.

ecFlow submits tasks (jobs) and receives acknowledgments from the tasks when they change status and when they send events, using child commands embedded in your scripts. ecFlow stores the relationships between tasks, and is able to submit tasks dependant on triggers, such as when a given task changes its status, for example when it finishes.  

ecFlow functionality is provided by the following executables and shared libraries

  • ecflow_client: This executable is a command-line program; it is used for all communication with the server. This executable implements the Command Level Interface (CLI). The bulk of this functionality is also provided by the python API
  • ecflow_ui: This is a specialized GUI client that monitors and visualizes the node tree hierarchy. Based on QT.
  • ecflow_server: This executable is the server. It is responsible for scheduling the jobs and responding to ecflow_client requests.
  • ecflow.so:    Python interface These shared libraries provide the python API for creating the suite definition and communication with the server.

ecFlow runs as a server receiving requests from clients. CLI, GUI, and suite jobs are clients. Communication is based on TCP/IP. Note that ecFlow is a scheduler and is not a queuing system such as NQS, SGE, Load leveler, or PBS. However, it can submit to queuing systems.

History

For almost three decades ECMWF used SMS (Supervisor Monitoring Scheduler) package to control the workflow for the Centre's operational models and systems. Written at ECMWF, it allows the design, submission, and monitoring of jobs both in the Research and Operations Departments, and provides common tools for scientists, analysts, and operators to cooperate. A large number of organizations also use SMS (from both Member and non-Member States). Development and support of SMS has now stopped. 

The replacement, ecFlow 4, has superseded SMS. It has been used for over a decade. It is a complete rewrite using an object-oriented methodology and modern standardized components; it acts as a comprehensive replacement for SMS. The rewrite helps improve maintainability, allows easier modification, and introduces object orientated features. The proprietary script language used by SMS, CDP, has been replaced by Python.

The development of ecflow 4 has now stopped. But support is still provided. It has been replaced by ecflow 5

You can find an introduction to ecFlow in the ECMWF Newsletter article  (Autumn 2011), starting on page 30.







Latest News

Go to Latest news to access older news items


Report a bug or issue

Please send an email to software.support@ecmwf.int or go to Issues if you have any suggestions for improvements or have discovered a bug with this software package.

Blog Posts

Releases

  File Modified
ZIP Archive ecFlow-5.12.0-Source.tar.gz Feb 13, 2024 by Marcos Bento
ZIP Archive ecFlow-5.11.4-Source.tar.gz Oct 12, 2023 by Iain Russell
ZIP Archive ecFlow-5.11.3-Source.tar.gz Jul 18, 2023 by Iain Russell
ZIP Archive ecFlow-5.11.1-Source.tar.gz Jul 04, 2023 by Iain Russell
ZIP Archive ecFlow-5.11.0-Source.tar.gz May 25, 2023 by Iain Russell
ZIP Archive ecFlow-5.10.0-Source.tar.gz Feb 09, 2023 by Iain Russell
File ecFlow-5.9.2-Source.tar.gz Nov 30, 2022 by Application deploy
File ecFlow-5.9.1-Source.tar.gz Nov 10, 2022 by Application deploy
File ecFlow-5.9.0-Source.tar.gz Oct 07, 2022 by Application deploy
File ecFlow-5.8.4-Source.tar.gz May 14, 2022 by Application deploy
File ecFlow-5.8.3-Source.tar.gz Mar 08, 2022 by Application deploy
File ecFlow-5.8.1-Source.tar.gz Jan 27, 2022 by Application deploy
File ecFlow-5.8.0-Source.tar.gz Jan 26, 2022 by Application deploy
File ecFlow-5.7.2-Source.tar.gz Hotfix release Sep 17, 2021 by Application deploy
File ecFlow-5.7.1-Source.tar.gz Sep 13, 2021 by Application deploy
File ecFlow-5.7.0-Source.tar.gz May 19, 2021 by Application deploy
File ecFlow-5.6.0-Source.tar.gz Nov 25, 2020 by Application deploy
File ecFlow-5.5.3-Source.tar.gz Sep 24, 2020 by Application deploy
File ecFlow-5.5.2-Source.tar.gz Sep 10, 2020 by Application deploy
File ecFlow-5.5.1-Source.tar.gz Jul 09, 2020 by Application deploy
File ecFlow-5.5.0-Source.tar.gz Jul 03, 2020 by Application deploy
File ecFlow-5.4.0-Source.tar.gz May 06, 2020 by Application deploy
ZIP Archive ecFlow-5.3.1-Source.tar.gz production release Apr 02, 2020 by Avi Bahra
File ecFlow-4.17.2-Source.tar.gz production release Mar 03, 2020 by Application deploy
File ecFlow-5.3.0-Source.tar.gz production release Mar 03, 2020 by Application deploy
File ecFlow-5.2.3-Source.tar.gz production release Feb 18, 2020 by Application deploy
File ecFlow-5.2.2-Source.tar.gz production release Feb 14, 2020 by Application deploy
File ecFlow-5.2.1-Source.tar.gz production release Jan 29, 2020 by Application deploy
File ecFlow-5.2.0-Source.tar.gz production release Jan 20, 2020 by Application deploy
File ecFlow-4.17.1-Source.tar.gz production release Jan 08, 2020 by Application deploy
File ecFlow-4.17.0-Source.tar.gz production release Oct 10, 2019 by Application deploy
File ecFlow-5.1.0-Source.tar.gz production release Oct 02, 2019 by Application deploy
File ecFlow-4.16.0-Source.tar.gz production release Aug 15, 2019 by Application deploy
File ecFlow-4.15.0-Source.tar.gz production release Jun 24, 2019 by Application deploy
File ecFlow-4.14.0-Source.tar.gz production release Mar 27, 2019 by Application deploy
File ecFlow-4.13.0-Source.tar.gz production release Feb 21, 2019 by Application deploy
File ecFlow-4.12.0-Source.tar.gz production release Jan 17, 2019 by Application deploy
File ecFlow-4.11.1-Source.tar.gz production release Oct 23, 2018 by Application deploy
File ecFlow-4.11.0-Source.tar.gz production release Oct 16, 2018 by Application deploy
ZIP Archive ecFlow-4.10.0-Source.tar.gz production release Jul 23, 2018 by Avi Bahra
File ecFlow-4.9.0-Source.tar.gz production release Apr 05, 2018 by Application deploy
ZIP Archive ecFlow-4.8.0-Source.tar.gz production release Jan 16, 2018 by Avi Bahra
ZIP Archive ecFlow-4.7.1-Source.tar.gz production release(hotfix) Nov 07, 2017 by Avi Bahra
ZIP Archive ecFlow-4.7.0-Source.tar.gz production release Oct 11, 2017 by Avi Bahra
ZIP Archive boost_1_53_0.tar.gz Boost libs for ecflow > 4.0.0 Nov 14, 2013 by Avi Bahra




Recently Updated