ecFlow's documentation is now on readthedocs!

Please go to https://ecflow.readthedocs.io/en/latest/index.html

Previous Up Next
It is sometimes useful to repeat the same task or family several times,
looping on a specific value. You can do that by defining a repeat attribute.
You can iterate over sequences of:
  • strings
  • integers
  • dates
A sequence of integers or dates is created by specifying the
first and the last element, with an optional increment (the default is one).

From ecflow5 we can also loop over an arbitrary list of dates.
datelist
repeat datelist YMD 20130101 20130102 20130103 20200101 20190101

An ecFlow variable, whose name corresponds to the name of the repeat,
will be generated. This can be used in scripts or trigger expressions.

If repeat date, or repeat datelist are used in trigger expressions, they will use date arithmetic.


Repeat with day/date

The behaviour of day/date attribute under a repeat has changed from ecflow 4.0. Take the example below.

In ecflow 4.0, if task t1 took longer than 1-hour task t2 would not run. (i.e. because we strayed over to the next day)

In ecflow 5.0, once the day monday is free on family f1, it stays free until the automatic re-queue caused by the parent repeat. The net effect being that task t2 will still run, even if we have strayed into Tuesday.

day/date remain free until re-queue
suite s1
  family f
    repeat integer rep 0 7
    family f1
       day monday  
       time 23:00
       task t1
       task t2 
            trigger t1 == complete
 ...

Repeat increment

A repeat under a family node will only increment when all the child nodes are complete.

In the example above, once task t1, and t2 are complete, the repeat integer rep will increment to the next value.

-

Ecf Script

We will add a new task /test/f4/f5/t1.
Create new ecf script file $HOME/course/test/f4/f5/t1.ecf to use these variables.


$HOME/course/test/f4/f5/t1.ecf 
%include <head.h>
ecflow_client --label=info "My name is '%NAME%' " " My value is '%VALUE%' " " My date is '%DATE%' "
# Note the use of repeat date generated variables DATE_YYYY, DATE_MM, DATE_DD, DATE_DOW to automatically reference year,month,day of the month,day of the week 
# These can also be used in trigger expression.
ecflow_client --label=date "year(%DATE_YYYY%) month(%DATE_MM%) day of month(%DATE_DD%) day of week(%DATE_DOW%)"  
sleep %SLEEP%
%include <tail.h>

Text

Let us modify the suite definition file again

# Definition of the suite test.
suite test
 edit ECF_INCLUDE "$HOME/course"
 edit ECF_HOME    "$HOME/course"
 family f4
     edit SLEEP 2
     repeat string NAME a b c d e f
     family f5
         repeat integer VALUE 1 10
         task t1
             repeat date DATE 20101230 20110105
             label info ""
             label date ""
     endfamily
 endfamily
endsuite

Python

$HOME/course/test.py
import os
from ecflow import Defs,Suite,Family,Task,Edit,Trigger,Complete,Event,Meter,Time,Day,Date,Label, \
                   RepeatString,RepeatInteger,RepeatDate
        
def create_family_f4():
    return Family("f4",
                Edit(SLEEP=2), 
                RepeatString("NAME", ["a", "b", "c", "d", "e", "f" ]),
                Family("f5",
                    RepeatInteger("VALUE", 1, 10),
                    Task("t1",
                        RepeatDate("DATE", 20101230, 20110105),
                        Label("info",""),
                        Label("date",""))))
 
print("Creating suite definition")  
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs( 
        Suite("test",
            Edit(ECF_INCLUDE=home, ECF_HOME=home),
            create_family_f4()))
print(defs) 
print("Checking job creation: .ecf -> .job0")  
print(defs.check_job_creation())

print("Checking trigger expressions")
assert len(defs.check()) == 0,defs.check() 

print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def")

What to do

  1. Type in the changes
  2. Replace the suite definition
  3. How many times will /test/f4/f5/t1 run?
  4. In ecflow_ui , try to modify the values of a repeat
  5. Since we are using a 2-second delay, remember to use f5(refresh) to see the intermediate values. The default is to refresh every 60 seconds.

  

Previous Up Next

7 Comments

  1. Axel Bonet

    repeat enumerated VAR a b c

    can also be used. It is widely appreciated when families must loop over an irregularly stepped list of dates (seasonal suites, eras)

    when a repeat variable is used in a trigger, its position is used for repeat string

    its value (when integer) of its position (otherwise) is used for repeat enumerated

    and its value is used for repeat date and integer.

  2. Avi Bahra

    Alternative styles

    import os
import ecflow as *

print("Creating suite definition")
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs(
        Suite("test",
            Edit(ECF_INCLUDE=home, ECF_HOME=home),
            Family("f4",
                Edit(SLEEP=2), 
                RepeatString("NAME", ["a", "b", "c", "d", "e", "f" ]),
                Family("f5",
                    RepeatInteger("VALUE", 1, 10),
                    Task("t1",
                        RepeatDate("DATE", 20101230, 20110105),
                        Label("info", ""),
                        Label("data",""))))))
print(defs)
print("Checking trigger expressions")
assert len(defs.check()) == 0,defs.check() 

print("Checking job creation: .ecf -> .job0")
result = defs.check_job_creation()

print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def")
defs.test.replace_on_server()
    import os
import ecflow as *

print("Creating suite definition")
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs().add( Suite("test") )
defs.test += [ {"ECF_INCLUDE":home,"ECF_HOME":home}, 
               Family("f4") ]
defs.test.f4 += [ Edit(SLEEP=2),
                  RepeatString("NAME", ["a", "b", "c", "d", "e", "f" ]),
                  Family("f5") ]
defs.test.f4.f5 += [ RepeatInteger("VALUE", 1, 10),
                     Task("t1")]
defs.test.f4.f5.t1 += [ RepeatDate("DATE", 20101230, 20110105),
                        Label("info", ""),
                        Label("data","") ]
print(defs)
print("Checking trigger expressions")
assert len(defs.check()) == 0,defs.check() 

print("Checking job creation: .ecf -> .job0")
result = defs.check_job_creation()

print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def")
defs.test.replace_on_server()

  3. Matthew Chantry

    A repeat under a family node will only increment when all the child nodes are complete.

    Is there a recommended way to avoid this behaviour? I have a repeat loop that I would like, for speed reasons, to run simultaneously. My current thinking is to unwrap the loop and use a python loop to create N tasks.

    1. Iain Russell

      Hi Matthew, yes, I think that in this case your approach of generating the tasks yourself with a Python loop would be best.

      1. Matthew Chantry

        Thanks Iain. 

        If I've understood, this can get quite messy as you'd need the filesystem to have folders for each member of the loop (even if the files themselves are symlinked to a master ecf file). This gets even worse if I want to unwrap nested loops.

        1. Iain Russell

          Well yes, you will need one directory per family and one .ecf file per task. As you say, you can use symlinks, so if you manually create one directory with a single master ecf file and all tasks symlinked to it, you can then create the other family directories as symlinks to that directory so there should not be much repetition.

    2. Axel Bonet

      well this is all the interest with suite design...
      the designer can "explode" the loop as multiple parallel families with higher impact on number of jobs submitted (a limit may refrain that)
      on disk occucation (WDIR)
      on disk IO

      the repeat comes with this simple rule.
      complete attribute can be useful in conjunction to fulfill the rule.

      when we want/can afford a suite to be more "agressive", we dedicate multiple families (with multiple repeat shifted to comply with the load we are allowed by system team, seasonal back in time run was managed that way)