In the previous exercise, we showed how ecflow provides simple load management.
The limits can still allow several hundred jobs to be submitted at once. This can cause problems:
- Excessive disk/io in job generation
- Server busy in job generation, and slow to respond to the GUI.
- Overload queuing systems like PBS/SLURM
Hence we need load management that can limit the number of submissions. When the Job becomes active the limit token is released.
This exercise shows the use of '-s' option. Which helps to control the number of tasks in the submitted state.
We can have more than 2 active jobs since we only control the number in the submitted state.
If we removed the -s then we can only have two active jobs running at one time
The use of -s allows the configuration of the suite, depending on the load the disk/io and queuing system can sustain.
Here is the simple illustration that modifies the previous example:
Text
Let us modify our suite definition file:
# Definition of the suite test. suite test edit ECF_INCLUDE "$HOME/course" edit ECF_HOME "$HOME/course" limit l1 2 family f5 inlimit -s l1 # by default consume 1 token from the limit l1 edit SLEEP 20 task t1 task t2 task t3 task t4 task t5 task t6 task t7 task t8 task t9 endfamily endsuite
Python
import os from ecflow import Defs,Suite,Family,Task,Edit,Trigger,Complete,Event,Meter,Time,Day,Date,Label, \ RepeatString,RepeatInteger,RepeatDate,InLimit,Limit def create_family_f5() : return Family("f5", # limit_name(l1),limit_path(""),no_of_tokens_to_consume(1),limit node(False), limit submission(True) InLimit("l1","",1,False,True), Edit(SLEEP=20), [ Task('t{}'.format(i)) for i in range(1,10) ] ) print("Creating suite definition") home = os.path.join(os.getenv("HOME"),"course") defs = Defs( Suite("test", Edit(ECF_INCLUDE=home,ECF_HOME=home), Limit("l1",2), create_family_f5())) print(defs) print("Checking job creation: .ecf -> .job0") print(defs.check_job_creation()) print("Checking trigger expressions and inlimits") assert len(defs.check()) == 0,defs.check() print("Saving definition to file 'test.def'") defs.save_as_defs("test.def")
What to do
- Edit the changes
- Replace the suite definition
- In ecflow_ui , observe the effects
- Change the value of the limit and inlimit, observe the effect.