Repository navigation
Add ecFlow suite generator for GFS forecast-only (C48 ATM) #5302
+3,371
−31
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Slurm-compatible environment header for Ursa | ||
| # Replaces envir-p1.h (PBS_JOBNAME/PBS_JOBID → Slurm equivalents) | ||
| export job=${job:-${SLURM_JOB_NAME:-$(basename "${ECF_JOB:-.}")}} | ||
| export jobid=${jobid:-${job}.${SLURM_JOB_ID:-$$}} | ||
|
|
||
| export RUN_ENVIR=${RUN_ENVIR:-emc} | ||
| export envir=%ENVIR% | ||
| export RUN=%RUN% |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| date | ||
| hostname | ||
| set -xe | ||
|
|
||
| export PS4='+ $SECONDS + ' | ||
|
|
||
| # Load ecflow module first so variable exports below take precedence | ||
| module load ecflow | ||
|
|
||
| # ecFlow communication variables | ||
| export ECF_NAME=%ECF_NAME% | ||
| export ECF_HOST=%ECF_HOST% | ||
| export ECF_PORT=%ECF_PORT% | ||
| export ECF_PASS=%ECF_PASS% | ||
| export ECF_TRYNO=%ECF_TRYNO% | ||
| export ECF_RID=${ECF_RID:-${SLURM_JOB_ID:-$(hostname -s).$$}} | ||
| export ECF_JOB=%ECF_JOB% | ||
| export ECF_JOBOUT=%ECF_JOBOUT% | ||
|
|
||
| # Preserve ECF_HOST/ECF_PORT for use after load_modules.sh resets modules | ||
| readonly _ECF_HOST_SAVED=%ECF_HOST% | ||
| readonly _ECF_PORT_SAVED=%ECF_PORT% | ||
|
|
||
| # Notify ecFlow that the task has started | ||
| timeout 300 ecflow_client --init=${ECF_RID} | ||
|
|
||
| # Error handler — restores ecflow_client and server coordinates | ||
| ERROR() { | ||
| set +ex | ||
| module load ecflow 2> /dev/null || true | ||
| export ECF_HOST="${_ECF_HOST_SAVED}" | ||
| export ECF_PORT="${_ECF_PORT_SAVED}" | ||
| if [ "$1" -eq 0 ]; then | ||
| msg="Killed by signal (likely via scancel)" | ||
| else | ||
| msg="Killed by signal $1" | ||
| fi | ||
| ecflow_client --abort="$msg" | ||
| echo "$msg" | ||
| trap 0 | ||
| exit $1 | ||
| } | ||
| trap 'ERROR $?' ERR EXIT |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| module load ecflow 2> /dev/null || true | ||
| export ECF_HOST="${_ECF_HOST_SAVED}" | ||
| export ECF_PORT="${_ECF_PORT_SAVED}" | ||
| timeout 300 ecflow_client --complete | ||
| trap 0 | ||
| exit 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
|
Contributor
There was a problem hiding this comment. This would work on all machines - but this still means for operations we have to go through and do the module loads for each job, etc. Is that just something we're accepting?
Contributor
Author
There was a problem hiding this comment. @JessicaMeixner-NOAA Since the module update hasn’t been brought into develop yet, I kept it as is for now. This is probably better handled as a separate PR to update all the ecFlow scripts accordingly?
Contributor
There was a problem hiding this comment. to be fair, this might be pretty hard to do without this and it work for all machines - but this is something that we should maybe think about moving forward for how we can have something that's ready for ops and that is machine portable for all machines.
Contributor
Author
There was a problem hiding this comment. @JessicaMeixner-NOAA Thanks for the insight. We can discuss this further and come up with a plan for how to proceed. |
||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="fcst" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
| export FCST_SEGMENT=0 | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_FCST" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| %include <tail.h> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="stage_ic" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_STAGE_IC" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
|
Comment on lines
+30
to
+31
Contributor
There was a problem hiding this comment. I see this in several |
||
| fi | ||
|
|
||
| %include <tail.h> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export TARBALL_TYPE=%TARBALL_TYPE% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="arch_tars" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_ARCHIVE_TARS" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| %include <tail.h> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="arch_vrfy" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_ARCHIVE_VRFY" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| %include <tail.h> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="cleanup" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_CLEANUP" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| %include <tail.h> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| # Rocoto runs one metp task per METplus use case; run them in sequence here | ||
| for METPCASE in g2g1 g2o1 pcp1; do | ||
| export METPCASE | ||
| export job="metp${METPCASE}" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGFS_ATMOS_VERIFICATION" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT (${METPCASE})***" | ||
| exit "${status}" | ||
| fi | ||
| done | ||
|
|
||
| %include <tail.h> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="atmos_products" | ||
|
|
||
| # Process the forecast hours assigned to this task via FHR_LIST. | ||
| # When run as a family child (generator .def), FHR_LIST is a | ||
| # comma-separated subset like "0,1,2,3,4". When run as a single | ||
| # task (static .def), set FHR_LIST to the full range externally. | ||
| export FORECAST_HOUR jobid | ||
| IFS=',' read -ra _fhrs <<< "%FHR_LIST%" | ||
| for fhr in "${_fhrs[@]}"; do | ||
| fhr3=$(printf '%%03d' "${fhr}") | ||
| export FORECAST_HOUR="${fhr}" | ||
| jobid="${job}_f${fhr3}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGLOBAL_ATMOS_PRODUCTS" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR at f${fhr3}***" | ||
| exit "${status}" | ||
| fi | ||
| done | ||
|
Comment on lines
+31
to
+43
Contributor
There was a problem hiding this comment. The suite generator groups forecast hours (e.g. What this script actually does is the opposite: it calls
Was the per-hour loop intentional, like is An alternative could look like:
Contributor
Author
There was a problem hiding this comment. @AnnaSmoot-NOAA Good catch. All of these issues were caused by setting |
||
|
|
||
| %include <tail.h> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/bash | ||
| # shellcheck disable=SC1009,SC1072,SC1073 | ||
| %include <head.h> | ||
| %include <envir.h> | ||
|
|
||
| set -x | ||
|
|
||
| export HOMEglobal=%HOMEglobal% | ||
| export EXPDIR=%EXPDIR% | ||
| export COMROOT=%COMROOT% | ||
| export DATAROOT=%DATAROOT% | ||
| export NET=%NET% | ||
| export PDY=%PDY% | ||
| export cyc=%CYC% | ||
| export machine=URSA | ||
|
|
||
| source "${HOMEglobal}/dev/ush/load_modules.sh" run | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR loading modules***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| export job="genesis" | ||
| export jobid="${job}.${SLURM_JOB_ID:-$$}" | ||
|
|
||
| "${HOMEglobal}/dev/jobs/JGFS_ATMOS_CYCLONE_GENESIS" | ||
| status=$? | ||
| if [[ ${status} -ne 0 ]]; then | ||
| ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT***" | ||
| exit "${status}" | ||
| fi | ||
|
|
||
| %include <tail.h> |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generate_workflows.py will not work with this test activated on Ursa. Suggest keeping this skip in place and removing from the gitlab hosts yaml for the time being.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TravisElless-NOAA This YAML file is intended for the ecFlow test only. A duplicate YAML file is currently used for Rocoto runs, but I’ll eliminate the duplication in the next PR for the Rocoto GEFS case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @AntonMFernando-NOAA, but I think we will need to address this here. As shown below running the generate_workflows script causes an error to occur and prevents things from running with the way things are currently setup. I don't want this to bring down the whole test system on Ursa
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TravisElless-NOAA Sorry, I think I misunderstood your concern. You are completely right. However, since we’re using the YAML files in
dev/ci/cases/prdirectly to run cases withgenerate_workflow, it would be better to move the YAML files used for this purpose to a separate directory, at least for the time being. Let me know if you have any other concerns. Made the changes.