Repository navigation
Add ecFlow suite generator for GFS forecast-only (C48 ATM) - #5302
Open
AntonMFernando-NOAA wants to merge 94 commits into
Open
AntonMFernando-NOAA wants to merge 94 commits into
AntonMFernando-NOAA wants to merge 94 commits into
Conversation
…ting - Add 9 new ECF job scripts (arch_tars, arch_vrfy, atmos_prod, cleanup, fcst, genesis, metp, stage_ic, tracker) - Add 4 utility header files for ECF environment (slurm.h, head.h, envir.h, tail.h) - Add load_ecflow_case.py for ECFlow case loading - Add sync_ecf_scripts.sh for synchronizing ECF scripts - Update gfs_forecast_only_ecflow.py to support new ECFlow workflow - Standardize ECF script structure with common includes and error handling
- Fix printf format string escaping in forecast hour loop - Change '%03d' to '%%03d' to properly escape the percent sign in ECF context - Prevents format string misinterpretation during ecflow script processing
…overwrite - Modify cleanup_stale_files() to return bool indicating cleanup success - Return True when no directories found or cleanup completed - Return False when user skips cleanup via prompt - Update cleanup skip message to clarify workflow continuation - Pass cleanup status to create_experiment() overwrite parameter - Only allow experiment overwrite if previous cleanup was successful - Prevents accidental overwrites when stale files remain uncleaned
…pproach - Change references from "symlink directory" to "script directory" throughout docstrings - Update method documentation to reflect that .ecf files are copied rather than symlinked - Clarify that ECF_FILES points to the script directory containing .ecf file copies - Update internal comments to accurately describe the copy-based approach instead of symlink references - Replace "symlink" terminology with "script" or "copy" in variable names and comments for consistency
… loading - Create c48_atm_ecflow.py as thin entry point for C48_ATM case loading - Refactor load_ecflow_case.py to accept optional default_yaml parameter - Extract DEFAULT_YAML constant into test-specific scripts for modularity - Update parse_args() to support optional vs required --yaml argument - Rename main() to run() for programmatic use; add new main() wrapper - Update module docstrings to document generic loader pattern and entry point delegation - Enables reusable ecFlow case loading framework for future test cases
- Add new C48ATMEcFlowSuite class that extends GFSForecastOnlyEcFlowSuite for atmosphere-only forecasts - Define comprehensive JJOB_MAP with 25+ task-to-J-Job script mappings for C48 ATM case - Implement RESOURCE_STEP_MAP to handle config step name overrides (upp, atmos_products, etc.) - Add SERVICE_TASKS and PRODUCT_TASKS class attributes for task classification - Implement _get_trigger() method with complete ATM-specific dependency chain mirroring rocoto/gfs_tasks.py - Register C48_ATM suite in ecflow_suite_factory.py with appropriate case loader - Update gfs_forecast_only_ecflow.py to support generalized case loading pattern
…attern - Delete monolithic C48_ATM suite class and extract task logic into reusable modules - Add ecflow_tasks.py with base EcFlowTask abstraction and common task definitions - Add ecflow_tasks_factory.py to generate task instances with dependency resolution - Add ecflow_workflow_tasks.py for workflow-level task orchestration - Add gfs_ecflow_tasks.py with GFS-specific task mappings and trigger logic - Update ecflow_suite_factory.py to use new modular task factory pattern - Update gfs_forecast_only_ecflow.py to integrate refactored task system - Improves maintainability by separating concerns and reducing code duplication across suite definitions
- Add --expdir argument to override experiment directory base path - Add --comroot argument to override COMROOT path - Add --pslot argument to override experiment name - Set environment variables from CLI arguments before loading case YAML - Ensure CLI overrides take precedence over YAML and environment defaults - Use Jinja2 getenv filters to apply overrides during case YAML rendering
- Add --stmp argument to load_ecflow_case.py CLI to allow overriding STMP (scratch/tmp) path - STMP is used to build DATAROOT = <stmp>/RUNDIRS/<pslot> - Set os.environ['STMP'] when --stmp argument is provided - Enables flexible experiment configuration for different scratch storage locations
- Rename `_symlink_map` instance variable to `_copy_map` throughout GFSForecastOnlyEcFlowSuite - Update loop variable names from `link_name, target_name` to `dest_name, src_name` for consistency with the copy operation semantics - Improve variable naming to better reflect that files are being copied, not symlinked - Update manifest generation and file counting logic to use new variable names
- Add Ursa to METplus verification exclusion list in config.base.j2 - Add HPSS archiving availability check for Ursa machine - Improve ecflow_client error reporting with detailed stdout/stderr output - Add try-catch block to capture and display subprocess errors with context - Update comment to reflect Ursa as unsupported platform for METplus and HPSS
- Add ECF_HOME to required environment variables for ecFlow job file storage - Enhance environment validation error message with setup instructions and Ursa example - Implement interactive suite deletion prompt instead of silent replacement - Check for existing suite before attempting deletion to prevent data loss - Add user confirmation dialog with abort option to preserve existing suites - Improve user guidance with module loading and variable export instructions
- Suspend suite before deletion to stop new task submissions - Add 120s timeout to delete operation for large suites with active tasks - Wrap deletion in try-except to handle timeout and error cases gracefully - Print warnings with manual recovery instructions if deletion fails - Prevents race conditions and provides better error feedback during suite reloading
…error handling - Add kill step after suspension to forcefully terminate active tasks - Insert 5-second sleep to allow task termination before deletion - Use --delete=force flag with explicit "yes" confirmation for more reliable deletion - Reduce delete timeout from 120s to 60s - Consolidate timeout and process error handling into single exception block - Simplify error messaging and provide updated manual deletion command - Update user-facing message from "Suspending" to "Stopping" for clarity
- Add detailed README covering prerequisites and system requirements - Document step-by-step Ursa connection setup for Windows, Mac, and Linux - Include environment configuration with ECF_HOST, ECF_PORT, and ECF_HOME - Provide ecFlow server startup, monitoring, and troubleshooting procedures - Document day-to-day operations workflow with quick reference commands - Add suite execution, task monitoring, and failure recovery instructions - Include cleanup procedures and .ecf script synchronization methods - Organize guide with clear sections for initial setup and operational tasks
- Add sys import and define _SCRIPT_DIR and _WORKFLOW_DIR path variables - Insert dev/workflow/ directory into sys.path to ensure ecflow.load_ecflow_case imports resolve to local package instead of system ecFlow Python bindings - Add noqa comment to suppress E402 import ordering violation after sys.path modification - Prevents import conflicts and ensures correct module resolution for ecflow suite execution
- Remove --load-only command-line argument from argument parser - Remove conditional logic that branched on load-only flag - Change suite loading to always stop after load step without beginning - Update user messaging to consistently show begin command after load - Simplify workflow by eliminating optional load-only behavior in favor of explicit two-step process
- Add guidance for deleting suite using ecflow_client --delete=force command - Improve user visibility into cleanup procedures after suite loading - Provide clear command example alongside existing begin and monitor instructions
…stances - Remove shared server configuration (uecflow01:23385) as deprecated - Require each user to run their own ecFlow server on unique port - Calculate ECF_PORT dynamically using UID offset (UID + 1500) to prevent collisions - Set ECF_HOST to local hostname instead of hardcoded shared server - Add echo statement to display assigned server details for user confirmation - Update both setup and Ursa remote connection sections for consistency - Simplifies multi-user workflows by eliminating shared server dependency
…ables list - Include STMP, PTMP, HOMEDIR, and NOSCRUB in the variables saved and removed from environment - Ensure temporary and home directory paths are properly managed during ecFlow suite loading - Prevent environment variable conflicts that could affect suite execution
- Remove `--load-only` CLI option documentation and section - Update suite loading description to clarify default behavior - Explain that suites load but do not start automatically - Document that users must run `ecflow_client --begin` manually - Simplify workflow instructions by removing redundant loading section
…tch.sh wrapper - Remove static slurm.h header file and its includes from all 9 .ecf scripts - Add ecf_sbatch.sh script to dynamically source config.resources for per-task Slurm resource allocation - Replace static SBATCH directive approach with runtime resource lookup via config.resources - Update README.md execution flow diagram to reflect envir.h and ecf_sbatch.sh submission model - Simplify resource management by centralizing task configuration in config.resources instead of hardcoded .def values - Enables dynamic resource allocation per task without modifying .def file or individual .ecf scripts
- Change cycle family identifier from full datetime (e.g. "2021032312") to hour only (e.g. "12") - Extract hour value once into `cyc` variable and reuse for both family name and CYC edit - Update comment to reflect new naming convention - Reduces redundancy and improves clarity of cycle family structure
- Add PDY and cyc environment variable stubs with defaults before sourcing config.base - Prevent "unbound variable" errors when config.base references runtime variables under set -eu - Update comment to clarify that config.base depends on PDY and cyc being defined - Ensures ecf_sbatch.sh can source config.base without requiring J-Job pre-initialization
- Change PDY default stub value from 19000101 to 20210323 - Maintains compatibility with config.base sourcing under set -eu - Aligns with more realistic test date for development and testing
…n ordering - Swap family hierarchy from run→cycle to cycle→run for improved organization - Change outer family from static RUN identifier to dynamic cycle timestamp (YYYYMMDDHH format) - Move RUN family inside cycle family as inner container - Set PDY and CYC variables at cycle family level instead of nested cycle level - Simplify cycle extraction by moving sdate calculation outside nested family blocks - Improves suite structure alignment with datetime-based cycle organization
- Add mkdir -p command to ensure ECF_JOBOUT directory exists - Prevent sbatch submission failures when parent directories are missing - Improves robustness of job submission workflow
- Create job output directory structure matching the suite tree hierarchy - Generate directories for each cycle and run under logs/suite_name/cycle/run - Create product family subdirectories for tasks marked as product_task - Ensure ecFlow can write .job files before ECF_JOB_CMD execution - Use os.makedirs with exist_ok=True to handle concurrent suite submissions safely
emcbot
removed
the
CI-Ursa-Running
|
C96_gcafs_cycled_noDA FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
… update references
|
C96_gcafs_cycled_noDA FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
|
C96_gcafs_cycled_noDA FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
|
C96_gcafs_cycled_noDA FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
|
C96_gcafs_cycled FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
|
C96_gcafs_cycled_noDA FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
|
C96_gcafs_cycled FAILED on Ursa (pipeline ID: 113066) In directory: Error Log Files: View Error Logs: (gcdas_fetch.log) (gcdas_fetch.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
AntonMFernando-NOAA
removed
the
CI-Ursa-Failed
emcbot
added
CI-Ursa-Building
| skip_ci_on_hosts: | ||
| - wcoss2 | ||
| - hera | ||
| - ursa |
Contributor
There was a problem hiding this comment.
generate_workflows.py will not work with this test activated on Ursa. Suggest keeping this skip in place and removing from the gitlab hosts yaml for the time being.
Contributor
Author
There was a problem hiding this comment.
@TravisElless-NOAA This YAML file is intended for the ecFlow test only. A duplicate YAML file is currently used for Rocoto runs, but I’ll eliminate the duplication in the next PR for the Rocoto GEFS case.
Contributor
There was a problem hiding this comment.
Thanks @AntonMFernando-NOAA, but I think we will need to address this here. As shown below running the generate_workflows script causes an error to occur and prevents things from running with the way things are currently setup. I don't want this to bring down the whole test system on Ursa
[Travis.J.Elless@ufe02 workflow]$ ./generate_workflows.sh -c -t test_5302 -GESC -A fv3-cpu /scratch4/NCEPDEV/stmp/Travis.J.Elless/tests/pr5302
Running all GEFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr
Running all GFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr
Running all SFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr
Running all GCAFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr
Running create_experiment.py for 18 cases
C48_S2SWA_gefs
C48_ATM
C48_ATM_ecflow
Could not find a crontab file for case C48_ATM_ecflow_test_5302!
Expected to find /scratch4/NCEPDEV/stmp/Travis.J.Elless/tests/pr5302/EXPDIR/C48_ATM_ecflow_test_5302/C48_ATM_ecflow_test_5302.crontab
Contributor
Author
There was a problem hiding this comment.
Thanks @AntonMFernando-NOAA, but I think we will need to address this here. As shown below running the generate_workflows script causes an error to occur and prevents things from running with the way things are currently setup. I don't want this to bring down the whole test system on Ursa
[Travis.J.Elless@ufe02 workflow]$ ./generate_workflows.sh -c -t test_5302 -GESC -A fv3-cpu /scratch4/NCEPDEV/stmp/Travis.J.Elless/tests/pr5302 Running all GEFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr Running all GFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr Running all SFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr Running all GCAFS cases in /scratch3/NCEPDEV/global/Travis.J.Elless/global-workflow/pr5302/global-workflow/dev/ci/cases/pr Running create_experiment.py for 18 cases C48_S2SWA_gefs C48_ATM C48_ATM_ecflow Could not find a crontab file for case C48_ATM_ecflow_test_5302! Expected to find /scratch4/NCEPDEV/stmp/Travis.J.Elless/tests/pr5302/EXPDIR/C48_ATM_ecflow_test_5302/C48_ATM_ecflow_test_5302.crontab
@TravisElless-NOAA Sorry, I think I misunderstood your concern. You are completely right. However, since we’re using the YAML files in dev/ci/cases/pr directly to run cases with generate_workflow, it would be better to move the YAML files used for this purpose to a separate directory, at least for the time being. Let me know if you have any other concerns. Made the changes.
- Introduced C48_ATM_ecflow.yaml for new experiment configuration. - Updated gitlab-ci-hosts.yml to remove reference to C48_ATM_ecflow. - Modified ci_cd_pipeline.rst to reflect changes in example paths. - Revised quick_start.rst to clarify ecFlow case handling.
…riable dependencies and enhance server discovery
…ML config, suite name, and server settings
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an ecFlow suite generator that mirrors the Rocoto architecture for GFS forecast-only workflows.
EcFlowTasks/GFSEcFlowTasks: per-task methods returning structured dicts, extending Rocoto's Tasks base for resource and config reuseGFSForecastOnlyEcFlowSuite: suite generator producing a .def file with category families (init, forecast, product, track, verf, post)#SBATCHdirectives baked into .ecf files at config time from get_resource() dataStructured ECF_FILES layout: {EXPDIR}/ecf_scripts/{include/, scripts/{category}/}
load_ecflow_case.py: CLI entry point with environment validationsync_ecf_scripts.sh: refreshes .ecf copies from repo sources preserving #SBATCH headersUrsa platform support in
detect_machine.shandconfig.base.j2Resolves ecFlow: C48 ATM forecast-only test on Ursa #5301
Refs ecFlow suite generator for global-workflow #5300
Documentation
dev/workflow/ecflow/README.mddev/workflow/ecflow/PROCESS_FLOW.mdType of change
Change characteristics
How has this been tested?
Include the output of
gw_cistat -r /path/to/RUNTESTS/directoryfor the tests you ran.gw_cistatis available after sourcing thegw_setup.shscript indev/ush/.Checklist