Skip to content

Add workflow option to stats command - #247

Open
Toby-Masters-SF wants to merge 19 commits into
mainfrom
241-workflow-test-statistics
Open

Add workflow option to stats command#247
Toby-Masters-SF wants to merge 19 commits into
mainfrom
241-workflow-test-statistics

Conversation

@Toby-Masters-SF

@Toby-Masters-SF Toby-Masters-SF commented Feb 15, 2026

Copy link
Copy Markdown
Contributor

Fixes # (link to the corresponding issue if applicable)

Description

Include a summary of the changes made

Testing Instructions

Steps:

  1. ...
  2. ...
  3. ...

Author Checklist

  • Skip bumping the CLI version

Reviewer Checklist

  • PR has a clear title and description
  • Manually tested the changes that the PR introduces
  • Changes introduced are covered by tests of acceptable quality

Notes

Plan

  • Create a new getWorkflowTemplateSummary(workflowHandle) function
  • Create a new listWorkflowTemplates function which can take different classes?

Testing notes

  • Reconciliations (cross-referenced using AI to interrogate). Reason for difference is multiple yml files in the same tests folder so may need to update the Regex expression: .*${FOLDER}/${templatePattern}/tests/.*_liquid_test.*.y(a)?ml. Resolved by temporarily deleting "extra" tests in test branch
    • Total - 58: CORRECT
    • Externally maanged - 58: CORRECT
    • YAML - at least 1 test - 49: TO CHECK? (AI GIVING 46)
    • YAML - 2+ tests - 45: TO CHECK? (AI GIVING 43)

Quick Notes

  • Plan is to create a series of parallel functions to avoid overcomplicating the existing functions
    • May also wish to consider moving to a separate file in the cli folder?
  • Review function getTemplatesSummary --> Can it be co-opted for the purposes of this workflow? Take the "Reconciliations" section as an example!
    • [TBC]

@Toby-Masters-SF Toby-Masters-SF self-assigned this Feb 15, 2026
@Toby-Masters-SF Toby-Masters-SF linked an issue Feb 15, 2026 that may be closed by this pull request
@coderabbitai

coderabbitai Bot commented Feb 15, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The stats CLI command now supports optional per-workflow reporting. Workflow definitions are loaded from JSON, template and YAML activity is scoped per workflow, summaries are displayed, and workflow-specific CSV files are generated.

Changes

Per-workflow statistics reporting

Layer / File(s) Summary
Workflow loading and fixtures
lib/utils/fsUtils.js, fixtures/market-repo/workflows/*
Adds workflow JSON loading with error handling and fixtures covering valid, empty, incomplete, and malformed definitions.
Workflow-scoped YAML matching
lib/cli/stats.js
Restricts YAML activity matching and counting to templates referenced by each workflow.
Per-workflow data aggregation
lib/cli/stats.js
Computes reconciliation, account, export, rollup, and YAML activity summaries for each workflow.
Workflow statistics output
lib/cli/stats.js
Displays workflow summaries, serializes CSV rows, and writes workflow-specific files under ./stats/.
Workflow overview orchestration
lib/cli/stats.js
Adds exported workflow overview generation across selected or discovered workflow handles.
CLI command routing and date validation
bin/cli.js, lib/cli/utils.js
Adds --workflow [handle], validates --since, and routes to workflow-specific or global statistics generation.
Workflow and statistics documentation
README.md
Documents workflow JSON structure, statistics commands, output paths, and workflow-loading behavior.

Estimated code review effort: 3 (Moderate) | ~22 minutes

Possibly related issues

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning Required sections exist, but Description and Testing Instructions are still placeholders rather than concrete details. Replace the placeholder summary and steps with the actual change description, testing steps, and any relevant issue reference.
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding a workflow option to the stats command.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 241-workflow-test-statistics

Comment @coderabbitai help to get the list of available commands.

@Toby-Masters-SF
Toby-Masters-SF force-pushed the 241-workflow-test-statistics branch 4 times, most recently from 3fb5199 to b134da4 Compare March 18, 2026 14:47
@Toby-Masters-SF
Toby-Masters-SF force-pushed the 241-workflow-test-statistics branch 3 times, most recently from 66ed189 to 51fc334 Compare March 31, 2026 08:24
@michieldegezelle
michieldegezelle marked this pull request as ready for review June 11, 2026 07:15
@michieldegezelle
michieldegezelle self-requested a review June 11, 2026 07:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
lib/cli/stats.js (2)

370-371: 💤 Low value

Misleading parameter name templatesInWorkflow.

The parameter is named templatesInWorkflow but it actually receives a full workflow object (which is then passed to yamlFilesActivity). The naming is inconsistent with the actual usage. Consider renaming to workflow for clarity.

-async function getYamlSummary(sinceDate, templatesInWorkflow) {
-  const yamlActivity = await yamlFilesActivity(sinceDate, templatesInWorkflow);
+async function getYamlSummary(sinceDate, workflow) {
+  const yamlActivity = await yamlFilesActivity(sinceDate, workflow);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/cli/stats.js` around lines 370 - 371, The parameter name
templatesInWorkflow in function getYamlSummary is misleading because it actually
expects a full workflow object passed into yamlFilesActivity; rename the
parameter to workflow (update the function signature of getYamlSummary and all
internal references) and update any call sites to pass the same workflow
variable name so usage is consistent with the actual object being passed to
yamlFilesActivity.

319-344: 💤 Low value

No defensive checks for missing workflow properties.

If a workflow JSON file is malformed or missing the templates property (or its sub-properties like reconciliations, exports, accounts), the code will throw with an unclear error. Consider adding defensive checks or clearer error messages.

💡 Optional: Add defensive validation
   // Fetch workflow
   // Assume no empty items if present within the workflow.
+  if (!workflow.templates) {
+    consola.error(`Workflow "${workflow.name || 'unknown'}" is missing 'templates' property`);
+    process.exit(1);
+  }
   summary.workflow_name = workflow.name;
 
   // Reconciliations
-  const reconciliationsInWorkflow = workflow.templates.reconciliations;
+  const reconciliationsInWorkflow = workflow.templates.reconciliations || [];
   // ...
-  const exportFilesInWorkflow = workflow.templates.exports;
+  const exportFilesInWorkflow = workflow.templates.exports || [];
   // ...
-  const accountTemplatesInWorkflow = workflow.templates.accounts;
+  const accountTemplatesInWorkflow = workflow.templates.accounts || [];
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/cli/stats.js` around lines 319 - 344, The code assumes workflow.templates
and its sub-properties exist (e.g., workflow.templates.reconciliations,
.exports, .accounts), which can throw on malformed workflow JSON; update the
start of this block to defensively validate workflow and its template arrays (or
normalize them) before using them: check that workflow is truthy, that
workflow.templates is an object, and that workflow.templates.reconciliations,
workflow.templates.exports, workflow.templates.accounts are arrays (fallback to
[] if missing) or throw a clear, descriptive error; ensure subsequent uses of
reconciliationsInWorkflow, exportFilesInWorkflow, accountTemplatesInWorkflow,
and calls to listExternallyManagedTemplates/countYamlFiles use these
validated/normalized variables so percentageRoundTwo and .length won't fail.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/cli/stats.js`:
- Around line 23-26: The code currently sets workflowsFolder and populates
workflowHandles using fs.readdirSync which will throw if the "workflows"
directory is missing; update the logic around the
workflowsFolder/workflowHandles assignment (the workflowsFolder const and
workflowHandles variable) to handle a missing folder by either checking
fs.existsSync(workflowsFolder) before calling fs.readdirSync or wrapping
readdirSync in a try/catch and falling back to an empty array, and ensure any
caught error is handled or logged rather than allowed to crash the process.
- Around line 55-58: The TEMPLATE_PATTERN is built from raw template names
(templatesInWorkflow) so regex metacharacters in names break the pattern; before
joining, escape each template string (e.g., implement or use an escapeRegExp
helper and map over templatesInWorkflow to escape chars like . * + ? ^ $ ( ) [ ]
{ } | \ /) then join the escaped names to build TEMPLATE_PATTERN and proceed to
construct YAML_EXPRESSION and fileTypeRegExp from that escaped pattern (refer to
templatesInWorkflow, TEMPLATE_PATTERN, YAML_EXPRESSION, fileTypeRegExp).

In `@lib/utils/fsUtils.js`:
- Around line 483-495: getWorkflow currently uses the raw CLI-provided
workflowHandle to build file paths allowing path traversal; update getWorkflow
to validate and sanitize workflowHandle (e.g., reject any path separators, "..",
or characters outside a safe whitelist like /^[A-Za-z0-9_-]+$/) and return a
clear error instead of accepting unsafe names, then apply the same sanitization
or safe-filename conversion in saveWorkflowOverviewToFile before constructing
CSV/output paths so both functions (getWorkflow and saveWorkflowOverviewToFile)
never join untrusted input into filesystem paths.

---

Nitpick comments:
In `@lib/cli/stats.js`:
- Around line 370-371: The parameter name templatesInWorkflow in function
getYamlSummary is misleading because it actually expects a full workflow object
passed into yamlFilesActivity; rename the parameter to workflow (update the
function signature of getYamlSummary and all internal references) and update any
call sites to pass the same workflow variable name so usage is consistent with
the actual object being passed to yamlFilesActivity.
- Around line 319-344: The code assumes workflow.templates and its
sub-properties exist (e.g., workflow.templates.reconciliations, .exports,
.accounts), which can throw on malformed workflow JSON; update the start of this
block to defensively validate workflow and its template arrays (or normalize
them) before using them: check that workflow is truthy, that workflow.templates
is an object, and that workflow.templates.reconciliations,
workflow.templates.exports, workflow.templates.accounts are arrays (fallback to
[] if missing) or throw a clear, descriptive error; ensure subsequent uses of
reconciliationsInWorkflow, exportFilesInWorkflow, accountTemplatesInWorkflow,
and calls to listExternallyManagedTemplates/countYamlFiles use these
validated/normalized variables so percentageRoundTwo and .length won't fail.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 25e76814-59ef-4417-9c64-b5134a0b6c9a

📥 Commits

Reviewing files that changed from the base of the PR and between 840b02d and e099926.

📒 Files selected for processing (3)
  • bin/cli.js
  • lib/cli/stats.js
  • lib/utils/fsUtils.js

Comment thread lib/cli/stats.js Outdated
Comment thread lib/cli/stats.js Outdated
Comment thread lib/utils/fsUtils.js
Comment thread bin/cli.js Outdated
Comment thread lib/cli/stats.js
fs.mkdirSync("stats");
}
if (!fs.existsSync(CSV_PATH)) {
fs.writeFileSync(CSV_PATH, ROW_HEADER, (err) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fs.writeFileSync silently ignores the callback passed as the third argument.

writeFileSync is synchronous and its third parameter is an options object (encoding, mode, flag) — not an error handler. The callback (err) => { consola.error(err); } is coerced to an options object and silently discarded. Any write failure (e.g. permission denied, disk full) throws an uncaught synchronous exception rather than being handled. The same issue exists at line 592 in saveOverviewToFile.

Remove the callback and either let the exception propagate naturally, or wrap the call in a try/catch.

@michieldegezelle michieldegezelle left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few additional code-review notes not covered by existing threads.

Can you also update the changelog, bump the version in package.json and do a npm update to update the package-lock? Thanks

Comment thread lib/cli/stats.js Outdated
Comment thread lib/cli/stats.js
const FOLDER = fsUtils.FOLDERS[templateType];
const YAML_EXPRESSION = `.*${FOLDER}/.*/tests/.*_liquid_test.*.y(a)?ml`;
const TEMPLATE_PATTERN = templatesInWorkflow ? `(${templatesInWorkflow.join("|")})` : `.*`;
// Issue with counting multiple yml files in the same tests folder?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover open question in a comment (Issue with counting multiple yml files in the same tests folder?). Either confirm this is not a problem and remove the comment, or document/fix the edge case.

Comment thread lib/cli/stats.js
};

// Fetch workflow
// Assume no empty items if present within the workflow.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Global vs workflow stats count different populations.

generateOverview counts only non-empty templates (listNonEmptyTemplates), while workflow mode counts every handle listed in the workflow JSON. That is probably fine, but worth documenting in the CLI help or PR description so users do not compare the two totals directly.

Comment thread lib/cli/stats.js

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
lib/cli/utils.js (1)

61-76: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add focused tests for date-validation boundaries.

Cover a valid leap day, malformed input, impossible dates such as 2024-02-31, and the process.exit(1) failure path. tests/lib/cli/utils.test.js already mocks consola, making it suitable for these cases.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/cli/utils.js` around lines 61 - 76, Add focused tests in utils.test.js
for checkDateFormat covering a valid leap day, malformed input, an impossible
date such as 2024-02-31, and the process.exit(1) failure path. Reuse the
existing consola mock and assert both validation outcomes and the expected exit
behavior without changing checkDateFormat.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@lib/cli/utils.js`:
- Around line 61-76: Add focused tests in utils.test.js for checkDateFormat
covering a valid leap day, malformed input, an impossible date such as
2024-02-31, and the process.exit(1) failure path. Reuse the existing consola
mock and assert both validation outcomes and the expected exit behavior without
changing checkDateFormat.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 41da8d3a-f73d-4466-a831-193bb82f0a10

📥 Commits

Reviewing files that changed from the base of the PR and between 9f40efb and 19a13f5.

📒 Files selected for processing (2)
  • bin/cli.js
  • lib/cli/utils.js
🚧 Files skipped from review as they are similar to previous changes (1)
  • bin/cli.js

@Toby-Masters-SF
Toby-Masters-SF force-pushed the 241-workflow-test-statistics branch from 1f856a2 to 6bf07a4 Compare August 11, 2026 15:35
@Toby-Masters-SF
Toby-Masters-SF force-pushed the 241-workflow-test-statistics branch from 6bf07a4 to 1817f89 Compare August 11, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow test statistics

2 participants