refactor: replace old tables with dynamic-mat-table and remove column icons

[abdimo101]

Description

This PR replaces old tables with dynamic-mat-table in datafiles.component and related-datasets.component and removes column header icons to keep table headers consistent.

Datafiles table:

Related Datasets table:

Motivation

Background on use case, changes needed

Fixes:

Please provide a list of the fixes implemented in this PR

• Items added

Changes:

Please provide a list of the changes implemented by this PR

• changes made

Tests included

• Included for each change/fix?
• Passing? (Merge will not be approved unless this is checked)

Documentation

• swagger documentation updated [required]
• official documentation updated [nice-to-have]

official documentation info

If you have updated the official documentation, please provide PR # and URL of the pages where the updates are included

Backend version

• Does it require a specific version of the backend
• which version of the backend is required:

Summary by Sourcery

Refactor dataset-related tables to use dynamic-mat-table with centralized pagination and selection handling while aligning table headers and visuals.

New Features:

• Introduce dynamic-mat-table data source, pagination, and row event handling for datafiles and related datasets views.

Bug Fixes:

• Ensure selected file size is accurately derived from the underlying files collection after row and master selection changes.
• Keep related datasets pagination and counts in sync with store-driven view model updates.

Enhancements:

• Replace legacy table components in datafiles and related datasets views with dynamic-mat-table, including server-side pagination and standardized empty states.
• Simplify selection logic in the datafiles component by delegating row and master selection handling to dynamic-mat-table events.
• Remove column header icons across multiple tables to maintain a consistent, cleaner table header appearance.
• Adjust datafiles header styling to support variable height content.

[sourcery-ai]

Hey - I've found 2 issues, and left some high level feedback:

• In DatafilesComponent, paginationMode is set to 'server-side' but there is no (paginationChange) handler wired up, so changing pages in the dynamic-mat-table will not update dataSource; either implement a paginationChange handler that slices/requests data and calls dataSource.next(...) or switch to 'client-side' pagination if all files are already loaded.
• In RelatedDatasetsComponent you are subscribing to vm$ and datasetsPerPage$ manually just to push into dataSource and update pagination, while also using vm$ with the async pipe in the template; consider binding the table directly to vm.relatedDatasets/vm.relatedDatasetsCount via the async pipe and deriving pagination properties in the template to avoid extra BehaviorSubjects and explicit subscriptions.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In DatafilesComponent, `paginationMode` is set to `'server-side'` but there is no `(paginationChange)` handler wired up, so changing pages in the dynamic-mat-table will not update `dataSource`; either implement a paginationChange handler that slices/requests data and calls `dataSource.next(...)` or switch to `'client-side'` pagination if all files are already loaded.
- In RelatedDatasetsComponent you are subscribing to `vm$` and `datasetsPerPage$` manually just to push into `dataSource` and update `pagination`, while also using `vm$` with the async pipe in the template; consider binding the table directly to `vm.relatedDatasets`/`vm.relatedDatasetsCount` via the async pipe and deriving pagination properties in the template to avoid extra BehaviorSubjects and explicit subscriptions.

## Individual Comments

### Comment 1
<location path="src/app/datasets/datafiles/datafiles.component.ts" line_range="126-128" />
<code_context>
+    DataFiles_File[]
+  >([]);
+
+  paginationMode: TablePaginationMode = "server-side";
+
+  pagination: TablePagination = {
+    pageSizeOptions: [10, 25, 50],
+    pageIndex: 0,
</code_context>
<issue_to_address>
**issue (bug_risk):** Table pagination is configured as server-side but the component never handles pagination changes or slices the data.

With `paginationMode: "server-side"` and `pagination.length = files.length`, the table is configured for server-side pagination, but the component never handles `paginationChange` and always passes the full `files` array to `dataSource`. Either implement `paginationChange` handling (and update `dataSource` / fetch new pages) to match server-side behavior, or change `paginationMode` to the client-side option if pagination should be done entirely in memory.
</issue_to_address>

### Comment 2
<location path="src/app/datasets/related-datasets/related-datasets.component.ts" line_range="83" />
<code_context>
+    },
+  };
+
+  dataSource: BehaviorSubject<object[]> = new BehaviorSubject<object[]>([]);
+
+  paginationMode: TablePaginationMode = "server-side";
</code_context>
<issue_to_address>
**suggestion:** Using `object[]` for the dataSource loses type safety for related datasets.

Given the table and view model both use `OutputDatasetObsoleteDto`/`DatasetClass`-shaped data, keeping `dataSource` as `BehaviorSubject<object[]>` weakens compile-time safety (e.g. when accessing `sender.row` in `onRowEvent` or formatting rows). Please narrow this to a concrete type such as `BehaviorSubject<DatasetClass[]>` (or the relevant DTO) so the table interactions remain type-safe.

Suggested implementation:

```typescript
  dataSource: BehaviorSubject<DatasetClass[]> = new BehaviorSubject<DatasetClass[]>([]);

```

1. Ensure `DatasetClass` (or the specific DTO you actually use for related datasets, e.g. `OutputDatasetObsoleteDto`) is imported at the top of `related-datasets.component.ts`:
   - For example:
     - `import { DatasetClass } from '...';`
   - Or replace `DatasetClass` with `OutputDatasetObsoleteDto` if that is the correct model:
     - `dataSource: BehaviorSubject<OutputDatasetObsoleteDto[]> = new BehaviorSubject<OutputDatasetObsoleteDto[]>([]);`
2. If the table template or other code assumes a different concrete type (e.g. `OutputDatasetObsoleteDto`), align the generic type argument here with whatever type is used in row event handlers (`sender.row`) and formatting logic, so the compiler enforces consistency across the component.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[Junjiequan]

lgtm, please merge it after dateformat update

[Junjiequan]

some copilot reviews need to be addressed

[nitrosx]

Code Review: Branch replace-old-tables-with-dynamic-mat-table

This document summarizes the changes introduced by the branch replace-old-tables-with-dynamic-mat-table compared to master, assesses their necessity, and identifies areas for improvement.

---

Assessment

This branch implements a major architectural improvement by replacing the legacy app-table component with the modern dynamic-mat-table component across dataset-related tables. The change brings server-side pagination, better separation of concerns, and alignment with the application's evolving table infrastructure.

Key Improvements Achieved:

1. Modern Table Infrastructure: Migration from custom app-table to the standardized dynamic-mat-table component, which provides:

   • Built-in server-side pagination support
   • Consistent event handling via IRowEvent and RowEventType enum
   • Better TypeScript typing with TableField, TablePagination, ITableSetting interfaces
   • Native Material Table integration

2. Performance Optimization: Server-side pagination for datafiles means the browser no longer receives all files at once, significantly improving performance for datasets with many files.

3. State Management Enhancement:

   • Added totalCount tracking for origdatablocks at the store level
   • Created proper view models (selectDatafilesPageViewModel) combining data, count, and related entities
   • Proper separation between paginated data and total count

4. Code Quality:

   • Removed ~100 lines of manual selection management code in datafiles component
   • Simplified event handling with unified onRowEvent() method
   • Better type safety throughout

5. UI Consistency: Removed redundant icon properties from column definitions across multiple components (files-dashboard, instruments-dashboard, related-proposals, sample-dashboard), cleaning up the codebase.

Files Changed (20 files, +471 -377 lines):

Core Table Migration:

• src/app/datasets/datafiles/datafiles.component.ts/html/spec.ts
• src/app/datasets/related-datasets/related-datasets.component.ts/html/spec.ts

State Management Updates:

• src/app/state-management/actions/datasets.actions.ts
• src/app/state-management/effects/datasets.effects.ts/spec.ts
• src/app/state-management/reducers/datasets.reducer.ts
• src/app/state-management/selectors/datasets.selectors.ts
• src/app/state-management/state/datasets.store.ts

Cleanup in Other Components:

• src/app/files/files-dashboard/files-dashboard.component.ts
• src/app/instruments/instruments-dashboard/instruments-dashboard.component.ts
• src/app/proposals/related-proposals/related-proposals.component.ts
• src/app/samples/sample-dashboard/sample-dashboard.component.ts/html/scss
• src/app/shared/modules/configurable-actions/configurable-actions.component.scss

---

Improvement Needed

Critical Issues:

1. Inconsistent Pagination Implementation:

   • related-datasets.component.ts still uses client-side filtering logic in formatTableData() but claims server-side pagination mode
   • The pagination configuration lacks proper initialization synchronization with actual data loading
   • vm$ observable transforms data with formatTableData() but the new table expects raw data with proper field mappings

2. Total Count Logic Problem:

   • The effect in datasets.effects.ts (fetchOrigDatablocksOfDataset$) uses forkJoin to fetch both paginated files AND all files just to get a count
   • This defeats the purpose of server-side pagination by downloading the entire dataset anyway
   • Creates unnecessary network overhead for large datasets
   • The API should return total count directly, or a separate count endpoint should be used

3. State Management Gaps:

   • selectDatafilesPageViewModel is created but datafiles.component.ts still maintains local files array and manually updates it
   • Redundant data flow: store → component → local state → table
   • Component manually calculates selectedFileSize instead of using a selector

4. **Missing Features (Feature Parity):

   • Sorting functionality is not implemented in the new tables (old table had sort: true/false)
   • Column customization (show/hide) appears to be partially implemented but not fully tested
   • The TableConfigService and DatasetsListService dependencies in related-datasets suggest planned features that aren't fully utilized

5. Testing Gaps:

   • Unit tests were updated but integration tests for the new pagination flow are minimal
   • No tests for error states or edge cases (empty results, network failures)
   • The onRowEvent tests in datafiles.spec.ts don't verify the actual selection state changes in the store

6. Code Cleanup Needed:

   • Dead code remains: currentPage$, datasetsPerPage$ in related-datasets are declared but the component uses vm$ with different structure
   • The formatTableData() method in related-datasets transforms data but may conflict with the new table's expectations
   • Unused imports in some test files

---

Verdict

Merge with Conditions: This branch represents a necessary and valuable modernization effort that should be merged, but requires the following conditions:

Must Fix Before Merge:

1. Remove the redundant "fetch all" query in datasets.effects.ts

   • The OrigdatablocksV4Service should be modified to return total count with paginated results, OR
   • Use a separate count endpoint to avoid downloading all files

2. Ensure related-datasets properly implements server-side pagination

   • Remove client-side data transformation that conflicts with server-side mode
   • Verify pagination actually fetches new data from the server

3. Clean up dead code and unused observables

   • Remove currentPage$, datasetsPerPage$ if not used
   • Ensure all declared properties are actually utilized

Recommended Improvements (Can be addressed in follow-up PRs):

1. Add sorting support to maintain feature parity with old tables
2. Complete the column customization feature (currently half-implemented)
3. Add comprehensive integration tests for pagination and selection workflows
4. Consider moving selection state management to the store instead of component-level

Acceptable Trade-offs:

• The temporary performance overhead of fetching counts is acceptable if addressed in follow-up PR
• Some UI polish (like the min-height fix in datafiles) can be refined post-merge
• The architectural direction is correct and the changes are largely well-implemented

Final Decision: APPROVE WITH CONDITIONS

The architectural direction is correct and the changes are largely well-implemented. The migration from the old table system to the dynamic mat-table is necessary for long-term maintainability. With the critical fixes mentioned above, this branch should be merged.

Generated with Mistral AI

[nitrosx]

Code Review: Branch replace-old-tables-with-dynamic-mat-table

This document provides a detailed analysis of the code changes introduced by the branch replace-old-tables-with-dynamic-mat-table compared to master, including logic correctness verification, edge case handling, code functionality summary, change rationale assessment, and unreachable code identification.

---

Executive Summary

The branch migrates dataset-related tables from the legacy app-table component to the modern dynamic-mat-table component. This is a necessary architectural modernization that improves maintainability, type safety, and performance through server-side pagination.

Total Changes: 20 files modified, +471 lines, -377 lines

---

1. What the Changed Code Does

Core Functionality

The changed code implements data display and user interaction for two main datasets:

A. Datafiles Component (datafiles.component.ts)

Purpose: Displays files within a dataset with selection and download capabilities

Key Operations:

• Fetches origdatablocks (file metadata) for a dataset from the API
• Displays files in a paginated table with path, size, and time columns
• Allows users to select individual files or all files (when download is enabled)
• Tracks selected file size for download warnings
• Supports server-side pagination (new in this branch)

Data Flow:

1. User navigates to dataset detail view
2. Component dispatches fetchOrigDatablocksAction with dataset PID
3. Effect calls OrigdatablocksV4Service.origDatablocksV4ControllerFindAllFilesV4() with pagination params
4. API returns paginated origdatablocks + total count
5. Reducer stores origdatablocks and count in state
6. Selector selectDatafilesPageViewModel combines datablocks, totalCount, and dataset
7. Component subscribes to view model, processes data, and populates the table
8. User interactions (pagination, selection) trigger appropriate actions

B. Related Datasets Component (related-datasets.component.ts)

Purpose: Displays datasets related to the current dataset

Key Operations:

• Fetches related datasets for a dataset
• Displays them in a paginated table
• Allows navigation to related datasets via row click
• Supports server-side pagination (new in this branch)

Data Flow:

1. Component dispatches fetchRelatedDatasetsAction
2. Effect fetches related datasets from API
3. Selector selectRelatedDatasetsPageViewModel provides data, count, and filters
4. Component formats data and populates the table
5. User pagination triggers changeRelatedDatasetsPageAction and refetch

Supporting State Management Changes

Actions: Added totalCount parameter to fetchOrigDatablocksCompleteAction

Effects: Modified fetchOrigDatablocksOfDataset$ to:

• Accept pagination filters (skip, limit)
• Use new OrigdatablocksV4Service endpoint
• Fetch both paginated data AND all data (for count)
• Return both origdatablocks and totalCount

Reducers: Store origDatablocksCount in DatasetState

Selectors: Added:

• selectCurrentOrigDatablocksCount: Returns total count of origdatablocks
• selectDatafilesPageViewModel: Combines datablocks, totalCount, and dataset

UI Component Cleanup

Removed unused icon properties from table column definitions in:

• files-dashboard.component.ts
• instruments-dashboard.component.ts
• related-proposals.component.ts
• sample-dashboard.component.ts

Minor CSS fixes in:

• datafiles.component.scss (min-height)
• sample-dashboard.component.scss (responsive layout)
• configurable-actions.component.scss (flexbox layout)

---

2. Logic Correctness Verification

Correct Implementations

1. Datafiles Selection Logic ✅

   • The new onRowEvent() handler correctly processes RowSelectionChange events
   • Sets row.selected = sender.checked for individual row selection
   • Handles MasterSelectionChange by iterating through all files and applying selection from selectionModel
   • Recalculates selectedFileSize by summing sizes of selected files
   • Logic is correct and equivalent to the old implementation

2. Pagination Logic ✅

   • onPaginationChange() correctly dispatches action with skip/limit calculated from pageIndex/pageSize
   • Effect properly constructs pagination query with skip and limit
   • Pagination state is properly updated in component

3. Data Transformation ✅

   • getAllFiles(): Correctly maps all files to their paths
   • getSelectedFiles(): Correctly filters and maps selected files
   • Column definitions properly use customRender for formatted display

4. State Management ✅

   • Reducer correctly stores totalCount
   • Selectors properly combine data and metadata
   • View models provide all necessary data to components

Logic Issues Found

1. CRITICAL: Inefficient Count Query in datasets.effects.ts ❌

   return forkJoin({
     paginated: this.origdatablocksService.origDatablocksV4ControllerFindAllFilesV4(
       JSON.stringify(paginatedQuery),
     ),
     all: this.origdatablocksService.origDatablocksV4ControllerFindAllFilesV4(
       JSON.stringify(allFilesQuery),  // Fetches ALL files just for count!
     ),
   })

   Problem: This fetches the entire dataset to get a count, defeating server-side pagination.

   Impact: For datasets with thousands of files, this causes massive unnecessary network transfer.

   Severity: HIGH - This is a performance regression, not an improvement.

2. INCONSISTENT: Related Datasets Server-Side Pagination ⚠️

   • Component declares paginationMode: TablePaginationMode = "server-side"
   • But formatTableData() performs client-side transformation of all data
   • The vm$ selector applies formatTableData to the entire dataset
   • This suggests the pagination might not be truly server-side

3. EDGE CASE: dataFileList Type Check ⚠️
   Old Code:

   block.dataFileList.map((file: DataFiles_File) => { ... })

   New Code:

   if (block.dataFileList && !Array.isArray(block.dataFileList)) {
     const file = block.dataFileList as DataFiles_File;
     // ...
   }

   Assessment: The new check handles the case where dataFileList might be a single object rather than an array. This is correct defensive programming.

4. POTENTIAL BUG: Missing null check in ngOnInit ⚠️
   In datafiles.component.ts:

   if (this.actionItems.datasets.length > 0) {
     this.actionItems.datasets[0].files = files;
   }

   Assessment: This is actually a fix - the old code didn't have this check and would fail if datasets array was empty.

---

3. Edge Cases Handling

Well-Handled Edge Cases ✅

1. Empty Data:

   • dataSource initialized as empty BehaviorSubject
   • Table shows empty message: "No datafiles available" or "No related datasets available"
   • Template condition: [ngIf]="files.length > 0 && maxFileSize" prevents rendering with no files

2. Null/Undefined Values:

   • totalCount defaults to 0: state.origDatablocksCount ?? 0
   • File size defaults to 0: file.size || 0
   • Proper null checks in data transformation

3. Selection Edge Cases:

   • selectedFileSize starts at 0
   • Reduce with initial value 0: reduce((sum, file) => sum + file.size, 0)
   • Handles empty array gracefully

4. Pagination Edge Cases:

   • Default values: filters?.skip || 0, filters?.limit || 5
   • Pagination config has sensible defaults

5. Type Safety:

   • Type assertions for API responses: paginated as any as OrigDatablock[]
   • Proper typing for table fields, events, and settings

Missing Edge Case Handling ⚠️

1. API Error Handling:

   • Effect has catchError(() => of(fetchOrigDatablocksFailedAction()))
   • But no error display in components
   • No user feedback when fetch fails

2. Pagination Boundary Conditions:

   • No handling for invalid pageIndex (negative, beyond total)
   • No handling for pageSize = 0
   • No handling for skip exceeding total count

3. Selection with Pagination:

   • CRITICAL ISSUE: When paginating, selections are maintained across pages in the files array
   • But the table only displays current page of files
   • If user selects files on page 1, then navigates to page 2, the selection state for page 1 files is preserved but not visible
   • This could lead to confusion: selectedFileSize shows count from all pages, but user only sees current page

4. Concurrent Requests:

   • No handling for multiple simultaneous pagination requests
   • Race conditions possible if user clicks pagination rapidly

5. Empty Selection Model:

   • No null check for sender.selectionModel in MasterSelectionChange handler
   • Could crash if selectionModel is undefined

6. File Size Calculation:

   • No handling for NaN file sizes
   • No handling for negative file sizes
   • No overflow protection for very large total sizes

---

4. Do the Changes Make Sense?

Yes, the Changes Make Sense ✅

Rationale for Migration:

1. Architectural Consistency: The application is standardizing on dynamic-mat-table as the table component. Migrating these two components aligns with this direction.

2. Server-Side Pagination: The old app-table did client-side pagination (slicing the full array), which is inefficient for large datasets. Server-side pagination is a genuine improvement.

3. Reduced Code Complexity: The old code had ~100 lines of manual selection state management (areAllSelected, isNoneSelected, updateSelectionStatus, updateSelectedInFiles). The new code delegates this to the table component, reducing component complexity.

4. Better Type Safety: The new code uses proper TypeScript interfaces (TableField, IRowEvent, etc.) instead of custom types from the old table component.

5. Consistent Event Model: Using IRowEvent with RowEventType enum provides a standardized way to handle table events across the application.

6. Modern Angular Patterns: Using BehaviorSubject for dataSource, proper Observable patterns, and better separation of concerns.

Questionable Decisions ⚠️

1. Fetching All Data for Count: As mentioned, this defeats the purpose of server-side pagination. This appears to be a workaround for API limitations.

2. Mixing Server and Client-Side Concepts: In related-datasets, the component uses server-side pagination mode but still transforms all data with formatTableData(). This is inconsistent.

3. Redundant View Model: The component creates vm$ that transforms data, but also has relatedDatasets$ that doesn't transform. This creates confusion about which data source to use.

4. Dead Code: currentPage$ and datasetsPerPage$ are declared but only used in the combineLatest subscription. They could be removed.

---

5. Unreachable Code Analysis

Confirmed Unreachable Code ❌

1. related-datasets.component.ts - currentPage$ and datasetsPerPage$

   currentPage$ = this.store.select(selectRelatedDatasetsCurrentPage);
   datasetsPerPage$ = this.store.select(selectRelatedDatasetsPerPage);

   Status: DEAD CODE

   These observables are only used in the combineLatest subscription in ngOnInit. They are never accessed directly. However, they are technically used, so not completely unreachable. But they serve no purpose since the component uses vm$ which already contains the data.

2. related-datasets.component.ts - relatedDatasets$

   relatedDatasets$ = this.store.select(selectRelatedDatasetsPageViewModel);

   Status: DEAD CODE

   This is declared but never used. The component uses vm$ instead which applies formatTableData transformation.

3. datafiles.component.ts - Removed Methods (No Longer Reachable)
   These were in master but removed in this branch:

   • onPageChange() - replaced by onPaginationChange()
   • onSelectOne() - replaced by onRowEvent()
   • onSelectAll() - replaced by onRowEvent()
   • getAreAllSelected() - no longer needed
   • getIsNoneSelected() - no longer needed
   • updateSelectionStatus() - no longer needed
   • updateSelectedInFiles() - no longer needed
   • tableData property - replaced by files + pagination

   Status: Correctly removed as part of refactoring

Potentially Unreachable Code ⚠️

1. datafiles.component.ts - dataset$ subscription
   The old code had:

   this.subscriptions.push(
     this.dataset$.subscribe((dataset) => {
       if (dataset) {
         this.actionItems.datasets = <ActionItemDataset[]>[dataset];
       }
     }),
   );

   The new code has this logic inside the vm$ subscription. The dataset$ observable is still defined but not directly subscribed to. However, vm$ includes dataset, so it's still accessible.

2. datafiles.component.ts - datablocks$ subscription
   Similar to above - the logic was consolidated into vm$ subscription.

Code That Appears Unused But Is Actually Used ✅

1. datafiles.component.ts - count property
   Still used in the template for display purposes (though not visible in the provided diff snippets)

2. datafiles.component.ts - tooLargeFile property
   Set in ngOnInit and likely used in template for warnings

---

6. Recommendations

Critical (Must Fix Before Merge)

1. Fix the Count Query Performance Issue

   • Modify OrigdatablocksV4Service to return total count with paginated results
   • OR implement a separate count endpoint
   • OR accept that this is a temporary workaround with a TODO comment explaining the limitation

2. Fix Selection State Across Pagination

   • Clarify the intended behavior: should selections persist across pages?
   • If yes: ensure selectedFileSize correctly reflects selections across all pages
   • If no: clear selections when page changes, or limit selections to current page only

3. Add Null Checks

   • Add null check for sender.selectionModel in onRowEvent MasterSelectionChange handler
   • Add validation for pagination parameters (pageIndex >= 0, pageSize > 0)

High Priority

4. Clean Up Dead Code

   • Remove relatedDatasets$ from related-datasets.component.ts
   • Remove currentPage$ and datasetsPerPage$ if not used elsewhere
   • Remove unused imports

5. Improve Error Handling

   • Add user feedback for API errors
   • Handle pagination boundary conditions

6. Fix Related Datasets Pagination

   • Decide: server-side or client-side?
   • If server-side: remove formatTableData transformation from vm$
   • If client-side: change paginationMode to "client-side"

Medium Priority

7. Add Missing Tests

   • Test pagination with edge cases (empty, single page, multiple pages)
   • Test selection across pagination
   • Test error states
   • Test with null/undefined data

8. Restore Feature Parity

   • Add sorting support
   • Verify all old features work (selection, download, warnings)

9. Type Safety Improvements

   • Remove as any type assertions where possible
   • Add proper types for API responses

---

Final Verdict

Overall Assessment: GOOD, BUT NEEDS WORK

Criteria	Rating	Notes
Logic Correctness	⚠️ Mostly Correct	Core logic works, but has critical performance issue with count query
Edge Case Handling	⚠️ Partial	Handles basic cases well, but missing several important edge cases
Code Makes Sense	✅ Yes	Architectural direction is sound and necessary
No Unreachable Code	⚠️ Mostly Clean	Some dead code remains, but major unreachable code was correctly removed
Production Ready	❌ No, not yet	Performance issue and edge cases need addressing

Decision: DO NOT MERGE YET

While the architectural direction is correct and the code is generally well-structured, there are critical issues that must be resolved:

1. The performance regression from fetching all files for count makes this worse than the old implementation for large datasets
2. The selection state behavior across pagination is unclear and potentially confusing
3. Missing edge case handling could cause runtime errors

Path Forward:

1. Fix the count query - This is the blocker. Options:

   • API change to return count with paginated results (preferred)
   • Use a separate count endpoint
   • Accept the workaround with clear documentation and TODO

2. Clarify and fix pagination selection behavior - Define the expected UX and implement consistently

3. Add null checks and validation - Prevent runtime errors

4. Clean up dead code - Remove unused observables and imports

5. Add comprehensive tests - Ensure robustness

Once these issues are addressed, the branch will be production-ready and represents a significant improvement over the current implementation.

---

Report generated by analyzing git diff between master and replace-old-tables-with-dynamic-mat-table branch

Generated with Mistral AI

[nitrosx]

Security Review: Branch replace-old-tables-with-dynamic-mat-table

This document provides a security-focused analysis of the code changes introduced by the branch replace-old-tables-with-dynamic-mat-table compared to master.

---

1. Injection Vulnerabilities

Assessment: No NEW injection vulnerabilities introduced ✅

Type	Status	Location	Risk	Notes
XSS via [innerHTML]	⚠️ Pre-existing	datafiles.component.html:5	Medium	[innerHTML]="hasFileAboveMaxSizeWarning()" - Template string from config is rendered as HTML. NOT introduced by this branch
Template Injection	✅ Safe	hasFileAboveMaxSizeWarning()	Low	String replacement of placeholders (<maxDirectDownloadSize>, <sftpHost>, <sourceFolder>) with sanitized values. Uses simple .replace() with controlled values
API Parameter Injection	✅ Safe	datasets.effects.ts	Low	Query parameters (pid, skip, limit) are properly typed and come from route params or controlled UI inputs
Code Injection	✅ Safe	All components	None	No eval(), new Function(), or dynamic code execution

Details:

The only potential injection point is the [innerHTML] binding in datafiles.component.html line 5:

<span [innerHTML]="hasFileAboveMaxSizeWarning()"></span>

This method constructs HTML by replacing placeholders in a template string:

warning = warning.replace(
  "<" + key + ">",
  `<strong>${valueMapping[key]}</strong>`,
);

Risk: If an administrator sets a malicious maxFileSizeWarning in the config, it could inject HTML/JS. However:

• This vulnerability existed in master before this branch
• It requires admin access to the config file
• The replacement values (sftpHost, sourceFolder, maxDirectDownloadSize) are controlled by the application and are sanitized

Verdict: No new injection vulnerabilities introduced by this branch.

---

2. Sensitive User Data Exposure

Assessment: No NEW sensitive data exposure ✅

Data Type	Status	Location	Risk	Notes
JWT Tokens	✅ No change	datafiles.component.ts	None	Same token handling as master (auth_token, jwt) for download forms
File Metadata	✅ Same as master	datafiles.component.ts	Low	Exposes: path, size, time, uid, gid, perm, hash - same fields as before
Dataset Metadata	✅ Same as master	related-datasets.component.ts	Low	Exposes: pid, name, sourceFolder, size, type, creationTime, owner - same fields as before
File Paths	✅ No change	Both components	Low	sourceFolder paths displayed, same as master
Owner Information	✅ No change	related-datasets.component.ts	Low	owner field displayed, same as master

Details:

The changes only affect how data is displayed (which table component is used), not what data is displayed. The same fields that were exposed in the old app-table are exposed in the new dynamic-mat-table.

• Datafiles: Same DataFiles_File interface fields (path, size, time, chk, uid, gid, perm, hash, selected)
• Related Datasets: Same OutputDatasetObsoleteDto fields (pid, datasetName, sourceFolder, size, type, creationTime, owner)

No new sensitive data is exposed through:

• Table columns
• API responses
• Error messages
• Logs

Verdict: No new sensitive user data exposure introduced by this branch.

---

3. Insecure API Usage

Assessment: No insecure API usage ✅

API	Status	Location	Notes
OrigdatablocksV4Service.origDatablocksV4ControllerFindAllFilesV4()	✅ Secure	datasets.effects.ts	Official SciCat SDK v4 endpoint, replaces v3 endpoint
DatasetsService.datasetsControllerFindAllOrigDatablocksV3()	❌ Removed	Was in master	Replaced by v4 service
DatasetsService.datasetsControllerFindAllDatablocksV3()	✅ Same	datasets.effects.ts	Still used for datablocks, unchanged

Details:

API Changes:

1. Old: datasetsService.datasetsControllerFindAllOrigDatablocksV3(pid) - v3 endpoint
2. New: origdatablocksService.origDatablocksV4ControllerFindAllFilesV4(JSON.stringify(query)) - v4 endpoint

Security Analysis:

• Both are official SciCat SDK methods
• Both require the same authentication
• The new v4 endpoint accepts a JSON query string, but the query is constructed from controlled parameters (pid from route, skip/limit from pagination)
• No hardcoded credentials or API keys
• No use of insecure protocols (no http:, only https: via SDK)
• No CORS bypass attempts
• No credential transmission in URLs

Query Construction:

const paginatedQuery = {
  where: { datasetId: pid },      // pid from route params
  limits: {
    skip: filters?.skip || 0,       // from pagination
    limit: filters?.limit || 5,     // from pagination
  },
};

The query is properly structured and uses controlled values. The JSON.stringify() is safe as it only serializes objects.

Verdict: No insecure API usage. The API migration from v3 to v4 is secure and follows the same patterns as the existing code.

---

4. Authentication Bypass

Assessment: No authentication bypass vulnerabilities ✅

Attack Vector	Status	Analysis
Token Manipulation	❌ Not possible	Tokens come from authService.getToken() and usersService.usersControllerGetUserJWTV3() - unchanged from master
PID Manipulation	❌ Not possible	Dataset PID comes from route parameters; users can only access datasets they have permission for (enforced server-side)
Filter Manipulation	❌ Not possible	Pagination filters (skip, limit) only affect client-side display; server enforces authorization
Route-based Access	✅ Same as master	No changes to route guards or authentication flows
Service Injection	✅ Safe	New OrigdatablocksV4Service is from official SDK, injected via DI like all other services

Details:

Authentication Flow:

• No changes to how users authenticate
• No changes to token storage or transmission
• No new ways to obtain or manipulate tokens

Authorization:

• The branch does not add or remove any authorization checks
• Server-side authorization (via SciCat API) remains the same
• Client-side, the same data access patterns are used

Data Access:

• Dataset PID comes from the route: this.datasetPid = dataset.pid
• The PID is used to fetch data: fetchOrigDatablocksAction({ pid: this.datasetPid, ... })
• Users can only navigate to datasets they have access to (enforced by route guards, not shown in this diff)
• No way to access other users' datasets through manipulation

JWT/Token Handling:
The token handling for file downloads is identical to master:

// Same in both master and this branch:
if (this.appConfig.multipleDownloadUseAuthToken) {
  this.auth_token = `Bearer ${this.authService.getToken().id}`;
  this[`${form}Element`].nativeElement.auth_token.value = this.auth_token;
}
if (!this.jwt) {
  this.subscriptions.push(
    this.usersService.usersControllerGetUserJWTV3().subscribe((jwt) => {
      this.jwt = jwt;
      this[`${form}Element`].nativeElement.jwt.value = jwt.jwt;
    }),
  );
}

Verdict: No authentication bypass vulnerabilities introduced by this branch.

---

Summary

Security Concern	Introduced by Branch?	Severity	Status
XSS via [innerHTML]	❌ No (pre-existing)	Medium	Not addressed
Template injection	❌ No	Low	Safe
Sensitive data exposure	❌ No	None	Safe
Insecure API usage	❌ No	None	Safe
Authentication bypass	❌ No	None	Safe

Overall Security Assessment: ✅ SECURE

Final Verdict: The branch introduces NO NEW security vulnerabilities.

The replace-old-tables-with-dynamic-mat-table branch maintains the same security posture as master. It is a pure refactoring that changes how tables are rendered (migrating from app-table to dynamic-mat-table) without changing:

• What data is displayed
• How data is fetched
• How authentication works
• What API endpoints are called

Pre-existing Security Issue (Not Introduced by This Branch):

1. XSS vulnerability via [innerHTML]="hasFileAboveMaxSizeWarning()" in datafiles.component.html - Requires malicious admin config

Recommendation: This branch is secure to merge from a security perspective. The pre-existing XSS issue should be addressed separately (e.g., by sanitizing the HTML or using [textContent] instead of [innerHTML]).

---

Security review conducted by analyzing git diff between master and replace-old-tables-with-dynamic-mat-table branch

Generated with Mistral AI

[abdimo101]

UPDATE:

Loading spinner has been added to datafiles & related datasets table paginators when the count request is in a pending state. Similar to how it was done in #2399

[Junjiequan]

Holding this PR until new SDK release

[abdellah257]

We can avoid fetching the count at each pagination change, only needed at first time.

[abdellah257]

fetching here without specifying the pagination parameters would result in using the default {skip: 0, limit:5} which is inconsistent with what we have in datafiles component {limit:10}. I suggest specifying these here.
Also I suggest we go for 25 for consistency through all tables