Repository navigation
Conversation
The list endpoint returns a `total` field: the SQL count of the links matching the current filters, minus the links the listing drops because neither their staging nor their final table exists. Orphans are counted across all pages, so the total is exact and identical on every page. The dashboard shows it next to "Filtrer vos données", taken from the first page, and decremented on delete. Also fix the frontend gateway path in the README: it is /dataset/, not /datafeeder/.
|
|
||
| # Total matching the filters regardless of pagination: SQL count minus the links the | ||
| # listing drops for lack of staging/final table (checked across all pages). | ||
| matching = session.execute( # type: ignore[reportDeprecated] |
There was a problem hiding this comment.
total is computed on every request, including "load more" pages with offset > 0, but the frontend only reads it from the first page. Each extra page re-runs the COUNT query and _count_orphans (a full scan) for nothing. Could we compute it only when offset == 0 and type it int | None?
|
|
||
| def _count_orphans(query: Any, session: Session, data_session: Session) -> int: | ||
| """Count links matching `query` that the listing drops for lack of staging/final table.""" | ||
| candidates = session.execute( # type: ignore[reportDeprecated] |
There was a problem hiding this comment.
_count_orphans runs the full query with no limit. That loads every matching IntegrityLink as an ORM entity and evaluates the access_level / public_access CASE + EXISTS for each row. It then sends a single information_schema IN (...) with every table name. On large instances this grows linearly with the number of links, whereas the endpoint used to be bounded by BATCH_SIZE. Selecting only staging_table_name, final_table_name and integrity_organization would avoid most of the cost.
| matching = session.execute( # type: ignore[reportDeprecated] | ||
| sa_select(func.count()).select_from(query.subquery()) | ||
| ).scalar_one() | ||
| total = matching - _count_orphans(query, session, data_session) |
There was a problem hiding this comment.
The total comes from two separate queries (COUNT, then the orphan candidates) that don't share a snapshot. If a link is created between them and its staging table doesn't exist yet, it is counted as an orphan without being in matching, so the total is too low. In the worst case it is negative, and IntegrityLinkListResponse.total has no ge=0 to catch that. At minimum, clamp it with max(0, ...).
Also, on the first page, _count_orphans re-checks table existence in information_schema for rows the pagination loop has just checked. We could reuse the per-chunk results instead of querying twice.
| ) | ||
|
|
||
|
|
||
| def _count_orphans(query: Any, session: Session, data_session: Session) -> int: |
There was a problem hiding this comment.
More of a design question: the root issue is that links are filtered after the query, based on table existence in another database. Adding a second full pass to count them works around this rather than fixing it. Persisting table existence on the link, or cleaning up orphaned links, would turn both the listing and the total into plain SQL with a WHERE filter. Probably out of scope here, but worth a follow-up issue.
The dashboard only lists a page of datasets at a time, so users could not tell how many datasets match their current search and filters. It now shows the total next to "Filtrer vos données", e.g. "Filtrer vos données - 1005 résultats".
totalfield: the SQL count of links matching the filters, minus the links the listing drops because neither their staging nor their final table exists. It is exact and identical on every page. This costs one extra query per list request, limited to non-empty links.integrityLinks.filters.resultCountkey is added to all translation files (fr and en filled in)./dataset/, not/datafeeder/.Checked in the browser against a dev database: the total matched the number of datasets reachable through "Charger plus" (386), and with a search term (32 results, 32 rows).