OpenAPI - paging iteration updade request - page limit & cursor

[rduyme]

Bug report Alexios

In the specs, it mentions the field "last page". I don't think this is a good idea to show to the user the last page.

Let me explain why:

In our Prod env, the total_product items is: 217_257_614
default page size is 10, but even if it's 50:
50 items/page → ~4_345_152 pages

If set page to this big number, this will cost in performance. Our search engine (Solr) will need to go all the way down in the index until it reaches this result set. On our Beta env, Solr went out of memory and crashed. So technically, this is not a good idea. For the user's point of view, this is also not very practical - it doesn't make sense to request such a big page. Just adjust the filters to get more accurate result.

For OpenAIRE APIs, we apply a limit for pagination with offset - up to 10000 results
If the user wants to get a bigger dataset, then cursor pagination should be used.
https://graph.openaire.eu/docs/next/apis/graph-api/searching-entities/sorting-and-paging#paging

So, I will remove the "last page" from the response (it is optional, right?) and will apply a validation on page max number.
If the user exceeds this limit, an error like this one will be thrown:

{
"message": "Page * pageSize must be less than or equal to 10000",
"error": "Bad Request",
"code": 400,
"timestamp": "2026-04-21T21:38:22.487+00:00",
"path": "/graph/v2/researchProducts?page=10000&pageSize=10&sortBy=relevance%20DESC"
}

I suggest to remove it from SKG-IF spec, also. And describe a cursor pagination, for the case when a big dataset is required by the user.

Regards,
Alexios

[rduyme]

Hi Alexios,
I agree,

• we can make the last page optional (it is mandatory at the moment)
• we can leave option to the server to return 400 as you suggest.

Regarding iteration, here is how it is defined at the moment.
https://elements-demo.stoplight.io/?spec=https://w3id.org/skg-if/api/skg-if-openapi.yaml

Product/Get list of product
query params:

search output

What did you have in mind for cursor pagination ?

[aisimeon349e6]

hi Renaud,

if the user wants to get results > 10000, then in the response a message like this should be returned:
"offset pagination max limit is 10000 results, to get more results, use cursor pagination"

and in the request, field for cursor should be exposed.
starting value for cursor is *
in the response's header cursor next value will be returned.
when cursor has value, offset pagination fields should be ignored (if they have value).

cursor pagination is already implemented for our APIs, so I will apply the same logic for SKG-IF API.

[rduyme]

@aisimeon349e6

Proposition 1 :

• part_of.last_page optional
• page query param description. ( page type number )
  to _page number . Note: As a client application you should build your iterating logic using the meta.next_page.local_identifier and meta.prev_page.local_identifier URLs
• add cursor query param
  cursor iteration token. Note: As a client application you should build your iterating logic using the meta.next_page.local_identifier and meta.prev_page.local_identifier URLs
  Do we want server to implement both. or leave simply cursor implementation as optional.

Proposition 2 :

• part_of.last_page optional
• page query param description. ( page type string )
  To
  Search page reference (page number or iteration cursor).
  Note: As a client application you should build your iterating logic using the meta.next_page.local_identifier and meta.prev_page.local_identifier URLs. Always leave empty for your first query

This would mean that for Proposition 2.

• for OpenAIRE you could just expose the cursor iteration logic.
• other implementers could use paging logic as well with the same params.

But... Proposition 2 is maybe a bit confusing regarding query attributes, and less standard, We see many API with both classic paging and cursor patterns exposed.

OpenAIRE specs refs : https://graph.openaire.eu/docs/next/apis/graph-api/searching-entities/sorting-and-paging/#paging

Note current page query param type is string and description is
Search page reference (page number or search iteration token).
Note: As a client application you should build your iterating logic using the meta.next_page.local_identifier and meta.prev_page.local_identifier URLs

[markusjt]

Adding optional cursor implementation and changing last_page to be optional sounds good to me. I think proposition 1 where cursor has it's own param key is better.
I'd leave the offset limit up to the implementer so it's not strictly required to implement cursor even if there are over 10000 results.

[menzowindhouwer]

I prefer proposition 1. Cursor should be optional.

But how to find out if cursor is supported by an endpoint:

1. trial & error
2. or we add cursor via an overlay, which means you can interrogate the openapi spec of the endpoint to find out if cursors are supported

I don't like the textual error message for the offset. The client than needs developer intervention to interpret the message, why not return a JSON object with the message but also the allowed maximum offset:

{
  err: "offset pagination max limit is 10000 results, to get more results, use cursor pagination",
  max: 10000
}

And what does this actually mean? can I switch to a cursor that skips the 10000 results or I've to start again getting all the results but then with a cursor and have to disgard the 10000 I already got? In that case we better be able to interrogate the openapi spec of the endpoint before we begin collecting results and choose cursor if its supported by the endpoint ...

[rduyme]

@menzowindhouwer

Hi Menzo,
I think : trial & error is enough. I suggest "HTTP 422 " if cursor is not implemented. With the same format we have at the moment for "Get List of products", ( when a filter is not implemented )