[ENG-9044] Add manage command to resync preprint dois v1 by Vlad0n20 · Pull Request #11617 · CenterForOpenScience/osf.io

Vlad0n20 · 2026-03-02T19:39:37Z

Ticket

ENG-9044

Purpose

Changes

Side Effects

QE Notes

CE Notes

Documentation

cslzchen

Looks good overall. In addition to my questions/comments:

Can we add logs of the output of the logs for you local run?
We should also work with CE to test this command with a copy of production DB.

cslzchen

Looks good 👍

Make sure we test this against a copy of prod DB before running this on prod.
- If there are errors, need look further on why they fail, and probably another fix.
- Test if the default batch size is good, how long does it take, and also see how many times we need to repeat this.
Make sure whoever uses this command knows that we need to repeatedly run this command.

cslzchen · 2026-03-13T16:44:51Z

+            help=(
+                'Maximum number of preprints to process per run (default: 500). '
+                'The command processes the first N eligible preprints and exits; '
+                're-run the command to continue with the next batch.'
+            ),


Oh, I see. This is an OK alternative to the loop I suggested. Just need to make sure whoever runs this command is aware of the fact that they need to keep running this command until none exists. Cc @adlius for your input on this.

antkryt · 2026-05-04T09:23:44Z

LGTM

mkovalua

LGTM, @Vlad0n20, please change target branch to current one feature/pbs-26-9, I have no permission to do it

…DOI resync

adlius · 2026-08-06T14:09:33Z


        user.date_last_logged_in = timezone.now()
        user.external_identity[provider][provider_id] = 'VERIFIED'
+        if provider.lower() in OSFUser.SOCIAL_FIELDS:


Is this part of this ticket?

adlius

So I have a couple of suggestions. Since Crossref has a limit of 10000 submissions from a single user, might as well set the default batch size to 1000 and no rate limit. Instead let's run the task every five minutes so that we can send about 12000 requests per hour.

Also there are some commits from other branches that is in this PR.

adlius · 2026-08-06T14:10:26Z

        assert len(user.system_tags) == 1

-        tag = Tag.all_tags.get(name=tag_name, system=True)
+        tag = Tag.all_tags.get(name=tag_name.lower(), system=True)


Same here. Is this part of this ticket?

adlius · 2026-08-06T14:19:16Z

+    return qs
+
+
+def resync_preprint_dois_v1(dry_run=True, batch_size=500, rate_limit=100, provider_id=None):


The rate limit of 100 is perhaps to restrictive. According to conversations with Crossref, they have a limit of 10000 pending items for each user.

A 500 batch size is also perhaps too small.

adlius · 2026-08-13T08:12:16Z

+
+        if rate_limit and not record_number % rate_limit:
+            logger.info(f'Rate limit reached at {record_number} preprints, sleeping {RATE_LIMIT_SLEEP}s')
+            time.sleep(RATE_LIMIT_SLEEP)


I am afraid this would clog the celery worker when this is run as a celery beat task.

brianjgeiger

Drive by:

What queue are the tasks going onto? The default queue? We should probably ensure that they go on the Low Priority queue to keep from clogging the pipes too much.
If we have a cron job that happens every five minutes, how are you going to run this in a dry-run scenario to make sure it's going to work before the cron job runs? We might want to rethink our strategy here.

adlius

Some issues found during local testing.

adlius

Some more issues to fix.

adlius · 2026-09-10T16:26:51Z

    def create_identifier(self, preprint, category, include_relation=True):
        if category == 'doi':
-            metadata = self.build_metadata(preprint, include_relation)
+            metadata = self.build_metadata(preprint, include_relation, include_unversioned_doi=True)


One more issue found during local testing: on an unmoderated preprint provider, when a user creates a new version but never publish that version, build_unversioned_posted_content would read preprint.get_guid().referent, which hands back that unpublished draft instead of the preprint actually being deposited. This is problematic because the unpublished draft will have date_published to be None and that would fail when building the payload.

adlius · 2026-09-11T17:51:26Z

@@ -48,6 +49,23 @@ def post(self, request):
                if record.get('status').lower() == 'success' and doi:
                    msg = record.find('msg').text
                    created = bool(msg == 'Successfully added')


So a DOI in the doi_unversioned category is only written when Crossref replies Successfully added. This could be a potential issue because if the DOI
is already registered, the email probably comes back with something like Successfully updated, so for those preprints
this branch never runs.

This would affect legacy preprints that exists before versioning was a thing. Their base guid is their original guid, so the DOI deposit here is the one registered years ago.

I checked against a local copy of prod db. One of preprint 22hq2 has 10.31219/osf.io/22hq2 already registered with crossref. So it's possible that the crossref response email wouldn't contain Successfully added but Successfully updated or some other language. Will need to confirm if that is indeed the case.

If that is the case, one possible fix is that for an unversioned DOI, store doi_unversioned on any successful record rather than only on Successfully added.

There is another problem: if we run the resync task every five minutes, and if the preprint that was resynced in the last run still haven't received an email back confirming the update of the DOI, then in the next run it will be included in the resync task again. So we'd have to rethink how we should do this. One idea suggested by @cslzchen is that we follow the example in this PR and monitory the number of active tasks and spawn new tasks if the number of active task falls below a certain number (say 1000). In this case, we don't have to worry about running the task every five minutes.

adlius · 2026-09-11T17:59:15Z

+    ).exclude(
+        tags__name='qatest',
+        tags__system=True,
+    ).select_related('provider').distinct()


we probably need an order_by here otherwise the ordering of the queryset would be undeterministic.

cslzchen requested changes Mar 5, 2026

View reviewed changes

Vlad0n20 changed the title ~~Add manage command to resync preprint dois v1~~ [ENG-9044] Add manage command to resync preprint dois v1 Mar 9, 2026

cslzchen approved these changes Mar 13, 2026

View reviewed changes

adlius changed the base branch from feature/pbs-26-2 to feature/pbs-26-6 April 7, 2026 13:13

Vlad0n20 force-pushed the fix/ENG-9044 branch 2 times, most recently from e803a0c to e1d4931 Compare April 14, 2026 11:46

mkovalua suggested changes May 12, 2026

View reviewed changes

Vlad0n20 changed the base branch from feature/pbs-26-6 to feature/pbs-26-9 May 13, 2026 09:02

Vlad0n20 force-pushed the fix/ENG-9044 branch from 7e0d863 to 9095a49 Compare May 13, 2026 09:02

mkovalua approved these changes May 13, 2026

View reviewed changes

Vlad0n20 force-pushed the fix/ENG-9044 branch from 9095a49 to 4855f92 Compare May 27, 2026 14:08

Vlad0n20 added 6 commits July 28, 2026 17:02

Add manage command to resync preprint dois v1

630c7ff

Update command

f943f05

Added a class-level docstring

9a50376

Register unversioned DOI alongside versioned on Crossref submissions

571020f

Add daily Celery beat task and unversioned DOI tracking for preprint …

928b8b7

…DOI resync

fix test

1973527

Vlad0n20 changed the base branch from feature/pbs-26-9 to feature/pbs-26-15 July 28, 2026 14:13

bodintsov force-pushed the fix/ENG-9044 branch from 4855f92 to 1973527 Compare July 28, 2026 14:49

adlius requested changes Aug 6, 2026

View reviewed changes

adlius requested changes Aug 13, 2026

View reviewed changes

bodintsov added 2 commits August 19, 2026 16:10

fix comments

7d513bc

fix tests

0a2b707

brianjgeiger reviewed Aug 27, 2026

View reviewed changes

adlius requested changes Aug 27, 2026

View reviewed changes

Comment thread website/identifiers/clients/crossref.py

add to low priority queue, fix short circuit

6ad8f6b

adlius requested changes Sep 11, 2026

View reviewed changes

		return qs


		def resync_preprint_dois_v1(dry_run=True, batch_size=500, rate_limit=100, provider_id=None):

Uh oh!

Conversation

Vlad0n20 commented Mar 2, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket

Purpose

Changes

Side Effects

QE Notes

CE Notes

Documentation

Uh oh!

cslzchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cslzchen left a comment • edited by adlius Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antkryt commented May 4, 2026

Uh oh!

mkovalua left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brianjgeiger left a comment

Choose a reason for hiding this comment

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Vlad0n20 commented Mar 2, 2026 •

edited by atlassian Bot

Loading

cslzchen left a comment •

edited by adlius

Loading

mkovalua left a comment •

edited

Loading