Skip to content

feat: add pilot database and router#570

Open
Robin-Van-de-Merghel wants to merge 18 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management
Open

feat: add pilot database and router#570
Robin-Van-de-Merghel wants to merge 18 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management

Conversation

@Robin-Van-de-Merghel

@Robin-Van-de-Merghel Robin-Van-de-Merghel commented Jun 13, 2025

Copy link
Copy Markdown
Contributor

Split of #421 , first part : pilot management

closes #960

@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from 8c655b0 to 2cfe44a Compare June 13, 2025 11:19
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel marked this pull request as draft June 13, 2025 11:36
Comment thread diracx-db/src/diracx/db/sql/utils/functions.py Outdated
Comment thread diracx-logic/src/diracx/logic/auth/token.py
Comment thread diracx-core/src/diracx/core/settings.py
Comment thread diracx-routers/src/diracx/routers/auth/token.py
Comment thread diracx-routers/src/diracx/routers/utils/users.py
Comment thread diracx-routers/tests/auth/test_standard.py
Comment thread diracx-testing/src/diracx/testing/utils.py
@Robin-Van-de-Merghel Robin-Van-de-Merghel marked this pull request as ready for review June 15, 2025 06:13
Comment thread diracx-logic/src/diracx/logic/pilots/query.py Outdated

@fstagni fstagni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review.

Comment thread diracx-db/src/diracx/db/sql/utils/base.py Outdated
Comment thread diracx-core/src/diracx/core/exceptions.py Outdated
Comment thread diracx-db/src/diracx/db/sql/job/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/tests/pilots/test_query.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/tests/pilots/test_pilot_creation.py Outdated
@Robin-Van-de-Merghel

Robin-Van-de-Merghel commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

Where an issue?

Whether we have a base router with require_auth or not, we won't be able to override it in its children (cf #417 ).
So for pilots we must have require_auth=False at the base router because of secret-exchange. But now every pilot endpoint is opened.

Possibilities

Splitting

Let's start with the routers themselves. We can separate pilots and users endpoints : /api/pilots (only for pilots) and ~/api/pilot_management (only for users).

That brings us:

  1. Cleaner: each their own territory, no overlapping, easier to read (you know that on /pilots its only pilot resources)
  2. More secure: we can add a dependency to the pilot router (same as verify_dirac_pilot_token but for pilots)
  3. We can have require_auth for /pilot_management, and enforce tokens

Using another approach

We could have a bare base router without dependency injection, with sub routers with verify_dirac_access_token:

# Authenticated parent router
protected_router = APIRouter(dependencies=[Depends(verify_dirac_access_token)])

@protected_router.get("/secure")
def secure_route():
    return {"msg": "secure"}

# Public sub-router (no auth)
public_router = APIRouter()

@public_router.get("/public")
def public_route():
    return {"msg": "public"}

Its goal would be to replacer DiracxRouter by fixing the issue, and keeping an explicit depedency injection.
Or have @router.authentificated.get(...) instead of @router.get, to explicitely say if we want auth or not

Comment thread diracx-routers/src/diracx/routers/pilots/query.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from a9b353d to 4ba2c9b Compare July 4, 2025 11:48
@aldbr aldbr force-pushed the robin-pilot-management branch from fbc9233 to c0c1ff2 Compare June 10, 2026 09:05
@aldbr

aldbr commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

ddev:
part of a bigger plan with 3 axis: pilot management (user), pilot security (pilot tokens), pilot transition in DIRAC
only missing piece is the getPilotOutput: could be handled with a DIRAC Agent to fetch the pilot output and put it in the DB for now
also the current DIRAC Webapp does not communicate to diracx

There was a plan (may be not written anywhere?):

  • pilot can talk to DIRAC and DiracX
  • DiracX can interact with pilot
    Either we revisit that plan or we write it down in an ADR

@fstagni will double check and see if getPilotOutput is the only issue. If it's the case, we just isolate it in DIRAC (outside of the PilotManagerHandler) and we move on with the legacy adaptor. Will closely be followed by issues if any problem occurs afterwards.

Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
@fstagni fstagni force-pushed the robin-pilot-management branch from 070de3e to afe2c12 Compare July 10, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pilot Manager service and DB in DiracX Open access and require auth not working inside a router

4 participants