Deploy validator traffic to a separate Cloud Foundry application

[FuhuXia]

The public DCAT-US schema validator currently runs in the same Cloud Foundry application as the Harvester administrative interface.

Validator-related paths include:

- /validate
- /validate/
- /api/validate
- /api/v1/validate

Problem

Keeping the validator and admin application together has several drawbacks:

• Validator requests consume the same CPU, memory, and Gunicorn workers as administrative operations.
• Large or expensive validation requests could reduce the availability of Harvester administration.
• The server-side URL-fetching feature increases the security exposure of the admin application.
• The validator cannot be scaled, restarted, or monitored independently.

Proposed approach

Deploy the existing codebase as an additional Cloud Foundry application named datagov-harvest-validator.

The existing datagov-harvest application will remain functionally unchanged and continue to contain the validator routes. Nginx will direct public validator traffic to the new application.

harvest.data.gov
        |
datagov-harvest-proxy
        |
        +-- validator routes
        |       -> datagov-harvest-validator
        |
        +-- all other routes
                -> datagov-harvest

This provides runtime and capacity isolation without requiring an immediate restructuring of the Flask application.

Things to consider

• Give the validator application its own internal route, instances, memory, and scaling configuration.
• Use a startup command that does not run database migrations.
• Route /validate, /validate/, /api/validate, and /api/v1/validate through nginx to the new application.
• Add the required proxy-to-validator network policy and deployment workflow steps.
• Keep the validator routes in the main application to provide a simple rollback path.
• Because the new application initially loads the full Flask application, it may still require existing service bindings and secrets. A validator-only application factory can be considered later.
• Consider additional protections for public validator traffic, including request timeouts, redirect validation, egress hostname restrictions, and rate limiting.

[SueValente]

metrics to support new system working better than old system.. large uploads will have failure isolation so it doesn't bring down harvester app.

eventually create validation only app. doesnt need DB, shed down to just this one route.. Reid's thoughts