Skip to content

fermi: migrate from LATDataQuery.cgi scraping to the new REST API - #3647

Draft
PazSheimy wants to merge 7 commits into
astropy:mainfrom
PazSheimy:fermi-rest-api
Draft

fermi: migrate from LATDataQuery.cgi scraping to the new REST API#3647
PazSheimy wants to merge 7 commits into
astropy:mainfrom
PazSheimy:fermi-rest-api

Conversation

@PazSheimy

Copy link
Copy Markdown

Fixes #3646
Closes #1849

Background

The Fermi LAT data server is replacing the LATDataQuery.cgi form endpoint
with a JSON REST API. Both will run in parallel for some time, but the REST
API will eventually replace LATDataQuery.cgi entirely, so this PR migrates
the module in advance for astroquery users.

  • Base URL: https://fermi.gsfc.nasa.gov/ssc/data/access/lat/query/api/v1
    (docs). I work on the
    Fermi data-server side and wrote this migration against the new API.

What changed

  • query_object_async() POSTs a JSON payload to {base}/query and returns
    the server-assigned query_id. Previously it returned the URL of an HTML
    results page
  • _parse_result() polls {base}/query/{id}/status and reads
    {base}/query/{id}/results, so the synchronous
    query_object() keeps its exact previous signature and return type
    (a list of FITS file URLs) — existing user scripts are unaffected.
  • All regex/HTML scraping is removed, along with the module's direct
    requests usage (everything now goes through BaseQuery._request).
  • New public methods: get_status(), list_results(),
    wait_for_completion() (with optional max_wait), and get_file_urls().
  • New query options exposed by the module: zenithangle,
    coordsystem (J2000/B1950/Galactic - previously only J2000), and all-sky queries
    (radius > 60 deg with an observation window <= 24 h); literal "RA,Dec"
    strings bypass name resolution.
  • Failed queries now raise RemoteServiceError;
    a query that never completes can be bounded with max_wait
    (raises TimeoutError).
  • Rejected queries surface the server's own error message: an HTTP error with
    a JSON body like {"error": "Invalid energy range ..."} is raised as a
    RemoteServiceError carrying that message (closes the old TODO from Fail with useful failure messages on genuine failures. #1849).
  • GetFermilatDatafile / get_fermilat_datafile kept as deprecated shims
    delegating to get_file_urls().
  • Removed the "Experimental" import-time warning; the module now follows the
    standard astroquery async/sync pattern.
  • Docs (docs/fermi/fermi.rst) and changelog updated.

Testing

  • Unit tests rewritten against mocked JSON responses (submit, status
    running/complete/failed, results), including payload-shape tests, error
    paths, and a forward-compatibility test for absolute file URLs:
    15 pass offline.
  • Remote tests (--remote-data) run against the live API on 2026-08-14:
    4/4 pass (basic query, async + status, zenith angle, all-sky).

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.07080% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.45%. Comparing base (70c49c0) to head (949a520).
⚠️ Report is 127 commits behind head on main.

Files with missing lines Patch % Lines
astroquery/fermi/core.py 83.78% 18 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3647      +/-   ##
==========================================
+ Coverage   73.17%   73.45%   +0.28%     
==========================================
  Files         227      230       +3     
  Lines       21078    21426     +348     
==========================================
+ Hits        15423    15739     +316     
- Misses       5655     5687      +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bsipocz bsipocz added the fermi label Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fermi: LATDataQuery.cgi is being replaced by a REST API — module update available Fail with useful failure messages on genuine failures.

2 participants