Feature: EventPreprocessor

[kosack]

Introduces an EventPreprocessor class that can generically transform an event table by applying the following steps:

• Generate new or rename existing columns with a FeatureGenerator
• Select "good" event rows with a QualityQuery
• Select which columns to output (by setting the features configuration attribute of the EventPreprocessor)

This is useful for doing the final steps of DL2 processing, and will eventually replace what is in DL2EventPreprocessor and DL2EventLoader, which will be deprecated in a future release.

The EventPreprocessor also includes the ability to pre-configure itself for specific use cases by setting the feature_set option. Currently only two ProcessingFeatureSet are implemented: feature_set=dl2_irf, which defines the transforms, event selection, and output features for processing simulated DL2 events, and feature_set=custom, which has no pre-configuration and requires all parmeters to be set in a config file.

The functionality of DL2EventLoader can be mimicked with the following:

from ctapipe.io import TableLoader, EventPreprocessor
from astropy.table import vstack

DL2FILE = "some_dl2_file.h5"
with  TableLoader(DL2FILE, dl2=True, simulated=True, observation_info=True) as loader:
    preprocess = EventPreprocessor(feature_set="dl2_irf") # or pass in a full config
    events = vstack(
        [
            preprocess(c.data) 
            for c in loader.read_subarray_events_chunked(chunk_size=100_000)
        ]
    )

tab = preprocess.quality_query.to_table(functions=True)
tab.reverse()
plt.barh(
    width=tab["cumulative_counts"],
    y=tab["criteria"],
)
plt.xlabel("Cumulative Counts")

this comes from the refactoring of DL2EventLoader in #2919

An example configuration, equivalent to the dl2_irf feature_set is:

 EventPreprocessor:
  FeatureGenerator:
    features:  # to generate
    - - reco_energy
      - RandomForestRegressor_energy
    - - reco_alt
      - HillasReconstructor_alt
    - - reco_az
      - HillasReconstructor_az
    - - gh_score
      - RandomForestClassifier_prediction
    - - theta
      - angular_separation(reco_az, reco_alt, true_az, true_alt)
    - - reco_fov_coord
      - altaz_to_fov(reco_az, reco_alt, subarray_pointing_lon, subarray_pointing_lat)
    - - reco_fov_lon
      - reco_fov_coord[:,0]
    - - reco_fov_lat
      - reco_fov_coord[:,1]
    - - true_fov_coord
      - altaz_to_fov(true_az, true_alt, subarray_pointing_lon, subarray_pointing_lat)
    - - true_fov_lon
      - true_fov_coord[:,0]
    - - true_fov_lat
      - true_fov_coord[:,1]
    - - true_fov_offset
      - angular_separation(reco_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
    - - reco_fov_offset
      - angular_separation(true_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
    - - multiplicity
      - np.count_nonzero(RandomForestClassifier_telescopes,axis=1)
  QualityQuery:
    quality_criteria:  # can use any input or generated feature
    - - Valid geometry
      - HillasReconstructor_is_valid
    - - valid energy
      - RandomForestRegressor_is_valid
    - - valid gammaness
      - RandomForestClassifier_is_valid
    - - sufficient multiplicity
      - multiplicity >= 4
  features: # to output after selection
  - event_id
  - obs_id
  - reco_energy
  - reco_alt
  - reco_az
  - gh_score
  - true_energy
  - true_alt
  - true_az
  - true_fov_offset
  - reco_fov_offset
  - theta
  - reco_fov_lat
  - true_fov_lat
  - reco_fov_lon
  - true_fov_lon
  - multiplicity

[maxnoe]

@mdebony This is intended for DL2 to DL3, so please take a look here.

I think it would be good to get this in and then use it in the DL2 to DL3 tool

[mexanick]

LGTM!

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[mdebony]

Seems very usefull for DL3 production, I've added a small suggestion in the comments.

[mexanick]

@mdebony could this be moved on?

[mdebony]

Yes, it's can move on.