diff --git a/docs/api/error_parity.binarize.rst b/docs/api/error_parity.binarize.rst new file mode 100644 index 0000000..d371a60 --- /dev/null +++ b/docs/api/error_parity.binarize.rst @@ -0,0 +1,12 @@ +error\_parity.binarize +====================== + +.. automodule:: error_parity.binarize + + + .. rubric:: Functions + + .. autosummary:: + + compute_binary_predictions + \ No newline at end of file diff --git a/docs/api/error_parity.classifiers.rst b/docs/api/error_parity.classifiers.rst new file mode 100644 index 0000000..1fe455f --- /dev/null +++ b/docs/api/error_parity.classifiers.rst @@ -0,0 +1,16 @@ +error\_parity.classifiers +========================= + +.. automodule:: error_parity.classifiers + + + .. rubric:: Classes + + .. autosummary:: + + BinaryClassifier + BinaryClassifierAtROCDiagonal + Classifier + EnsembleGroupwiseClassifiers + RandomizedClassifier + \ No newline at end of file diff --git a/docs/api/error_parity.cvxpy_utils.rst b/docs/api/error_parity.cvxpy_utils.rst new file mode 100644 index 0000000..5c7c59a --- /dev/null +++ b/docs/api/error_parity.cvxpy_utils.rst @@ -0,0 +1,16 @@ +error\_parity.cvxpy\_utils +========================== + +.. automodule:: error_parity.cvxpy_utils + + + .. rubric:: Functions + + .. autosummary:: + + compute_fair_optimum + compute_halfspace_inequality + compute_line + make_cvxpy_halfspace_inequality + make_cvxpy_point_in_polygon_constraints + \ No newline at end of file diff --git a/docs/api/error_parity.evaluation.rst b/docs/api/error_parity.evaluation.rst new file mode 100644 index 0000000..04b075f --- /dev/null +++ b/docs/api/error_parity.evaluation.rst @@ -0,0 +1,16 @@ +error\_parity.evaluation +======================== + +.. automodule:: error_parity.evaluation + + + .. rubric:: Functions + + .. autosummary:: + + eval_accuracy_and_equalized_odds + evaluate_fairness + evaluate_performance + evaluate_predictions + evaluate_predictions_bootstrap + \ No newline at end of file diff --git a/docs/api/error_parity.pareto_curve.rst b/docs/api/error_parity.pareto_curve.rst new file mode 100644 index 0000000..dd20a79 --- /dev/null +++ b/docs/api/error_parity.pareto_curve.rst @@ -0,0 +1,15 @@ +error\_parity.pareto\_curve +=========================== + +.. automodule:: error_parity.pareto_curve + + + .. rubric:: Functions + + .. autosummary:: + + compute_inner_and_outer_adjustment_ci + compute_postprocessing_curve + fit_and_evaluate_postprocessing + get_envelope_of_postprocessing_frontier + \ No newline at end of file diff --git a/docs/api/error_parity.plotting.rst b/docs/api/error_parity.plotting.rst new file mode 100644 index 0000000..7e8cdb7 --- /dev/null +++ b/docs/api/error_parity.plotting.rst @@ -0,0 +1,14 @@ +error\_parity.plotting +====================== + +.. automodule:: error_parity.plotting + + + .. rubric:: Functions + + .. autosummary:: + + plot_polygon_edges + plot_postprocessing_frontier + plot_postprocessing_solution + \ No newline at end of file diff --git a/docs/api/error_parity.roc_utils.rst b/docs/api/error_parity.roc_utils.rst new file mode 100644 index 0000000..33b30ce --- /dev/null +++ b/docs/api/error_parity.roc_utils.rst @@ -0,0 +1,15 @@ +error\_parity.roc\_utils +======================== + +.. automodule:: error_parity.roc_utils + + + .. rubric:: Functions + + .. autosummary:: + + calc_cost_of_point + compute_global_roc_from_groupwise + compute_roc_point_from_predictions + roc_convex_hull + \ No newline at end of file diff --git a/docs/api/error_parity.threshold_optimizer.rst b/docs/api/error_parity.threshold_optimizer.rst new file mode 100644 index 0000000..9c39ccd --- /dev/null +++ b/docs/api/error_parity.threshold_optimizer.rst @@ -0,0 +1,12 @@ +error\_parity.threshold\_optimizer +================================== + +.. automodule:: error_parity.threshold_optimizer + + + .. rubric:: Classes + + .. autosummary:: + + RelaxedThresholdOptimizer + \ No newline at end of file diff --git a/docs/concepts.rst b/docs/concepts.rst new file mode 100644 index 0000000..7defcea --- /dev/null +++ b/docs/concepts.rst @@ -0,0 +1,33 @@ +Core concepts +============= + +ROC curves and convex hulls +--------------------------- + +For each group, the model induces an ROC curve of achievable (FPR, TPR) pairs by thresholding scores. The convex hull of this curve characterizes all achievable points by mixing thresholds and, when necessary, randomization. + +Randomized thresholds +--------------------- + +When the optimal target lies strictly inside a segment of the ROC hull, it is realized by a randomized classifier that mixes at most two deterministic thresholds, and, if needed, the diagonal (random) classifier. See :class:`error_parity.classifiers.RandomizedClassifier`. + +Fairness constraints +-------------------- + +Let groups be indexed by ``a, b``. We support constraints expressed on group-specific rates. Examples: + +- Equalized odds: constrain distances between (TPR, FPR) pairs across groups. +- Equal opportunity: constrain TPR parity. +- Predictive equality: constrain FPR parity. +- Demographic parity: constrain PPR (positive prediction rate) parity. + +Relaxations and \(\ell_p\) norms +---------------------------------- + +For equalized odds, distances between group ROC points are measured with an \(\ell_p\) norm, e.g., \(\ell_\infty\) (default), \(\ell_1\) (sum of absolute differences), or \(\ell_2\). + +Costs and performance +--------------------- + +The optimizer can compute theoretical cost at the global solution point for user-specified false positive and false negative costs. With unit costs, cost equals error rate. See :meth:`error_parity.threshold_optimizer.RelaxedThresholdOptimizer.cost`. + diff --git a/docs/conf.py b/docs/conf.py index 404a7a5..6ad468b 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -28,6 +28,7 @@ extensions = [ 'sphinx_rtd_theme', 'sphinx.ext.autodoc', + 'sphinx.ext.intersphinx', 'sphinx.ext.coverage', 'sphinx.ext.napoleon', 'sphinx.ext.autosummary', @@ -48,6 +49,24 @@ exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] autoclass_content = 'both' +autosummary_generate = True +autodoc_typehints = 'description' +napoleon_numpy_docstring = True +napoleon_google_docstring = False +napoleon_use_param = True +napoleon_use_rtype = True + +autosectionlabel_prefix_document = True + +myst_heading_anchors = 3 + +intersphinx_mapping = { + 'python': ('https://docs.python.org/3', None), + 'numpy': ('https://numpy.org/doc/stable/', None), + 'scipy': ('https://docs.scipy.org/doc/scipy/', None), + 'sklearn': ('https://scikit-learn.org/stable/', None), + 'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None), +} # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output diff --git a/docs/constraints.rst b/docs/constraints.rst new file mode 100644 index 0000000..0efaa76 --- /dev/null +++ b/docs/constraints.rst @@ -0,0 +1,38 @@ +Constraints +=========== + +This page summarizes the implemented constraints and how to select them. + +Available constraints +--------------------- + +- **Equalized odds** (default): equalize both TPR and FPR across groups, up to a tolerance. + + - Select with ``constraint="equalized_odds"``. + - Relaxation via \(\ell_p\) norm between group ROC points; choose ``l_p_norm``. + +- **Equal opportunity**: equalize TPR across groups. + + - Select with ``constraint="true_positive_rate_parity"``. + +- **Predictive equality**: equalize FPR across groups. + + - Select with ``constraint="false_positive_rate_parity"``. + +- **Demographic parity**: equalize positive prediction rate (PPR) across groups. + + - Select with ``constraint="demographic_parity"``. + +Tolerance +--------- + +All constraints accept a nonnegative ``tolerance`` parameter specifying the maximum allowed disparity according to the constraint's metric. ``tolerance=0.0`` enforces strict parity. + +Practical guidance +------------------ + +- Use equalized odds when both types of errors matter and the base rates differ by group. +- Use equal opportunity when minimizing false negatives for positives is paramount. +- Use predictive equality when minimizing false positives for negatives is paramount. +- Use demographic parity when the rate of positive decisions itself should be similar across groups. + diff --git a/docs/error_parity.rst b/docs/error_parity.rst deleted file mode 100644 index 0387a8d..0000000 --- a/docs/error_parity.rst +++ /dev/null @@ -1,67 +0,0 @@ -:code:`error\_parity` package -============================= - -error\_parity.threshold\_optimizer module ------------------------------------------ - -.. automodule:: error_parity.threshold_optimizer - :members: - :undoc-members: - :show-inheritance: - -error\_parity.pareto\_curve module ----------------------------------- - -.. automodule:: error_parity.pareto_curve - :members: - :undoc-members: - :show-inheritance: - -error\_parity.plotting module ------------------------------ - -.. automodule:: error_parity.plotting - :members: - :undoc-members: - :show-inheritance: - - -error\_parity.evaluation module -------------------------------- - -.. automodule:: error_parity.evaluation - :members: - :undoc-members: - :show-inheritance: - -error\_parity.binarize module ------------------------------ - -.. automodule:: error_parity.binarize - :members: - :undoc-members: - :show-inheritance: - -error\_parity.classifiers module --------------------------------- - -.. automodule:: error_parity.classifiers - :members: - :undoc-members: - :show-inheritance: - -error\_parity.cvxpy\_utils module ---------------------------------- - -.. automodule:: error_parity.cvxpy_utils - :members: - :undoc-members: - :show-inheritance: - -error\_parity.roc\_utils module -------------------------------- - -.. automodule:: error_parity.roc_utils - :members: - :undoc-members: - :show-inheritance: diff --git a/docs/faq.rst b/docs/faq.rst new file mode 100644 index 0000000..5c31953 --- /dev/null +++ b/docs/faq.rst @@ -0,0 +1,28 @@ +FAQ +=== + +Do I need to retrain my model? +------------------------------ + +No. ``error-parity`` is a postprocessing method that wraps a score-based predictor. + +What if my groups are not 0..G-1? +--------------------------------- + +Encode them to integers starting at 0 before calling ``fit`` or prediction. Noncontiguous encodings raise a ``ValueError``. + +Can I use decision_function instead of predict_proba? +----------------------------------------------------- + +Yes, pass ``predictor=lambda X: model.decision_function(X)``. Ensure higher values indicate higher likelihood of the positive class. + +What tolerance should I use? +---------------------------- + +Start with ``tolerance=0.0`` (strict). Increase gradually to explore trade-offs using the postprocessing curve utilities. + +How do I get uncertainty estimates? +----------------------------------- + +Use :func:`error_parity.evaluation.evaluate_predictions_bootstrap` and the plotting utilities for confidence intervals on frontiers. + diff --git a/docs/glossary.rst b/docs/glossary.rst new file mode 100644 index 0000000..02b9047 --- /dev/null +++ b/docs/glossary.rst @@ -0,0 +1,21 @@ +Glossary +======== + +ROC + Receiver Operating Characteristic; plots TPR vs. FPR as threshold varies. + +ROC hull + The convex hull of achievable ROC points; mixing thresholds (and randomized classifiers) realizes any point on the hull. + +TPR / FNR + True positive rate / false negative rate; ``FNR = 1 - TPR``. + +FPR / TNR + False positive rate / true negative rate; ``TNR = 1 - FPR``. + +PPR + Positive prediction rate; fraction of predictions that are positive. + +Tolerance + Maximum allowed disparity under a chosen fairness constraint. + diff --git a/docs/index.rst b/docs/index.rst index 27a8636..9f6d005 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,12 +6,16 @@ Welcome to error-parity's documentation! ======================================== -The :code:`error-parity` package allows you to easily achieve error-rate +.. important:: + + This documentation was authored and extensively updated with assistance from GPT-5. + +The :code:`error-parity` package allows you to easily achieve error-rate fairness between societal groups. -It's compatible with any score-based predictor, and can map out all of its +It's compatible with any score-based predictor, and can map out all of its attainable fairness-accuracy trade-offs. -Full code available on the `GitHub repository`_, +Full code available on the `GitHub repository`_, including various `jupyter notebook examples`_ . Check out the following sub-pages: @@ -19,9 +23,21 @@ Check out the following sub-pages: .. toctree:: :maxdepth: 1 - Readme file + Overview + Installation + Quickstart + Concepts + Constraints + Usage: Threshold optimizer + Usage: Postprocessing frontier + Usage: Evaluation + Usage: Plotting + Metrics reference API reference - Example notebooks + Notebooks Gallery + FAQ + Glossary + Readme file Citing diff --git a/docs/installation.rst b/docs/installation.rst new file mode 100644 index 0000000..c4c8598 --- /dev/null +++ b/docs/installation.rst @@ -0,0 +1,30 @@ +Installation +============ + +From PyPI +--------- + +.. code-block:: bash + + pip install error-parity + +From source (development) +------------------------- + +.. code-block:: bash + + git clone https://github.com/socialfoundations/error-parity.git + pip install ./error-parity + +Requirements +------------ + +The package depends on standard scientific Python libraries (NumPy, SciPy, scikit-learn, pandas) and ``cvxpy`` for the optimization step. For plotting, ``matplotlib`` and ``seaborn`` are used. + +For building documentation and rendering notebooks locally, see ``requirements/docs.txt``. + +Supported Python versions +------------------------- + +See the PyPI classifiers badge and ``pyproject.toml`` for the supported versions. + diff --git a/docs/metrics_reference.rst b/docs/metrics_reference.rst new file mode 100644 index 0000000..7aa1dcf --- /dev/null +++ b/docs/metrics_reference.rst @@ -0,0 +1,38 @@ +Metrics reference +================= + +This page summarizes metrics produced by :mod:`error_parity.evaluation`. + +Performance metrics +------------------- + +- **accuracy**: fraction of correct predictions. +- **tpr (recall)**: true positive rate. +- **fnr**: false negative rate (1 - TPR). +- **fpr**: false positive rate. +- **tnr**: true negative rate (1 - FPR). +- **precision**: TP / predicted positives. +- **ppr**: positive prediction rate. +- **squared_loss**: mean squared error on scores vs. labels. +- **log_loss**: logistic loss on scores vs. labels. + +Fairness aggregations +--------------------- + +For each metric ``m`` and groups ``a, b``, we compute: + +- ``m_ratio = min(m_a, m_b, ...) / max(m_a, m_b, ...)`` +- ``m_diff = max(m_a, m_b, ...) - min(m_a, m_b, ...)`` + +Equalized odds +-------------- + +- ``equalized_odds_ratio = min(fnr_ratio, fpr_ratio)`` +- ``equalized_odds_diff = max(tpr_diff, fpr_diff)`` +- ``equalized_odds_diff_l{1,2,inf}`` via \(\ell_1\), \(\ell_2\), \(\ell_\infty\) norms over (TPR, FPR) pairwise differences. + +Groupwise outputs +----------------- + +Set ``return_groupwise_metrics=True`` in :func:`error_parity.evaluation.evaluate_fairness` or :func:`error_parity.evaluation.evaluate_predictions` to include per-group metrics like ``tpr_group=0``, ``fpr_group=1``, etc. + diff --git a/docs/modules.rst b/docs/modules.rst index c4eb9ae..ea25390 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -1,7 +1,16 @@ API reference ============= -.. toctree:: - :maxdepth: 2 +.. autosummary:: + :toctree: api/ + :recursive: + :caption: Public modules - error_parity + error_parity.threshold_optimizer + error_parity.pareto_curve + error_parity.plotting + error_parity.evaluation + error_parity.binarize + error_parity.classifiers + error_parity.cvxpy_utils + error_parity.roc_utils diff --git a/docs/overview.rst b/docs/overview.rst new file mode 100644 index 0000000..54c0558 --- /dev/null +++ b/docs/overview.rst @@ -0,0 +1,50 @@ +Overview +======== + +The ``error-parity`` package provides fast post-processing for any score-based classifier to satisfy user-chosen fairness constraints with optional relaxation. It works by: + +- computing group-wise ROC curves and their convex hulls; +- solving a small convex optimization to find the optimal fairness–accuracy operating point under a constraint and tolerance; +- realizing that solution as group-specific binary classifiers, which can include carefully randomized thresholds when the target point lies in the interior of a group's ROC hull. + +Key features +------------ + +- **Drop-in postprocessing**: Wrap any predictor that outputs scores in ``[0, 1]`` or probabilities. +- **Multiple constraints**: Equalized odds (with configurable \(\ell_p\) metric), equal opportunity, predictive equality, demographic parity. +- **Tunable tolerance**: Explore strict (``tolerance=0``) to relaxed constraints (e.g., ``tolerance=0.05``). +- **Pareto frontier**: Compute and visualize fairness–performance trade-offs across tolerances. +- **Metrics and evaluation**: Built-in evaluation and bootstrap uncertainty. +- **Plotting utilities**: Visualize ROC hulls, chosen operating points, and postprocessing frontiers. + +Typical workflow +---------------- + +1. Choose a fairness constraint and tolerance. +2. Fit a ``RelaxedThresholdOptimizer`` on data with labels and group membership. +3. Use the fitted object to predict on new data (requires group membership). +4. Optionally compute a postprocessing curve across tolerances and plot results. + +When to use error-parity +------------------------ + +- You have a trained model that outputs scores (e.g., ``predict_proba``) and you want to enforce fairness constraints without retraining. +- You need to compare models at the same fairness level. +- You want to quantify the fairness–performance frontier under different tolerances. + +What this is not +---------------- + +- ``error-parity`` is a postprocessing library. It does not change the model training objective. It complements in-processing or pre-processing approaches. + +See also +-------- + +- :doc:`quickstart` +- :doc:`concepts` +- :doc:`constraints` +- :doc:`usage_threshold_optimizer` +- :doc:`usage_pareto_curve` +- :doc:`usage_evaluation` +- :doc:`usage_plotting` + diff --git a/docs/quickstart.rst b/docs/quickstart.rst new file mode 100644 index 0000000..07b421d --- /dev/null +++ b/docs/quickstart.rst @@ -0,0 +1,41 @@ +Quickstart +========== + +This guide demonstrates postprocessing a trained model to satisfy a fairness constraint. + +Minimal example +--------------- + +.. code-block:: python + + import numpy as np + from sklearn.linear_model import LogisticRegression + from error_parity import RelaxedThresholdOptimizer + + # Assume X, y, group are numpy arrays + # group must be encoded as integers 0..G-1 + model = LogisticRegression().fit(X, y) + + fair_clf = RelaxedThresholdOptimizer( + predictor=lambda X: model.predict_proba(X)[:, -1], + constraint="equalized_odds", + tolerance=0.05, + ) + + fair_clf.fit(X=X, y=y, group=group) + y_pred = fair_clf(X=X_test, group=group_test) + +Notes +----- + +- ``group`` indexing must be contiguous starting at 0. If ``np.max(group) > n_groups-1``, a ``ValueError`` is raised. +- For callable predictors, pass the callable directly as ``predictor``. For scikit-learn API, pass a ``lambda`` to ``predict_proba`` or ``decision_function`` as appropriate. +- Use ``tolerance=0.0`` for strict parity; higher tolerance allows more disparity. + +Next steps +---------- + +- Explore :doc:`usage_threshold_optimizer` for options (costs, norms, diagnostics). +- Compute a :doc:`usage_pareto_curve` to visualize the fairness–performance frontier. +- Evaluate metrics and uncertainty with :doc:`usage_evaluation`. + diff --git a/docs/usage_evaluation.rst b/docs/usage_evaluation.rst new file mode 100644 index 0000000..2fa4301 --- /dev/null +++ b/docs/usage_evaluation.rst @@ -0,0 +1,25 @@ +Evaluation and metrics +====================== + +This page covers performance and fairness evaluation helpers and their bootstrap variants. + +Performance metrics +------------------- + +.. autofunction:: error_parity.evaluation.evaluate_performance + +Fairness metrics +---------------- + +.. autofunction:: error_parity.evaluation.evaluate_fairness + +End-to-end evaluation +--------------------- + +.. autofunction:: error_parity.evaluation.evaluate_predictions + +Bootstrap estimates +------------------- + +.. autofunction:: error_parity.evaluation.evaluate_predictions_bootstrap + diff --git a/docs/usage_pareto_curve.rst b/docs/usage_pareto_curve.rst new file mode 100644 index 0000000..2656102 --- /dev/null +++ b/docs/usage_pareto_curve.rst @@ -0,0 +1,23 @@ +Postprocessing frontier (Pareto curve) +====================================== + +Use :func:`error_parity.pareto_curve.compute_postprocessing_curve` to compute the fairness–performance frontier across tolerances. + +API +--- + +.. autofunction:: error_parity.pareto_curve.compute_postprocessing_curve + +Interpretation +-------------- + +- The returned ``pandas.DataFrame`` has one row per tolerance tick and columns for each metric and dataset split. +- Use :func:`error_parity.plotting.plot_postprocessing_frontier` to visualize the envelope of the frontier and optional bootstrap confidence intervals. + +Related utilities +----------------- + +.. autofunction:: error_parity.pareto_curve.fit_and_evaluate_postprocessing +.. autofunction:: error_parity.pareto_curve.get_envelope_of_postprocessing_frontier +.. autofunction:: error_parity.pareto_curve.compute_inner_and_outer_adjustment_ci + diff --git a/docs/usage_plotting.rst b/docs/usage_plotting.rst new file mode 100644 index 0000000..c9cdfd9 --- /dev/null +++ b/docs/usage_plotting.rst @@ -0,0 +1,9 @@ +Plotting utilities +================== + +Plot postprocessing solutions and frontiers. + +.. autofunction:: error_parity.plotting.plot_postprocessing_solution + +.. autofunction:: error_parity.plotting.plot_postprocessing_frontier + diff --git a/docs/usage_threshold_optimizer.rst b/docs/usage_threshold_optimizer.rst new file mode 100644 index 0000000..71a7029 --- /dev/null +++ b/docs/usage_threshold_optimizer.rst @@ -0,0 +1,22 @@ +Using the RelaxedThresholdOptimizer +=================================== + +The :class:`error_parity.threshold_optimizer.RelaxedThresholdOptimizer` wraps a score-based predictor and realizes a fairness-constrained classifier. + +Constructor +----------- + +.. autoclass:: error_parity.threshold_optimizer.RelaxedThresholdOptimizer + :members: + :undoc-members: + :show-inheritance: + +Tips +---- + +- Ensure ``group`` values are integers ``0..G-1``. +- If your model returns a 2-D array of probabilities, the optimizer will use the last column (``[:, -1]``). +- Control the solution search resolution with ``max_roc_ticks`` if your ROC arrays are large. +- Use ``l_p_norm`` with ``constraint="equalized_odds"`` to pick \(\ell_1\), \(\ell_2\), or \(\ell_\infty\). +- Use ``false_pos_cost`` and ``false_neg_cost`` to reflect asymmetric error costs; the method ``cost`` reports the theoretical cost at the global solution point. +