From cde6d904975a461c87c4104fa45764d94906f99c Mon Sep 17 00:00:00 2001 From: Janos Gabler Date: Sat, 4 Jul 2026 17:59:07 +0200 Subject: [PATCH] Convert all algorithm documentation to the new format. --- CLEANUP_NOTES.MD | 20 + CONSTRAINTS_REFACTOR_REVIEW.md | 180 + REPORT.md | 230 + docs/source/algorithms.md | 4813 +++++------------ .../source/how_to/how_to_add_optimizers.ipynb | 6 +- docs/source/refs.bib | 178 + src/optimagic/optimizers/bhhh.py | 37 + src/optimagic/optimizers/fides.py | 150 + src/optimagic/optimizers/ipopt.py | 1446 +++++ src/optimagic/optimizers/nag_optimizers.py | 733 +++ src/optimagic/optimizers/neldermead.py | 96 +- src/optimagic/optimizers/nlopt_optimizers.py | 1040 +++- src/optimagic/optimizers/pounders.py | 151 + src/optimagic/optimizers/pygmo_optimizers.py | 1227 +++++ src/optimagic/optimizers/scipy_optimizers.py | 1277 ++++- src/optimagic/optimizers/tao_optimizers.py | 96 +- src/optimagic/optimizers/tranquilo.py | 813 +++ 17 files changed, 9115 insertions(+), 3378 deletions(-) create mode 100644 CLEANUP_NOTES.MD create mode 100644 CONSTRAINTS_REFACTOR_REVIEW.md create mode 100644 REPORT.md diff --git a/CLEANUP_NOTES.MD b/CLEANUP_NOTES.MD new file mode 100644 index 000000000..bc50b4003 --- /dev/null +++ b/CLEANUP_NOTES.MD @@ -0,0 +1,20 @@ +Here we collect task to work on after the main refactoring of constraints is done. + + +## Bug Fixes + + + +## Improvements + +- Check whether there are more cases like diagonal covariance matrices or 1x1 covariance matrices that can actually be covered without A reparametrization. A good candidate are probability constraints with 2 or less entries. I haven't checked if this is already exploited. +- Update and extend the documentation (how constraints are implemented) +- Do we really need the flat param names for error messages in ResolutionContext if we have the source and can apply selector to params? + + + +## Clean UP + +- Remove all mentions of the constraints refactoring (e.g. in test_constraint_pipeline_invariants) +- The IntArray and FloatArray types in optimagic.parameters.constraints.types should actually be defined in optimagic.typing and used throughout. +- Split up constraints.py into one file per constraint type that also contains the transformations diff --git a/CONSTRAINTS_REFACTOR_REVIEW.md b/CONSTRAINTS_REFACTOR_REVIEW.md new file mode 100644 index 000000000..5826d368e --- /dev/null +++ b/CONSTRAINTS_REFACTOR_REVIEW.md @@ -0,0 +1,180 @@ +# Review guide: constraints refactoring stack + +Context document for Claude sessions that review one PR of the constraints +refactoring. Read this first, then check out the branch under review. This file is +deliberately untracked; delete it when the whole stack is merged. + +## Status (2026-07-03, evening) + +- PRs 1-3 are **merged into `main`** (squash: #686, #687, #688). `main` is the base + of the remaining stack. +- Branches 4-6 have been rebased onto the new `main` (each squash drops the original + commits, so this used `git rebase --onto origin/main ` + per branch). All remaining branches contain their predecessor and are green on + the targeted test loop, mypy, and pre-commit; the full fast suite passed on the + last branch. +- PR 3 review landed major design changes (now in `main`): selector resolution is a + `Constraint._resolve(context)` method on each user constraint class; the + `Resolved*Constraint` dataclasses live in `optimagic.constraints` directly after + their user-facing counterparts (`parameters/constraints/types.py` no longer + exists); resolved class names exactly match their user counterparts + (e.g. `ResolvedFlatCovConstraint`); resolving a `NonlinearConstraint` raises + `NotImplementedError`. Constraint-type-specific behavior is unit tested in + `tests/optimagic/test_constraints.py`; `test_resolution.py` covers only general + end-to-end resolution behavior. + +## What this refactoring is + +The code that implements constraints via reparametrization (formerly +`process_constraints.py`, `consolidate_constraints.py`, `check_constraints.py`, +`process_selectors.py`, `kernel_transformations.py`, `space_conversion.py` in +`src/optimagic/parameters/`) was rewritten in 5 stacked PRs. Goals: typed internals +(frozen dataclasses instead of the mutable `constr_info` dict and constraint dicts), +less entanglement, provenance-based error messages that cite the user's original +constraints, and extension slots for kernel second derivatives, jax derivatives, +probability-with-fixed-params, and native linear-constraint pass-through (none of +these extensions are implemented; do not add them during review). + +The full design and rationale are in the plan file +`~/.claude/plans/i-want-to-refactor-moonlit-nygaard.md`. The math is unchanged and +documented in `docs/source/explanation/implementation_of_constraints.md`. + +## The branch stack (review in this order) + +Each branch is one PR, sits on top of the previous one, and is green on the full +fast test suite, mypy, and pre-commit. + +1. `constraints-refactor-1-characterization-tests` — **MERGED (squash, #686)**; now + the base of the stack. + Test-only safety net: `tests/optimagic/parameters/test_constraint_pipeline_invariants.py` + pins golden internal params for 26 constraint sets, round trips, the feasibility + invariant, derivatives, and first-error types at the `get_converter` seam. During + review the corpus grew (shifted covariance blocks, sdcorr simplified to bounds, + uncorrelated sdcorr) and the fixtures were made typed/frozen (a `Case` dataclass + and an `ExpectedInternal` dataclass instead of dicts). + Also carries two behavior fixes: `check_fixes_and_bounds` crashed with `TypeError` + instead of raising `InvalidConstraintError`; and the uncorrelated-covariance + simplification now works for covariance blocks that do not start at position 0 of + the flat vector (the off-diagonal indices are selected by block-local position, not + by matching global index values). The second fix was moved up from PR 6. + +2. `constraints-refactor-2-typed-boundary` + `deprecations.pre_process_constraints` inverted: legacy dict constraints + (incl. `loc/locs/query/queries`, which become selector closures) are converted + INTO `Constraint` objects; `OptimizationProblem.constraints` is + `list[Constraint]`; nonlinear split via `isinstance`. Temporary `_to_dict` seams + keep the old dict pipeline running (removed in PR 5). + New tests were added to `tests/optimagic/test_deprecations.py` (not a separate + file: they cover `deprecations.pre_process_constraints`, so they can be deleted + together with the rest of the deprecation-era tests once dict constraints are + removed in 0.6.0). + +3. `constraints-refactor-3-typed-resolution` — **MERGED (squash, #688)**. + Selector resolution as typed `Constraint._resolve(context)` methods; the + `Resolved*Constraint` dataclasses and `ConstraintSource` provenance live in + `optimagic.constraints`; `resolution.py` holds the loop and `ResolutionContext`. + `process_selectors.py` deleted. A temporary `to_legacy_dicts` seam feeds the + still-dict-based consolidation (removed in PR 5). + +4. `constraints-refactor-4-numpy-linear-consolidation` + Shape-preserving pandas -> numpy rewrite of the linear consolidation, verified + by a 200-scenario differential test against a verbatim copy of the pandas code + (`tests/optimagic/parameters/test_linear_consolidation_differential.py`, + deleted again in PR 5). Preserves first-occurrence dedup order and lb/ub swaps + under negative rescaling; linear constraints no longer get their index rewritten + during equality plugging (the rewritten index was unused before but would have + corrupted the numpy weight scatter). + +5. `constraints-refactor-5-typed-core` + The big one: `validation.py`, `consolidation.py`, `consolidate_linear.py`, + `transforms.py`, `kernels.py`; `SpaceConversionSpec` replaces `constr_info`; + `SpaceConverter` gets real methods; `get_space_converter` orchestrates + validate -> normalize -> consolidate. Old modules, all seams, and + `Constraint._to_dict` are deleted. Test retargeting changed only constraint + construction lines and imports — no expected value changed. + +6. `constraints-refactor-6-review-fixes` + Collects genuine behavior bugs found during review, one commit + regression test + each. The uncorrelated-covariance-on-shifted-blocks fix now lands in PR 1 (`main`) + and is carried through the typed core in PR 5, so this PR now only adds the + dedicated consolidation-stage regression test (`test_consolidation.py`). + +## Ground rules for review sessions + +- **Golden values are the contract.** Never edit expected numbers in + `test_constraint_pipeline_invariants.py` or `test_space_conversion.py`. If a + change would require that, it changes the reparametrization and needs explicit + discussion, not a test update. +- **Deliberate behavior changes — do not revert them:** + - fixes/bounds colliding with cov/sdcorr/probability raise + `InvalidConstraintError` (was `TypeError`, PR 1), + - conflicting fixes on equality-constrained params raise + `InvalidConstraintError` (was `AssertionError`, PR 5), + - conflicting linear fixed values raise `InvalidConstraintError` + (was `ValueError`, PR 5), + - misaligned linear weights raise `InvalidConstraintError` at resolution time + (was `ValueError` later, PR 3), + - resolving a `NonlinearConstraint` raises `NotImplementedError` (was + `InvalidConstraintError`, PR 3): nonlinear constraints are passed directly to + optimizers and are split off before resolution, so this is an internal error, + not invalid user input, + - post-consolidation error messages cite the originating user constraints. +- **Where changes go:** + - Review edits to PR N (naming, docstrings, structure, behavior-preserving + refactors) -> new commit on branch N. + - Genuine behavior bugs (pre-existing or introduced) -> branch + `constraints-refactor-6-review-fixes`, one commit per fix with a regression + test. Not on the PR branches, so those stay behavior-preserving. + +## Propagation rule (important) + +Every change committed to a branch during review MUST be merged into all later +branches of the stack by rebasing the rest of the stack. It is enough to do this +ONCE at the end of the session (not after every commit): + +```bash +# after all review changes on constraints-refactor--... are committed: +git switch constraints-refactor--... +git rebase constraints-refactor--... +# repeat for each later branch in order, up to and including +# constraints-refactor-6-review-fixes +``` + +After rebasing, rerun the verification (below) on the LAST branch of the stack, not +just the branch under review. Never leave the session with the stack in a state +where a later branch does not contain an earlier branch. + +## Verification commands + +```bash +# fast targeted loop (seconds) +pixi run pytest tests/optimagic/parameters tests/optimagic/test_constraints.py \ + tests/optimagic/test_deprecations.py \ + tests/optimagic/optimization/test_with_constraints.py \ + tests/optimagic/optimization/test_with_advanced_constraints.py \ + tests/estimagic -m "not slow" + +# full fast suite (~4-5 min), strict typing, hooks +pixi run tests-fast +pixi run mypy +pre-commit run --all-files +``` + +All new modules under `src/optimagic/parameters/constraints/` and the rewritten +`space_conversion.py` are strictly type checked (not in the mypy override exempt +list in `pyproject.toml`) — keep it that way. + +## Known context that saves time + +- Legacy dict constraints are deprecated (FutureWarning, removal in 0.6.0). The + adapter in `deprecations.py` (incl. `FixedValueConstraint` and the loc/query + selector closures) exists only to support them and dies with them. +- `InternalParams` and `SpaceConverter` still live in + `parameters/space_conversion.py` on purpose: import paths for estimagic, the + slice plots and `constraint_tools` are unchanged. +- Consolidation order in `consolidation.py::consolidate_constraints` is load + bearing (equality merging -> fix propagation -> cov/sdcorr simplification -> + bound tightening -> replacements -> dedup -> linear bundling -> checks -> spec + assembly). Do not reorder while reviewing for style. +- The tests directory has no `__init__.py` files: test module basenames must be + unique across the whole tests tree. diff --git a/REPORT.md b/REPORT.md new file mode 100644 index 000000000..738c84df5 --- /dev/null +++ b/REPORT.md @@ -0,0 +1,230 @@ +# Report: Algorithm documentation standardization + +Branch `standardize-algo-docs`, 2026-07-04. All ~60 remaining algorithms were migrated +to the documentation system described in +`docs/source/how_to/how_to_document_optimizers.md`. During the migration, every claim in +the old `algorithms.md` text was checked against the optimagic code and the wrapped +libraries' documentation/source. This file records (1) errors in the old documentation +that were corrected, (2) code bugs discovered along the way that were deliberately NOT +fixed (documentation-focused branch), and (3) the one behavioral fix that was applied. + +## 1. Errors in the old documentation (corrected) + +### scipy + +- **Stale constraint-support notes.** The old text said "SLSQP's / COBYLA's / + trust_constr's general nonlinear constraints are not supported yet by optimagic". + All three have `supports_nonlinear_constraints=True` today. The new docstrings + instead explain how optimagic converts equality constraints to inequality pairs + for COBYLA (which natively only handles inequalities). +- **Copy-paste error in `scipy_newton_cg`.** The warning advised refining "an optimum + found with Powell" with another optimizer — it meant an optimum found with + newton_cg. +- **Wrong default for `scipy_direct`.** The old text claimed the volume tolerance + `eps` defaults to 1e-6, "differing from scipy's default 1e-4". The actual optimagic + default is `CONVERGENCE_FTOL_REL = 2e-9`. +- **Nonexistent options documented.** `scipy_truncated_newton` was documented with + `stopping.maxiter`, but the class only has `stopping_maxfun`; `scipy_ls_lm` was + documented with `tr_solver`/`tr_solver_options`, which only exist on the trf and + dogbox least-squares solvers. Dropped. +- Typo `finitie_difference_precision`; several places where optimagic defaults + silently differ from SciPy are now stated explicitly (Nelder-Mead ftol 1e-8 vs + SciPy 1e-4, SLSQP 1e-8 vs 1e-6, BFGS xrtol 1e-5 vs 0, trust_constr xtol 1e-5 vs + 1e-8, dual_annealing maxfun 1e6 vs 1e7, differential_evolution atol 1e-8 vs 0). + +### nlopt + +- **MMA/CCSAQ constraint support was backwards.** The old text said these support + "nonlinear equality (but not inequality) constraints". NLopt supports the + opposite: nonlinear *inequality* (not equality) constraints. In optimagic, MMA + gets equalities via conversion to inequality pairs, and CCSAQ currently exposes no + nonlinear constraints at all (`supports_nonlinear_constraints=False`). +- **Wrong dropdown title.** The section was titled `nlopt_lbfgs`, but the registered + algorithm name is `nlopt_lbfgsb`. +- **`nlopt_tnewton` is not preconditioned.** The old text described it as + "preconditioned truncated Newton"; the wrapped variant is `LD_TNEWTON`, the plain + variant without preconditioning or restarting (those are separate NLopt variants). +- **Wrong DIRECT variant list.** The old text offered a nonexistent "DIRECT_RAND" + and omitted "DIRECT_NOSCAL". The code maps six combinations, and + `random_search=True` only takes effect together with `locally_biased=True` (now + documented on the field). +- **Outdated PRAXIS bound-handling claim.** "Returns infinity when constraints are + violated" no longer matches NLopt's documentation ("bounds emulated by variable + transformation"). The warning that optimagic disables bounds for PRAXIS is kept. +- "Globally convergent" for MMA/CCSAQ clarified per NLopt's own note (convergence to + a *local* optimum from any feasible starting point); removed an unsourced "<10 + past updates" claim for LBFGS storage; numerous typos (Nedler-Mead, alggorithm, + faunction, a `:cite:` role missing its colon). + +### pygmo + +- **Wrong default population size formula.** The old text said "twice the number of + parameters but at least X". The code (`get_population_size`) uses + `10 * (n_params + 1)`, clipped below at the algorithm-specific minimum. +- **`ftol`/`xtol` descriptions swapped** for sade, cmaes and xnes — a known pagmo + documentation bug that had been copied into algorithms.md. Verified against + `cmaes.cpp`/`de.cpp`: ftol is flatness of the population's fitness values, xtol is + flatness in parameter space. +- **Wrong option ranges/defaults** (all verified against the pagmo C++ source): + `pygmo_de.weight_coefficient` documented as range [0, 2], pagmo enforces [0, 1]; + `pygmo_sga.mutation_polynomial_distribution_index` documented "[0, 1], default 1", + pagmo enforces [1, 100]; `pygmo_sga.selection_tournament_size` default is 2, not 1. +- **Undocumented deviations from pygmo defaults** now stated: bee_colony + `max_n_trials` (optimagic 1 vs pygmo 20), mbh + `stopping_max_inner_runs_without_improvement` (30 vs 5), simulated_annealing + end temperature (0.01 vs 0.1, must be below the start temperature). +- Broken `.. note:` directive (single colon); GWO section typos ("usinng", "pased", + "shokingly") and its criticism of the algorithm is now attributed to the pagmo + developers rather than "our opinion"; `batch_evaluator` bullets dropped for gaco + and pso_gen (the classes only have `n_cores`). + +### ipopt + +- **Contradictory `linear_solver` default.** The old text said the default is + "ma27" while its own value list said "mumps (default)". The code default is + `"mumps"`. +- **`acceptable_dual_inf_tol` mismatch.** The old text documented Ipopt's default + 1e+10, but the code passes 1e-10 (see code bugs below). The docstring now + documents the actual behavior and flags the deviation. +- **Mangled bullets un-fused.** `theta_max_fact` had been fused into the + `watchdog_trial_iter_max` bullet; `recalc_y_feas_tol` was mislabeled. +- **Wrong default for `resto_failure_feasibility_threshold`.** Old text said 0; the + wrapper maps `None` to `100 * convergence_ftol_rel`. +- Value typos: "10-05" → 1e-5 (`filter_margin_fact`), "10-06" → 1e-6 (`sigma_min`), + `"bound_mult"` → `"bound-mult"` (the code's Literal), + `adaptive_mu_kkterror_red_iters` labeled float but is an int; word typos + ("lwer", "bunded", "brrier"). Deviations from Ipopt defaults now flagged + (`convergence_ftol_rel` 2e-9 vs Ipopt `tol` 1e-8; `stopping_maxiter` 1,000,000 vs + Ipopt 3000). + +### NAG (DFO-LS, Py-BOBYQA) + +- **`seek_global_optimum` restriction was wrong.** Old text: "Only applies for noisy + criterion functions." Py-BOBYQA's documentation demonstrates it on noise-free + functions; the actual requirement is finite lower and upper bounds. +- **Fast-start default condition inverted.** Old text said "jacobian" is the default + fast-start method "if `len(x) >= number of root contributions`". DFO-LS documents + the opposite (`use_full_rank_interp` defaults to True iff m >= n), i.e. "jacobian" + is default when the number of residuals is at least the number of parameters. +- **Wrong option value in prose.** "trustregion_step" — the value accepted by the + code is `"trustregion"`. +- Stale `:ref:` pointers for options whose default constants no longer live in + `algo_options.py`; descriptions are now inline on the fields. + +### fides + +- **Wrong documented defaults.** `trustregion.subspace_dimension` was documented as + defaulting to "2D"; the optimagic default is `"full"` (the fides package itself + defaults to "2D"). Similarly, optimagic defaults `stepback_strategy` to + `"truncate"` while fides defaults to `"reflect"`. Both deviations are now stated. +- **Nonexistent options dropped.** `trustregion.refine_stepback` and + `trustregion.scaled_gradient_as_possible_stepback` were documented but are not + fields of the class. +- Convergence criteria formulas verified against fides 0.7.4 + (`Optimizer.check_convergence`) and written as math blocks. + +### TAO pounders + +- **Outdated link.** The old text linked github.com/erdc/petsc4py, an unofficial and + outdated mirror; petsc4py is developed as part of PETSc. +- **Wrong disabling semantics.** "Set to False to disable" a tolerance — the fields + are floats and the code disables on any falsy value; correct advice is "set to + zero to disable". +- LaTeX typos (missing backslash on `\epsilon`, `X0` → `X_0`). + +### Own optimizers (bhhh, neldermead_parallel, pounders) + +- **bhhh:** outdated interface description (fun returning a dict with a + "contributions" entry) replaced by the current `@om.mark.likelihood` interface; + "only vaid" typo. +- **neldermead_parallel:** the class docstring was a broken numpydoc stub + documenting nonexistent `criterion`/`x` parameters; `adaptive` adapts the simplex + parameters to the problem *dimension* (Gao–Han), not to "simplex size"; the code's + capping of `n_cores` at `n_params - 1` is now documented. +- **pounders:** `convergence.gtol_rel` default written as "1-8" instead of 1e-8; + `trustregion_expansion_factor_successful` was described as a "Shrinking factor" — + it is an expansion factor; the documented literal "steihaug-toint" is actually + `"steihaug_toint"`; old option names `trustregion_threshold_successful` / + `..._very_successful` are now `trustregion_threshold_acceptance` / + `..._successful` and are documented under the current names; the documented + `batch_evaluator` option no longer exists (only `n_cores`). + +### Other documentation fixes + +- **`nevergrad_wizard` / `nevergrad_portfolio` dropdowns were broken.** They + referenced classes that no longer exist (`NevergradWizard`, `Wizard`, + `NevergradPortfolio`, `Portfolio`, including `from ... import Wizard` usage + examples). The algorithms are registered as `nevergrad_ngopt` and + `nevergrad_meta`; dropdowns renamed and examples rewritten using the string-based + `optimizer` option. +- **`how_to_add_optimizers.ipynb` crashed the docs build.** Its example + `@mark.minimizer` call was missing the now-required `needs_bounds` and + `supports_infinite_bounds` arguments. Also fixed its dead cross-reference + `algo_options_docs` → `algo_options`. +- **tranquilo was not documented anywhere** (no docstrings, no algorithms.md + section). Documented from scratch based on the tranquilo paper (Gabler, Gsell, + Mensinger, Petrosyan 2024) and the tranquilo 0.1.1 source. + +## 2. Code bugs discovered but NOT fixed here + +These surfaced while verifying documentation against the code. They are behavioral +issues and deserve their own PRs. + +- **`BHHH.converence_gtol_abs`** (bhhh.py): the field name is misspelled + ("converence"). Renaming is user-facing (algo_options key), so it needs a + deprecation path. +- **`Ipopt.acceptable_dual_inf_tol = 1e-10`** (ipopt.py): Ipopt's default is 1e+10. + A "desired" threshold of 1e-10 makes the acceptable-point heuristic essentially + impossible to trigger via dual infeasibility — this looks like a sign/typo bug + when the default was transcribed. +- **NAG `noise_n_evals_per_point`** (nag_optimizers.py): annotated + `NonNegativeInt | None`, but `_change_evals_per_point_interface` only works with a + *callable*; passing an int would crash inside the solver. +- **NAG fast-start dict key `"min_inital_points"`** (nag_optimizers.py): the key + validated by `_build_options_dict` is literally misspelled; the correctly spelled + key raises. Additionally, the old constant docstring mentioned + `scale_of_jacobian_singular_value_floor`, which is not an accepted key. +- **`PygmoSga.mutation_strategy`** (pygmo_optimizers.py): the Literal type lacks + `"gaussian"` although the implementation handles it — as typed, the + `mutation_gaussian_width` option is unreachable. +- **`PygmoXnes.population_size`** (pygmo_optimizers.py): typed `float | None`; + should presumably be an int type. +- **`TAOPounders` convergence criteria may be inert** (tao_optimizers.py): the + wrapper always installs a user-defined maxiter convergence test (since + `stopping_maxiter` is never None), which per TAO semantics replaces the default + gradient-tolerance tests — so the documented gatol/grtol/gttol criteria appear to + have no effect through this interface. +- **Tranquilo dead options** (tranquilo.py, verified against tranquilo 0.1.1): + `batch_evaluator` is never read by the wrapper (the internal evaluator is + hardcoded to "joblib"); `stopping_maxtime` is passed but never checked in + tranquilo's main loop; `convergence_min_trust_region_radius` maps to + `ConvOptions.min_radius`, which `_is_converged` never reads; `functype` is + hardcoded to "scalar" by the wrapper. +- **scipy `relative_step_size_diff_approx` is effectively dead** (scipy_optimizers.py, + trf/dogbox): optimagic always passes a Jacobian callable to + `scipy.optimize.least_squares`, so SciPy's internal finite differencing (which + this option controls) is never used. +- **refs.bib duplicate**: `Cartis2018` and `Cartis2018b` are two entries for the same + arXiv paper (1804.00154). Both keys are cited; consolidating requires touching the + citations. + +## 3. Behavioral fix applied in this branch + +**`from __future__ import annotations` silently disabled algo-option coercion.** +`Algorithm.__post_init__` converts/validates option values by looking up each +dataclass field's type in `TYPE_CONVERTERS` (keyed by type objects such as +`PositiveInt`). With the future import — which the documentation how-to guide +*requires* so that autodoc renders type aliases readably — `field.type` is the +annotation *string* (`"PositiveInt"`), so no lookup ever matched and coercion was +silently skipped. This already affected every module that had the import before this +branch (scipy, gfo, nevergrad, pyswarms, pygad, bayesian, pygmo, tranquilo, ...) and +surfaced here as a test failure when ipopt gained the import +(`test_ipopt_algo_options` passes 5.5 for an integer-typed Ipopt option and relies on +coercion to int). + +Fix: `TYPE_CONVERTERS_BY_NAME` in `src/optimagic/type_conversion.py` (string-keyed +companion table) and a two-branch lookup in `Algorithm.__post_init__`. Coercion now +works identically whether or not a module uses the future import. Full fast test +suite passes (3055 passed). If this branch should stay documentation-only, the fix +extracts cleanly into a separate commit/PR — but note the ipopt options test fails +without it. diff --git a/docs/source/algorithms.md b/docs/source/algorithms.md index c8c379b25..c4c687424 100644 --- a/docs/source/algorithms.md +++ b/docs/source/algorithms.md @@ -45,775 +45,576 @@ automatically installed when you install optimagic. ```{eval-rst} .. dropdown:: scipy_slsqp - .. code-block:: - - "scipy_slsqp" + **How to use this algorithm:** - Minimize a scalar function of one or more variables using the SLSQP algorithm. + .. code-block:: python - SLSQP stands for Sequential Least Squares Programming. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_slsqp(stopping_maxiter=1_000, ...), + ) - SLSQP is a line search algorithm. It is well suited for continuously - differentiable scalar optimization problems with up to several hundred parameters. + or using the string interface: - The optimizer is taken from scipy which wraps the SLSQP optimization subroutine - originally implemented by :cite:`Kraft1988`. + .. code-block:: python - .. note:: - SLSQP's general nonlinear constraints are not supported yet by optimagic. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_slsqp", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - - **convergence.ftol_abs** (float): Precision goal for the value of - f in the stopping criterion. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **display** (bool): Set to True to print convergence messages. Default is False. Scipy name: **disp**. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipySLSQP ``` ```{eval-rst} .. dropdown:: scipy_neldermead - .. code-block:: + **How to use this algorithm:** - "scipy_neldermead" - - Minimize a scalar function using the Nelder-Mead algorithm. - - The Nelder-Mead algorithm is a direct search method (based on function comparison) - and is often applied to nonlinear optimization problems for which derivatives are - not known. - Unlike most modern optimization methods, the Nelder–Mead heuristic can converge to - a non-stationary point, unless the problem satisfies stronger conditions than are - necessary for modern methods. - - Nelder-Mead is never the best algorithm to solve a problem but rarely the worst. - Its popularity is likely due to historic reasons and much larger than its - properties warrant. - - The argument `initial_simplex` is not supported by optimagic as it is not - compatible with optimagic's handling of constraints. - - - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, - but we do not count this as convergence. - - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, - the optimization stops but we do not count this as convergence. - - **convergence.xtol_abs** (float): Absolute difference in parameters between iterations - that is tolerated to declare convergence. As no relative tolerances can be passed to Nelder-Mead, - optimagic sets a non zero default for this. - - **convergence.ftol_abs** (float): Absolute difference in the criterion value between - iterations that is tolerated to declare convergence. As no relative tolerances can be passed to Nelder-Mead, - optimagic sets a non zero default for this. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. - - **adaptive** (bool): Adapt algorithm parameters to dimensionality of problem. - Useful for high-dimensional minimization (:cite:`Gao2012`, p. 259-277). scipy's default is False. + .. code-block:: python -``` + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_neldermead(stopping_maxiter=1_000, ...), + ) -```{eval-rst} -.. dropdown:: scipy_powell + or using the string interface: - .. code-block:: + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_neldermead", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - "scipy_powell" + **Description and available options:** - Minimize a scalar function using the modified Powell method. + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyNelderMead +``` - .. warning:: - In our benchmark using a quadratic objective function, the Powell algorithm - did not find the optimum very precisely (less than 4 decimal places). - If you require high precision, you should refine an optimum found with Powell - with another local optimizer. +```{eval-rst} +.. dropdown:: scipy_powell - The criterion function need not be differentiable. + **How to use this algorithm:** - Powell's method is a conjugate direction method, minimizing the function by a - bi-directional search in each parameter's dimension. + .. code-block:: python - The argument ``direc``, which is the initial set of direction vectors and which - is part of the scipy interface is not supported by optimagic because it is - incompatible with how optimagic handles constraints. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_powell(stopping_maxiter=1_000, ...), + ) - - **convergence.xtol_rel (float)**: Stop when the relative movement between parameter - vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative improvement between two - iterations is smaller than this. More formally, this is expressed as + or using the string interface: - .. math:: + .. code-block:: python - \frac{(f^k - f^{k+1})}{\\max{{\{|f^k|, |f^{k+1}|, 1\}}}} \leq - \text{relative_criterion_tolerance} + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_powell", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, - the optimization stops but we do not count thisas convergence. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, - but we do not count this as convergence. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyPowell ``` ```{eval-rst} .. dropdown:: scipy_bfgs - .. code-block:: - - "scipy_bfgs" + **How to use this algorithm:** - Minimize a scalar function of one or more variables using the BFGS algorithm. + .. code-block:: python - BFGS stands for Broyden-Fletcher-Goldfarb-Shanno algorithm. It is a quasi-Newton - method that can be used for solving unconstrained nonlinear optimization problems. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_bfgs(stopping_maxiter=1_000, ...), + ) - BFGS is not guaranteed to converge unless the function has a quadratic Taylor - expansion near an optimum. However, BFGS can have acceptable performance even - for non-smooth optimization instances. + or using the string interface: - - **convergence.gtol_abs** (float): Stop if all elements of the gradient are smaller than this. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, - but we do not count this as convergence. - - **norm** (float): Order of the vector norm that is used to calculate the gradient's "score" that - is compared to the gradient tolerance to determine convergence. Default is infinite which means that - the largest entry of the gradient vector is compared to the gradient tolerance. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. - - **convergence_xtol_rel** (float): Relative tolerance for `x`. Terminate successfully if step size is less than `xk * xrtol` where `xk` is the current parameter vector. Default is 1e-5. SciPy name: **xrtol**. - - **armijo_condition** (float): Parameter for Armijo condition rule. Default is 1e-4. Ensures + .. code-block:: python - .. math:: + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_bfgs", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - f(x_k+\alpha p_k) \le f(x_k) \;+\mathrm{armijo\_condition}\,\cdot\,\alpha\,\nabla f(x_k)^\top p_k, - - so each step yields at least a fraction **armijo_condition** of the predicted decrease. Smaller ⇒ more aggressive steps, larger ⇒ more conservative ones. SciPy name: **c1**. - - **curvature_condition** (float): Parameter for curvature condition rule. Default is 0.9. Ensures - - .. math:: + **Description and available options:** - \nabla f(x_k+\alpha p_k)^\top p_k \ge \mathrm{curvature\_condition}\,\cdot\,\nabla f(x_k)^\top p_k, - - so the new slope isn’t too negative. Smaller ⇒ stricter curvature reduction (smaller steps), larger ⇒ looser (bigger steps). SciPy name: **c2**. + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyBFGS ``` ```{eval-rst} .. dropdown:: scipy_conjugate_gradient - .. code-block:: - - "scipy_conjugate_gradient" + **How to use this algorithm:** - Minimize a function using a nonlinear conjugate gradient algorithm. + .. code-block:: python - The conjugate gradient method finds functions' local optima using just the gradient. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_conjugate_gradient(stopping_maxiter=1_000, ...), + ) - This conjugate gradient algorithm is based on that of Polak and Ribiere, detailed - in :cite:`Nocedal2006`, pp. 120-122. + or using the string interface: - Conjugate gradient methods tend to work better when: + .. code-block:: python - - the criterion has a unique global minimizing point, and no local minima or - other stationary points. - - the criterion is, at least locally, reasonably well approximated by a - quadratic function. - - the criterion is continuous and has a continuous gradient. - - the gradient is not too large, e.g., has a norm less than 1000. - - The initial guess is reasonably close to the criterion's global minimizer. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_conjugate_gradient", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - - **convergence.gtol_abs** (float): Stop if all elements of the - gradient are smaller than this. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **norm** (float): Order of the vector norm that is used to calculate the gradient's - "score" that is compared to the gradient tolerance to determine convergence. - Default is infinite which means that the largest entry of the gradient vector - is compared to the gradient tolerance. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyConjugateGradient ``` ```{eval-rst} .. dropdown:: scipy_newton_cg - .. code-block:: - - "scipy_newton_cg" - - Minimize a scalar function using Newton's conjugate gradient algorithm. - - .. warning:: - In our benchmark using a quadratic objective function, the truncated newton - algorithm did not find the optimum very precisely (less than 4 decimal places). - If you require high precision, you should refine an optimum found with Powell - with another local optimizer. - - Newton's conjugate gradient algorithm uses an approximation of the Hessian to find - the minimum of a function. It is practical for small and large problems - (see :cite:`Nocedal2006`, p. 140). - - Newton-CG methods are also called truncated Newton methods. This function differs - scipy_truncated_newton because - - - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy - and scipy while ``scipy_truncated_newton``'s algorithm calls a C function. + **How to use this algorithm:** - - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization - while ``scipy_truncated_newton``'s algorithm supports bounds. + .. code-block:: python - Conjugate gradient methods tend to work better when: + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_newton_cg(stopping_maxiter=1_000, ...), + ) - - the criterion has a unique global minimizing point, and no local minima or - other stationary points. - - the criterion is, at least locally, reasonably well approximated by a - quadratic function. - - the criterion is continuous and has a continuous gradient. - - the gradient is not too large, e.g., has a norm less than 1000. - - The initial guess is reasonably close to the criterion's global minimizer. + or using the string interface: - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. Newton CG uses the average - relative change in the parameters for determining the convergence. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + .. code-block:: python + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_newton_cg", + algo_options={"stopping_maxiter": 1_000, ...}, + ) + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyNewtonCG ``` ```{eval-rst} .. dropdown:: scipy_cobyla - .. code-block:: - - "scipy_cobyla" + **How to use this algorithm:** - Minimize a scalar function of one or more variables using the COBYLA algorithm. + .. code-block:: python - COBYLA stands for Constrained Optimization By Linear Approximation. - It is derivative-free and supports nonlinear inequality and equality constraints. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_cobyla(stopping_maxiter=1_000, ...), + ) - .. note:: - Cobyla's general nonlinear constraints is not supported yet by optimagic. + or using the string interface: - Scipy's implementation wraps the FORTRAN implementation of the algorithm. + .. code-block:: python - For more information on COBYLA see :cite:`Powell1994`, :cite:`Powell1998` and - :cite:`Powell2007`. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_cobyla", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. In case of COBYLA this is - a lower bound on the size of the trust region and can be seen as the - required accuracy in the variables but this accuracy is not guaranteed. - - **trustregion.initial_radius** (float): Initial value of the trust region radius. - Since a linear approximation is likely only good near the current simplex, - the linear program is given the further requirement that the solution, - which will become the next evaluation point must be within a radius - RHO_j from x_j. RHO_j only decreases, never increases. The initial RHO_j is - the `trustregion.initial_radius`. In this way COBYLA's iterations behave - like a trust region algorithm. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyCOBYLA ``` ```{eval-rst} .. dropdown:: scipy_truncated_newton - .. code-block:: + **How to use this algorithm:** - "scipy_truncated_newton" - - Minimize a scalar function using truncated Newton algorithm. - - This function differs from scipy_newton_cg because - - - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy - and scipy while ``scipy_truncated_newton``'s algorithm calls a C function. - - - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization - while ``scipy_truncated_newton``'s algorithm supports bounds. - - Conjugate gradient methods tend to work better when: - - - the criterion has a unique global minimizing point, and no local minima or - other stationary points. - - the criterion is, at least locally, reasonably well approximated by a - quadratic function. - - the criterion is continuous and has a continuous gradient. - - the gradient is not too large, e.g., has a norm less than 1000. - - The initial guess is reasonably close to the criterion's global minimizer. - - optimagic does not support the ``scale`` nor ``offset`` argument as they are not - compatible with the way optimagic handles constraints. It also does not support - ``messg_num`` which is an additional way to control the verbosity of the optimizer. - - - **func_min_estimate** (float): Minimum function value estimate. Defaults to 0. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this as - convergence. - - **convergence.xtol_abs** (float): Absolute difference in parameters - between iterations after scaling that is tolerated to declare convergence. - - **convergence.ftol_abs** (float): Absolute difference in the - criterion value between iterations after scaling that is tolerated - to declare convergence. - - **convergence.gtol_abs** (float): Stop if the value of the - projected gradient (after applying x scaling factors) is smaller than this. - If convergence.gtol_abs < 0.0, - convergence.gtol_abs is set to - 1e-2 * sqrt(accuracy). - - **max_hess_evaluations_per_iteration** (int): Maximum number of hessian*vector - evaluations per main iteration. If ``max_hess_evaluations == 0``, the - direction chosen is ``- gradient``. If ``max_hess_evaluations < 0``, - ``max_hess_evaluations`` is set to ``max(1,min(50,n/2))`` where n is the - length of the parameter vector. This is also the default. - - **max_step_for_line_search** (float): Maximum step for the line search. - It may be increased during the optimization. If too small, it will be set - to 10.0. By default we use scipy's default. - - **line_search_severity** (float): Severity of the line search. If < 0 or > 1, - set to 0.25. optimagic defaults to scipy's default. - - **finitie_difference_precision** (float): Relative precision for finite difference - calculations. If <= machine_precision, set to sqrt(machine_precision). - optimagic defaults to scipy's default. - - **criterion_rescale_factor** (float): Scaling factor (in log10) used to trigger - criterion rescaling. If 0, rescale at each iteration. If a large value, - never rescale. If < 0, rescale is set to 1.3. optimagic defaults to scipy's - default. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + .. code-block:: python + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_truncated_newton(stopping_maxfun=100_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_truncated_newton", + algo_options={"stopping_maxfun": 100_000, ...}, + ) + + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyTruncatedNewton ``` ```{eval-rst} .. dropdown:: scipy_trust_constr - .. code-block:: + **How to use this algorithm:** - "scipy_trust_constr" + .. code-block:: python - Minimize a scalar function of one or more variables subject to constraints. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_trust_constr(stopping_maxiter=1_000, ...), + ) - .. warning:: - In our benchmark using a quadratic objective function, the trust_constr - algorithm did not find the optimum very precisely (less than 4 decimal places). - If you require high precision, you should refine an optimum found with trust_constr - with another local optimizer. + or using the string interface: - .. note:: - Its general nonlinear constraints' handling is not supported yet by optimagic. - - It switches between two implementations depending on the problem definition. - It is the most versatile constrained minimization algorithm - implemented in SciPy and the most appropriate for large-scale problems. - For equality constrained problems it is an implementation of Byrd-Omojokun - Trust-Region SQP method described in :cite:`Lalee1998` and in :cite:`Conn2000`, - p. 549. When inequality constraints are imposed as well, it switches to the - trust-region interior point method described in :cite:`Byrd1999`. - This interior point algorithm in turn, solves inequality constraints by - introducing slack variables and solving a sequence of equality-constrained - barrier problems for progressively smaller values of the barrier parameter. - The previously described equality constrained SQP method is - used to solve the subproblems with increasing levels of accuracy - as the iterate gets closer to a solution. - - It approximates the Hessian using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) - Hessian update strategy. - - - **convergence.gtol_abs** (float): Tolerance for termination - by the norm of the Lagrangian gradient. The algorithm will terminate - when both the infinity norm (i.e., max abs value) of the Lagrangian - gradient and the constraint violation are smaller than the - convergence.gtol_abs. - For this algorithm we use scipy's gradient tolerance for trust_constr. - This smaller tolerance is needed for the sum of squares tests to pass. - - **stopping.maxiter** (int): If the maximum number of iterations is reached, - the optimization stops, but we do not count this as convergence. - - **convergence.xtol_rel** (float): Tolerance for termination by - the change of the independent variable. The algorithm will terminate when - the radius of the trust region used in the algorithm is smaller than the - convergence.xtol_rel. - - **trustregion.initial_radius** (float): Initial value of the trust region radius. - The trust radius gives the maximum distance between solution points in - consecutive iterations. It reflects the trust the algorithm puts in the - local approximation of the optimization problem. For an accurate local - approximation the trust-region should be large and for an approximation - valid only close to the current point it should be a small one. - The trust radius is automatically updated throughout the optimization - process, with ``trustregion_initial_radius`` being its initial value. - - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_trust_constr", + algo_options={"stopping_maxiter": 1_000, ...}, + ) + + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyTrustConstr ``` ```{eval-rst} .. dropdown:: scipy_ls_dogbox - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + + + @om.mark.least_squares + def fun(x): + return x + + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_ls_dogbox(stopping_maxfun=1_000, ...), + ) + + or using the string interface: - "scipy_ls_dogbox" - - Minimize a nonlinear least squares problem using a rectangular trust region method. - - Typical use case is small problems with bounds. Not recommended for problems with - rank-deficient Jacobian. - - The algorithm supports the following options: - - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is below this. - - **convergence.gtol_rel** (float): Stop when the gradient, - divided by the absolute value of the criterion function is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this as - convergence. - - **tr_solver** (str): Method for solving trust-region subproblems, relevant only - for 'trf' and 'dogbox' methods. - - - 'exact' is suitable for not very large problems with dense - Jacobian matrices. The computational complexity per iteration is - comparable to a singular value decomposition of the Jacobian - matrix. - - 'lsmr' is suitable for problems with sparse and large Jacobian - matrices. It uses the iterative procedure - `scipy.sparse.linalg.lsmr` for finding a solution of a linear - least-squares problem and only requires matrix-vector product - evaluations. - If None (default), the solver is chosen based on the type of Jacobian - returned on the first iteration. - - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - - - ``tr_solver='exact'``: `tr_options` are ignored. - - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. + .. code-block:: python + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="scipy_ls_dogbox", + algo_options={"stopping_maxfun": 1_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyLSDogbox ``` ```{eval-rst} .. dropdown:: scipy_ls_trf - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "scipy_ls_trf" - - Minimize a nonlinear least squares problem using a trustregion reflective method. - - Trust Region Reflective algorithm, particularly suitable for large sparse problems - with bounds. Generally robust method. - - The algorithm supports the following options: - - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is below this. - - **convergence.gtol_rel** (float): Stop when the gradient, - divided by the absolute value of the criterion function is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this as - convergence. - - **tr_solver** (str): Method for solving trust-region subproblems, relevant only - for 'trf' and 'dogbox' methods. - - - 'exact' is suitable for not very large problems with dense - Jacobian matrices. The computational complexity per iteration is - comparable to a singular value decomposition of the Jacobian - matrix. - - 'lsmr' is suitable for problems with sparse and large Jacobian - matrices. It uses the iterative procedure - `scipy.sparse.linalg.lsmr` for finding a solution of a linear - least-squares problem and only requires matrix-vector product - evaluations. - If None (default), the solver is chosen based on the type of Jacobian - returned on the first iteration. - - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - - - ``tr_solver='exact'``: `tr_options` are ignored. - - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. + import optimagic as om + + + @om.mark.least_squares + def fun(x): + return x + + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_ls_trf(stopping_maxfun=1_000, ...), + ) + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="scipy_ls_trf", + algo_options={"stopping_maxfun": 1_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyLSTRF ``` ```{eval-rst} .. dropdown:: scipy_ls_lm - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om - "scipy_ls_lm" - - Minimize a nonlinear least squares problem using a Levenberg-Marquardt method. - - Does not handle bounds and sparse Jacobians. Usually the most efficient method for - small unconstrained problems. - - The algorithm supports the following options: - - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is below this. - - **convergence.gtol_rel** (float): Stop when the gradient, - divided by the absolute value of the criterion function is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this as - convergence. - - **tr_solver** (str): Method for solving trust-region subproblems, relevant only - for 'trf' and 'dogbox' methods. - - - 'exact' is suitable for not very large problems with dense - Jacobian matrices. The computational complexity per iteration is - comparable to a singular value decomposition of the Jacobian - matrix. - - 'lsmr' is suitable for problems with sparse and large Jacobian - matrices. It uses the iterative procedure - `scipy.sparse.linalg.lsmr` for finding a solution of a linear - least-squares problem and only requires matrix-vector product - evaluations. - If None (default), the solver is chosen based on the type of Jacobian - returned on the first iteration. - - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - - - ``tr_solver='exact'``: `tr_options` are ignored. - - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. + @om.mark.least_squares + def fun(x): + return x + + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_ls_lm(stopping_maxfun=1_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="scipy_ls_lm", + algo_options={"stopping_maxfun": 1_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyLSLM ``` ```{eval-rst} .. dropdown:: scipy_basinhopping - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_basinhopping(n_local_optimizations=10, ...), + ) - "scipy_basinhopping" - - Find the global minimum of a function using the basin-hopping algorithm which combines a global stepping algorithm with local minimization at each step. - - Basin-hopping is a two-phase method that combines a global stepping algorithm with local minimization at each step. Designed to mimic the natural process of energy minimization of clusters of atoms, it works well for similar problems with “funnel-like, but rugged” energy landscapes. - - This is mainly supported for completeness. Consider optimagic's built in multistart - optimization for a similar approach that can run multiple optimizations in parallel, - supports all local algorithms in optimagic (as opposed to just those from scipy) - and allows for a better visualization of the multistart history. - - When provided the derivative is passed to the local minimization method. - - The algorithm supports the following options: - - - **local_algorithm** (str/callable): Any scipy local minimizer: valid options are. - "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA". - "SLSQP". "trust-constr". "dogleg". "trust-ncg". "trust-exact". "trust-krylov". - or a custom function for local minimization, default is "L-BFGS-B". - - **n_local_optimizations**: (int) The number local optimizations. Default is 100 as - in scipy's default. - - **temperature**: (float) Controls the randomness in the optimization process. - Higher the temperatures the larger jumps in function value will be accepted. - Default is 1.0 as in scipy's default. - - **stepsize**: (float) Maximum step size. Default is 0.5 as in scipy's default. - - **local_algo_options**: (dict) Additional keyword arguments for the local - minimizer. Check the documentation of the local scipy algorithms for details on - what is supported. - - **take_step**: (callable) Replaces the default step-taking routine. Default is - None as in scipy's default. - - **accept_test**: (callable) Define a test to judge the acception of steps. Default - is None as in scipy's default. - - **interval**: (int) Determined how often the step size is updated. Default is 50 - as in scipy's default. - - **convergence.n_unchanged_iterations**: (int) Number of iterations the global - minimum estimate stays the same to stops the algorithm. Default is None as in - scipy's default. - - **seed**: (None, int, numpy.random.Generator,numpy.random.RandomState)Default is - None as in scipy's default. - - **target_accept_rate**: (float) Adjusts the step size. Default is 0.5 as in scipy's default. - - **stepwise_factor**: (float) Step size multiplier upon each step. Lies between (0,1), default is 0.9 as in scipy's default. + or using the string interface: + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_basinhopping", + algo_options={"n_local_optimizations": 10, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyBasinhopping ``` ```{eval-rst} .. dropdown:: scipy_brute - .. code-block:: - - "scipy_brute" + **How to use this algorithm:** - Find the global minimum of a fuction over a given range by brute force. + .. code-block:: python - Brute force evaluates the criterion at each point and that is why better suited for problems with very few parameters. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_brute(n_grid_points=50, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - The start values are not actually used because the grid is only defined by bounds. - It is still necessary for optimagic to infer the number and format of the - parameters. + or using the string interface: - Due to the parallelization, this algorithm cannot collect a history of parameters - and criterion evaluations. + .. code-block:: python - The algorithm supports the following options: + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_brute", + algo_options={"n_grid_points": 50, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - - **n_grid_points** (int): the number of grid points to use for the brute force - search. Default is 20 as in scipy. - - **polishing_function** (callable): Function to seek a more precise minimum near - brute-force' best gridpoint taking brute-force's result at initial guess as a - positional argument. Default is None providing no polishing. - - **n_cores** (int): The number of cores on which the function is evaluated in - parallel. Default 1. - - **batch_evaluator** (str or callable). An optimagic batch evaluator. Default - 'joblib'. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyBrute ``` ```{eval-rst} .. dropdown:: scipy_differential_evolution - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_differential_evolution( + population_size_multiplier=10, ... + ), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - "scipy_differential_evolution" - - Find the global minimum of a multivariate function using differential evolution (DE). DE is a gradient-free method. - - Due to optimagic's general parameter format the integrality and vectorized - arguments are not supported. - - The algorithm supports the following options: - - - **strategy** (str): Measure of quality to improve a candidate solution, can be one - of the following keywords (default 'best1bin'.) - - ‘best1bin’ - - ‘best1exp’ - - ‘rand1exp’ - - ‘randtobest1exp’ - - ‘currenttobest1exp’ - - ‘best2exp’ - - ‘rand2exp’ - - ‘randtobest1bin’ - - ‘currenttobest1bin’ - - ‘best2bin’ - - ‘rand2bin’ - - ‘rand1bin’ - - - **stopping.maxiter** (int): The maximum number of criterion evaluations - without polishing is(stopping.maxiter + 1) * population_size * number of - parameters - - **population_size_multiplier** (int): A multiplier setting the population size. - The number of individuals in the population is population_size * number of - parameters. The default 15. - - **convergence.ftol_rel** (float): Default 0.01. - - **mutation_constant** (float/tuple): The differential weight denoted by F in - literature. Should be within 0 and 2. The tuple form is used to specify - (min, max) dithering which can help speed convergence. Default is (0.5, 1). - - **recombination_constant** (float): The crossover probability or CR in the - literature determines the probability that two solution vectors will be combined - to produce a new solution vector. Should be between 0 and 1. The default is 0.7. - - **seed** (int): DE is stochastic. Define a seed for reproducability. - - **polish** (bool): Uses scipy's L-BFGS-B for unconstrained problems and - trust-constr for constrained problems to slightly improve the minimization. - Default is True. - - **sampling_method** (str/np.array): Specify the sampling method for the initial - population. It can be one of the following options - - "latinhypercube" - - "sobol" - - "halton" - - "random" - - an array specifying the initial population of shape (total population size, - number of parameters). The initial population is clipped to bounds before use. - Default is 'latinhypercube' - - - **convergence.ftol_abs** (float): - CONVERGENCE_SECOND_BEST_ABSOLUTE_CRITERION_TOLERANCE - - **n_cores** (int): The number of cores on which the function is evaluated in - parallel. Default 1. - - **batch_evaluator** (str or callable). An optimagic batch evaluator. Default - 'joblib'. + or using the string interface: + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_differential_evolution", + algo_options={"population_size_multiplier": 10, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyDifferentialEvolution ``` ```{eval-rst} .. dropdown:: scipy_shgo - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "scipy_shgo" - - Find the global minimum of a fuction using simplicial homology global optimization. - - The algorithm supports the following options: - - - **local_algorithm** (str): The local optimization algorithm to be used. Only - COBYLA and SLSQP supports constraints. Valid options are - "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA". - "SLSQP". "trust-constr". "dogleg". "trust-ncg". "trust-exact". "trust-krylov" - or a custom function for local minimization, default is "L-BFGS-B". - - **local_algo_options**: (dict) Additional keyword arguments for the local - minimizer. Check the documentation of the local scipy algorithms for details on - what is supported. - - **n_sampling_points** (int): Specify the number of sampling points to construct - the simplical complex. - - **n_simplex_iterations** (int): Number of iterations to construct the simplical - complex. Default is 1 as in scipy. - - **sampling_method** (str/callable): The method to use for sampling the search - space. Default 'simplicial'. - - **max_sampling_evaluations** (int): The maximum number of evaluations of the - criterion function in the sampling phase. - - **convergence.minimum_criterion_value** (float): Specify the global minimum when - it is known. Default is - np.inf. For maximization problems, flip the sign. - - **convergence.minimum_criterion_tolerance** (float): Specify the relative error - between the current best minimum and the supplied global criterion_minimum - allowed. Default is scipy's default, 1e-4. - - **stopping.maxiter** (int): The maximum number of iterations. - - **stopping.maxfun** (int): The maximum number of criterion - evaluations. - - **stopping.max_processing_time** (int): The maximum time allowed for the - optimization. - - **minimum_homology_group_rank_differential** (int): The minimum difference in the - rank of the homology group between iterations. - - **symmetry** (bool): Specify whether the criterion contains symetric variables. - - **minimize_every_iteration** (bool): Specify whether the gloabal sampling points - are passed to the local algorithm in every iteration. - - **max_local_minimizations_per_iteration** (int): The maximum number of local - optimizations per iteration. Default False, i.e. no limit. - - **infinity_constraints** (bool): Specify whether to save the sampling points - outside the feasible domain. Default is True. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_shgo(n_sampling_points=256, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_shgo", + algo_options={"n_sampling_points": 256, ...}, + ) + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipySHGO ``` ```{eval-rst} .. dropdown:: scipy_dual_annealing - .. code-block:: + **How to use this algorithm:** - "scipy_dual_annealing" - - Find the global minimum of a function using dual annealing for continuous variables. - - The algorithm supports the following options: - - - **stopping.maxiter** (int): Specify the maximum number of global searh - iterations. - - **local_algorithm** (str): The local optimization algorithm to be used. valid - options are: "Nelder-Mead", "Powell", "CG", "BFGS", "Newton-CG", "L-BFGS-B", - "TNC", "COBYLA", "SLSQP", "trust-constr", "dogleg", "trust-ncg", "trust-exact", - "trust-krylov", Default "L-BFGS-B". - - **local_algo_options**: (dict) Additional keyword arguments for the local - minimizer. Check the documentation of the local scipy algorithms for details on - what is supported. - - **initial_temperature** (float): The temparature algorithm starts with. The higher values lead to a wider search space. The range is (0.01, 5.e4] and default is 5230.0. - - **restart_temperature_ratio** (float): Reanneling starts when the algorithm is decreased to initial_temperature * restart_temperature_ratio. Default is 2e-05. - - **visit** (float): Specify the thickness of visiting distribution's tails. Range is (1, 3] and default is scipy's default, 2.62. - - **accept** (float): Controls the probability of acceptance. Range is (-1e4, -5] and default is scipy's default, -5.0. Smaller values lead to lower acceptance probability. - - **stopping.maxfun** (int): soft limit for the number of criterion evaluations. - - **seed** (int, None or RNG): Dual annealing is a stochastic process. Seed or - random number generator. Default None. - - **no_local_search** (bool): Specify whether to apply a traditional Generalized Simulated Annealing with no local search. Default is False. + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_dual_annealing(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_dual_annealing", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyDualAnnealing ``` ```{eval-rst} .. dropdown:: scipy_direct - .. code-block:: + **How to use this algorithm:** - "scipy_direct" + .. code-block:: python - Find the global minimum of a function using dividing rectangles method. It is not necessary to provide an initial guess. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.scipy_direct(stopping_maxfun=10_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - The algorithm supports the following options: + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="scipy_direct", + algo_options={"stopping_maxfun": 10_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - - **eps** (float): Specify the minimum difference of the criterion values between the current best hyperrectangle and the next potentially best hyperrectangle to be divided determining the trade off between global and local search. Default is 1e-6 differing from scipy's default 1e-4. - - **stopping.maxfun** (int/None): Maximum number of criterion evaluations allowed. Default is None which caps the number of evaluations at 1000 * number of dimentions automatically. - - **stopping.maxiter** (int): Maximum number of iterations allowed. - - **locally_biased** (bool): Determine whether to use the locally biased variant of the algorithm DIRECT_L. Default is True. - - **convergence.minimum_criterion_value** (float): Specify the global minimum when it is known. Default is minus infinity. For maximization problems, flip the sign. - - **convergence.minimum_criterion_tolerance** (float): Specify the relative error between the current best minimum and the supplied global criterion_minimum allowed. Default is scipy's default, 1e-4. - - **volume_hyperrectangle_tolerance** (float): Specify the smallest volume of the hyperrectangle containing the lowest criterion value allowed. Range is (0,1). Default is 1e-16. - - **length_hyperrectangle_tolerance** (float): Depending on locally_biased it can refer to normalized side (True) or diagonal (False) length of the hyperrectangle containing the lowest criterion value. Range is (0,1). Default is scipy's default, 1e-6. + **Description and available options:** + .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyDirect ``` (own-algorithms)= @@ -823,168 +624,107 @@ automatically installed when you install optimagic. We implement a few algorithms from scratch. They are currently considered experimental. ```{eval-rst} -.. dropdown:: bhhh +.. dropdown:: bhhh + + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om - .. code-block:: - "bhhh" + @om.mark.likelihood + def fun(x): + return x**2 - Minimize a likelihood function using the BHHH algorithm. - BHHH (:cite:`Berndt1974`) can - and should ONLY - be used for minimizing - (or maximizing) a likelihood. It is similar to the Newton-Raphson - algorithm, but replaces the Hessian matrix with the outer product of the - gradient. This approximation is based on the information matrix equality - (:cite:`Halbert1982`) and is thus only vaid when minimizing (or maximizing) - a likelihood. + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.bhhh(stopping_maxiter=1_000), + ) - The criterion function :func:`func` should return a dictionary with - at least the entry ``{"contributions": array_or_pytree}`` where ``array_or_pytree`` - contains the likelihood contributions of each individual. + or using the string interface: - bhhh supports the following options: + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="bhhh", + algo_options={"stopping_maxiter": 1_000}, + ) - - **convergence.gtol_abs** (float): Stopping criterion for the - gradient tolerance. Default is 1e-8. - - **stopping.maxiter** (int): Maximum number of iterations. - If reached, terminate. Default is 200. + **Description and available options:** + .. autoclass:: optimagic.optimizers.bhhh.BHHH ``` ```{eval-rst} -.. dropdown:: neldermead_parallel +.. dropdown:: neldermead_parallel - .. code-block:: + **How to use this algorithm:** - "neldermead_parallel" + .. code-block:: python - Minimize a function using the neldermead_parallel algorithm. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.neldermead_parallel(n_cores=2), + ) - This is a parallel Nelder-Mead algorithm following Lee D., Wiswall M., A parallel - implementation of the simplex function minimization routine, - Computational Economics, 2007. + or using the string interface: - The algorithm was implemented by Jacek Barszczewski + .. code-block:: python - The algorithm supports the following options: + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="neldermead_parallel", + algo_options={"n_cores": 2}, + ) - - **init_simplex_method** (string or callable): Name of the method to create initial - simplex or callable which takes as an argument initial value of parameters - and returns initial simplex as j+1 x j array, where j is length of x. - The default is "gao_han". - - **n_cores** (int): Degree of parallization. The default is 1 (no parallelization). + **Description and available options:** - - **adaptive** (bool): Adjust parameters of Nelder-Mead algorithm to account - for simplex size. The default is True. + .. autoclass:: optimagic.optimizers.neldermead.NelderMeadParallel +``` - - **stopping.maxiter** (int): Maximum number of algorithm iterations. - The default is STOPPING_MAX_ITERATIONS. +```{eval-rst} +.. dropdown:: pounders - - **convergence.ftol_abs** (float): maximal difference between - function value evaluated on simplex points. - The default is CONVERGENCE_SECOND_BEST_ABSOLUTE_CRITERION_TOLERANCE. + **How to use this algorithm:** - - **convergence.xtol_abs** (float): maximal distance between points - in the simplex. The default is CONVERGENCE_SECOND_BEST_ABSOLUTE_PARAMS_TOLERANCE. + .. code-block:: python - - **batch_evaluator** (string or callable): See :ref:`batch_evaluators` for - details. Default "joblib". + import optimagic as om -``` -```{eval-rst} -.. dropdown:: pounders + @om.mark.least_squares + def fun(x): + return x - .. code-block:: - "pounders" - - Minimize a function using the POUNDERS algorithm. - - POUNDERs (:cite:`Benson2017`, :cite:`Wild2015`, `GitHub repository - `_) - - can be a useful tool for economists who estimate structural models using - indirect inference, because unlike commonly used algorithms such as Nelder-Mead, - POUNDERs is tailored for minimizing a non-linear sum of squares objective function, - and therefore may require fewer iterations to arrive at a local optimum than - Nelder-Mead. - - Scaling the problem is necessary such that bounds correspond to the unit hypercube - :math:`[0, 1]^n`. For unconstrained problems, scale each parameter such that unit - changes in parameters result in similar order-of-magnitude changes in the criterion - value(s). - - pounders supports the following options: - - - - **convergence.gtol_abs**: Convergence tolerance for the - absolute gradient norm. Stop if norm of the gradient is less than this. - Default is 1e-8. - - **convergence.gtol_rel**: Convergence tolerance for the - relative gradient norm. Stop if norm of the gradient relative to the criterion - value is less than this. Default is 1-8. - - **convergence.gtol_scaled**: Convergence tolerance for the - scaled gradient norm. Stop if norm of the gradient divided by norm of the - gradient at the initial parameters is less than this. - Disabled, i.e. set to False, by default. - - **max_interpolation_points** (int): Maximum number of interpolation points. - Default is `2 * n + 1`, where `n` is the length of the parameter vector. - - **stopping.maxiter** (int): Maximum number of iterations. - If reached, terminate. Default is 2000. - - **trustregion_initial_radius (float)**: Delta, initial trust-region radius. - 0.1 by default. - - **trustregion_minimal_radius** (float): Minimal trust-region radius. - 1e-6 by default. - - **trustregion_maximal_radius** (float): Maximal trust-region radius. - 1e6 by default. - - **trustregion_shrinking_factor_not_successful** (float): Shrinking factor of - the trust-region radius in case the solution vector of the suproblem - is not accepted, but the model is fully linear (i.e. "valid"). - Defualt is 0.5. - - **trustregion_expansion_factor_successful** (float): Shrinking factor of - the trust-region radius in case the solution vector of the suproblem - is accepted. Default is 2. - - **theta1** (float): Threshold for adding the current x candidate to the - model. Function argument to find_affine_points(). Default is 1e-5. - - **theta2** (float): Threshold for adding the current x candidate to the model. - Argument to get_interpolation_matrices_residual_model(). Default is 1e-4. - - **trustregion_threshold_successful** (float): First threshold for accepting the - solution vector of the subproblem as the best x candidate. Default is 0. - - **trustregion_threshold_very_successful** (float): Second threshold for accepting - the solution vector of the subproblem as the best x candidate. Default is 0.1. - - **c1** (float): Treshold for accepting the norm of our current x candidate. - Function argument to find_affine_points() for the case where input array - *model_improving_points* is zero. - - **c2** (int): Treshold for accepting the norm of our current x candidate. - Equal to 10 by default. Argument to *find_affine_points()* in case - the input array *model_improving_points* is not zero. - - **trustregion_subproblem_solver** (str): Solver to use for the trust-region - subproblem. Two internal solvers are supported: - - "bntr": Bounded Newton Trust-Region (default, supports bound constraints) - - "gqtpar": (does not support bound constraints) - - **trustregion_subsolver_options** (dict): Options dictionary containing - the stopping criteria for the subproblem. It takes different keys depending - on the type of subproblem solver used. With the exception of the stopping criterion - "maxiter", which is always included. - - If the subsolver "bntr" is used, the dictionary also contains the tolerance levels - "gtol_abs", "gtol_rel", and "gtol_scaled". Moreover, the "conjugate_gradient_method" - can be provided. Available conjugate gradient methods are: - - "cg". In this case, two additional stopping criteria are "gtol_abs_cg" and "gtol_rel_cg" - - "steihaug-toint" - - "trsbox" (default) - - If the subsolver "gqtpar" is employed, the two stopping criteria are - "k_easy" and "k_hard". - - None of the dictionary keys need to be specified by default, but can be. - - **batch_evaluator** (str or callable): Name of a pre-implemented batch evaluator - (currently "joblib" and "pathos_mp") or callable with the same interface - as the optimagic batch_evaluators. Default is "joblib". - - **n_cores (int)**: Number of processes used to parallelize the function - evaluations. Default is 1. + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pounders(stopping_maxiter=1_000), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="pounders", + algo_options={"stopping_maxiter": 1_000}, + ) + + **Description and available options:** + .. autoclass:: optimagic.optimizers.pounders.Pounders ``` (tao-algorithms)= @@ -997,68 +737,38 @@ need to have [petsc4py](https://pypi.org/project/petsc4py/) installed. ```{eval-rst} .. dropdown:: tao_pounders - .. code-block:: - - "tao_pounders" - - Minimize a function using the POUNDERs algorithm. - - POUNDERs (:cite:`Benson2017`, :cite:`Wild2015`, `GitHub repository - `_) - - can be a useful tool for economists who estimate structural models using - indirect inference, because unlike commonly used algorithms such as Nelder-Mead, - POUNDERs is tailored for minimizing a non-linear sum of squares objective function, - and therefore may require fewer iterations to arrive at a local optimum than - Nelder-Mead. - - Scaling the problem is necessary such that bounds correspond to the unit hypercube - :math:`[0, 1]^n`. For unconstrained problems, scale each parameter such that unit - changes in parameters result in similar order-of-magnitude changes in the criterion - value(s). - - POUNDERs has several convergence criteria. Let :math:`X` be the current parameter - vector, :math:`X_0` the initial parameter vector, :math:`g` the gradient, and - :math:`f` the criterion function. - - ``absolute_gradient_tolerance`` stops the optimization if the norm of the gradient - falls below :math:`\epsilon`. + **How to use this algorithm:** - .. math:: + .. code-block:: python - ||g(X)|| < \epsilon + import optimagic as om - ``relative_gradient_tolerance`` stops the optimization if the norm of the gradient - relative to the criterion value falls below :math:`epsilon`. - .. math:: + @om.mark.least_squares + def fun(x): + return x - \frac{||g(X)||}{|f(X)|} < \epsilon - ``scaled_gradient_tolerance`` stops the optimization if the norm of the gradient is - lower than some fraction :math:`epsilon` of the norm of the gradient at the initial - parameters. + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.tao_pounders(stopping_maxiter=1_000, ...), + ) - .. math:: + or using the string interface: - \frac{||g(X)||}{||g(X0)||} < \epsilon + .. code-block:: python - - **convergence.gtol_abs** (float): Stop if norm of gradient is less than this. - If set to False the algorithm will not consider convergence.gtol_abs. - - **convergence.gtol_rel** (float): Stop if relative norm of gradient is less - than this. If set to False the algorithm will not consider - convergence.gtol_rel. - - **convergence.scaled_gradient_tolerance** (float): Stop if scaled norm of gradient is smaller - than this. If set to False the algorithm will not consider - convergence.scaled_gradient_tolerance. - - **trustregion.initial_radius** (float): Initial value of the trust region radius. - It must be :math:`> 0`. - - **stopping.maxiter** (int): Alternative Stopping criterion. - If set the routine will stop after the number of specified iterations or - after the step size is sufficiently small. If the variable is set the - default criteria will all be ignored. + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="tao_pounders", + algo_options={"stopping_maxiter": 1_000, ...}, + ) + **Description and available options:** + .. autoclass:: optimagic.optimizers.tao_optimizers.TAOPounders ``` (nag-algorithms)= @@ -1074,263 +784,70 @@ install each of them separately: ```{eval-rst} .. dropdown:: nag_dfols - *Note*: We recommend to install `DFO-LS` version 1.5.3 or higher. Versions of 1.5.0 or lower also work but the versions `1.5.1` and `1.5.2` contain bugs that can lead to errors being raised. + **How to use this algorithm:** - .. code-block:: + .. code-block:: python + + import optimagic as om + + + @om.mark.least_squares + def fun(x): + return x + + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nag_dfols(stopping_maxfun=10_000, ...), + ) - "nag_dfols" - - Minimize a function with least squares structure using DFO-LS. - - The DFO-LS algorithm :cite:`Cartis2018b` is designed to solve the nonlinear - least-squares minimization problem (with optional bound constraints). - Remember to cite :cite:`Cartis2018b` when using DF-OLS in addition to optimagic. - - .. math:: - - \min_{x\in\mathbb{R}^n} &\quad f(x) := \sum_{i=1}^{m}r_{i}(x)^2 \\ - \text{s.t.} &\quad \text{lower_bounds} \leq x \leq \text{upper_bounds} - - The :math:`r_{i}` are called root contributions in optimagic. - - DFO-LS is a derivative-free optimization algorithm, which means it does not require - the user to provide the derivatives of f(x) or :math:`r_{i}(x)`, nor does it - attempt to estimate them internally (by using finite differencing, for instance). - - There are two main situations when using a derivative-free algorithm - (such as DFO-LS) is preferable to a derivative-based algorithm (which is the vast - majority of least-squares solvers): - - 1. If the residuals are noisy, then calculating or even estimating their derivatives - may be impossible (or at least very inaccurate). By noisy, we mean that if we - evaluate :math:`r_{i}(x)` multiple times at the same value of x, we get different - results. This may happen when a Monte Carlo simulation is used, for instance. - - 2. If the residuals are expensive to evaluate, then estimating derivatives - (which requires n evaluations of each :math:`r_{i}(x)` for every point of - interest x) may be prohibitively expensive. Derivative-free methods are designed - to solve the problem with the fewest number of evaluations of the criterion as - possible. - - To read the detailed documentation of the algorithm `click here - `_. - - There are four possible convergence criteria: - - 1. when the lower trust region radius is shrunk below a minimum - (``convergence.minimal_trustregion_radius_tolerance``). - - 2. when the improvements of iterations become very small - (``convergence.slow_progress``). This is very similar to - ``relative_criterion_tolerance`` but ``convergence.slow_progress`` is more - general allowing to specify not only the threshold for convergence but also - a period over which the improvements must have been very small. - - 3. when a sufficient reduction to the criterion value at the start parameters - has been reached, i.e. when - :math:`\frac{f(x)}{f(x_0)} \leq - \text{convergence.ftol_scaled}` - - 4. when all evaluations on the interpolation points fall within a scaled version of - the noise level of the criterion function. This is only applicable if the - criterion function is noisy. You can specify this criterion with - ``convergence.noise_corrected_criterion_tolerance``. - - DF-OLS supports resetting the optimization and doing a fast start by - starting with a smaller interpolation set and growing it dynamically. - For more information see `their detailed documentation - `_ and :cite:`Cartis2018b`. - - - **clip_criterion_if_overflowing** (bool): see :ref:`algo_options`. - convergence.minimal_trustregion_radius_tolerance (float): see - :ref:`algo_options`. - - **convergence.noise_corrected_criterion_tolerance** (float): Stop when the - evaluations on the set of interpolation points all fall within this factor - of the noise level. - The default is 1, i.e. when all evaluations are within the noise level. - If you want to not use this criterion but still flag your - criterion function as noisy, set this tolerance to 0.0. - - .. warning:: - Very small values, as in most other tolerances don't make sense here. - - - **convergence.ftol_scaled** (float): - Terminate if a point is reached where the ratio of the criterion value - to the criterion value at the start params is below this value, i.e. if - :math:`f(x_k)/f(x_0) \leq - \text{convergence.ftol_scaled}`. Note this is - deactivated unless the lowest mathematically possible criterion value (0.0) - is actually achieved. - - **convergence.slow_progress** (dict): Arguments for converging when the evaluations - over several iterations only yield small improvements on average, see - see :ref:`algo_options` for details. - - **initial_directions (str)**: see :ref:`algo_options`. - - **interpolation_rounding_error** (float): see :ref:`algo_options`. - - **noise_additive_level** (float): Used for determining the presence of noise - and the convergence by all interpolation points being within noise level. - 0 means no additive noise. Only multiplicative or additive is supported. - - **noise_multiplicative_level** (float): Used for determining the presence of noise - and the convergence by all interpolation points being within noise level. - 0 means no multiplicative noise. Only multiplicative or additive is - supported. - - **noise_n_evals_per_point** (callable): How often to evaluate the criterion - function at each point. - This is only applicable for criterion functions with noise, - when averaging multiple evaluations at the same point produces a more - accurate value. - The input parameters are the ``upper_trustregion_radius`` (:math:`\Delta`), - the ``lower_trustregion_radius`` (:math:`\rho`), - how many iterations the algorithm has been running for, ``n_iterations`` - and how many resets have been performed, ``n_resets``. - The function must return an integer. - Default is no averaging (i.e. - ``noise_n_evals_per_point(...) = 1``). - - **random_directions_orthogonal** (bool): see :ref:`algo_options`. - - **stopping.maxfun** (int): see :ref:`algo_options`. - - **threshold_for_safety_step** (float): see :ref:`algo_options`. - - **trustregion.expansion_factor_successful** (float): see :ref:`algo_options`. - - **trustregion.expansion_factor_very_successful** (float): see :ref:`algo_options`. - - **trustregion.fast_start_options** (dict): see :ref:`algo_options`. - - **trustregion.initial_radius** (float): Initial value of the trust region radius. - - **trustregion.method_to_replace_extra_points (str)**: If replacing extra points in - successful iterations, whether to use geometry improving steps or the - momentum method. Can be "geometry_improving" or "momentum". - - **trustregion.n_extra_points_to_replace_successful** (int): The number of extra - points (other than accepting the trust region step) to replace. Useful when - ``trustregion.n_interpolation_points > len(x) + 1``. - - **trustregion.n_interpolation_points** (int): The number of interpolation points to - use. The default is :code:`len(x) + 1`. If using resets, this is the - number of points to use in the first run of the solver, before any resets. - - **trustregion.precondition_interpolation** (bool): see :ref:`algo_options`. - - **trustregion.shrinking_factor_not_successful** (float): see :ref:`algo_options`. - - **trustregion.shrinking_factor_lower_radius** (float): see :ref:`algo_options`. - - **trustregion.shrinking_factor_upper_radius** (float): see :ref:`algo_options`. - - **trustregion.threshold_successful** (float): Share of the predicted improvement - that has to be achieved for a trust region iteration to count as successful. - - **trustregion.threshold_very_successful** (float): Share of the predicted - improvement that has to be achieved for a trust region iteration to count - as very successful. + or using the string interface: + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="nag_dfols", + algo_options={"stopping_maxfun": 10_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nag_optimizers.NagDFOLS ``` ```{eval-rst} .. dropdown:: nag_pybobyqa - .. code-block:: + **How to use this algorithm:** - "nag_pybobyqa" - - Minimize a function using the BOBYQA algorithm. - - BOBYQA (:cite:`Powell2009`, :cite:`Cartis2018`, :cite:`Cartis2018a`) is a - derivative-free trust-region method. It is designed to solve nonlinear local - minimization problems. - - Remember to cite :cite:`Powell2009` and :cite:`Cartis2018` when using pybobyqa in - addition to optimagic. If you take advantage of the ``seek_global_optimum`` option, - cite :cite:`Cartis2018a` additionally. - - There are two main situations when using a derivative-free algorithm like BOBYQA - is preferable to derivative-based algorithms: - - 1. The criterion function is not deterministic, i.e. if we evaluate the criterion - function multiple times at the same parameter vector we get different results. - - 2. The criterion function is very expensive to evaluate and only finite differences - are available to calculate its derivative. - - The detailed documentation of the algorithm can be found `here - `_. - - There are four possible convergence criteria: - - 1. when the trust region radius is shrunk below a minimum. This is - approximately equivalent to an absolute parameter tolerance. - - 2. when the criterion value falls below an absolute, user-specified value, - the optimization terminates successfully. - - 3. when insufficient improvements have been gained over a certain number of - iterations. The (absolute) threshold for what constitutes an insufficient - improvement, how many iterations have to be insufficient and with which - iteration to compare can all be specified by the user. - - 4. when all evaluations on the interpolation points fall within a scaled version of - the noise level of the criterion function. This is only applicable if the - criterion function is noisy. - - - **clip_criterion_if_overflowing** (bool): see :ref:`algo_options`. - - **convergence.criterion_value** (float): Terminate successfully if - the criterion value falls below this threshold. This is deactivated - (i.e. set to -inf) by default. - - **convergence.minimal_trustregion_radius_tolerance** (float): Minimum allowed - value of the trust region radius, which determines when a successful - termination occurs. - - **convergence.noise_corrected_criterion_tolerance** (float): Stop when the - evaluations on the set of interpolation points all fall within this - factor of the noise level. - The default is 1, i.e. when all evaluations are within the noise level. - If you want to not use this criterion but still flag your - criterion function as noisy, set this tolerance to 0.0. - - .. warning:: - Very small values, as in most other tolerances don't make sense here. - - - **convergence.slow_progress** (dict): Arguments for converging when the evaluations - over several iterations only yield small improvements on average, see - see :ref:`algo_options` for details. - - **initial_directions** (str)``: see :ref:`algo_options`. - - **interpolation_rounding_error** (float): see :ref:`algo_options`. - - **noise_additive_level** (float): Used for determining the presence of noise - and the convergence by all interpolation points being within noise level. - 0 means no additive noise. Only multiplicative or additive is supported. - - **noise_multiplicative_level** (float): Used for determining the presence of noise - and the convergence by all interpolation points being within noise level. - 0 means no multiplicative noise. Only multiplicative or additive is - supported. - - **noise_n_evals_per_point** (callable): How often to evaluate the criterion - function at each point. - This is only applicable for criterion functions with noise, - when averaging multiple evaluations at the same point produces a more - accurate value. - The input parameters are the ``upper_trustregion_radius`` (``delta``), - the ``lower_trustregion_radius`` (``rho``), - how many iterations the algorithm has been running for, ``n_iterations`` - and how many resets have been performed, ``n_resets``. - The function must return an integer. - Default is no averaging (i.e. ``noise_n_evals_per_point(...) = 1``). - - **random_directions_orthogonal** (bool): see :ref:`algo_options`. - - **seek_global_optimum** (bool): whether to apply the heuristic to escape local - minima presented in :cite:`Cartis2018a`. Only applies for noisy criterion - functions. - - **stopping.maxfun** (int): see :ref:`algo_options`. - - **threshold_for_safety_step** (float): see :ref:`algo_options`. - - **trustregion.expansion_factor_successful** (float): see :ref:`algo_options`. - - **trustregion.expansion_factor_very_successful** (float): see :ref:`algo_options`. - - **trustregion.initial_radius** (float): Initial value of the trust region radius. - - **trustregion.minimum_change_hession_for_underdetermined_interpolation** (bool): - Whether to solve the underdetermined quadratic interpolation problem by - minimizing the Frobenius norm of the Hessian, or change in Hessian. - - **trustregion.n_interpolation_points** (int): The number of interpolation points to - use. With $n=len(x)$ the default is $2n+1$ if the criterion is not noisy. - Otherwise, it is set to $(n+1)(n+2)/2)$. - - Larger values are particularly useful for noisy problems. - Py-BOBYQA requires - - .. math:: - n + 1 \leq \text{trustregion.n_interpolation_points} \leq (n+1)(n+2)/2. - - **trustregion.precondition_interpolation** (bool): see :ref:`algo_options`. - - **trustregion.reset_options** (dict): Options for resetting the optimization, - see :ref:`algo_options` for details. - - **trustregion.shrinking_factor_not_successful** (float): see :ref:`algo_options`. - - **trustregion.shrinking_factor_upper_radius** (float): see :ref:`algo_options`. - - **trustregion.shrinking_factor_lower_radius** (float): see :ref:`algo_options`. - - **trustregion.threshold_successful** (float): see :ref:`algo_options`. - - **trustregion.threshold_very_successful** (float): see :ref:`algo_options`. + .. code-block:: python + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nag_pybobyqa( + stopping_max_criterion_evaluations=10_000, ... + ), + ) + + or using the string interface: + .. code-block:: python + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nag_pybobyqa", + algo_options={"stopping_max_criterion_evaluations": 10_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nag_optimizers.NagPyBOBYQA ``` (pygmo-algorithms)= @@ -1343,721 +860,529 @@ supports the following [pygmo2](https://esa.github.io/pygmo2) optimizers. ```{eval-rst} .. dropdown:: pygmo_gaco - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_gaco(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - "pygmo_gaco" - - Minimize a scalar function using the generalized ant colony algorithm. - - The version available through pygmo is an generalized version of the - original ant colony algorithm proposed by :cite:`Schlueter2009`. - - This algorithm can be applied to box-bounded problems. - - Ant colony optimization is a class of optimization algorithms modeled on the - actions of an ant colony. Artificial "ants" (e.g. simulation agents) locate - optimal solutions by moving through a parameter space representing all - possible solutions. Real ants lay down pheromones directing each other to - resources while exploring their environment. The simulated "ants" similarly - record their positions and the quality of their solutions, so that in later - simulation iterations more ants locate better solutions. - - The generalized ant colony algorithm generates future generations of ants by - using a multi-kernel gaussian distribution based on three parameters (i.e., - pheromone values) which are computed depending on the quality of each - previous solution. The solutions are ranked through an oracle penalty - method. - - - **population_size** (int): Size of the population. If None, it's twice the - number of parameters but at least 64. - - **batch_evaluator** (str or Callable): Name of a pre-implemented batch - evaluator (currently 'joblib' and 'pathos_mp') or Callable with the same - interface as the optimagic batch_evaluators. See :ref:`batch_evaluators`. - - **n_cores** (int): Number of cores to use. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed - to be part of the initial population. This saves one criterion function - evaluation that cannot be done in parallel with other evaluations. Default - False. - - - **stopping.maxiter** (int): Number of generations to evolve. - - **kernel_size** (int): Number of solutions stored in the solution archive. - - **speed_parameter_q** (float): This parameter manages the convergence speed - towards the found minima (the smaller the faster). In the pygmo - documentation it is referred to as $q$. It must be positive and can be - larger than 1. The default is 1.0 until **threshold** is reached. Then it - is set to 0.01. - - **oracle** (float): oracle parameter used in the penalty method. - - **accuracy** (float): accuracy parameter for maintaining a minimum penalty - function's values distances. - - **threshold** (int): when the iteration counter reaches the threshold the - convergence speed is set to 0.01 automatically. To deactivate this effect - set the threshold to stopping.maxiter which is the largest allowed - value. - - **speed_of_std_values_convergence** (int): parameter that determines the - convergence speed of the standard deviations. This must be an integer - (`n_gen_mark` in pygmo and pagmo). - - **stopping.max_n_without_improvements** (int): if a positive integer is - assigned here, the algorithm will count the runs without improvements, if - this number exceeds the given value, the algorithm will be stopped. - - **stopping.maxfun** (int): maximum number of function - evaluations. - - **focus** (float): this parameter makes the search for the optimum greedier - and more focused on local improvements (the higher the greedier). If the - value is very high, the search is more focused around the current best - solutions. Values larger than 1 are allowed. - - **cache** (bool): if True, memory is activated in the algorithm for multiple calls. + or using the string interface: + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_gaco", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoGaco ``` ```{eval-rst} .. dropdown:: pygmo_bee_colony - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_bee_colony(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python - "pygmo_bee_colony" - - Minimize a scalar function using the artifical bee colony algorithm. - - The Artificial Bee Colony Algorithm was originally proposed by - :cite:`Karaboga2007`. The implemented version of the algorithm is proposed - in :cite:`Mernik2015`. The algorithm is only suited for bounded parameter - spaces. - - - **stopping.maxiter** (int): Number of generations to evolve. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed - to be part of the initial population. This saves one criterion function - evaluation that cannot be done in parallel with other evaluations. Default - False. - - **max_n_trials** (int): Maximum number of trials for abandoning a source. - Default is 1. - - **population_size** (int): Size of the population. If None, it's twice the - number of parameters but at least 20. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_bee_colony", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoBeeColony ``` ```{eval-rst} .. dropdown:: pygmo_de - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "pygmo_de" - - Minimize a scalar function using the differential evolution algorithm. - - Differential Evolution is a heuristic optimizer originally presented in - :cite:`Storn1997`. The algorithm is only suited for bounded parameter - spaces. - - - **population_size** (int): Size of the population. If None, it's twice the - number of parameters but at least 10. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed - to be part of the initial population. This saves one criterion function - evaluation that cannot be done in parallel with other evaluations. Default - False. - - **stopping.maxiter** (int): Number of generations to evolve. - - **weight_coefficient** (float): Weight coefficient. It is denoted by $F$ in - the main paper and must lie in [0, 2]. It controls the amplification of - the differential variation $(x_{r_2, G} - x_{r_3, G})$. - - **crossover_probability** (float): Crossover probability. - - **mutation_variant (str or int)**: code for the mutation variant to create a - new candidate individual. The default is . The following are available: - - - "best/1/exp" (1, when specified as int) - - "rand/1/exp" (2, when specified as int) - - "rand-to-best/1/exp" (3, when specified as int) - - "best/2/exp" (4, when specified as int) - - "rand/2/exp" (5, when specified as int) - - "best/1/bin" (6, when specified as int) - - "rand/1/bin" (7, when specified as int) - - "rand-to-best/1/bin" (8, when specified as int) - - "best/2/bin" (9, when specified as int) - - "rand/2/bin" (10, when specified as int) - - **convergence.criterion_tolerance**: stopping criteria on the criterion - tolerance. Default is 1e-6. It is not clear whether this is the absolute - or relative criterion tolerance. - - **convergence.xtol_rel**: stopping criteria on the x - tolerance. In pygmo the default is 1e-6 but we use our default value of - 1e-5. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_de(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_de", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoDe ``` ```{eval-rst} .. dropdown:: pygmo_sea - .. code-block:: + **How to use this algorithm:** - "pygmo_sea" + .. code-block:: python - Minimize a scalar function using the (N+1)-ES simple evolutionary algorithm. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_sea(stopping_maxiter=5_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - This algorithm represents the simplest evolutionary strategy, where a population of - $\lambda$ individuals at each generation produces one offspring by mutating its best - individual uniformly at random within the bounds. Should the offspring be better - than the worst individual in the population it will substitute it. + or using the string interface: - See :cite:`Oliveto2007`. + .. code-block:: python - The algorithm is only suited for bounded parameter spaces. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_sea", + algo_options={"stopping_maxiter": 5_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 10. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): number of generations to consider. Each generation - will compute the objective function once. + **Description and available options:** + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoSea ``` ```{eval-rst} .. dropdown:: pygmo_sga - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_sga(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_sga", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - "pygmo_sga" - - Minimize a scalar function using a simple genetic algorithm. - - A detailed description of the algorithm can be found `in the pagmo2 documentation - `_. - - See also :cite:`Oliveto2007`. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. - - **crossover_probability** (float): Crossover probability. - - **crossover_strategy** (str): the crossover strategy. One of “exponential”,“binomial”, - “single” or “sbx”. Default is "exponential". - - **eta_c** (float): distribution index for “sbx” crossover. This is an inactive - parameter if other types of crossovers are selected. Can be in [1, 100]. - - **mutation_probability** (float): Mutation probability. - - **mutation_strategy** (str): Mutation strategy. Must be "gaussian", "polynomial" or - "uniform". Default is "polynomial". - - **mutation_polynomial_distribution_index** (float): Must be in [0, 1]. Default is 1. - - **mutation_gaussian_width** (float): Must be in [0, 1]. Default is 1. - - **selection_strategy (str)**: Selection strategy. Must be "tournament" or "truncated". - - **selection_truncated_n_best** (int): number of best individuals to use in the - "truncated" selection mechanism. - - **selection_tournament_size** (int): size of the tournament in the "tournament" - selection mechanism. Default is 1. + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoSga ``` ```{eval-rst} .. dropdown:: pygmo_sade - .. code-block:: + **How to use this algorithm:** - "pygmo_sade" - - Minimize a scalar function using Self-adaptive Differential Evolution. - - The original Differential Evolution algorithm (pygmo_de) can be significantly - improved introducing the idea of parameter self-adaptation. - - Many different proposals have been made to self-adapt both the crossover and the - F parameters of the original differential evolution algorithm. pygmo's - implementation supports two different mechanisms. The first one, proposed by - :cite:`Brest2006`, does not make use of the differential evolution operators to - produce new values for the weight coefficient $F$ and the crossover probability - $CR$ and, strictly speaking, is thus not self-adaptation, rather parameter control. - The resulting differential evolution variant is often referred to as jDE. - The second variant is inspired by the ideas introduced by :cite:`Elsayed2011` and - uses a variaton of the selected DE operator to produce new $CR$ anf $F$ parameters - for each individual. This variant is referred to iDE. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - jde (bool): Whether to use the jDE self-adaptation variant to control the $F$ and - $CR$ parameter. If True jDE is used, else iDE. - - **stopping.maxiter** (int): Number of generations to evolve. - - **mutation_variant** (int or str): code for the mutation variant to create a new - candidate individual. The default is "rand/1/exp". The first ten are the - classical mutation variants introduced in the orginal DE algorithm, the remaining - ones are, instead, considered in the work by :cite:`Elsayed2011`. - The following are available: - - - "best/1/exp" or 1 - - "rand/1/exp" or 2 - - "rand-to-best/1/exp" or 3 - - "best/2/exp" or 4 - - "rand/2/exp" or 5 - - "best/1/bin" or 6 - - "rand/1/bin" or 7 - - "rand-to-best/1/bin" or 8 - - "best/2/bin" or 9 - - "rand/2/bin" or 10 - - "rand/3/exp" or 11 - - "rand/3/bin" or 12 - - "best/3/exp" or 13 - - "best/3/bin" or 14 - - "rand-to-current/2/exp" or 15 - - "rand-to-current/2/bin" or 16 - - "rand-to-best-and-current/2/exp" or 17 - - "rand-to-best-and-current/2/bin" or 18 - - - **keep_adapted_params** (bool): when true the adapted parameters $CR$ anf $F$ are - not reset between successive calls to the evolve method. Default is False. - - ftol (float): stopping criteria on the x tolerance. - - xtol (float): stopping criteria on the f tolerance. + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_sade(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + or using the string interface: + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_sade", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoSade ``` ```{eval-rst} .. dropdown:: pygmo_cmaes - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_cmaes(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python - "pygmo_cmaes" - - Minimize a scalar function using the Covariance Matrix Evolutionary Strategy. - - CMA-ES is one of the most successful algorithm, classified as an Evolutionary - Strategy, for derivative-free global optimization. The version supported by - optimagic is the version described in :cite:`Hansen2006`. - - In contrast to the pygmo version, optimagic always sets force_bounds to True. This - avoids that ill defined parameter values are evaluated. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - - **stopping.maxiter** (int): Number of generations to evolve. - - **backward_horizon** (float): backward time horizon for the evolution path. It must - lie betwen 0 and 1. - - **variance_loss_compensation** (float): makes partly up for the small variance loss in - case the indicator is zero. `cs` in the MATLAB Code of :cite:`Hansen2006`. It must - lie between 0 and 1. - - **learning_rate_rank_one_update** (float): learning rate for the rank-one update of - the covariance matrix. `c1` in the pygmo and pagmo documentation. It must lie - between 0 and 1. - - **learning_rate_rank_mu_update** (float): learning rate for the rank-mu update of the - covariance matrix. `cmu` in the pygmo and pagmo documentation. It must lie between - 0 and 1. - - **initial_step_size** (float): initial step size, :math:`\sigma^0` in the original - paper. - - **ftol** (float): stopping criteria on the x tolerance. - - **xtol** (float): stopping criteria on the f tolerance. - - **keep_adapted_params** (bool): when true the adapted parameters are not reset - between successive calls to the evolve method. Default is False. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_cmaes", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoCmaes ``` ```{eval-rst} .. dropdown:: pygmo_simulated_annealing - .. code-block:: + **How to use this algorithm:** - "pygmo_simulated_annealing" - - Minimize a function with the simulated annealing algorithm. - - This version of the simulated annealing algorithm is, essentially, an iterative - random search procedure with adaptive moves along the coordinate directions. It - permits uphill moves under the control of metropolis criterion, in the hope to avoid - the first local minima encountered. This version is the one proposed in - :cite:`Corana1987`. - - .. note: When selecting the starting and final temperature values it helps to think - about the tempertaure as the deterioration in the objective function value that - still has a 37% chance of being accepted. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **start_temperature** (float): starting temperature. Must be > 0. - - **end_temperature** (float): final temperature. Our default (0.01) is lower than in - pygmo and pagmo. The final temperature must be positive. - - **n_temp_adjustments** (int): number of temperature adjustments in the annealing - schedule. - - **n_range_adjustments** (int): number of adjustments of the search range performed at - a constant temperature. - - **bin_size** (int): number of mutations that are used to compute the acceptance rate. - - **start_range** (float): starting range for mutating the decision vector. It must lie - between 0 and 1. + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_simulated_annealing(start_temperature=5.0, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_simulated_annealing", + algo_options={"start_temperature": 5.0, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoSimulatedAnnealing ``` ```{eval-rst} .. dropdown:: pygmo_pso - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_pso(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_pso", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** - "pygmo_pso" - - Minimize a scalar function using Particle Swarm Optimization. - - Particle swarm optimization (PSO) is a population based algorithm inspired by the - foraging behaviour of swarms. In PSO each point has memory of the position where it - achieved the best performance xli (local memory) and of the best decision vector - :math:`x^g` in a certain neighbourhood, and uses this information to update its - position. - - For a survey on particle swarm optimization algorithms, see :cite:`Poli2007`. - - Each particle determines its future position :math:`x_{i+1} = x_i + v_i` where - - .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - x^{l}_i) + - \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 10. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. - - - **omega** (float): depending on the variant chosen, :math:`\omega` is the particles' - inertia weight or the construction coefficient. It must lie between 0 and 1. - - **force_of_previous_best** (float): :math:`\eta_1` in the equation above. It's the - magnitude of the force, applied to the particle’s velocity, in the direction of - its previous best position. It must lie between 0 and 4. - - **force_of_best_in_neighborhood** (float): :math:`\eta_2` in the equation above. It's - the magnitude of the force, applied to the particle’s velocity, in the direction - of the best position in its neighborhood. It must lie between 0 and 4. - - **max_velocity** (float): maximum allowed particle velocity as fraction of the box - bounds. It must lie between 0 and 1. - - **algo_variant (int or str)**: algorithm variant to be used: - - 1 or "canonical_inertia": Canonical (with inertia weight) - - 2 or "social_and_cog_rand": Same social and cognitive rand. - - 3 or "all_components_rand": Same rand. for all components - - 4 or "one_rand": Only one rand. - - 5 or "canonical_constriction": Canonical (with constriction fact.) - - 6 or "fips": Fully Informed (FIPS) - - - **neighbor_definition (int or str)**: swarm topology that defines each particle's - neighbors that is to be used: - - - 1 or "gbest" - - 2 or "lbest" - - 3 or "Von Neumann" - - 4 or "Adaptive random" - - - **neighbor_param** (int): the neighbourhood parameter. If the lbest topology is - selected (neighbor_definition=2), it represents each particle's indegree (also - outdegree) in the swarm topology. Particles have neighbours up to a radius of k = - neighbor_param / 2 in the ring. If the Randomly-varying neighbourhood topology is - selected (neighbor_definition=4), it represents each particle’s maximum outdegree - in the swarm topology. The minimum outdegree is 1 (the particle always connects - back to itself). If neighbor_definition is 1 or 3 this parameter is ignored. - - **keep_velocities** (bool): when true the particle velocities are not reset between - successive calls to `evolve`. + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoPso ``` ```{eval-rst} .. dropdown:: pygmo_pso_gen - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_pso_gen(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - "pygmo_pso_gen" - - Minimize a scalar function with generational Particle Swarm Optimization. - - Particle Swarm Optimization (generational) is identical to pso, but does update the - velocities of each particle before new particle positions are computed (taking into - consideration all updated particle velocities). Each particle is thus evaluated on - the same seed within a generation as opposed to the standard PSO which evaluates - single particle at a time. Consequently, the generational PSO algorithm is suited - for stochastic optimization problems. - - For a survey on particle swarm optimization algorithms, see :cite:`Poli2007`. - - Each particle determines its future position :math:`x_{i+1} = x_i + v_i` where - - .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - x^{l}_i) + - \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 10. - - **batch_evaluator (str or Callable)**: Name of a pre-implemented batch evaluator - (currently 'joblib' and 'pathos_mp') or Callable with the same interface as the - optimagic batch_evaluators. See :ref:`batch_evaluators`. - - **n_cores** (int): Number of cores to use. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. - - - **omega** (float): depending on the variant chosen, :math:`\omega` is the particles' - inertia weight or the constructuion coefficient. It must lie between 0 and 1. - - **force_of_previous_best** (float): :math:`\eta_1` in the equation above. It's the - magnitude of the force, applied to the particle’s velocity, in the direction of - its previous best position. It must lie between 0 and 4. - - **force_of_best_in_neighborhood** (float): :math:`\eta_2` in the equation above. It's - the magnitude of the force, applied to the particle’s velocity, in the direction - of the best position in its neighborhood. It must lie between 0 and 4. - - **max_velocity** (float): maximum allowed particle velocity as fraction of the box - bounds. It must lie between 0 and 1. - - **algo_variant** (int): code of the algorithm's variant to be used: - - - 1 or "canonical_inertia": Canonical (with inertia weight) - - 2 or "social_and_cog_rand": Same social and cognitive rand. - - 3 or "all_components_rand": Same rand. for all components - - 4 or "one_rand": Only one rand. - - 5 or "canonical_constriction": Canonical (with constriction fact.) - - 6 or "fips": Fully Informed (FIPS) - - - **neighbor_definition** (int): code for the swarm topology that defines each - particle's neighbors that is to be used: - - - 1 or "gbest" - - 2 or "lbest" - - 3 or "Von Neumann" - - 4 or "Adaptive random" - - - **neighbor_param** (int): the neighbourhood parameter. If the lbest topology is - selected (neighbor_definition=2), it represents each particle's indegree (also - outdegree) in the swarm topology. Particles have neighbours up to a radius of k = - neighbor_param / 2 in the ring. If the Randomly-varying neighbourhood topology is - selected (neighbor_definition=4), it represents each particle’s maximum outdegree - in the swarm topology. The minimum outdegree is 1 (the particle always connects - back to itself). If neighbor_definition is 1 or 3 this parameter is ignored. - - **keep_velocities** (bool): when true the particle velocities are not reset between - successive calls to `evolve`. + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_pso_gen", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoPsoGen ``` ```{eval-rst} .. dropdown:: pygmo_mbh - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_mbh(perturbation=0.05, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: - "pygmo_mbh" - - Minimize a scalar function using generalized Monotonic Basin Hopping. - - Monotonic basin hopping, or simply, basin hopping, is an algorithm rooted in the - idea of mapping the objective function $f(x_0)$ into the local minima found starting - from $x_0$. This simple idea allows a substantial increase of efficiency in solving - problems, such as the Lennard-Jones cluster or the MGA-1DSM interplanetary - trajectory problem that are conjectured to have a so-called funnel structure. - - See :cite:`Wales1997` for the paper introducing the basin hopping idea for a - Lennard-Jones cluster optimization. - - pygmo provides an original generalization of this concept resulting in a - meta-algorithm that operates on a population. When a population containing a single - individual is used the original method is recovered. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 250. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **inner_algorithm** (pygmo.algorithm): an pygmo algorithm or a user-defined algorithm, - either C++ or Python. If None the `pygmo.compass_search` algorithm will be used. - - **stopping.max_inner_runs_without_improvement** (int): consecutive runs of the inner - algorithm that need to result in no improvement for mbh to stop. - - **perturbation** (float): the perturbation to be applied to each component. + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_mbh", + algo_options={"perturbation": 0.05, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoMbh ``` ```{eval-rst} .. dropdown:: pygmo_xnes - .. code-block:: + **How to use this algorithm:** - "pygmo_xnes" - - Minimize a scalar function using Exponential Evolution Strategies. - - Exponential Natural Evolution Strategies is an algorithm closely related to CMAES - and based on the adaptation of a gaussian sampling distribution via the so-called - natural gradient. Like CMAES it is based on the idea of sampling new trial vectors - from a multivariate distribution and using the new sampled points to update the - distribution parameters. Naively this could be done following the gradient of the - expected fitness as approximated by a finite number of sampled points. While this - idea offers a powerful lead on algorithmic construction it has some major drawbacks - that are solved in the so-called Natural Evolution Strategies class of algorithms by - adopting, instead, the natural gradient. xNES is one of the most performing variants - in this class. - - See :cite:`Glasmachers2010` and the `pagmo documentation on xNES - `_ - for details. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. - - - **learning_rate_mean_update** (float): learning rate for the mean update - (:math:`\eta_\mu`). It must be between 0 and 1 or None. - - **learning_rate_step_size_update** (float): learning rate for the step-size update. It - must be between 0 and 1 or None. - - **learning_rate_cov_matrix_update** (float): learning rate for the covariance matrix - update. It must be between 0 and 1 or None. - - **initial_search_share** (float): share of the given search space that will be - initally searched. It must be between 0 and 1. Default is 1. - - **ftol** (float): stopping criteria on the x tolerance. - - **xtol** (float): stopping criteria on the f tolerance. - - **keep_adapted_params** (bool): when true the adapted parameters are not reset between - successive calls to the evolve method. Default is False. + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_xnes(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_xnes", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoXnes ``` ```{eval-rst} .. dropdown:: pygmo_gwo - .. code-block:: + **How to use this algorithm:** - "pygmo_gwo" + .. code-block:: python - Minimize a scalar function usinng the Grey Wolf Optimizer. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_gwo(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - The grey wolf optimizer was proposed by :cite:`Mirjalili2014`. The pygmo - implementation that is wrapped by optimagic is pased on the pseudo code provided in - that paper. + or using the string interface: - This algorithm is a classic example of a highly criticizable line of search that led - in the first decades of our millenia to the development of an entire zoo of - metaphors inspiring optimzation heuristics. In our opinion they, as is the case for - the grey wolf optimizer, are often but small variations of already existing - heuristics rebranded with unnecessray and convoluted biological metaphors. In the - case of GWO this is particularly evident as the position update rule is shokingly - trivial and can also be easily seen as a product of an evolutionary metaphor or a - particle swarm one. Such an update rule is also not particulary effective and - results in a rather poor performance most of times. + .. code-block:: python - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_gwo", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoGwo ``` ```{eval-rst} .. dropdown:: pygmo_compass_search - .. code-block:: + **How to use this algorithm:** - "pygmo_compass_search" + .. code-block:: python - Minimize a scalar function using compass search. + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_compass_search(stopping_maxfun=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: - The algorithm is described in :cite:`Kolda2003`. + .. code-block:: python - It is considered slow but reliable. It should not be used for stochastic problems. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_compass_search", + algo_options={"stopping_maxfun": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** - - **population_size** (int): Size of the population. Even though the algorithm is not - population based the population size does affect the results of the algorithm. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxfun** (int): maximum number of function evaluations. - - **start_range** (float): the start range. Must be in (0, 1]. - - **stop_range** (float): the stop range. Must be in (0, start_range]. - - **reduction_coeff** (float): the range reduction coefficient. Must be in (0, 1). + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoCompassSearch ``` ```{eval-rst} .. dropdown:: pygmo_ihs - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_ihs(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: - "pygmo_ihs" - - Minimize a scalar function using the improved harmony search algorithm. - - Improved harmony search (IHS) was introduced by :cite:`Mahdavi2007`. - IHS supports stochastic problems. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **stopping.maxiter** (int): Number of generations to evolve. - - **choose_from_memory_probability** (float): probability of choosing from memory - (similar to a crossover probability). - - **min_pitch_adjustment_rate** (float): minimum pitch adjustment rate. (similar to a - mutation rate). It must be between 0 and 1. - - **max_pitch_adjustment_rate** (float): maximum pitch adjustment rate. (similar to a - mutation rate). It must be between 0 and 1. - - **min_distance_bandwidth** (float): minimum distance bandwidth. (similar to a mutation - width). It must be positive. - - **max_distance_bandwidth** (float): maximum distance bandwidth. (similar to a mutation - width). + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_ihs", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoIhs ``` ```{eval-rst} .. dropdown:: pygmo_de1220 - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.pygmo_de1220(stopping_maxiter=1_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="pygmo_de1220", + algo_options={"stopping_maxiter": 1_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - "pygmo_de1220" - - Minimize a scalar function using Self-adaptive Differential Evolution, pygmo flavor. - - See `the PAGMO documentation for details - `_. - - - **population_size** (int): Size of the population. If None, it's twice the number of - parameters but at least 64. - - **seed** (int): seed used by the internal random number generator. - - **discard_start_params** (bool): If True, the start params are not guaranteed to be - part of the initial population. This saves one criterion function evaluation that - cannot be done in parallel with other evaluations. Default False. - - **jde** (bool): Whether to use the jDE self-adaptation variant to control the $F$ and - $CR$ parameter. If True jDE is used, else iDE. - - **stopping.maxiter** (int): Number of generations to evolve. - - **allowed_variants** (array-like object): allowed mutation variants (can be codes - or strings). Each code refers to one mutation variant to create a new candidate - individual. The first ten refer to the classical mutation variants introduced in - the original DE algorithm, the remaining ones are, instead, considered in the work - by :cite:`Elsayed2011`. The default is ["rand/1/exp", "rand-to-best/1/exp", - "rand/1/bin", "rand/2/bin", "best/3/exp", "best/3/bin", "rand-to-current/2/exp", - "rand-to-current/2/bin"]. The following are available: - - - 1 or "best/1/exp" - - 2 or "rand/1/exp" - - 3 or "rand-to-best/1/exp" - - 4 or "best/2/exp" - - 5 or "rand/2/exp" - - 6 or "best/1/bin" - - 7 or "rand/1/bin" - - 8 or "rand-to-best/1/bin" - - 9 or "best/2/bin" - - 10 or "rand/2/bin" - - 11 or "rand/3/exp" - - 12 or "rand/3/bin" - - 13 or "best/3/exp" - - 14 or "best/3/bin" - - 15 or "rand-to-current/2/exp" - - 16 or "rand-to-current/2/bin" - - 17 or "rand-to-best-and-current/2/exp" - - 18 or "rand-to-best-and-current/2/bin" - - - **keep_adapted_params** (bool): when true the adapted parameters $CR$ anf $F$ are not - reset between successive calls to the evolve method. Default is False. - - **ftol** (float): stopping criteria on the x tolerance. - - **xtol** (float): stopping criteria on the f tolerance. + **Description and available options:** + .. autoclass:: optimagic.optimizers.pygmo_optimizers.PygmoDe1220 ``` (ipopt-algorithm)= @@ -2074,1185 +1399,33 @@ To use ipopt, you need to have (`conda install cyipopt`). ```{eval-rst} -.. dropdown:: ipopt +.. dropdown:: ipopt - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.ipopt(stopping_maxiter=1_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="ipopt", + algo_options={"stopping_maxiter": 1_000, ...}, + ) - "ipopt" - - Minimize a scalar function using the Interior Point Optimizer. - - This implementation of the Interior Point Optimizer (:cite:`Waechter2005`, - :cite:`Waechter2005a`, :cite:`Waechter2005b`, :cite:`Nocedal2009`) relies on - `cyipopt `_, a Python - wrapper for the `Ipopt optimization package - `_. - - There are two levels of termination criteria. If the usual "desired" - tolerances (see tol, dual_inf_tol etc) are satisfied at an iteration, the - algorithm immediately terminates with a success message. On the other hand, - if the algorithm encounters "acceptable_iter" many iterations in a row that - are considered "acceptable", it will terminate before the desired - convergence tolerance is met. This is useful in cases where the algorithm - might not be able to achieve the "desired" level of accuracy. - - The options are analogous to the ones in the `ipopt documentation - `_ with the exception of the - linear solver options which are here bundled into a dictionary. Any argument - that takes "yes" and "no" in the ipopt documentation can also be passed as a - `True` and `False`, respectively. and any option that accepts "none" in - ipopt accepts a Python `None`. - - The following options are not supported: - - `num_linear_variables`: since optimagic may reparametrize your problem - and this changes the parameter problem, we do not support this option. - - derivative checks - - print options. - - - - **convergence.ftol_rel** (float): The algorithm - terminates successfully, if the (scaled) non linear programming error - becomes smaller than this value. - - - **mu_target** (float): Desired value of complementarity. Usually, the barrier - parameter is driven to zero and the termination test for complementarity - is measured with respect to zero complementarity. However, in some cases - it might be desired to have Ipopt solve barrier problem for strictly - positive value of the barrier parameter. In this case, the value of - "mu_target" specifies the final value of the barrier parameter, and the - termination tests are then defined with respect to the barrier problem for - this value of the barrier parameter. The valid range for this real option - is 0 ≤ mu_target and its default value is 0. - - - **s_max** (float): Scaling threshold for the NLP error. - - - **stopping.maxiter** (int): If the maximum number of iterations is - reached, the optimization stops, but we do not count this as successful - convergence. The difference to ``max_criterion_evaluations`` is that one - iteration might need several criterion evaluations, for example in a line - search or to determine if the trust region radius has to be shrunk. - - **stopping.max_wall_time_seconds** (float): Maximum number of walltime clock seconds. - - **stopping.max_cpu_time** (float): Maximum number of CPU seconds. - A limit on CPU seconds that Ipopt can use to solve one problem. - If during the convergence check this limit is exceeded, Ipopt will - terminate with a corresponding message. The valid range for this - real option is 0 < max_cpu_time and its default value is :math:`1e+20` . - - - **dual_inf_tol** (float): Desired threshold for the dual infeasibility. - Absolute tolerance on the dual infeasibility. Successful termination - requires that the max-norm of the (unscaled) dual infeasibility is less - than this threshold. The valid range for this real option is 0 < - dual_inf_tol and its default value is 1. - - **constr_viol_tol** (float): Desired threshold for the constraint and bound - violation. Absolute tolerance on the constraint and variable bound - violation. Successful termination requires that the max-norm of the - (unscaled) constraint violation is less than this threshold. - If option ``bound_relax_factor`` is not zero 0, then Ipopt relaxes given variable bounds. - The value of constr_viol_tol is used to restrict the absolute amount of this bound - relaxation. The valid range for this real option is 0 < constr_viol_tol - and its default value is 0.0001. - - **compl_inf_tol** (float): Desired threshold for the complementarity conditions. - Absolute tolerance on the complementarity. Successful termination - requires that the max-norm of the (unscaled) complementarity is - less than this threshold. The valid range for this real option is - 0 < text{compl_inf_tol and its default is 0.0001. - - **acceptable_iter** (int): Number of "acceptable" iterates before termination. - If the algorithm encounters this many successive "acceptable" - iterates (see above on the acceptable heuristic), it terminates, assuming - that the problem has been solved to best possible accuracy given - round-off. If it is set to zero, this heuristic is disabled. The valid - range for this integer option is 0 ≤ acceptable_iter. - - **acceptable_tol** (float):"Acceptable" convergence tolerance (relative). - Determines which (scaled) overall optimality error is considered to be "acceptable". - The valid range for this real option is 0 < acceptable_tol. - - **acceptable_dual_inf_tol** (float): "Acceptance" threshold for the dual - infeasibility. Absolute tolerance on the dual infeasibility. "Acceptable" - termination requires that the (max-norm of the unscaled) dual - infeasibility is less than this threshold; see also ``acceptable_tol`` . The - valid range for this real option is 0 < acceptable_dual_inf_tol and its - default value is :math:`1e+10.` - - **acceptable_constr_viol_tol** (float): "Acceptance" threshold for the constraint violation. - Absolute tolerance on the constraint violation. - "Acceptable" termination requires that the max-norm - of the (unscaled) constraint violation is less than this threshold; see - also ``acceptable_tol`` . The valid range for this real option is 0 < - acceptable_constr_viol_tol and its default value is 0.01. - - **acceptable_compl_inf_tol** (float): "Acceptance" threshold for the - complementarity conditions. Absolute tolerance on the complementarity. - "Acceptable" termination requires that the max-norm of the (unscaled) - complementarity is less than this threshold; see also ``acceptable_tol`` . The - valid range for this real option is 0 < text{acceptable_compl_inf_tol and its - default value is 0.01. - - **acceptable_obj_change_tol** (float): "Acceptance" stopping criterion based on - objective function change. If the relative - change of the objective function (scaled by :math:`max(1,|f(x)|)` ) is less than - this value, this part of the acceptable tolerance termination is - satisfied; see also ``acceptable_tol`` . This is useful for the quasi-Newton - option, which has trouble to bring down the dual infeasibility. The valid - range for this real option is 0 ≤ acceptable_obj_change_tol and its - default value is :math:`1e+20` . - - - **diverging_iterates_tol** (float): Threshold for maximal value of primal iterates. - If any component of the primal iterates exceeded this value (in - absolute terms), the optimization is aborted with the exit message that - the iterates seem to be diverging. The valid range for this real option is - 0 < diverging_iterates_tol and its default value is :math:`1e+20` . - - **nlp_lower_bound_inf** (float): any bound less or equal this value will be - considered -inf (i.e. not lwer bounded). The valid range for this real - option is unrestricted and its default value is :math:`-1e+19` . - - **nlp_upper_bound_inf** (float): any bound greater or this value will be - considered :math:`+\inf` (i.e. not upper bunded). The valid range for this real - option is unrestricted and its default value is :math:`1e+19` . - - **fixed_variable_treatment (str)**: Determines how fixed variables should be - handled. The main difference between those options is that the starting - point in the "make_constraint" case still has the fixed variables at their - given values, whereas in the case "make_parameter(_nodual)" the functions - are always evaluated with the fixed values for those variables. Also, for - "relax_bounds", the fixing bound constraints are relaxed (according to - ``bound_relax_factor`` ). For all but "make_parameter_nodual", bound - multipliers are computed for the fixed variables. The default value for - this string option is "make_parameter". Possible values: - - - "make_parameter": Remove fixed variable from optimization variables - - "make_parameter_nodual": Remove fixed variable from optimization - variables and do not compute bound multipliers for fixed variables - - "make_constraint": Add equality constraints fixing variables - - "relax_bounds": Relax fixing bound constraints - - **dependency_detector (str)**: Indicates which linear solver - should be used to detect linearly dependent equality constraints. This is - experimental and does not work well. The default value for this string - option is "none". Possible values: - - - "none" or None: don't check; no extra work at beginning - - "mumps": use MUMPS - - "wsmp": use WSMP - - "ma28": use MA28 - - **dependency_detection_with_rhs (str or bool)**: Indicates if the right hand - sides of the constraints should be considered in addition to gradients - during dependency detection. The default value for this string option is - "no". Possible values: 'yes', 'no', True, False. - - - **kappa_d** (float): Weight for linear damping term (to handle one-sided bounds). - See Section 3.7 in implementation paper. The valid range for this - real option is 0 ≤ kappa_d and its default value is :math:`1e-05` . - - **bound_relax_factor** (float): Factor for initial relaxation of the bounds. - Before start of the optimization, the bounds given by the user are - relaxed. This option sets the factor for this relaxation. Additional, the - constraint violation tolerance ``constr_viol_tol`` is used to bound the - relaxation by an absolute value. If it is set to zero, then then bounds - relaxation is disabled. See Eqn.(35) in implementation paper. Note that - the constraint violation reported by Ipopt at the end of the solution - process does not include violations of the original (non-relaxed) variable - bounds. See also option honor_original_bounds. The valid range for this - real option is 0 ≤ bound_relax_factor and its default value is :math:`1e-08` . - - **honor_original_bounds** (str or bool): Indicates whether final points should - be projected into original bunds. Ipopt might relax the bounds during the - optimization (see, e.g., option ``bound_relax_factor`` ). This option - determines whether the final point should be projected back into the - user-provide original bounds after the optimization. Note that violations - of constraints and complementarity reported by Ipopt at the end of the - solution process are for the non-projected point. The default value for - this string option is "no". Possible values: 'yes', 'no', True, False - - - **check_derivatives_for_naninf (str)**: whether to check for NaN / inf in the - derivative matrices. - Activating this option will cause an error if an - invalid number is detected in the constraint Jacobians or the Lagrangian - Hessian. If this is not activated, the test is skipped, and the algorithm - might proceed with invalid numbers and fail. If test is activated and an - invalid number is detected, the matrix is written to output with - print_level corresponding to J_MORE_DETAILED; so beware of large output! - The default value for this string option is "no". - - **jac_c_constant (str or bool)**: Indicates whether to assume that all equality - constraints are linear Activating this option will cause Ipopt to ask - for the Jacobian of the equality constraints only once from the NLP and - reuse this information later. The default value for this string option - is "no". Possible values: yes, no, True, False. - - **jac_d_constant (str or bool)**: Indicates whether to - assume that all inequality constraints are linear Activating this option - will cause Ipopt to ask for the Jacobian of the inequality constraints - only once from the NLP and reuse this information later. The default value - for this string option is "no". Possible values: yes, no, True, False - - **hessian_constant (str or bool)**: Indicates whether to assume the problem is a QP - (quadratic objective, linear constraints). Activating this option will - cause Ipopt to ask for the Hessian of the Lagrangian function only once - from the NLP and reuse this information later. The default value for this - string option is "no". Possible values: yes, no, True, False. - - - **nlp_scaling_method (str)**: Select the technique used for scaling the NLP. - Selects the technique used for scaling the problem internally before it is - solved. For user-scaling, the parameters come from the NLP. If you are - using AMPL, they can be specified through suffixes ("scaling_factor") The - default value for this string option is "gradient-based". Possible values: - - - "none": no problem scaling will be performed - "user-scaling": scaling - parameters will come from the user - "gradient-based": - scale the problem so the maximum gradient at the starting point is - ``nlp_scaling_max_gradient`` . - - "equilibration-based": scale the problem so that first derivatives are - of order 1 at random points (uses Harwell routine MC19) - - **obj_scaling_factor** (float): Scaling factor for the objective function. - This option sets a scaling factor for the objective function. The - scaling is seen internally by Ipopt but the unscaled objective is - reported in the console output. If additional scaling parameters are - computed (e.g. user-scaling or gradient-based), both factors are - multiplied. If this value is chosen to be negative, Ipopt will maximize - the objective function instead of minimizing it. The valid range for - this real option is unrestricted and its default value is 1. - - **nlp_scaling_max_gradient** (float): Maximum gradient after NLP scaling. - This is the gradient scaling cut-off. If the maximum gradient is above - this value, then gradient based scaling will be performed. Scaling - parameters are calculated to scale the maximum gradient back to this - value. (This is g_max in Section 3.8 of the implementation paper.) Note: - This option is only used if ``nlp_scaling_method`` is chosen as - "gradient-based". The valid range for this real option is :math:`0 < - \text{nlp_scaling_max_gradient}` and its default value is 100. - - **nlp_scaling_obj_target_gradient** (float): advanced! Target value for - objective function gradient size. If a positive number is chosen, the - scaling factor for the objective function is computed so that the - gradient has the max norm of the given size at the starting point. This - overrides ``nlp_scaling_max_gradient`` for the objective function. The valid - range for this real option is 0 ≤ nlp_scaling_obj_target_gradient and - its default value is 0. - - **nlp_scaling_constr_target_gradient** (float): arget value for constraint function gradient size. If a positive number is chosen, the scaling factors for the constraint functions are computed so that the gradient has the max norm of the given size at the starting point. This overrides nlp_scaling_max_gradient for the constraint functions. The valid range for this real option is 0 ≤ nlp_scaling_constr_target_gradient and its default value is 0. - - **nlp_scaling_min_value** (float): Minimum value of - gradient-based scaling values. This is the lower bound for the scaling - factors computed by gradient-based scaling method. If some derivatives - of some functions are huge, the scaling factors will otherwise become - very small, and the (unscaled) final constraint violation, for example, - might then be significant. Note: This option is only used if - ``nlp_scaling_method`` is chosen as "gradient-based". The valid range for - this real option is 0 ≤ nlp_scaling_min_value and its default value is - :math:`1e-08`. - - - **bound_push** (float): Desired minimum absolute distance from the initial - point to bound. Determines how much the initial point might have to be - modified in order to be sufficiently inside the bounds (together with - ``bound_frac`` ). (This is kappa_1 in Section 3.6 of implementation paper.) - The valid range for this real option is 0 < bound_push and its default - value is 0.01. - - **bound_frac** (float): Desired minimum relative distance - from the initial point to bound. Determines how much the initial point - might have to be modified in order to be sufficiently inside the bounds - (together with "bound_push"). (This is kappa_2 in Section 3.6 of - implementation paper.) The valid range for this real option is 0 < - bound_frac ≤ 0.5 and its default value is 0.01. - - **slack_bound_push** (float): Desired minimum absolute distance from the - initial slack to bound. Determines how much the initial slack - variables might have to be modified in order to be sufficiently inside the inequality bounds - (together with ``slack_bound_frac`` ). (This is kappa_1 in Section 3.6 of - implementation paper.) The valid range for this real option is 0 < - slack_bound_push and its default value is 0.01. - - **slack_bound_frac** (float): Desired minimum relative distance from the - initial slack to bound. Determines how much the initial slack - variables might have to be modified in order to be sufficiently inside the inequality bounds - (together with ``slack_bound_push`` ). (This is kappa_2 in Section 3.6 of - implementation paper.) The valid range for this real option is 0 < - slack_bound_frac ≤ 0.5 and its default value is 0.01. - - **constr_mult_init_max** (float): Maximum allowed least-square guess of - constraint multipliers. Determines how large the initial least-square - guesses of the constraint multipliers are allowed to be (in max-norm). - If the guess is larger than this value, it is discarded and all - constraint multipliers are set to zero. This options is also used when - initializing the restoration phase. By default, - "resto.constr_mult_init_max" (the one used in RestoIterateInitializer) - is set to zero. The valid range for this real option is 0 ≤ - constr_mult_init_max and its default value is 1000. - - **bound_mult_init_val** (float): Initial value for the bound multipliers. - All dual variables corresponding to bound constraints are initialized - to this value. The valid range for this real option is - 0 < bound_mult_init_val and its default value is 1. - - **bound_mult_init_method (str)**: Initialization method - for bound multipliers This option defines how the iterates for the bound - multipliers are initialized. If "constant" is chosen, then all bound - multipliers are initialized to the value of ``bound_mult_init_val``. If - "mu-based" is chosen, the each value is initialized to the the value of - "mu_init" divided by the corresponding slack variable. This latter - option might be useful if the starting point is close to the optimal - solution. The default value for this string option is "constant". - Possible values: - - - "constant": set all bound multipliers to the value of ``bound_mult_init_val`` - - "mu-based": initialize to mu_init/x_slack - - **least_square_init_primal (str or bool)**: - Least square initialization of the primal variables. If set to - yes, Ipopt ignores the user provided point and solves a least square - problem for the primal variables (x and s) to fit the linearized - equality and inequality constraints.This might be useful if the user - doesn't know anything about the starting point, or for solving an LP or - QP. The default value for this string option is "no". Possible values: - - - "no": take user-provided point - - "yes": overwrite user-provided point with least-square estimates - - **least_square_init_duals (str or bool)**: Least square - initialization of all dual variables If set to yes, Ipopt tries to - compute least-square multipliers (considering ALL dual variables). If - successful, the bound multipliers are possibly corrected to be at - least ``bound_mult_init_val`` . This might be useful if the user doesn't - know anything about the starting point, or for solving an LP or QP. - This overwrites option ``bound_mult_init_method`` . The default value for - this string option is "no". Possible values: - - - "no": use ``bound_mult_init_val`` and least-square equality constraint multipliers - - "yes": overwrite user-provided point with least-square estimates - - **warm_start_init_point (str or bool)**: Warm-start for initial point - Indicates whether this optimization should use a warm start - initialization, where values of primal and dual variables are given - (e.g., from a previous optimization of a related problem.) The default - value for this string option is "no". Possible values: - - - "no" or False: do not use the warm start initialization - - "yes" or True: use the warm start initialization - - **warm_start_same_structure (str or bool)**: - Advanced feature! Indicates whether a problem with a structure - identical t the previous one is to be solved. If enabled, then the - algorithm assumes that an NLP is now to be solved whose structure is - identical to one that already was considered (with the same NLP - object). The default value for this string option is "no". Possible - values: yes, no, True, False. - - **warm_start_bound_push** (float): same as - ``bound_push`` for the regular initializer. The valid range for this real - option is 0 < warm_start_bound_push and its default value is 0.001. - - **warm_start_bound_frac** (float): same as ``bound_frac`` for the regular - initializer The valid range for this real option is 0 < - warm_start_bound_frac ≤ 0.5 and its default value is 0.001. - - **warm_start_slack_bound_push** (float): same as ``slack_bound_push`` for the - regular initializer The valid range for this real option is 0 < - warm_start_slack_bound_push and its default value is 0.001. - - **warm_start_slack_bound_frac** (float): same as ``slack_bound_frac`` for the - regular initializer The valid range for this real option is 0 < - warm_start_slack_bound_frac ≤ 0.5 and its default value is 0.001. - - **warm_start_mult_bound_push** (float): same as ``mult_bound_push`` for the - regular initializer The valid range for this real option is 0 < - warm_start_mult_bound_push and its default value is 0.001. - - **warm_start_mult_init_max** (float): Maximum initial value for the - equality multipliers. The valid range for this real option is - unrestricted and its default value is :math:`1e+06` . - - **warm_start_entire_iterate (str or bool)**: Tells algorithm whether to use the GetWarmStartIterate - method in the NLP. The default value for this string option is "no". - Possible values: - - - "no": call GetStartingPoint in the NLP - - "yes": call GetWarmStartIterate in the NLP - - **warm_start_target_mu** (float): Advanced and experimental! The valid range - for this real option is unrestricted and its default value is 0. - - - **option_file_name (str)**: File name of options file. By default, the name - of the Ipopt options file is "ipopt.opt" - or something else if - specified in the IpoptApplication::Initialize call. If this option is - set by SetStringValue BEFORE the options file is read, it specifies the - name of the options file. It does not make any sense to specify this - option within the options file. Setting this option to an empty string - disables reading of an options file. - - **replace_bounds (bool or str)**: - Whether all variable bounds should be replaced by inequality - constraints. This option must be set for the inexact algorithm. The - default value for this string option is "no". Possible values: "yes", - "no", True, False. - - **skip_finalize_solution_call (str or bool)**: Whether a - call to NLP::FinalizeSolution after optimization should be suppressed. - In some Ipopt applications, the user might want to call the - FinalizeSolution method separately. Setting this option to "yes" will - cause the IpoptApplication object to suppress the default call to that - method. The default value for this string option is "no". Possible - values: "yes", "no", True, False - - **timing_statistics (str or bool)**: - Indicates whether to measure time spend in components of Ipopt and NLP - evaluation. The overall algorithm time is unaffected by this option. - The default value for this string option is "no". Possible values: - "yes", "no", True, False - - - **mu_max_fact** (float): Factor for initialization of maximum value for - barrier parameter. This option determines the upper bound on the barrier - parameter. This upper bound is computed as the average complementarity - at the initial point times the value of this option. (Only used if - option "mu_strategy" is chosen as "adaptive".) The valid range for this - real option is 0 < mu_max_fact and its default value is 1000. - - **mu_max** (float): Maximum value for barrier parameter. This option specifies an - upper bound on the barrier parameter in the adaptive mu selection mode. - If this option is set, it overwrites the effect of mu_max_fact. (Only - used if option "mu_strategy" is chosen as "adaptive".) The valid range - for this real option is 0 < mu_max and its default value is - 100000. - - **mu_min** (float): Minimum value for barrier parameter. This option - specifies the lower bound on the barrier parameter in the adaptive mu - selection mode. By default, it is set to the minimum of :math:`1e-11` and - min( ``tol`` , ``compl_inf_tol`` )/( ``barrier_tol_factor`` +1), which should be a - reasonable value. (Only used if option ``mu_strategy`` is chosen as - "adaptive".) The valid range for this real option is 0 < mu_min and its - default value is :math:`1e-11` . - - **adaptive_mu_globalization (str)**: Globalization - strategy for the adaptive mu selection mode. To achieve global - convergence of the adaptive version, the algorithm has to switch to the - monotone mode (Fiacco-McCormick approach) when convergence does not seem - to appear. This option sets the criterion used to decide when to do this - switch. (Only used if option "mu_strategy" is chosen as "adaptive".) The - default value for this string option is "obj-constr-filter". Possible - values: - - - "kkt-error": nonmonotone decrease of kkt-error - - "obj-constr-filter": 2-dim filter for objective and constraint violation - - "never-monotone-mode": disables globalization. - - **adaptive_mu_kkterror_red_iters** (float): advanced feature! Maximum - number of iterations requiring sufficient progress. For the - "kkt-error" based globalization strategy, sufficient progress must be - made for "adaptive_mu_kkterror_red_iters" iterations. If this number - of iterations is exceeded, the globalization strategy switches to the - monotone mode. The valid range for this integer option is 0 ≤ - adaptive_mu_kkterror_red_iters and its default value is 4. - - **adaptive_mu_kkterror_red_fact** (float): advanced feature! Sufficient - decrease factor for "kkt-error" globalization strategy. For the - "kkt-error" based globalization strategy, the error must decrease by - this factor to be deemed sufficient decrease. The valid range for this - real option is 0 < adaptive_mu_kkterror_red_fact < 1 and its default - value is 0.9999. - - **filter_margin_fact** (float): advanced feature! Factor - determining width of margin for obj-constr-filter adaptive - globalization strategy. When using the adaptive globalization - strategy, "obj-constr-filter", sufficient progress for a filter entry - is defined as follows: (new obj) < (filter obj) - - filter_margin_fact*(new constr-viol) OR (new constr-viol) < (filter - constr-viol) - filter_margin_fact*(new constr-viol). For the - description of the "kkt-error-filter" option see ``filter_max_margin`` . - The valid range for this real option is 0 < filter_margin_fact < 1 and - its default value is :math:`10-05` . - - **filter_max_margin** (float): advanced - feature! Maximum width of margin in obj-constr-filter adaptive - globalization strategy. The valid range for this real option is 0 < - filter_max_margin and its default value is 1. - - **adaptive_mu_restore_previous_iterate (str or bool)**: advanced feature! - Indicates if the previous accepted iterate should be restored if the - monotone mode is entered. When the globalization strategy for the - adaptive barrier algorithm switches to the monotone mode, it can - either start from the most recent iterate (no), or from the last - iterate that was accepted (yes). The default value for this string - option is "no". Possible values: "yes", "no", True, False - - **adaptive_mu_monotone_init_factor** (float): advanced feature! Determines - the initial value of the barrier parameter when switching to the - monotone mode. When the globalization strategy for the adaptive - barrier algorithm switches to the monotone mode and fixed_mu_oracle is - chosen as "average_compl", the barrier parameter is set to the current - average complementarity times the value of - "adaptive_mu_monotone_init_factor". The valid range for this real - option is 0 < adaptive_mu_monotone_init_factor and its default value - is 0.8. - - **adaptive_mu_kkt_norm_type (str)**: advanced! Norm used for the KKT - error in the adaptive mu globalization strategies. When computing the - KKT error for the globalization strategies, the norm to be used is - specified with this option. Note, this option is also used in the - QualityFunctionMuOracle. The default value for this string option is - "2-norm-squared". Possible values: - - - "1-norm": use the 1-norm (abs sum) - - "2-norm-squared": use the 2-norm squared (sum of squares) - - "max-norm": use the infinity norm (max) - - "2-norm": use 2-norm - - **mu_strategy (str)**: Update strategy for barrier - parameter. Determines which barrier parameter update strategy is to be - used. The default value for this string option is "monotone". Possible values: - - - "monotone": use the monotone (Fiacco-McCormick) strategy - - "adaptive": use the adaptive update strategy - - **mu_oracle (str)**: Oracle for a new barrier parameter in the adaptive strategy. - Determines how a new barrier parameter is computed in each "free-mode" iteration of the - adaptive barrier parameter strategy. (Only considered if "adaptive" is - selected for option "mu_strategy"). The default value for this string - option is "quality-function". Possible values: - - - "probing": Mehrotra's probing heuristic - - "loqo": LOQO's centrality rule - - "quality-function": minimize a quality function - - **fixed_mu_oracle (str)**: - Oracle for the barrier parameter when switching to fixed mode. - Determines how the first value of the barrier parameter should be - computed when switching to the "monotone mode" in the adaptive - strategy. (Only considered if "adaptive" is selected for option - "mu_strategy".) The default value for this string option is - "average_compl". Possible values: - - - "probing": Mehrotra's probing heuristic - - "loqo": LOQO's centrality rule - - "quality-function": minimize a quality function - - "average_compl": base on current average complementarity - - **mu_init** (float): Initial value for the barrier parameter. This option - determines the initial value for the barrier parameter (mu). It is - only relevant in the monotone, Fiacco-McCormick version of the - algorithm. (i.e., if "mu_strategy" is chosen as "monotone") The valid - range for this real option is 0 < mu_init and its default value is 0.1. - - **barrier_tol_factor** (float): Factor for mu in barrier stop test. - The convergence tolerance for each barrier problem in the monotone - mode is the value of the barrier parameter times "barrier_tol_factor". - This option is also used in the adaptive mu strategy during the - monotone mode. This is kappa_epsilon in implementation paper. The - valid range for this real option is 0 < barrier_tol_factor and its - default value is 10. - - **mu_linear_decrease_factor** (float): Determines - linear decrease rate of barrier parameter. For the Fiacco-McCormick - update procedure the new barrier parameter mu is obtained by taking - the minimum of mu*"mu_linear_decrease_factor" and - mu^"superlinear_decrease_power". This is kappa_mu in implementation - paper. This option is also used in the adaptive mu strategy during the - monotone mode. The valid range for this real option is 0 < - mu_linear_decrease_factor < 1 and its default value is 0.2. - - **mu_superlinear_decrease_power** (float): Determines superlinear decrease - rate of barrier parameter. For the Fiacco-McCormick update procedure - the new barrier parameter mu is obtained by taking the minimum of - mu*"mu_linear_decrease_factor" and mu^"superlinear_decrease_power". - This is theta_mu in implementation paper. This option is also used in - the adaptive mu strategy during the monotone mode. The valid range for - this real option is 1 < mu_superlinear_decrease_power < 2 and its - default value is 1.5. - - **mu_allow_fast_monotone_decrease (str or bool)**: - Advanced feature! Allow skipping of barrier problem if barrier test i - already met. The default value for this string option is "yes". - Possible values: - - - "no": Take at least one iteration per barrier problem even if the - barrier test is already met for the updated barrier parameter - - "yes": Allow fast decrease of mu if barrier test it met - - **tau_min** (float): Advanced feature! Lower bound on fraction-to-the-boundary - parameter tau. This is tau_min in the implementation paper. This - option is also used in the adaptive mu strategy during the monotone - mode. The valid range for this real option is 0 < tau_min < 1 and its - default value is 0.99. - - **sigma_max** (float): Advanced feature! Maximum - value of the centering parameter. This is the upper bound for the - centering parameter chosen by the quality function based barrier - parameter update. Only used if option "mu_oracle" is set to - "quality-function". The valid range for this real option is 0 < - sigma_max and its default value is 100. - - - **sigma_min** (float): Advanced - feature! Minimum value of the centering parameter. This is the lower - bound for the centering parameter chosen by the quality function based - barrier parameter update. Only used if option "mu_oracle" is set to - "quality-function". The valid range for this real option is 0 ≤ - sigma_min and its default value is :math:`10-06` . - - **quality_function_norm_type (str)**: Advanced feature. - Norm used for components of the quality - function. Only used if option "mu_oracle" is set to - "quality-function". The default value for this string option is - "2-norm-squared". Possible values: - - - "1-norm": use the 1-norm (abs sum) - - "2-norm-squared": use the 2-norm squared (sum of squares) - - "max-norm": use the infinity norm (max) - - "2-norm": use 2-norm - - **quality_function_centrality (str)**: Advanced - feature. The penalty term for centrality that is included in quality - function. This determines whether a term is added to the quality - function to penalize deviation from centrality with respect to - complementarity. The complementarity measure here is the xi in the - Loqo update rule. Only used if option "mu_oracle" is set to - "quality-function". The default value for this string option is - "none". Possible values: - - - "none": no penalty term is added - - "log": complementarity * the log of the centrality measure - - "reciprocal": complementarity * the reciprocal of the centrality - measure - - "cubed-reciprocal": complementarity * the reciprocal of the centrality - measure cubed - - **quality_function_balancing_term (str)**: Advanced - feature. The balancing term included in the quality function for - centrality. This determines whether a term is added to the quality - function that penalizes situations where the complementarity is much - smaller than dual and primal infeasibilities. Only used if option - "mu_oracle" is set to "quality-function". The default value for this - string option is "none". Possible values: - - - "none": no balancing term is adde - - "cubic": :math:`max(0,\max(\text{dual_inf},\text{primal_inf})-\text{compl})^3` - - **quality_function_max_section_steps** (int): Maximum number of search - steps during direct search procedure determining the optimal centering - parameter. The golden section search is performed for the quality - function based mu oracle. Only used if option "mu_oracle" is set to - "quality-function". The valid range for this integer option is 0 ≤ - quality_function_max_section_steps and its default value is 8. - - **quality_function_section_sigma_tol** (float): advanced feature! - Tolerance for the section search procedure determining the optimal - centering parameter (in sigma space). The golden section search is - performed for the quality function based mu oracle. Only used if - option "mu_oracle" is set to "quality-function". The valid range for - this real option is 0 ≤ quality_function_section_sigma_tol < 1 and its - default value is 0.01. - - **quality_function_section_qf_tol** (float): - advanced feature! Tolerance for the golden section search procedure - determining the optimal centering parameter (in the function value - space). The golden section search is performed for the quality - function based mu oracle. Only used if option "mu_oracle" is set to - "quality-function". The valid range for this real option is 0 ≤ - quality_function_section_qf_tol < 1 and its default value is 0. - - - **line_search_method (str)**: Advanced feature. Globalization method used in - backtracking line search. Only the "filter" choice is officially - supported. But sometimes, good results might be obtained with the other - choices. The default value for this string option is "filter". Possible values: - - - "filter": Filter method - - "cg-penalty": Chen-Goldfarb penalty function - - "penalty": Standard penalty function - - **alpha_red_factor** (float): Advanced feature. - Fractional reduction of the trial step size - in the backtracking lne search. At every step of the backtracking line - search, the trial step size is reduced by this factor. The valid range - for this real option is 0 < alpha_red_factor < 1 and its default value - is 0.5. - - **accept_every_trial_step (str or bool)**: Always accept the first - trial step. Setting this option to "yes" essentially disables the line - search and makes the algorithm take aggressive steps, without global - convergence guarantees. The default value for this string option is - "no". Possible values: "yes", "no", True, False. - - **accept_after_max_steps** (float): advanced feature. - Accept a trial point after maximal this - number of steps een if it does not satisfy line search conditions. - Setting this to -1 disables this option. The valid range for this - integer option is -1 ≤ accept_after_max_steps and its default value is -1. - - **alpha_for_y (str)**: Method to determine the step size for constraint - multipliers (alpha_y) . The default value for this string option is - "primal". Possible values: - - - "primal": use primal step size - - "bound-mult": use step size for the bound multipliers (good for LPs) - - "min": use the min of primal and bound multipliers - - "max": use the max of primal and bound multipliers - - "full": take a full step of size one - - "min-dual-infeas": choose step size minimizing new dual infeasibility - - "safer-min-dual-infeas": like "min_dual_infeas", but safeguarded by - "min" and "max" - - "primal-and-full": use the primal step size, and full step if - delta_x <= alpha_for_y_tol - - "dual-and-full": use the dual step size, and full step if - delta_x <= alpha_for_y_tol - - "acceptor": Call LSAcceptor to get step size for y - - **alpha_for_y_tol** (float): Tolerance for - switching to full equality multiplier steps. This is only relevant if - "alpha_for_y" is chosen "primal-and-full" or "dual-and-full". The step - size for the equality constraint multipliers is taken to be one if the - max-norm of the primal step is less than this tolerance. The valid range - for this real option is 0 ≤ alpha_for_y_tol and its default value is 10. - - **tiny_step_tol** (float): Advanced feature. Tolerance for detecting - numerically insignificant steps. If the search direction in the primal - variables (x and s) is, in relative terms for each component, less than - this value, the algorithm accepts the full step without line search. If - this happens repeatedly, the algorithm will terminate with a - corresponding exit message. The default value is 10 times machine - precision. The valid range for this real option is 0 ≤ tiny_step_tol and - its default value is 2.22045 · :math:`1e-15`. - - **tiny_step_y_tol** (float): Advanced - feature. Tolerance for quitting because of numerically insignificant - steps. If the search direction in the primal variables (x and s) is, in - relative terms for each component, repeatedly less than tiny_step_tol, - and the step in the y variables is smaller than this threshold, the - algorithm will terminate. The valid range for this real option is 0 ≤ - tiny_step_y_tol and its default value is 0.01. - - - **watchdog_shortened_iter_trigger** (int): Number of shortened iterations - that trigger the watchdog. If the number of successive iterations in - which the backtracking line search did not accept the first trial point - exceeds this number, the watchdog procedure is activated. Choosing "0" - here disables the watchdog procedure. The valid range for this integer - option is 0 ≤ watchdog_shortened_iter_trigger and its default value is - 10. - - **watchdog_trial_iter_max** (int): Maximum number of watchdog - iterations. This option determines the number of trial iterations - allowed before the watchdog procedure is aborted and the algorithm - returns to the stored point. The valid range for this integer option - is 1 ≤ watchdog_trial_iter_max and its default value is 3. - theta_max_fact (float): Advanced feature. Determines upper bound for - constraint violation in the filter. The algorithmic parameter - theta_max is determined as theta_max_fact times the maximum of 1 and - the constraint violation at initial point. Any point with a - constraint violation larger than theta_max is unacceptable to the - filter (see Eqn. (21) in the implementation paper). The valid range - for this real option is 0 < theta_max_fact and its default value is - 10000. - - **theta_min_fact** (float): advanced feature. Determines - constraint violation threshold in the switching rule. The - algorithmic parameter theta_min is determined as - theta_min_fact times the maximum of 1 and the constraint - violation at initial point. The switching rules treats an - iteration as an h-type iteration whenever the current - constraint violation is larger than theta_min (see paragraph - before Eqn. (19) in the implementation paper). The valid - range for this real option is 0 < theta_min_fact and its - default value is 0.0001. - - **eta_phi** (float): advanced! - Relaxation factor in the Armijo condition. See Eqn. (20) in - the implementation paper. The valid range for this real - option is 0 < eta_phi < 0.5 and its default value is :math:`1e-08`. - - **delta** (float): advanced! Multiplier for constraint violation - in the switching rule. See Eqn. (19) in the implementation - paper. The valid range for this real option is 0 < delta and - its default value is 1. - - **s_phi** (float): advanced! Exponent for - linear barrier function model in the switching rule. See Eqn. - (19) in the implementation paper. The valid range for this - real option is 1 < s_phi and its default value is 2.3. - - **s_theta** (float): advanced! Exponent for current constraint - violation in the switching rule. See Eqn. (19) in the - implementation paper. The valid range for this real option is - 1 < s_theta and its default value is 1.1. - - **gamma_phi** (float): - advanced! Relaxation factor in the filter margin for the - barrier function. See Eqn. (18a) in the implementation paper. - The valid range for this real option is 0 < gamma_phi < 1 and - its default value is :math:`1e-08`. - - **gamma_theta** (float): advanced! - Relaxation factor in the filter margin for the constraint - violation. See Eqn. (18b) in the implementation paper. The - valid range for this real option is 0 < gamma_theta < 1 and - its default value is :math:`1e-05`. - - **alpha_min_frac** (float): advanced! - Safety factor for the minimal step size (before switching to - restoration phase). This is gamma_alpha in Eqn. (20) in the - implementation paper. The valid range for this real option is - 0 < alpha_min_frac < 1 and its default value is 0.05. - - **max_soc** (int): Maximum number of second order correction trial steps - at each iteration. Choosing 0 disables the second order - corrections. This is p^{max} of Step A-5.9 of Algorithm A in - the implementation paper. The valid range for this integer - option is 0 ≤ max_soc and its default value is 4. - - **kappa_soc** (float): advanced! Factor in the sufficient reduction rule - for second order correction. This option determines how much - a second order correction step must reduce the constraint - violation so that further correction steps are attempted. See - Step A-5.9 of Algorithm A in the implementation paper. The - valid range for this real option is 0 < kappa_soc and its - default value is 0.99. - - **obj_max_inc** (float): advanced! - Determines the upper bound on the acceptable increase of - barrier objective function. Trial points are rejected if they - lead to an increase in the barrier objective function by more - than obj_max_inc orders of magnitude. The valid range for - this real option is 1 < obj_max_inc and its default value is 5. - - **max_filter_resets** (int): advanced! Maximal allowed number - of filter resets. A positive number enables a heuristic - that resets the filter, whenever in more than - "filter_reset_trigger" successive iterations the last - rejected trial steps size was rejected because of the - filter. This option determine the maximal number of resets - that are allowed to take place. The valid range for this - integer option is 0 ≤ max_filter_resets and its default - value is 5. - - **filter_reset_trigger** (int): Advanced! Number - of iterations that trigger the filter reset. If the filter - reset heuristic is active and the number of successive - iterations in which the last rejected trial step size was - rejected because of the filter, the filter is reset. The - valid range for this integer option is 1 ≤ - filter_reset_trigger and its default value is 5. - - **corrector_type (str)**: advanced! The type of corrector steps that should - be taken. If "mu_strategy" is "adaptive", this option determines what - kind of corrector steps should be tried. Changing this option is - experimental. The default value for this string option is "none". - Possible values: - - - "none" or None: no corrector - - "affine": corrector step towards mu=0 - - "primal-dual": corrector step towards current mu - - **skip_corr_if_neg_curv (str or bool)**: advanced! - Whether to skip the corrector step in negative curvature - iteration. The corrector step is not tried if negative curvature has been - encountered during the computation of the search direction in the current - iteration. This option is only used if "mu_strategy" is "adaptive". - Changing this option is experimental. The default value for this string - option is "yes". Possible values: "yes", "no", True, False. - - **skip_corr_in_monotone_mode (str or bool)**: Advanced! Whether to skip the - corrector step during monotone brrier parameter mode. The corrector step - is not tried if the algorithm is currently in the monotone mode (see also - option "barrier_strategy"). This option is only used if "mu_strategy" is - "adaptive". Changing this option is experimental. The default value for - this string option is "yes". Possible values: "yes", "no", True, False - - **corrector_compl_avrg_red_fact** (float): advanced! Complementarity tolerance - factor for accepting corrector step. This option determines the factor by - which complementarity is allowed to increase for a corrector step to be - accepted. Changing this option is experimental. The valid range for this - real option is 0 < corrector_compl_avrg_red_fact and its default value is - 1. - - **soc_method** (int): Ways to apply second order correction. This option - determines the way to apply second order correction, 0 is the method - described in the implementation paper. 1 is the modified way which adds - alpha on the rhs of x and s rows. Officially, the valid range for this - integer option is 0 ≤ soc_method ≤ 1 and its default value is 0 but only 0 - and 1 are allowed. - - - **nu_init** (float): advanced! Initial value of the penalty parameter. The - valid range for this real option is 0 < nu_init and its default value is - :math:`1e-06`. - - **nu_inc** (float): advanced! Increment of the penalty parameter. The - valid range for this real option is 0 < nu_inc and its default value is - 0.0001. - - **rho** (float): advanced! Value in penalty parameter update formula. - The valid range for this real option is 0 < rho < 1 and its default value - is 0.1. - - **kappa_sigma** (float): advanced! Factor limiting the deviation of - dual variables from primal estimates. If the dual variables deviate from - their primal estimates, a correction is performed. See Eqn. (16) in the - implementation paper. Setting the value to less than 1 disables the - correction. The valid range for this real option is 0 < kappa_sigma and - its default value is :math:`1e+10`. - - **recalc_y (str or bool)**: Tells the algorithm to - recalculate the equality and inequality multipliers as least square - estimates. This asks the algorithm to recompute the multipliers, whenever - the current infeasibility is less than recalc_y_feas_tol. Choosing yes - might be helpful in the quasi-Newton option. However, each recalculation - requires an extra factorization of the linear system. If a limited memory - quasi-Newton option is chosen, this is used by default. The default value - for this string option is "no". Possible values: - - - "no" or False: use the Newton step to update the multipliers - - "yes" or True: use least-square multiplier - - **estimates recalc_y_feas_tol** (float): Feasibility threshold for - recomputation of multipliers. If recalc_y is chosen and the current - infeasibility is less than this value, then the multipliers are - recomputed. The valid range for this real option is 0 < recalc_y_feas_tol - and its default value is :math:`1e-06`. - - **slack_move** (float): advanced! Correction - size for very small slacks. Due to numerical issues or the lack of an - interior, the slack variables might become very small. If a slack becomes - very small compared to machine precision, the corresponding bound is moved - slightly. This parameter determines how large the move should be. Its - default value is mach_eps^{3/4}. See also end of Section 3.5 in - implementation paper - but actual implementation might be somewhat - different. The valid range for this real option is 0 ≤ slack_move and its - default value is 1.81899 · :math:`1e-12`. - - **constraint_violation_norm_type (str)**: advanced! - Norm to be used for the constraint violation in te line search. - Determines which norm should be used when the algorithm computes the - constraint violation in the line search. The default value for this string - option is "1-norm". Possible values: - - - "1-norm": use the 1-norm - - "2-norm": use the 2-norm - - "max-norm": use the infinity norm - - - **mehrotra_algorithm (str or bool)**: Indicates whether to do Mehrotra's - predictor-corrector algorithm. If enabled, line search is disabled and the - (unglobalized) adaptive mu strategy is chosen with the "probing" oracle, - and "corrector_type=affine" is used without any safeguards; you should not - set any of those options explicitly in addition. Also, unless otherwise - specified, the values of ``bound_push`` , ``bound_frac`` , and - ``bound_mult_init_val`` are set more aggressive, and sets - "alpha_for_y=bound_mult". The Mehrotra's predictor-corrector algorithm - works usually very well for LPs and convex QPs. The default value for this - string option is "no". Possible values: "yes", "no", True, False. - - **fast_step_computation (str or bool)**: Indicates if the linear system should - be solved quickly. If enabled, the algorithm assumes that the linear - system that is solved to obtain the search direction is solved - sufficiently well. In that case, no residuals are computed to verify the - solution and the computation of the search direction is a little faster. - The default value for this string option is "no". Possible values: "yes", - "no", True, False. - - **min_refinement_steps** (int): Minimum number of iterative - refinement steps per linear system solve. Iterative refinement (on the - full asymmetric system) is performed for each right hand side. This - option determines the minimum number of iterative refinements (i.e. at - least "min_refinement_steps" iterative refinement steps are enforced per - right hand side.) The valid range for this integer option is 0 ≤ - min_refinement_steps and its default value is 1. - - **max_refinement_steps** (int): Maximum number of iterative refinement - steps per linear system - solve. Iterative refinement (on the full unsymmetric system) is performed - for each right hand side. This option determines the maximum number of - iterative refinement steps. The valid range for this integer option is 0 ≤ - max_refinement_steps and its default value is 10. - - **residual_ratio_max** (float): advanced! Iterative refinement tolerance. - Iterative refinement is - performed until the residual test ratio is less than this tolerance (or - until "max_refinement_steps" refinement steps are performed). The valid - range for this real option is 0 < residual_ratio_max and its default value - is :math:`1e-10`. - - **residual_ratio_singular** (float): advanced! Threshold for - declaring linear system singular after filed iterative refinement. If the - residual test ratio is larger than this value after failed iterative - refinement, the algorithm pretends that the linear system is singular. The - valid range for this real option is 0 < residual_ratio_singular and its - default value is :math:`1e-05`. - - **residual_improvement_factor** (float): advanced! - Minimal required reduction of residual test ratio in iterative refinement. - If the improvement of the residual test ratio made by one iterative - refinement step is not better than this factor, iterative refinement is - aborted. The valid range for this real option is 0 < - residual_improvement_factor and its default value is 1. - - - **neg_curv_test_tol** (float): Tolerance for heuristic to ignore wrong - inertia. If nonzero, incorrect inertia in the augmented system is ignored, - and Ipopt tests if the direction is a direction of positive curvature. - This tolerance is alpha_n in the paper by :cite:`Chiang2014` and it - determines when the direction is considered to be sufficiently positive. A - value in the range of [1e-12, 1e-11] is recommended. The valid range for - this real option is 0 ≤ neg_curv_test_tol and its default value is 0. - - **neg_curv_test_reg (str or bool)**: Whether to do the curvature test with the - primal regularization (see :cite:`Chiang2014`). The default value for - this string option is "yes". Possible values: - - - "yes" or True: use primal regularization with the - inertia-free curvature test - - "no" or False: use original IPOPT approach, in which the - primal regularization is ignored - - **max_hessian_perturbation** (float): Maximum value of regularization - parameter for handling negative curvature. In order to guarantee that the - search directions are indeed proper descent directions, Ipopt requires - that the inertia of the (augmented) linear system for the step computation - has the correct number of negative and positive eigenvalues. The idea is - that this guides the algorithm away from maximizers and makes Ipopt more - likely converge to first order optimal points that are minimizers. If the - inertia is not correct, a multiple of the identity matrix is added to the - Hessian of the Lagrangian in the augmented system. This parameter gives - the maximum value of the regularization parameter. If a regularization of - that size is not enough, the algorithm skips this iteration and goes to - the restoration phase. This is delta_w^max in the implementation paper. - The valid range for this real option is 0 < max_hessian_perturbation and - its default value is :math:`1e+20`. - - **min_hessian_perturbation** (float): Smallest - perturbation of the Hessian block. The size of the perturbation of the - Hessian block is never selected smaller than this value, unless no - perturbation is necessary. This is delta_w^min in implementation paper. - The valid range for this real option is 0 ≤ min_hessian_perturbation and - its default value is :math:`1e-20`. - - **perturb_inc_fact_first** (float): Increase - factor for x-s perturbation for very first perturbation. The factor by - which the perturbation is increased when a trial value was not sufficient - - this value is used for the computation of the very first perturbation - and allows a different value for the first perturbation than that used - for the remaining perturbations. This is bar_kappa_w^+ in the - implementation paper. The valid range for this real option is 1 < - perturb_inc_fact_first and its default value is 100. - - **perturb_inc_fact** (float): Increase factor for x-s perturbation. The factor - by which the perturbation is increased when a trial value was not - sufficient - this value is used for the computation of all - perturbations except for - the first. This is kappa_w^+ in the implementation paper. The valid - range for this real option is 1 < perturb_inc_fact and its default value - is 8. - - **perturb_dec_fact** (float): Decrease factor for x-s perturbation. - The factor by which the perturbation is decreased when a trial value is - deduced from the size of the most recent successful perturbation. This - is kappa_w^- in the implementation paper. The valid range for this real - option is 0 < perturb_dec_fact < 1 and its default value is 0.333333. - - **first_hessian_perturbation** (float): Size of first x-s perturbation - tried. The first value tried for the x-s perturbation in the inertia - correction scheme. This is delta_0 in the implementation paper. The - valid range for this real option is 0 < first_hessian_perturbation and - its default value is 0.0001. - - **jacobian_regularization_value** (float): Size - of the regularization for rank-deficient constraint Jacobians. This is - bar delta_c in the implementation paper. The valid range for this real - option is 0 ≤ jacobian_regularization_value and its default value is - :math:`1e-08`. - - **jacobian_regularization_exponent** (float): advanced! Exponent for - mu in the regularization for rnk-deficient constraint Jacobians. This is - kappa_c in the implementation paper. The valid range for this real - option is 0 ≤ jacobian_regularization_exponent and its default value is - 0.25. - - **perturb_always_cd (str or bool)**: advanced! Active permanent - perturbation of constraint linearization. Enabling this option leads to - using the delta_c and delta_d perturbation for the computation of every - search direction. Usually, it is only used when the iteration matrix is - singular. The default value for this string option is "no". Possible - values: "yes", "no", True, False. - - - **expect_infeasible_problem (str or bool)**: Enable heuristics to quickly - detect an infeasible problem. This options is meant to activate - heuristics that may speed up the infeasibility determination if you - expect that there is a good chance for the problem to be infeasible. In - the filter line search procedure, the restoration phase is called more - quickly than usually, and more reduction in the constraint violation is - enforced before the restoration phase is left. If the problem is square, - this option is enabled automatically. The default value for this string - option is "no". Possible values: "yes", "no", True, False. - - **expect_infeasible_problem_ctol** (float): Threshold for disabling - "expect_infeasible_problem" option. If the constraint violation becomes - smaller than this threshold, the "expect_infeasible_problem" heuristics - in the filter line search are disabled. If the problem is square, this - options is set to 0. The valid range for this real option is 0 ≤ - expect_infeasible_problem_ctol and its default value is 0.001. - - **expect_infeasible_problem_ytol** (float): Multiplier threshold for - activating "xpect_infeasible_problem" option. If the max norm of the - constraint multipliers becomes larger than this value and - "expect_infeasible_problem" is chosen, then the restoration phase is - entered. The valid range for this real option is 0 < - expect_infeasible_problem_ytol and its default value is :math:`1e+08`. - - **start_with_resto (str or bool)**: Whether to switch to restoration phase - in first iteration.Setting this option to "yes" forces the algorithm to - switch to the feasibility restoration phase in the first iteration. If - the initial point is feasible, the algorithm will abort with a failure. - The default value for this string option is "no". Possible values: - "yes", "no", True, False - - **soft_resto_pderror_reduction_factor** (float): - Required reduction in primal-dual error in the soft restoration phase. - The soft restoration phase attempts to reduce the primal-dual error with - regular steps. If the damped primal-dual step (damped only to satisfy - the fraction-to-the-boundary rule) is not decreasing the primal-dual - error by at least this factor, then the regular restoration phase is - called. Choosing "0" here disables the soft restoration phase. The valid - range for this real option is 0 ≤ soft_resto_pderror_reduction_factor - and its default value is 0.9999. - - **max_soft_resto_iters** (int): advanced! - Maximum number of iterations performed successively in soft rstoration - phase. If the soft restoration phase is performed for more than so many - iterations in a row, the regular restoration phase is called. The valid - range for this integer option is 0 ≤ max_soft_resto_iters and its - default value is 10. - - **required_infeasibility_reduction** (float): Required - reduction of infeasibility before leaving restoration phase. The - restoration phase algorithm is performed, until a point is found that is - acceptable to the filter and the infeasibility has been reduced by at - least the fraction given by this option. The valid range for this real - option is 0 ≤ required_infeasibility_reduction < 1 and its default value - is 0.9. - - **max_resto_iter** (int): advanced! Maximum number of successive - iterations in restoration phase.The algorithm terminates with an error - message if the number of iterations successively taken in the - restoration phase exceeds this number. The valid range for this integer - option is 0 ≤ max_resto_iter and its default value is 3000000. - - **evaluate_orig_obj_at_resto_trial (str or bool)**: Determines if the - original objective function should be evaluated at restoration phase - trial points. Enabling this option makes the restoration phase algorithm - evaluate the objective function of the original problem at every trial - point encountered during the restoration phase, even if this value is - not required. In this way, it is guaranteed that the original objective - function can be evaluated without error at all accepted iterates; - otherwise the algorithm might fail at a point where the restoration - phase accepts an iterate that is good for the restoration phase problem, - but not the original problem. On the other hand, if the evaluation of - the original objective is expensive, this might be costly. The default - value for this string option is "yes". Possible values: "yes", "no", - True, False - - **resto_penalty_parameter** (float): advanced! Penalty parameter - in the restoration phase objective function. This is the parameter rho in - equation (31a) in the Ipopt implementation paper. The valid range for - this real option is 0 < resto_penalty_parameter and its default value is - 1000. - - **resto_proximity_weight** (float): advanced! Weighting factor for the - proximity term in restoration pase objective. This determines how - the parameter zeta in equation (29a) in the implementation paper - is computed. zeta here is resto_proximity_weight*sqrt(mu), where - mu is the current barrier parameter. The valid range for this real - option is 0 ≤ resto_proximity_weight and its default value is 1. - - **bound_mult_reset_threshold** (float): Threshold for resetting bound - multipliers after the restoration pase. After returning from the - restoration phase, the bound multipliers are updated with a Newton - step for complementarity. Here, the change in the primal variables - during the entire restoration phase is taken to be the - corresponding primal Newton step. However, if after the update the - largest bound multiplier exceeds the threshold specified by this - option, the multipliers are all reset to 1. - The valid range for this real option is 0 ≤ bound_mult_reset_threshold - and its default value is 1000. - - **constr_mult_reset_threshold** (float): - Threshold for resetting equality and inequality multipliers ater - restoration phase. After returning from the restoration phase, the - constraint multipliers are recomputed by a least square estimate. This - option triggers when those least-square estimates should be ignored. - The valid range for this real option is 0 ≤ constr_mult_reset_threshold - and its default value is 0. - - **resto_failure_feasibility_threshold** (float): advanced! - Threshold for primal infeasibility to declare failure - of restoration phase. If the restoration phase is terminated because of - the "acceptable" termination criteria and the primal infeasibility is - smaller than this value, the restoration phase is declared to have - failed. The default value is actually 1e2*tol, where tol is the general - termination tolerance. The valid range for this real option is 0 ≤ - resto_failure_feasibility_threshold and its default value is 0. - - - **limited_memory_aug_solver (str)**: advanced! Strategy for solving the - augmented system for low-rank Hessian. - The default value for this string option is "sherman-morrison". - Possible values: - - - "sherman-morrison": use Sherman-Morrison formula - - "extended": use an extended augmented system - - **limited_memory_max_history** (int): Maximum size of the history for the - limited quasi-Newton Hessian approximation. This option determines the - number of most recent iterations that are taken into account for the - limited-memory quasi-Newton approximation. The valid range for this - integer option is 0 ≤ limited_memory_max_history and - its default value is 6. - - **limited_memory_update_type (str)**: Quasi-Newton update formula for the - limited memory quasi-Newton approximation. The default value for this - string option is "bfgs". Possible values: - - - "bfgs": BFGS update (with skipping) - - "sr1": SR1 (not working well) - - **limited_memory_initialization (str)**: - Initialization strategy for the limited memory quasi-Newton - aproximation. Determines how the diagonal Matrix B_0 as the first term in - the limited memory approximation should be computed. The default value for - this string option is "scalar1". Possible values: - - - "scalar1": sigma = s^Ty/s^Ts - - "scalar2": sigma = y^Ty/s^Ty - - "scalar3": arithmetic average of scalar1 and scalar2 - - "scalar4": geometric average of scalar1 and scalar2 - - "constant": sigma = limited_memory_init_val - - **limited_memory_init_val** (float): Value for B0 in low-rank update. The - starting matrix in the low rank update, B0, is chosen to be this multiple - of the identity in the first iteration (when no updates have been - performed yet), and is constantly chosen as this value, if - "limited_memory_initialization" is "constant". The valid range for this - real option is 0 < limited_memory_init_val and its default value is 1. - - **limited_memory_init_val_max** (float): Upper bound on value for B0 in - low-rank update. The starting matrix in the low rank update, B0, is chosen - to be this multiple of the identity in the first iteration (when no - updates have been performed yet), and is constantly chosen as this value, - if "limited_memory_initialization" is "constant". The valid range for this - real option is 0 < limited_memory_init_val_max and its default value is - :math:`1e+08`. - - **limited_memory_init_val_min** (float): Lower bound on value for B0 in - low-rank update. The starting matrix in the low rank update, B0, is chosen - to be this multiple of the identity in the first iteration (when no - updates have been performed yet), and is constantly chosen as this value, - if "limited_memory_initialization" is "constant". The valid range for this - real option is 0 < limited_memory_init_val_min and its default value is - :math:`1e-08`. - - **limited_memory_max_skipping** (int): Threshold for successive - iterations where update is skipped. If the update is skipped more than - this number of successive iterations, the quasi-Newton approximation is - reset. The valid range for this integer option is 1 ≤ - limited_memory_max_skipping and its default value is 2. - - **limited_memory_special_for_resto (str or bool)**: Determines if the - quasi-Newton updates should be special dring the restoration phase. Until - Nov 2010, Ipopt used a special update during the restoration phase, but it - turned out that this does not work well. The new default uses the regular - update procedure and it improves results. If for some reason you want to - get back to the original update, set this option to "yes". The default - value for this string option is "no". Possible values: "yes", "no", True, - False. - - **hessian_approximation (str)**: Indicates what Hessian information is - to be used. This determines which kind of information for the Hessian of - the Lagrangian function is used by the algorithm. The default value for - this string option is "limited-memory". Possible values: - "exact": Use - second derivatives provided by the NLP. - "limited-memory": Perform a - limited-memory quasi-Newton approximation - - **hessian_approximation_space (str)**: advanced! - Indicates in which subspace the Hessian information is to - be approximated. The default value for this string option is - "nonlinear-variables". Possible values: - "nonlinear-variables": only in - space of nonlinear variables. - "all-variables": in space of all variables - (without slacks) - - **linear_solver (str)**: Linear solver used for step - computations. Determines which linear algebra package is to be used for - the solution of the augmented linear system (for obtaining the search - directions). The default value for this string option is "ma27". Possible - values: - - - "mumps" (use the Mumps package, default) - - "ma27" (load the Harwell routine MA27 from library at runtime) - - "ma57" (load the Harwell routine MA57 from library at runtime) - - "ma77" (load the Harwell routine HSL_MA77 from library at runtime) - - "ma86" (load the Harwell routine MA86 from library at runtime) - - "ma97" (load the Harwell routine MA97 from library at runtime) - - "pardiso" (load the Pardiso package from pardiso-project.org - from user-provided library at runtime) - - "custom" (use custom linear solver (expert use)) - - **linear_solver_options** (dict or None): dictionary with the - linear solver options, possibly including `linear_system_scaling`, - `hsllib` and `pardisolib`. See the `ipopt documentation for details - `_. The linear solver - options are not automatically converted to float at the moment.] + **Description and available options:** + .. autoclass:: optimagic.optimizers.ipopt.Ipopt ``` (fides-algorithm)= @@ -3265,106 +1438,33 @@ need to have [the fides package](https://github.com/fides-dev/fides) installed (`pip install fides>=0.7.4`, make sure you have at least 0.7.1). ```{eval-rst} -.. dropdown:: fides - - .. code-block:: - - "fides" - - `Fides `_ implements an Interior - Trust Region Reflective for boundary costrained optimization problems based on the - papers :cite:`Coleman1994` and :cite:`Coleman1996`. Accordingly, Fides is named after - the Roman goddess of trust and reliability. In contrast to other optimizers, Fides - solves the full trust-region subproblem exactly, which can yields higher quality - proposal steps, but is computationally more expensive. This makes Fides particularly - attractive for optimization problems with objective functions that are computationally - expensive to evaluate and the computational cost of solving the trust-region - subproblem is negligible. - - - **hessian_update_strategy** (str): Hessian Update Strategy to employ. You can provide - a lowercase or uppercase string or a - fides.hession_approximation.HessianApproximation class instance. FX, SSM, TSSM and - GNSBFGS are not supported by optimagic. The available update strategies are: - - - **bb**: Broydens "bad" method as introduced :cite:`Broyden1965`. - - **bfgs**: Broyden-Fletcher-Goldfarb-Shanno update strategy. - - **bg**: Broydens "good" method as introduced in :cite:`Broyden1965`. - - You can use a general BroydenClass Update scheme using the Broyden class from - `fides.hessian_approximation`. This is a generalization of BFGS/DFP methods - where the parameter :math:`phi` controls the convex combination between the - two. This is a rank 2 update strategy that preserves positive-semidefiniteness - and symmetry (if :math:`\phi \in [0,1]`). It is described in - :cite:`Nocedal1999`, Chapter 6.3. - - **dfp**: Davidon-Fletcher-Powell update strategy. - - **sr1**: Symmetric Rank 1 update strategy as described in :cite:`Nocedal1999`, - Chapter 6.2. - - - **convergence.ftol_abs** (float): absolute convergence criterion - tolerance. This is only the interpretation of this parameter if the relative - criterion tolerance is set to 0. Denoting the absolute criterion tolerance by - :math:`\alpha` and the relative criterion tolerance by :math:`\beta`, the - convergence condition on the criterion improvement is - :math:`|f(x_k) - f(x_{k-1})| < \alpha + \beta \cdot |f(x_{k-1})|` - - **convergence.ftol_rel** (float): relative convergence criterion - tolerance. This is only the interpretation of this parameter if the absolute - criterion tolerance is set to 0 (as is the default). Denoting the absolute - criterion tolerance by :math:`\alpha` and the relative criterion tolerance by - :math:`\beta`, the convergence condition on the criterion improvement is - :math:`|f(x_k) - f(x_{k-1})| < \alpha + \beta \cdot |f(x_{k-1})|` - - **convergence.xtol_abs** (float): The optimization terminates - successfully when the step size falls below this number, i.e. when - :math:`||x_{k+1} - x_k||` is smaller than this tolerance. - - **convergence.gtol_abs** (float): The optimization terminates - successfully when the gradient norm is less or equal than this tolerance. - - **convergence.gtol_rel** (float): The optimization terminates - successfully when the norm of the gradient divided by the absolute function value - is less or equal to this tolerance. - - - **stopping.maxiter** (int): maximum number of allowed iterations. - - **stopping.max_seconds** (int): maximum number of walltime seconds, deactivated by - default. - - - **trustregion.initial_radius** (float): Initial trust region radius. Default is 1. - - **trustregion.stepback_strategy** (str): search refinement strategy if proposed step - reaches a parameter bound. The default is "truncate". The available options are: - - - "reflect": recursive reflections at boundary. - - "reflect_single": single reflection at boundary. - - "truncate": truncate step at boundary and re-solve the restricted subproblem - - "mixed": mix reflections and truncations - - - **trustregion.subspace_dimension** (str): Subspace dimension in which the subproblem - will be solved. The default is "2D". The following values are available: - - - "2D": Two dimensional Newton/Gradient subspace - - "full": full dimensionality - - "scg": Conjugated Gradient subspace via Steihaug's method - - - **trustregion.max_stepback_fraction** (float): Stepback parameter that controls how - close steps are allowed to get to the boundary. It is the maximal fraction of a - step to take if full step would reach breakpoint. - - - **trustregion.decrease_threshold** (float): Acceptance threshold for trust region - ratio. The default is 0.25 (:cite:`Nocedal2006`). The radius is decreased if the - trust region ratio is below this value. This is denoted by :math:`\\mu` in - algorithm 4.1 in :cite:`Nocedal2006`. - - **trustregion.increase_threshold** (float): Threshold for the trust region radius - ratio above which the trust region radius can be increased. This is denoted by - :math:`\eta` in algorithm 4.1 in :cite:`Nocedal2006`. The default is 0.75 - (:cite:`Nocedal2006`). - - **trustregion.decrease_factor** (float): factor by which trust region radius will be - decreased in case it is decreased. This is denoted by :math:`\gamma_1` in - algorithm 4.1 in :cite:`Nocedal2006` and its default is 0.25. - - **trustregion.increase_factor** (float): factor by which trust region radius will be - increase in case it is increase. This is denoted by :math:`\gamma_2` in algorithm - 4.1 in :cite:`Nocedal2006` and its default is 2.0. - - - **trustregion.refine_stepback** (bool): whether to refine stepbacks via optimization. - Default is False. - - **trustregion.scaled_gradient_as_possible_stepback** (bool): whether the scaled - gradient should be added to the set of possible stepback proposals. Default is - False. +.. dropdown:: fides + + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.fides(hessian_update_strategy="bfgs"), + ) + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="fides", + algo_options={"hessian_update_strategy": "bfgs"}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.fides.Fides ``` ## The NLOPT Optimizers (nlopt) @@ -3376,544 +1476,495 @@ addition to optimagic when using an NLOPT algorithm. To install nlopt run `conda install nlopt`. ```{eval-rst} -.. dropdown:: nlopt_bobyqa +.. dropdown:: nlopt_bobyqa - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "nlopt_bobyqa" + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_bobyqa(stopping_maxfun=10_000, ...), + ) - Minimize a scalar function using the BOBYQA algorithm. + or using the string interface: - The implementation is derived from the BOBYQA subroutine of M. J. D. Powell. + .. code-block:: python - The algorithm performs derivative free bound-constrained optimization using - an iteratively constructed quadratic approximation for the objective function. - Due to its use of quadratic appoximation, the algorithm may perform poorly - for objective functions that are not twice-differentiable. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_bobyqa", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - For details see :cite:`Powell2009`. + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptBOBYQA ``` ```{eval-rst} -.. dropdown:: nlopt_neldermead +.. dropdown:: nlopt_neldermead - .. code-block:: + **How to use this algorithm:** - "nlopt_neldermead" + .. code-block:: python - Minimize a scalar function using the Nelder-Mead simplex algorithm. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_neldermead(stopping_maxfun=10_000, ...), + ) - The basic algorithm is described in :cite:`Nelder1965`. + or using the string interface: - The difference between the nlopt implementation an the original implementation is - that the nlopt version supports bounds. This is done by moving all new points that - would lie outside the bounds exactly on the bounds. + .. code-block:: python - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. -``` + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_neldermead", + algo_options={"stopping_maxfun": 10_000, ...}, + ) -```{eval-rst} -.. dropdown:: nlopt_praxis + **Description and available options:** - .. code-block:: + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptNelderMead +``` - "nlopt_praxis" +```{eval-rst} +.. dropdown:: nlopt_praxis - Minimize a scalar function using principal-axis method. + **How to use this algorithm:** - This is a gradient-free local optimizer originally described in :cite:`Brent1972`. - It assumes quadratic form of the optimized function and repeatedly updates a set of conjugate - search directions. + .. code-block:: python - The algorithm is not invariant to scaling of the objective function and may - fail under its certain rank-preserving transformations (e.g., will lead to - a non-quadratic shape of the objective function). + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_praxis(stopping_maxfun=10_000, ...), + ) - The algorithm is not determenistic and it is not possible to achieve - detereminancy via seed setting. + or using the string interface: - The algorithm failed on a simple benchmark function with finite parameter bounds. - Passing arguments `lower_bounds` and `upper_bounds` has been disabled for this - algorithm. + .. code-block:: python - The difference between the nlopt implementation an the original implementation is - that the nlopt version supports bounds. This is done by returning infinity (Inf) - when the constraints are violated. The implementation of bound constraints - is achieved at the const of significantly reduced speed of convergence. - In case of bounded constraints, this method is dominated by `nlopt_bobyqa` - and `nlopt_cobyla`. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_praxis", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + **Description and available options:** + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptPRAXIS ``` ```{eval-rst} -.. dropdown:: nlopt_cobyla +.. dropdown:: nlopt_cobyla - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_cobyla(stopping_maxfun=10_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_cobyla", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - "nlopt_cobyla" - - Minimize a scalar function using the cobyla method. - - The alggorithm is derived from Powell's Constrained Optimization BY Linear - Approximations (COBYLA) algorithm. It is a derivative-free optimizer with - nonlinear inequality and equality constrains, described in :cite`Powell1994`. - - It constructs successive linear approximations of the objective function and - constraints via a simplex of n+1 points (in n dimensions), and optimizes these - approximations in a trust region at each step. - - The the nlopt implementation differs from the original implementation in a - a few ways: - - Incorporates all of the NLopt termination criteria. - - Adds explicit support for bound constraints. - - Allows the algorithm to increase the trust-reion radius if the predicted - imptoovement was approximately right and the simplex is satisfactory. - - Pseudo-randomizes simplex steps in the algorithm, aimproving robustness by - avoiding accidentally taking steps that don't improve conditioning, preserving - the deterministic nature of the algorithm. - - Supports unequal initial-step sizes in the different parameters. - - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptCOBYLA ``` ```{eval-rst} -.. dropdown:: nlopt_sbplx +.. dropdown:: nlopt_sbplx - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_sbplx(stopping_maxfun=10_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_sbplx", + algo_options={"stopping_maxfun": 10_000, ...}, + ) + + **Description and available options:** - "nlopt_sbplx" - - Minimize a scalar function using the "Subplex" algorithm. - - The alggorithm is a reimplementation of Tom Rowan's "Subplex" algorithm. - See :cite:`Rowan1990`. - Subplex is a variant of Nedler-Mead that uses Nedler-Mead on a sequence of - subspaces. It is climed to be more efficient and robust than the original - Nedler-Mead algorithm. - - The difference between this re-implementation and the original algorithm - of Rowan, is that it explicitly supports bound constraints providing big - improvement in the case where the optimum lies against one of the constraints. - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptSbplx ``` ```{eval-rst} -.. dropdown:: nlopt_newuoa +.. dropdown:: nlopt_newuoa - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_newuoa(stopping_maxfun=10_000, ...), + ) + + or using the string interface: - "nlopt_newuoa" - - Minimize a scalar function using the NEWUOA algorithm. - - The algorithm is derived from the NEWUOA subroutine of M.J.D Powell which - uses iteratively constructed quadratic approximation of the objctive fucntion - to perform derivative-free unconstrained optimization. Fore more details see: - :cite:`Powell2004`. - - The algorithm in `nlopt` has been modified to support bound constraints. If all - of the bound constraints are infinite, this function calls the `nlopt.LN_NEWUOA` - optimizers for uncsonstrained optimization. Otherwise, the `nlopt.LN_NEWUOA_BOUND` - optimizer for constrained problems. - - `NEWUOA` requires the dimension n of the parameter space to be `≥ 2`, i.e. the - implementation does not handle one-dimensional optimization problems. - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_newuoa", + algo_options={"stopping_maxfun": 10_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptNEWUOA ``` ```{eval-rst} -.. dropdown:: nlopt_tnewton +.. dropdown:: nlopt_tnewton - .. code-block:: + **How to use this algorithm:** - "nlopt_tnewton" + .. code-block:: python - Minimize a scalar function using the "TNEWTON" algorithm. + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_tnewton(stopping_maxfun=10_000, ...), + ) + + or using the string interface: - The alggorithm is based on a Fortran implementation of a preconditioned - inexact truncated Newton algorithm written by Prof. Ladislav Luksan. + .. code-block:: python - Truncated Newton methods are a set of algorithms designed to solve large scale - optimization problems. The algorithms use (inaccurate) approximations of the - solutions to Newton equations, using conjugate gradient methodds, to handle the - expensive calculations of derivatives during each iteration. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_tnewton", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - Detailed description of algorithms is given in :cite:`Dembo1983`. + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptTNewton ``` ```{eval-rst} -.. dropdown:: nlopt_lbfgs +.. dropdown:: nlopt_lbfgsb - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "nlopt_lbfgs" + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_lbfgsb(stopping_maxfun=10_000, ...), + ) - Minimize a scalar function using the "LBFGS" algorithm. + or using the string interface: - The alggorithm is based on a Fortran implementation of low storage BFGS algorithm - written by Prof. Ladislav Luksan. + .. code-block:: python - LFBGS is an approximation of the original Broyden–Fletcher–Goldfarb–Shanno algorithm - based on limited use of memory. Memory efficiency is obtained by preserving a limi- - ted number (<10) of past updates of candidate points and gradient values and using - them to approximate the hessian matrix. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_lbfgsb", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - Detailed description of algorithms is given in :cite:`Nocedal1989`, :cite:`Nocedal1980`. + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptLBFGSB ``` ```{eval-rst} -.. dropdown:: nlopt_ccsaq +.. dropdown:: nlopt_ccsaq - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_ccsaq(stopping_maxfun=10_000, ...), + ) + + or using the string interface: - "nlopt_ccsaq" - - Minimize a scalar function using CCSAQ algorithm. - - CCSAQ uses the quadratic variant of the conservative convex separable approximation. - The algorithm performs gradient based local optimization with equality (but not - inequality) constraints. At each candidate point x, a quadratic approximation - to the criterion faunction is computed using the value of gradient at point x. A - penalty term is incorporated to render optimizaion convex and conservative. The - algorithm is "globally convergent" in the sense that it is guaranteed to con- - verge to a local optimum from any feasible starting point. - - The implementation is based on CCSA algorithm described in :cite:`Svanberg2002`. - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_ccsaq", + algo_options={"stopping_maxfun": 10_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptCCSAQ ``` ```{eval-rst} -.. dropdown:: nlopt_mma +.. dropdown:: nlopt_mma - .. code-block:: + **How to use this algorithm:** - "nlopt_mma" + .. code-block:: python - Minimize a scalar function using the method of moving asymptotes (MMA). + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_mma(stopping_maxfun=10_000, ...), + ) + + or using the string interface: - The implementation is based on an algorithm described in :cite:`Svanberg2002`. + .. code-block:: python - The algorithm performs gradient based local optimization with equality (but - not inequality) constraints. At each candidate point x, an approximation to the - criterion faunction is computed using the value of gradient at point x. A quadratic - penalty term is incorporated to render optimizaion convex and conservative. The - algorithm is "globally convergent" in the sense that it is guaranteed to con- - verge to a local optimum from any feasible starting point. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_mma", + algo_options={"stopping_maxfun": 10_000, ...}, + ) + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptMMA ``` ```{eval-rst} -.. dropdown:: nlopt_var +.. dropdown:: nlopt_var - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_var(rank_1_update=False, ...), + ) + + or using the string interface: + + .. code-block:: python - "nlopt_var" - - Minimize a scalar function limited memory switching variable-metric method. - - The algorithm relies on saving only limited number M of past updates of the - gradient to approximate the inverse hessian. The large is M, the more memory is - consumed - - Detailed explanation of the algorithm, including its two variations of rank-2 and - rank-1 methods can be found in the following paper :cite:`Vlcek2006` . - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. - - **rank_1_update** (bool): Whether I rank-1 or rank-2 update is used. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_var", + algo_options={"rank_1_update": False, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptVAR ``` ```{eval-rst} -.. dropdown:: nlopt_slsqp +.. dropdown:: nlopt_slsqp - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "nlopt_slsqp" + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_slsqp(stopping_maxfun=10_000, ...), + ) + + or using the string interface: - Optimize a scalar function based on SLSQP method. + .. code-block:: python - SLSQP solves gradient based nonlinearly constrained optimization problems. - The algorithm treats the optimization problem as a sequence of constrained - least-squares problems. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_slsqp", + algo_options={"stopping_maxfun": 10_000, ...}, + ) - The implementation is based on the procedure described in :cite:`Kraft1988` - and :cite:`Kraft1994` . + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptSLSQP ``` ```{eval-rst} -.. dropdown:: nlopt_direct +.. dropdown:: nlopt_direct - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_direct(locally_biased=True, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: + + .. code-block:: python - "nlopt_direct" - - Optimize a scalar function based on DIRECT method. - - DIRECT is the DIviding RECTangles algorithm for global optimization, described - in :cite:`Jones1993` . - - Variations of the algorithm include locally biased routines (distinguished by _L - suffix) that prove to be more efficients for functions that have few local minima. - See the following for the DIRECT_L variant :cite:`Gablonsky2001` . - - Locally biased algorithms can be implmented both with deterministic and random - (distinguished by _RAND suffix) search algorithm. - - Finally, both original and locally biased variants can be implemented with and - without the rescaling of the bound constraints. - - Boolean arguments `locally_biased`, 'random_search', and 'unscaled_bouds' can be - set to `True` or `False` to determine which method is run. The comprehensive list - of available methods are: - - "DIRECT" - - "DIRECT_L" - - "DIRECT_L_NOSCAL" - - "DIRECT_L_RAND" - - "DIRECT_L_RAND_NOSCAL" - - "DIRECT_RAND" - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. - - **locally_biased** (bool): Whether the "L" version of the algorithm is selected. - - **random_search** (bool): Whether the randomized version of the algorithm is selected. - - **unscaled_bounds** (bool): Whether the "NOSCAL" version of the algorithm is selected. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_direct", + algo_options={"locally_biased": True, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptDirect ``` ```{eval-rst} -.. dropdown:: nlopt_esch +.. dropdown:: nlopt_esch - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "nlopt_esch" + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_esch(stopping_maxfun=10_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: - Optimize a scalar function using the ESCH algorithm. + .. code-block:: python - ESCH is an evolutionary algorithm that supports bound constraints only. Specifi - cally, it does not support nonlinear constraints. + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_esch", + algo_options={"stopping_maxfun": 10_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - More information on this method can be found in - :cite:`DaSilva2010` , :cite:`DaSilva2010a` , :cite:`Beyer2002` and :cite:`Vent1975` . + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this - as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptESCH ``` ```{eval-rst} -.. dropdown:: nlopt_isres +.. dropdown:: nlopt_isres - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python - "nlopt_isres" + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_isres(stopping_maxfun=10_000, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) - Optimize a scalar function using the ISRES algorithm. + or using the string interface: - ISRES is an implementation of "Improved Stochastic Evolution Strategy" - written for solving optimization problems with non-linear constraints. The - algorithm is supposed to be a global method, in that it has heuristics to - avoid local minima. However, no convergence proof is available. + .. code-block:: python - The original method and a refined version can be found, respecively, in - :cite:`PhilipRunarsson2005` and :cite:`Thomas2000` . + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_isres", + algo_options={"stopping_maxfun": 10_000, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + **Description and available options:** - - **convergence.xtol_rel** (float): Stop when the relative - movement between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute - movement between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of - the criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of - function evaluation is reached, the optimization stops but we do not count - this as convergence. + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptISRES ``` ```{eval-rst} -.. dropdown:: nlopt_crs2_lm +.. dropdown:: nlopt_crs2_lm - .. code-block:: + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + import numpy as np + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.nlopt_crs2_lm(population_size=100, ...), + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + or using the string interface: - "nlopt_crs2_lm" - - Optimize a scalar function using the CRS2_LM algorithm. - - This implementation of controlled random search method with local mutation is based - on :cite:`Kaelo2006` . - - The original CRS method is described in :cite:`Price1978` and :cite:`Price1983` . - - CRS class of algorithms starts with random population of points and evolves the - points "randomly". The size of the initial population can be set via the param- - meter population_size. If the user doesn't specify a value, it is set to the nlopt - default of 10*(n+1). - - - **convergence.xtol_rel** (float): Stop when the relative movement - between parameter vectors is smaller than this. - - **convergence.xtol_abs** (float): Stop when the absolute movement - between parameter vectors is smaller than this. - - **convergence.ftol_rel** (float): Stop when the relative - improvement between two iterations is smaller than this. - - **convergence.ftol_abs** (float): Stop when the change of the - criterion function between two iterations is smaller than this. - - **stopping.maxfun** (int): If the maximum number of function - evaluation is reached, the optimization stops but we do not count this as - convergence. - - **population_size** (int): Size of the population. If None, it's set to be - 10 * (number of parameters + 1). + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="nlopt_crs2_lm", + algo_options={"population_size": 100, ...}, + bounds=om.Bounds(lower=np.array([-5, -5, -5]), upper=np.array([5, 5, 5])), + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.nlopt_optimizers.NloptCRS2LM ``` ## Optimizers from iminuit @@ -4280,17 +2331,16 @@ and hence imprecise.\ ``` ```{eval-rst} -.. dropdown:: nevergrad_wizard +.. dropdown:: nevergrad_ngopt **How to use this algorithm:** .. code-block:: import optimagic as om - from optimagic.optimizers.nevergrad_optimizers import Wizard om.minimize( ..., - algorithm=om.algos.nevergrad_wizard(optimizer= Wizard.NGOptRW, ...) + algorithm=om.algos.nevergrad_ngopt(optimizer="NGOptRW", ...) ) or @@ -4299,28 +2349,26 @@ and hence imprecise.\ om.minimize( ..., - algorithm="nevergrad_wizard", + algorithm="nevergrad_ngopt", algo_options={"optimizer": "NGOptRW", ...} ) **Description and available options:** - .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradWizard - .. autoclass:: optimagic.optimizers.nevergrad_optimizers.Wizard + .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradNGOpt ``` ```{eval-rst} -.. dropdown:: nevergrad_portfolio +.. dropdown:: nevergrad_meta **How to use this algorithm:** .. code-block:: import optimagic as om - from optimagic.optimizers.nevergrad_optimizers import Portfolio om.minimize( ..., - algorithm=om.algos.nevergrad_portfolio(optimizer= Portfolio.BFGSCMAPlus, ...) + algorithm=om.algos.nevergrad_meta(optimizer="BFGSCMAPlus", ...) ) or @@ -4329,14 +2377,13 @@ and hence imprecise.\ om.minimize( ..., - algorithm="nevergrad_portfolio", + algorithm="nevergrad_meta", algo_options={"optimizer": "BFGSCMAPlus", ...} ) **Description and available options:** - .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradPortfolio - .. autoclass:: optimagic.optimizers.nevergrad_optimizers.Portfolio + .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradMeta ``` ## Bayesian Optimization @@ -4958,6 +3005,84 @@ LocalBestPSO, GeneralOptimizerPSO). To use these optimizers, you need to have ``` +## The Tranquilo Optimizer + +optimagic supports [tranquilo](https://github.com/OpenSourceEconomics/tranquilo), a +trust-region optimizer for noisy and/or computationally expensive black-box problems +that was developed by the optimagic developers. To use it, you need to have the +tranquilo package (version 0.1.0 or newer) installed, e.g. via `pip install tranquilo` +or `conda install -c conda-forge tranquilo`. `tranquilo` is the scalar version of the +algorithm; `tranquilo_ls` exploits the least-squares structure of the objective function +and should be preferred whenever your objective function can be expressed as a sum of +squared residuals. + +```{eval-rst} +.. dropdown:: tranquilo + + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.tranquilo(stopping_maxfun=5_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=lambda x: x @ x, + params=[1.0, 2.0, 3.0], + algorithm="tranquilo", + algo_options={"stopping_maxfun": 5_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.tranquilo.Tranquilo +``` + +```{eval-rst} +.. dropdown:: tranquilo_ls + + **How to use this algorithm:** + + .. code-block:: python + + import optimagic as om + + + @om.mark.least_squares + def fun(x): + return x + + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm=om.algos.tranquilo_ls(stopping_maxfun=5_000, ...), + ) + + or using the string interface: + + .. code-block:: python + + om.minimize( + fun=fun, + params=[1.0, 2.0, 3.0], + algorithm="tranquilo_ls", + algo_options={"stopping_maxfun": 5_000, ...}, + ) + + **Description and available options:** + + .. autoclass:: optimagic.optimizers.tranquilo.TranquiloLS +``` + ## References ```{eval-rst} diff --git a/docs/source/how_to/how_to_add_optimizers.ipynb b/docs/source/how_to/how_to_add_optimizers.ipynb index 86f926743..766a5fb49 100644 --- a/docs/source/how_to/how_to_add_optimizers.ipynb +++ b/docs/source/how_to/how_to_add_optimizers.ipynb @@ -421,10 +421,14 @@ " needs_jac=False,\n", " # does the optimizer need the hessian? -> no, gaco is derivative free\n", " needs_hess=False,\n", + " # does the optimizer need bounds? -> yes, gaco needs finite bounds\n", + " needs_bounds=True,\n", " # does the optimizer support parallelism? -> yes\n", " supports_parallelism=True,\n", " # does the optimizer support bounds? -> yes\n", " supports_bounds=True,\n", + " # does the optimizer support infinite bounds? -> no, bounds must be finite\n", + " supports_infinite_bounds=False,\n", " # does the optimizer support linear constraints? -> no\n", " supports_linear_constraints=False,\n", " # does the optimizer support nonlinear constraints? -> no\n", @@ -603,7 +607,7 @@ "of commonly used convergence and stopping criteria. We also align the default values for \n", "stopping and convergence criteria as much as possible. \n", "\n", - "You can find the harmonized names and value [here](algo_options_docs). \n", + "You can find the harmonized names and value {ref}`here `. \n", "\n", "To align the names of other tuning parameters as much as possible with what is already \n", "there, simple have a look at the optimizers we already wrapped. For example, if you are \n", diff --git a/docs/source/refs.bib b/docs/source/refs.bib index 215f3471e..d02e32488 100644 --- a/docs/source/refs.bib +++ b/docs/source/refs.bib @@ -1114,4 +1114,182 @@ @article{Ni2013 year = {2013} } +@Article{Froehlich2022, + author = {Fabian Fr{\"o}hlich and Peter K. Sorger}, + journal = {PLOS Computational Biology}, + title = {Fides: Reliable trust-region optimization for parameter estimation of ordinary differential equation models}, + year = {2022}, + month = {jul}, + number = {7}, + pages = {e1010322}, + volume = {18}, + doi = {10.1371/journal.pcbi.1010322}, + publisher = {Public Library of Science ({PLoS})}, +} + +@Misc{Johnson2007, + author = {Johnson, Steven G.}, + title = {The NLopt nonlinear-optimization package}, + year = {2007}, + url = {https://github.com/stevengj/nlopt}, +} + +@Article{Svanberg1987, + author = {Svanberg, Krister}, + journal = {International Journal for Numerical Methods in Engineering}, + title = {The method of moving asymptotes---a new method for structural optimization}, + year = {1987}, + number = {2}, + pages = {359--373}, + volume = {24}, + doi = {10.1002/nme.1620240207}, +} + +@Article{LeeWiswall2007, + author = {Lee, Donghoon and Wiswall, Matthew}, + journal = {Computational Economics}, + title = {A Parallel Implementation of the Simplex Function Minimization Routine}, + year = {2007}, + number = {2}, + pages = {171--187}, + volume = {30}, + doi = {10.1007/s10614-007-9094-2}, +} + +@Article{Wessing2019, + author = {Wessing, Simon}, + journal = {Optimization Letters}, + title = {Proper initialization is crucial for the Nelder--Mead simplex search}, + year = {2019}, + pages = {847--856}, + volume = {13}, + doi = {10.1007/s11590-018-1284-4}, +} + +@Book{Nash1990, + author = {Nash, John C.}, + publisher = {Adam Hilger}, + title = {Compact Numerical Methods for Computers: Linear Algebra and Function Minimisation}, + year = {1990}, + address = {Bristol}, + edition = {2nd}, +} + +@Misc{Varadhan2016, + author = {Varadhan, Ravi and Borchers, Hans W.}, + title = {dfoptim: Derivative-Free Optimization}, + year = {2016}, + note = {R package version 2016.7-1}, + url = {https://CRAN.R-project.org/package=dfoptim}, +} + +@Article{Powell1964, + author = {Powell, M. J. D.}, + journal = {The Computer Journal}, + title = {An efficient method for finding the minimum of a function of several variables without calculating derivatives}, + year = {1964}, + number = {2}, + pages = {155--162}, + volume = {7}, + doi = {10.1093/comjnl/7.2.155}, +} + +@Article{Branch1999, + author = {Branch, Mary Ann and Coleman, Thomas F. and Li, Yuying}, + journal = {SIAM Journal on Scientific Computing}, + title = {A Subspace, Interior, and Conjugate Gradient Method for Large-Scale Bound-Constrained Minimization Problems}, + year = {1999}, + number = {1}, + pages = {1--23}, + volume = {21}, + doi = {10.1137/S1064827595289108}, +} + +@InProceedings{Voglis2004, + author = {Voglis, C. and Lagaris, I. E.}, + booktitle = {WSEAS International Conference on Applied Mathematics}, + title = {A Rectangular Trust Region Dogleg Approach for Unconstrained and Bound Constrained Nonlinear Optimization}, + year = {2004}, + address = {Corfu, Greece}, + url = {https://www.cs.uoi.gr/~lagaris/papers/PREPRINTS/dogbox.pdf}, +} + +@InCollection{More1978, + author = {Mor{\'e}, Jorge J.}, + booktitle = {Numerical Analysis}, + editor = {Watson, G. A.}, + pages = {105--116}, + publisher = {Springer}, + series = {Lecture Notes in Mathematics}, + title = {The Levenberg-Marquardt algorithm: Implementation and theory}, + volume = {630}, + year = {1978}, + doi = {10.1007/BFb0067700}, +} + +@Article{Nash1984, + author = {Nash, Stephen G.}, + journal = {SIAM Journal on Numerical Analysis}, + title = {Newton-Type Minimization via the Lanczos Method}, + year = {1984}, + number = {4}, + pages = {770--788}, + volume = {21}, + doi = {10.1137/0721052}, +} + +@Article{Endres2018, + author = {Endres, Stefan C. and Sandrock, Carl and Focke, Walter W.}, + journal = {Journal of Global Optimization}, + title = {A simplicial homology algorithm for {Lipschitz} optimisation}, + year = {2018}, + number = {2}, + pages = {181--217}, + volume = {72}, + doi = {10.1007/s10898-018-0645-y}, +} + +@Article{Tsallis1988, + author = {Tsallis, Constantino}, + journal = {Journal of Statistical Physics}, + title = {Possible generalization of {Boltzmann-Gibbs} statistics}, + year = {1988}, + number = {1-2}, + pages = {479--487}, + volume = {52}, + doi = {10.1007/BF01016429}, +} + +@Article{TsallisStariolo1996, + author = {Tsallis, Constantino and Stariolo, Daniel A.}, + journal = {Physica A: Statistical Mechanics and its Applications}, + title = {Generalized simulated annealing}, + year = {1996}, + number = {1-2}, + pages = {395--406}, + volume = {233}, + doi = {10.1016/S0378-4371(96)00271-3}, +} + +@Article{Xiang1997, + author = {Xiang, Y. and Sun, D. Y. and Fan, W. and Gong, X. G.}, + journal = {Physics Letters A}, + title = {Generalized simulated annealing algorithm and its application to the {Thomson} model}, + year = {1997}, + number = {3}, + pages = {216--220}, + volume = {233}, + doi = {10.1016/S0375-9601(97)00474-X}, +} + +@techreport{Gabler2024, + author = {Gabler, Jano{\'s} and Gsell, Sebastian and Mensinger, Tim and Petrosyan, Mariam}, + title = {Tranquilo: An Optimizer for the Method of Simulated Moments}, + institution = {CRC TR 224, University of Bonn and University of Mannheim}, + type = {Discussion Paper}, + number = {522}, + year = {2024}, + url = {https://www.crctr224.de/research/discussion-papers/archive/dp522} +} + @Comment{jabref-meta: databaseType:bibtex;} diff --git a/src/optimagic/optimizers/bhhh.py b/src/optimagic/optimizers/bhhh.py index c7e3a1539..aaa45bc6a 100644 --- a/src/optimagic/optimizers/bhhh.py +++ b/src/optimagic/optimizers/bhhh.py @@ -1,5 +1,7 @@ """Implement Berndt-Hall-Hall-Hausman (BHHH) algorithm.""" +from __future__ import annotations + from dataclasses import dataclass from typing import Callable, cast @@ -31,9 +33,44 @@ ) @dataclass(frozen=True) class BHHH(Algorithm): + """Minimize a likelihood function using the BHHH algorithm. + + BHHH (:cite:`Berndt1974`) can - and should ONLY - be used for minimizing (or + maximizing) a likelihood function. It is similar to the Newton-Raphson algorithm, + but replaces the Hessian matrix with the outer product of the gradients of the + likelihood contributions. This approximation is based on the information matrix + equality (:cite:`Halbert1982`) and is thus only valid when minimizing (or + maximizing) a likelihood function. In exchange, the approximated Hessian is + always positive semidefinite and no second derivatives are needed. + + To use bhhh, the objective function must be a likelihood function, i.e. a + function that is decorated with ``om.mark.likelihood`` and returns the + likelihood contributions of each observation rather than their sum. + + The algorithm is a local optimizer that uses first derivatives of the + likelihood contributions. It does not support bounds or constraints. + + .. note:: + This is a pure-Python implementation within optimagic. It is currently + considered experimental. + + """ + converence_gtol_abs: NonNegativeFloat = 1e-8 + """Stopping criterion for the gradient tolerance. + + The algorithm converges when the inner product of the aggregated gradient and + the candidate search direction falls below this value. + + """ + # TODO: Why is this 200? stopping_maxiter: PositiveInt = 200 + """Maximum number of iterations. + + If reached, terminate. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/fides.py b/src/optimagic/optimizers/fides.py index 4d90814c0..761eef91f 100644 --- a/src/optimagic/optimizers/fides.py +++ b/src/optimagic/optimizers/fides.py @@ -1,5 +1,7 @@ """Implement the fides optimizer.""" +from __future__ import annotations + import logging from dataclasses import dataclass from typing import Callable, Literal, cast @@ -47,6 +49,33 @@ ) @dataclass(frozen=True) class Fides(Algorithm): + """Minimize a scalar function using the Fides trust-region optimizer. + + Fides implements an Interior Trust-Region Reflective method for bound-constrained + optimization, following the approach of :cite:`Coleman1994` and + :cite:`Coleman1996`. Accordingly, Fides is named after the Roman goddess of trust + and reliability. The optimizer is taken from the ``fides`` package + (:cite:`Froehlich2022`), which must be installed separately + (``pip install fides>=0.7.4``). + + Fides is a gradient-based local optimizer for smooth, differentiable scalar + objective functions. It supports lower and upper bounds (including infinite + bounds) but no linear or nonlinear constraints. It requires first derivatives; the + Hessian is approximated internally using one of several quasi-Newton update + strategies, so no user-provided Hessian is needed. It is well suited for problems + with up to several hundred parameters. + + In contrast to many other trust-region methods, Fides solves the full trust-region + subproblem exactly rather than approximately. This can yield higher-quality + proposal steps at a higher per-iteration cost, which makes Fides especially + attractive when the objective function is expensive to evaluate and the cost of + solving the trust-region subproblem is negligible in comparison. + + .. note:: + General linear and nonlinear constraints are not supported by optimagic. + + """ + hessian_update_strategy: Literal[ "bfgs", "bb", @@ -54,30 +83,151 @@ class Fides(Algorithm): "dfp", "sr1", ] = "bfgs" + """Quasi-Newton strategy used to approximate the Hessian of the objective. + + The available strategies are: + + - ``"bfgs"``: the Broyden-Fletcher-Goldfarb-Shanno update, a rank-2 update that + preserves symmetry and positive definiteness. This is the default. + - ``"dfp"``: the Davidon-Fletcher-Powell update. + - ``"sr1"``: the Symmetric Rank 1 update, described in :cite:`Nocedal1999`, + Chapter 6.2. + - ``"bb"``: Broyden's "bad" method, introduced in :cite:`Broyden1965`. + - ``"bg"``: Broyden's "good" method, introduced in :cite:`Broyden1965`. + + The general Broyden class update, a convex combination of the BFGS and DFP updates + controlled by a parameter :math:`\\phi` (:cite:`Nocedal1999`, Chapter 6.3), and the + residual-based approximations ``FX``, ``SSM``, ``TSSM`` and ``GNSBFGS`` provided by + the ``fides`` package are not available through this option, because they require + access to least-squares residuals or an interpolation parameter that optimagic does + not pass through. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + r"""Stop when the absolute change in the objective becomes small. + + Denoting the absolute criterion tolerance by :math:`\alpha` (this parameter) and + the relative criterion tolerance by :math:`\beta` + (``convergence_ftol_rel``), Fides stops successfully when + + .. math:: + + |f_k - f_{k-1}| < \alpha + \beta \, |f_{k-1}|. + + This parameter therefore governs convergence on its own only when + ``convergence_ftol_rel`` is set to 0. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + r"""Stop when the relative change in the objective becomes small. + + This is the relative criterion tolerance :math:`\beta` in the joint convergence + condition documented under ``convergence_ftol_abs``. It governs convergence on its + own only when ``convergence_ftol_abs`` is set to 0 (as is the default). + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + r"""Stop when the step size becomes small. + + The optimization terminates successfully when the norm of the step + :math:`\|x_{k} - x_{k-1}\|` falls below this tolerance. + + """ + convergence_gtol_abs: NonNegativeFloat = CONVERGENCE_GTOL_ABS + """Stop when the gradient norm is less than or equal to this tolerance.""" + convergence_gtol_rel: NonNegativeFloat = CONVERGENCE_GTOL_REL + """Stop when the gradient norm divided by the absolute function value is less than + or equal to this tolerance.""" + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """Maximum number of iterations.""" + stopping_max_seconds: float = np.inf + """Maximum number of wall-clock seconds. Deactivated (infinite) by default.""" + trustregion_initial_radius: PositiveFloat = 1.0 + """Initial trust-region radius.""" + trustregion_stepback_strategy: Literal[ "truncate", "reflect", "reflect_single", "mixed", ] = "truncate" + """Refinement strategy applied when a proposed step reaches a parameter bound. + + The available options are: + + - ``"truncate"``: truncate the step at the boundary and re-solve the restricted + subproblem. This is optimagic's default. + - ``"reflect"``: recursive reflections at the boundary. + - ``"reflect_single"``: a single reflection at the boundary. + - ``"mixed"``: mix reflections and truncations. + + Note that optimagic defaults to ``"truncate"``, whereas the ``fides`` package + itself defaults to ``"reflect"``. + + """ + trustregion_subspace_dimension: Literal[ "full", "2D", "scg", ] = "full" + """Dimension of the subspace in which the trust-region subproblem is solved. + + The available options are: + + - ``"full"``: use the full parameter dimensionality. This is optimagic's default. + - ``"2D"``: a two-dimensional Newton/gradient subspace. + - ``"scg"``: a conjugate-gradient subspace via Steihaug's method. + + Note that optimagic defaults to ``"full"``, whereas the ``fides`` package itself + defaults to ``"2D"``. + + """ + trustregion_max_stepback_fraction: float = 0.95 + """Controls how close steps are allowed to get to the boundary. + + It is the maximal fraction of a step to take if the full step would reach a + breakpoint (the bound). + + """ + trustregion_decrease_threshold: float = 0.25 + r"""Acceptance threshold for the trust-region ratio. + + The trust-region radius is decreased when the trust-region ratio falls below this + value. It is denoted by :math:`\mu` in algorithm 4.1 of :cite:`Nocedal2006`. + + """ + trustregion_increase_threshold: float = 0.75 + r"""Threshold for the trust-region ratio above which the radius may be increased. + + It is denoted by :math:`\eta` in algorithm 4.1 of :cite:`Nocedal2006`. + + """ + trustregion_decrease_factor: float = 0.25 + r"""Factor by which the trust-region radius is multiplied when it is decreased. + + It is denoted by :math:`\gamma_1` in algorithm 4.1 of :cite:`Nocedal2006`. + + """ + trustregion_increase_factor: float = 2.0 + r"""Factor by which the trust-region radius is multiplied when it is increased. + + It is denoted by :math:`\gamma_2` in algorithm 4.1 of :cite:`Nocedal2006`. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/ipopt.py b/src/optimagic/optimizers/ipopt.py index ba23074c0..d383b0776 100644 --- a/src/optimagic/optimizers/ipopt.py +++ b/src/optimagic/optimizers/ipopt.py @@ -1,5 +1,7 @@ """Implement cyipopt's Interior Point Optimizer.""" +from __future__ import annotations + from dataclasses import dataclass from typing import Any, Literal @@ -48,45 +50,318 @@ ) @dataclass(frozen=True) class Ipopt(Algorithm): + """Minimize a scalar function using the Interior Point Optimizer (Ipopt). + + Ipopt (:cite:`Waechter2005b`) is an open-source solver for large-scale smooth + nonlinear optimization developed within the COIN-OR project. It implements a + primal-dual interior point method with a filter line search + (:cite:`Waechter2005`, :cite:`Waechter2005a`) and, optionally, an adaptive + update strategy for the barrier parameter (:cite:`Nocedal2009`). optimagic's + support for Ipopt is built on `cyipopt + `_, a Python wrapper for + the `Ipopt optimization package + `_. + + Ipopt is a local optimizer for smooth (ideally twice continuously + differentiable) scalar objective functions. It supports bounds as well as + nonlinear equality and inequality constraints and is designed to scale to very + large problems. It requires first derivatives of the objective function and + the constraints. By default, second derivatives are approximated with a + limited-memory quasi-Newton method, so no Hessian has to be provided. Ipopt is + not suitable for noisy or discontinuous objective functions. + + There are two levels of termination criteria. If the usual "desired" + tolerances (see ``convergence_ftol_rel``, ``dual_inf_tol`` etc.) are satisfied + at an iteration, the algorithm immediately terminates with a success message. + On the other hand, if the algorithm encounters ``acceptable_iter`` many + iterations in a row that are considered "acceptable", it will terminate before + the desired convergence tolerance is met. This is useful in cases where the + algorithm might not be able to achieve the "desired" level of accuracy. + + The options are analogous to the ones in the `Ipopt options reference + `_, with the exception of the + linear solver options, which are here bundled into the dictionary + ``linear_solver_options``. Any option that takes "yes" and "no" in the Ipopt + documentation can also be passed as ``True`` and ``False``, respectively, and + any option that accepts "none" in Ipopt accepts a Python ``None``. + + The following options are not supported: + + - ``num_linear_variables``: since optimagic may reparametrize your problem and + this changes the parameter problem, we do not support this option + - derivative checks + - print options. + + .. note:: + To use this algorithm you need to have `cyipopt installed + `_ + (``conda install -c conda-forge cyipopt``). + + """ + # convergence criteria convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Relative convergence tolerance of the algorithm. + + The algorithm terminates successfully if the (scaled) nonlinear programming + error becomes smaller than this value. This is passed to Ipopt as the ``tol`` + option. optimagic's default (2e-9) is stricter than Ipopt's own default of + 1e-8. + + """ + dual_inf_tol: PositiveFloat = 1.0 + """Desired threshold for the dual infeasibility. + + Successful termination requires that the max-norm of the (unscaled) dual + infeasibility is less than this threshold. + + """ + constr_viol_tol: PositiveFloat = 0.0001 + """Desired threshold for the constraint and variable bound violation. + + Successful termination requires that the max-norm of the (unscaled) constraint + violation is less than this threshold. If ``bound_relax_factor`` is not zero, + Ipopt relaxes the given variable bounds; the value of ``constr_viol_tol`` is + then used to restrict the absolute amount of this bound relaxation. + + """ + compl_inf_tol: PositiveFloat = 0.0001 + """Desired threshold for the complementarity conditions. + + Successful termination requires that the max-norm of the (unscaled) + complementarity is less than this threshold. + + """ + s_max: float = 100 + """Scaling threshold for the NLP error. + + See paragraph after Eqn. (6) in :cite:`Waechter2005b`. + + """ + mu_target: NonNegativeFloat = 0.0 + """Desired value of the complementarity. + + Usually, the barrier parameter is driven to zero and the termination test for + complementarity is measured with respect to zero complementarity. However, in + some cases it might be desirable to have Ipopt solve the barrier problem for a + strictly positive value of the barrier parameter. In this case, ``mu_target`` + specifies the final value of the barrier parameter and the termination tests + are then defined with respect to the barrier problem for this value. + + """ + # stopping criteria stopping_maxiter: PositiveInt = STOPPING_MAXITER + """Maximum number of iterations. + + If the maximum number of iterations is reached, the optimization stops, but we + do not count this as successful convergence. One iteration might need several + criterion evaluations, e.g. in a line search. optimagic's default (1,000,000) + is much larger than Ipopt's own default of 3000. + + """ + stopping_max_wall_time_seconds: PositiveFloat = 1e20 + """Maximum number of wall clock seconds Ipopt may use to solve one problem. + + If this limit is exceeded during the convergence check, Ipopt terminates with + a corresponding message. + + """ + stopping_max_cpu_time: PositiveFloat = 1e20 + """Maximum number of CPU seconds Ipopt may use to solve one problem. + + If this limit is exceeded during the convergence check, Ipopt terminates with + a corresponding message. + + """ + # acceptable criteria acceptable_iter: NonNegativeInt = 15 + """Number of successive "acceptable" iterates before termination. + + If the algorithm encounters this many successive "acceptable" iterates, it + terminates, assuming that the problem has been solved to the best possible + accuracy given round-off. If set to zero, this heuristic is disabled. + + """ + acceptable_tol: PositiveFloat = 1e-6 + """"Acceptable" convergence tolerance (relative). + + Determines which (scaled) overall optimality error is considered to be + "acceptable". Must be larger than ``convergence_ftol_rel``. + + """ + acceptable_dual_inf_tol: PositiveFloat = 1e-10 + """"Acceptance" threshold for the dual infeasibility. + + "Acceptable" termination requires that the max-norm of the (unscaled) dual + infeasibility is less than this threshold; see also ``acceptable_tol``. Note + that optimagic's default (1e-10) is much stricter than Ipopt's own default of + 1e10. + + """ + acceptable_constr_viol_tol: PositiveFloat = 0.01 + """"Acceptance" threshold for the constraint violation. + + "Acceptable" termination requires that the max-norm of the (unscaled) + constraint violation is less than this threshold; see also + ``acceptable_tol``. + + """ + acceptable_compl_inf_tol: PositiveFloat = 0.01 + """"Acceptance" threshold for the complementarity conditions. + + "Acceptable" termination requires that the max-norm of the (unscaled) + complementarity is less than this threshold; see also ``acceptable_tol``. + + """ + acceptable_obj_change_tol: PositiveFloat = 1e20 + """"Acceptance" stopping criterion based on the objective function change. + + If the relative change of the objective function (scaled by + ``max(1, |f(x)|)``) is less than this value, this part of the acceptable + tolerance termination is satisfied; see also ``acceptable_tol``. This is + useful for the quasi-Newton option, which has trouble bringing down the dual + infeasibility. + + """ + diverging_iterates_tol: PositiveFloat = 1e20 + """Threshold for the maximal value of the primal iterates. + + If any component of the primal iterates exceeds this value in absolute terms, + the optimization is aborted with the exit message that the iterates seem to be + diverging. + + """ + nlp_lower_bound_inf: float = -1e19 + """Any bound less or equal to this value will be considered minus infinity + (i.e. the variable is treated as not lower bounded).""" + nlp_upper_bound_inf: float = 1e19 + """Any bound greater or equal to this value will be considered plus infinity + (i.e. the variable is treated as not upper bounded).""" + fixed_variable_treatment: Literal[ "make_parameter", "make_parameter_nodual", "relax_bounds", "make_constraint", ] = "make_parameter" + """Determines how fixed variables are handled. + + Possible values: + + - "make_parameter": remove fixed variables from the optimization variables + - "make_parameter_nodual": as "make_parameter", but do not compute bound + multipliers for fixed variables + - "make_constraint": add equality constraints fixing the variables + - "relax_bounds": relax the fixing bound constraints (according to + ``bound_relax_factor``) + + The main difference is that in the "make_constraint" case the starting point + still has the fixed variables at their given values, whereas with + "make_parameter(_nodual)" the functions are always evaluated with the fixed + values for those variables. For all but "make_parameter_nodual", bound + multipliers are computed for the fixed variables. + + """ + dependency_detector: Literal["none", "mumps", "wsmp", "ma28"] | None = None + """Which linear solver should be used to detect linearly dependent equality + constraints. + + ``None`` (or "none") means no check is performed. This feature is experimental + and does not work well. + + """ + dependency_detection_with_rhs: YesNoBool = False + """Whether the right hand sides of the constraints should be considered in + addition to the gradients during dependency detection.""" + # bounds kappa_d: NonNegativeFloat = 1e-5 + """Weight for the linear damping term (to handle one-sided bounds). + + See Section 3.7 in :cite:`Waechter2005b`. + + """ + bound_relax_factor: NonNegativeFloat = 1e-8 + """Factor for the initial relaxation of the bounds. + + Before the start of the optimization, the bounds given by the user are relaxed + by this relative factor. In addition, ``constr_viol_tol`` is used to bound the + relaxation by an absolute value. If set to zero, the bound relaxation is + disabled. See Eqn. (35) in :cite:`Waechter2005b`. Note that the constraint + violation reported by Ipopt at the end of the solution process does not + include violations of the original (non-relaxed) variable bounds. See also + ``honor_original_bounds``. + + """ + honor_original_bounds: YesNoBool = False + """Whether the final point should be projected back into the original bounds. + + Ipopt might relax the bounds during the optimization (see + ``bound_relax_factor``). This option determines whether the final point is + projected back into the user-provided original bounds after the optimization. + Note that violations of constraints and complementarity reported by Ipopt at + the end of the solution process are for the non-projected point. + + """ + # derivatives check_derivatives_for_naninf: YesNoBool = False + """Whether to check for NaN or Inf in the derivative matrices. + + Activating this option will cause an error if an invalid number is detected in + the constraint Jacobians or the Lagrangian Hessian. If it is not activated, + the test is skipped and the algorithm might proceed with invalid numbers and + fail. + + """ + # not sure if we should support the following: jac_c_constant: YesNoBool = False + """Whether to assume that all equality constraints are linear. + + Activating this option will cause Ipopt to ask for the Jacobian of the + equality constraints only once and reuse this information later. + + """ + jac_d_constant: YesNoBool = False + """Whether to assume that all inequality constraints are linear. + + Activating this option will cause Ipopt to ask for the Jacobian of the + inequality constraints only once and reuse this information later. + + """ + hessian_constant: YesNoBool = False + """Whether to assume that the problem is a QP (quadratic objective, linear + constraints). + + Activating this option will cause Ipopt to ask for the Hessian of the + Lagrangian function only once and reuse this information later. + + """ + # scaling nlp_scaling_method: ( Literal[ @@ -97,95 +372,508 @@ class Ipopt(Algorithm): ] | None ) = "gradient-based" + """Technique used for scaling the problem internally before it is solved. + + Possible values: + + - ``None`` or "none": no problem scaling is performed + - "user-scaling": the scaling parameters come from the user + - "gradient-based": scale the problem so that the maximum gradient at the + starting point is ``nlp_scaling_max_gradient`` + - "equilibration-based": scale the problem so that first derivatives are of + order 1 at random points (uses the Harwell routine MC19) + + """ + obj_scaling_factor: float = 1 + """Scaling factor for the objective function. + + The scaling is only seen internally by Ipopt, the unscaled objective is + reported in the output. If additional scaling parameters are computed (e.g. + with user-scaling or gradient-based scaling), both factors are multiplied. If + this value is negative, Ipopt maximizes the objective function instead of + minimizing it. + + """ + nlp_scaling_max_gradient: PositiveFloat = 100 + """Maximum gradient after NLP scaling. + + This is the gradient scaling cut-off. If the maximum gradient is above this + value, gradient based scaling is performed, with scaling parameters calculated + to scale the maximum gradient back to this value. This is g_max in Section 3.8 + of :cite:`Waechter2005b`. Only used if ``nlp_scaling_method`` is + "gradient-based". + + """ + nlp_scaling_obj_target_gradient: NonNegativeFloat = 0.0 + """Advanced feature. Target value for the objective function gradient size. + + If a positive number is chosen, the scaling factor for the objective function + is computed so that the gradient has the max-norm of the given size at the + starting point. This overrides ``nlp_scaling_max_gradient`` for the objective + function. + + """ + nlp_scaling_constr_target_gradient: NonNegativeFloat = 0.0 + """Advanced feature. Target value for the constraint function gradient size. + + If a positive number is chosen, the scaling factors for the constraint + functions are computed so that the gradient has the max-norm of the given size + at the starting point. This overrides ``nlp_scaling_max_gradient`` for the + constraint functions. + + """ + nlp_scaling_min_value: NonNegativeFloat = 1e-8 + """Minimum value of gradient-based scaling values. + + This is the lower bound for the scaling factors computed by the gradient-based + scaling method. If some derivatives of some functions are huge, the scaling + factors will otherwise become very small, and the (unscaled) final constraint + violation, for example, might then be significant. Only used if + ``nlp_scaling_method`` is "gradient-based". + + """ + # initialization bound_push: PositiveFloat = 0.01 + """Desired minimum absolute distance from the initial point to the bounds. + + Determines how much the initial point might have to be modified in order to be + sufficiently inside the bounds (together with ``bound_frac``). This is kappa_1 + in Section 3.6 of :cite:`Waechter2005b`. + + """ + # TODO: refine type to fix the range (0,0.5] bound_frac: PositiveFloat = 0.01 + """Desired minimum relative distance from the initial point to the bounds. + + Determines how much the initial point might have to be modified in order to be + sufficiently inside the bounds (together with ``bound_push``). This is kappa_2 + in Section 3.6 of :cite:`Waechter2005b`. The valid range is (0, 0.5]. + + """ + slack_bound_push: PositiveFloat = 0.01 + """Desired minimum absolute distance from the initial slack to the bounds. + + Determines how much the initial slack variables might have to be modified in + order to be sufficiently inside the inequality bounds (together with + ``slack_bound_frac``). This is kappa_1 in Section 3.6 of + :cite:`Waechter2005b`. + + """ + # TODO: refine type to fix the range (0,0.5] slack_bound_frac: PositiveFloat = 0.01 + """Desired minimum relative distance from the initial slack to the bounds. + + Determines how much the initial slack variables might have to be modified in + order to be sufficiently inside the inequality bounds (together with + ``slack_bound_push``). This is kappa_2 in Section 3.6 of + :cite:`Waechter2005b`. The valid range is (0, 0.5]. + + """ + constr_mult_init_max: NonNegativeFloat = 1000 + """Maximum allowed least-squares guess of the constraint multipliers. + + Determines how large the initial least-squares guesses of the constraint + multipliers are allowed to be (in max-norm). If the guess is larger than this + value, it is discarded and all constraint multipliers are set to zero. This + option is also used when initializing the restoration phase. + + """ + bound_mult_init_val: PositiveFloat = 1 + """Initial value for the bound multipliers. + + All dual variables corresponding to bound constraints are initialized to this + value. + + """ + bound_mult_init_method: Literal[ "constant", "mu-based", ] = "constant" + """Initialization method for the bound multipliers. + + If "constant" is chosen, all bound multipliers are initialized to the value of + ``bound_mult_init_val``. If "mu-based" is chosen, each value is initialized to + ``mu_init`` divided by the corresponding slack variable. The latter might be + useful if the starting point is close to the optimal solution. + + """ + least_square_init_primal: YesNoBool = False + """Least-squares initialization of the primal variables. + + If enabled, Ipopt ignores the user-provided point and solves a least-squares + problem for the primal variables (x and s) to fit the linearized equality and + inequality constraints. This might be useful if the user doesn't know anything + about the starting point, or for solving an LP or QP. + + """ + least_square_init_duals: YesNoBool = False + """Least-squares initialization of all dual variables. + + If enabled, Ipopt tries to compute least-squares multipliers, considering all + dual variables. If successful, the bound multipliers are possibly corrected to + be at least ``bound_mult_init_val``. This might be useful if the user doesn't + know anything about the starting point, or for solving an LP or QP. This + overwrites the option ``bound_mult_init_method``. + + """ + # warm start warm_start_init_point: YesNoBool = False + """Whether this optimization should use a warm start initialization, where + values of the primal and dual variables are given (e.g. from a previous + optimization of a related problem).""" + warm_start_same_structure: YesNoBool = False + """Advanced feature. + + Indicates whether a problem with a structure identical to the previous one is + to be solved. If enabled, the algorithm assumes that an NLP is now to be + solved whose structure is identical to one that was already considered (with + the same NLP object). + + """ + warm_start_bound_push: PositiveFloat = 0.001 + """Same as ``bound_push`` for the warm start initializer.""" + + # TODO: refine type to fix the range (0,0.5] warm_start_bound_frac: PositiveFloat = 0.001 + """Same as ``bound_frac`` for the warm start initializer. + + The valid range is (0, 0.5]. + + """ + warm_start_slack_bound_push: PositiveFloat = 0.001 + """Same as ``slack_bound_push`` for the warm start initializer.""" + # TODO: refine type to fix the range (0,0.5]) warm_start_slack_bound_frac: PositiveFloat = 0.001 + """Same as ``slack_bound_frac`` for the warm start initializer. + + The valid range is (0, 0.5]. + + """ + warm_start_mult_bound_push: PositiveFloat = 0.001 + """Same as ``bound_push`` for the bound multipliers in the warm start + initializer.""" + warm_start_mult_init_max: float = 1e6 + """Maximum initial value for the equality constraint multipliers when using a + warm start.""" + warm_start_entire_iterate: YesNoBool = False + """Whether to use the GetWarmStartIterate method in the NLP instead of + GetStartingPoint when using a warm start.""" + warm_start_target_mu: float = 0.0 + """Advanced and experimental feature.""" + # miscellaneous option_file_name: str = "" + """File name of an Ipopt options file. + + By default, optimagic passes an empty string, which disables reading of an + options file. If a name is given, Ipopt reads additional options from that + file. + + """ + replace_bounds: YesNoBool = False + """Whether all variable bounds should be replaced by inequality constraints. + + This option must be set for the inexact algorithm. + + """ + skip_finalize_solution_call: YesNoBool = False + """Whether the call to NLP::FinalizeSolution after the optimization should be + suppressed.""" + timing_statistics: YesNoBool = False + """Whether to measure the time spent in the components of Ipopt and the NLP + evaluation. + + The overall algorithm time is unaffected by this option. + + """ + # barrier parameter update mu_max_fact: PositiveFloat = 1000 + """Factor for the initialization of the maximum value of the barrier + parameter. + + The upper bound on the barrier parameter is computed as the average + complementarity at the initial point times the value of this option. Only used + if ``mu_strategy`` is "adaptive". + + """ + mu_max: PositiveFloat = 100_000 + """Maximum value for the barrier parameter. + + This option specifies an upper bound on the barrier parameter in the adaptive + mu selection mode. If this option is set, it overwrites the effect of + ``mu_max_fact``. Only used if ``mu_strategy`` is "adaptive". + + """ + mu_min: PositiveFloat = 1e-11 + """Minimum value for the barrier parameter. + + This option specifies the lower bound on the barrier parameter in the adaptive + mu selection mode. In Ipopt, it is by default set to the minimum of 1e-11 and + ``min(tol, compl_inf_tol) / (barrier_tol_factor + 1)``, which should be a + reasonable value. Only used if ``mu_strategy`` is "adaptive". + + """ + adaptive_mu_globalization: Literal[ "obj-constr-filter", "kkt-error", "never-monotone-mode", ] = "obj-constr-filter" + """Globalization strategy for the adaptive mu selection mode. + + To achieve global convergence of the adaptive version, the algorithm has to + switch to the monotone mode (Fiacco-McCormick approach) when convergence does + not seem to appear. This option sets the criterion used to decide when to make + this switch. Possible values: + + - "kkt-error": nonmonotone decrease of the KKT error + - "obj-constr-filter": two-dimensional filter for objective and constraint + violation + - "never-monotone-mode": disables globalization + + Only used if ``mu_strategy`` is "adaptive". + + """ + adaptive_mu_kkterror_red_iters: NonNegativeInt = 4 + """Advanced feature. Maximum number of iterations requiring sufficient + progress. + + For the "kkt-error" based globalization strategy, sufficient progress must be + made for this many iterations. If this number of iterations is exceeded, the + globalization strategy switches to the monotone mode. + + """ + # TODO: refine type to fix the range (0,1) adaptive_mu_kkterror_red_fact: PositiveFloat = 0.9999 + """Advanced feature. Sufficient decrease factor for the "kkt-error" + globalization strategy. + + For the "kkt-error" based globalization strategy, the error must decrease by + this factor to be deemed sufficient decrease. The valid range is (0, 1). + + """ + # TODO: refine type to fix the range (0,1) filter_margin_fact: PositiveFloat = 1e-5 + """Advanced feature. Factor determining the width of the margin for the + "obj-constr-filter" adaptive globalization strategy. + + Sufficient progress for a filter entry is defined as: (new obj) < (filter obj) + - filter_margin_fact * (new constr-viol) or (new constr-viol) < (filter + constr-viol) - filter_margin_fact * (new constr-viol). For the description of + the "kkt-error-filter" option see ``filter_max_margin``. The valid range is + (0, 1). + + """ + filter_max_margin: PositiveFloat = 1 + """Advanced feature. + + Maximum width of the margin in the "obj-constr-filter" adaptive globalization + strategy. + + """ + adaptive_mu_restore_previous_iterate: YesNoBool = False + """Advanced feature. Whether the previous accepted iterate should be restored + if the monotone mode is entered. + + When the globalization strategy for the adaptive barrier algorithm switches to + the monotone mode, it can either start from the most recent iterate (False) or + from the last iterate that was accepted (True). + + """ + adaptive_mu_monotone_init_factor: PositiveFloat = 0.8 + """Advanced feature. Determines the initial value of the barrier parameter + when switching to the monotone mode. + + When the globalization strategy for the adaptive barrier algorithm switches to + the monotone mode and ``fixed_mu_oracle`` is chosen as "average_compl", the + barrier parameter is set to the current average complementarity times the + value of this option. + + """ + adaptive_mu_kkt_norm_type: Literal[ "max-norm", "2-norm-squared", "1-norm", "2-norm", ] = "2-norm-squared" + """Advanced feature. Norm used for the KKT error in the adaptive mu + globalization strategies. + + When computing the KKT error for the globalization strategies, the norm to be + used is specified with this option. Note that this option is also used in the + quality function based mu oracle. + + """ + mu_strategy: Literal["monotone", "adaptive"] = "monotone" + """Update strategy for the barrier parameter. + + "monotone" uses the monotone (Fiacco-McCormick) strategy; "adaptive" uses the + adaptive update strategy from :cite:`Nocedal2009`. + + """ + mu_oracle: Literal[ "probing", "quality-function", "loqo", ] = "quality-function" + """Oracle for a new barrier parameter in the adaptive strategy. + + Determines how a new barrier parameter is computed in each "free-mode" + iteration of the adaptive barrier parameter strategy. Possible values: + + - "probing": Mehrotra's probing heuristic + - "loqo": LOQO's centrality rule + - "quality-function": minimize a quality function + + Only used if ``mu_strategy`` is "adaptive". + + """ + fixed_mu_oracle: Literal[ "probing", "loqo", "quality-function", "average_compl", ] = "average_compl" + """Oracle for the barrier parameter when switching to the fixed mode. + + Determines how the first value of the barrier parameter should be computed + when switching to the "monotone mode" in the adaptive strategy. In addition to + the choices for ``mu_oracle``, "average_compl" bases the value on the current + average complementarity. Only used if ``mu_strategy`` is "adaptive". + + """ + mu_init: PositiveFloat = 0.1 + """Initial value for the barrier parameter. + + This option determines the initial value for the barrier parameter mu. It is + only relevant in the monotone, Fiacco-McCormick version of the algorithm, + i.e. if ``mu_strategy`` is "monotone". + + """ + barrier_tol_factor: PositiveFloat = 10 + """Factor for mu in the barrier stop test. + + The convergence tolerance for each barrier problem in the monotone mode is the + value of the barrier parameter times this factor. This option is also used in + the adaptive mu strategy during the monotone mode. This is kappa_epsilon in + :cite:`Waechter2005b`. + + """ + # TODO: refine type to fix the range (0,1) mu_linear_decrease_factor: PositiveFloat = 0.2 + """Determines the linear decrease rate of the barrier parameter. + + For the Fiacco-McCormick update procedure, the new barrier parameter mu is + obtained by taking the minimum of ``mu * mu_linear_decrease_factor`` and + ``mu ** mu_superlinear_decrease_power``. This is kappa_mu in + :cite:`Waechter2005b`. This option is also used in the adaptive mu strategy + during the monotone mode. The valid range is (0, 1). + + """ + # TODO: refine type to fix the range (1,2) mu_superlinear_decrease_power: GtOneFloat = 1.5 + """Determines the superlinear decrease rate of the barrier parameter. + + For the Fiacco-McCormick update procedure, the new barrier parameter mu is + obtained by taking the minimum of ``mu * mu_linear_decrease_factor`` and + ``mu ** mu_superlinear_decrease_power``. This is theta_mu in + :cite:`Waechter2005b`. This option is also used in the adaptive mu strategy + during the monotone mode. The valid range is (1, 2). + + """ + mu_allow_fast_monotone_decrease: YesNoBool = True + """Advanced feature. Allow skipping of the barrier problem if the barrier test + is already met. + + If False, the algorithm takes at least one iteration per barrier problem, even + if the barrier test is already met for the updated barrier parameter. + + """ + # TODO: refine type to fix the range (0,1) tau_min: PositiveFloat = 0.99 + """Advanced feature. Lower bound on the fraction-to-the-boundary parameter + tau. + + This is tau_min in :cite:`Waechter2005b`. This option is also used in the + adaptive mu strategy during the monotone mode. The valid range is (0, 1). + + """ + sigma_max: PositiveFloat = 100 + """Advanced feature. Maximum value of the centering parameter. + + This is the upper bound for the centering parameter chosen by the quality + function based barrier parameter update. Only used if ``mu_oracle`` is + "quality-function". + + """ + sigma_min: NonNegativeFloat = 1e-6 + """Advanced feature. Minimum value of the centering parameter. + + This is the lower bound for the centering parameter chosen by the quality + function based barrier parameter update. Only used if ``mu_oracle`` is + "quality-function". + + """ + quality_function_norm_type: Literal[ "max-norm", "2-norm-squared", "1-norm", "2-norm", ] = "2-norm-squared" + """Advanced feature. Norm used for the components of the quality function. + + Only used if ``mu_oracle`` is "quality-function". + + """ + quality_function_centrality: ( Literal[ "none", @@ -195,22 +883,105 @@ class Ipopt(Algorithm): ] | None ) = None + """Advanced feature. The penalty term for centrality that is included in the + quality function. + + This determines whether a term is added to the quality function to penalize + deviation from centrality with respect to complementarity. The complementarity + measure here is the xi in the LOQO update rule. Possible values: + + - ``None`` or "none": no penalty term is added + - "log": complementarity times the log of the centrality measure + - "reciprocal": complementarity times the reciprocal of the centrality measure + - "cubed-reciprocal": complementarity times the reciprocal of the centrality + measure cubed + + Only used if ``mu_oracle`` is "quality-function". + + """ + quality_function_balancing_term: Literal["none", "cubic"] | None = None + """Advanced feature. The balancing term included in the quality function for + centrality. + + This determines whether a term is added to the quality function that penalizes + situations where the complementarity is much smaller than the dual and primal + infeasibilities. ``None`` (or "none") adds no balancing term; "cubic" adds + ``max(0, max(dual_inf, primal_inf) - compl)**3``. Only used if ``mu_oracle`` + is "quality-function". + + """ + quality_function_max_section_steps: NonNegativeInt = 8 + """Maximum number of search steps during the direct search procedure + determining the optimal centering parameter. + + The golden section search is performed for the quality function based mu + oracle. Only used if ``mu_oracle`` is "quality-function". + + """ + # TODO: refine type to fix the range [0,1) quality_function_section_sigma_tol: NonNegativeFloat = 0.01 + """Advanced feature. Tolerance for the section search procedure determining + the optimal centering parameter (in sigma space). + + The golden section search is performed for the quality function based mu + oracle. Only used if ``mu_oracle`` is "quality-function". The valid range is + [0, 1). + + """ + # TODO: refine type to fix the range [0,1) quality_function_section_qf_tol: NonNegativeFloat = 0.0 + """Advanced feature. Tolerance for the golden section search procedure + determining the optimal centering parameter (in the function value space). + + Only used if ``mu_oracle`` is "quality-function". The valid range is [0, 1). + + """ + # line search line_search_method: Literal[ "filter", "penalty", "cg-penalty", ] = "filter" + """Advanced feature. Globalization method used in the backtracking line + search. + + Only the "filter" choice is officially supported, but sometimes good results + might be obtained with "cg-penalty" (Chen-Goldfarb penalty function) or + "penalty" (standard penalty function). + + """ + # TODO: refine type to fix the range (0,1) alpha_red_factor: PositiveFloat = 0.5 + """Advanced feature. Fractional reduction of the trial step size in the + backtracking line search. + + At every step of the backtracking line search, the trial step size is reduced + by this factor. The valid range is (0, 1). + + """ + accept_every_trial_step: YesNoBool = False + """Always accept the first trial step. + + Setting this option to True essentially disables the line search and makes the + algorithm take aggressive steps, without global convergence guarantees. + + """ + accept_after_max_steps: Literal[-1] | NonNegativeInt = -1 + """Advanced feature. Accept a trial point after at most this number of steps, + even if it does not satisfy the line search conditions. + + Setting this to -1 disables this heuristic. + + """ + alpha_for_y: Literal[ "primal", "bound-mult", @@ -223,29 +994,206 @@ class Ipopt(Algorithm): "dual-and-full", "acceptor", ] = "primal" + """Method to determine the step size for the equality constraint multipliers + (alpha_y). + + Possible values: + + - "primal": use the primal step size + - "bound-mult": use the step size for the bound multipliers (good for LPs) + - "min": use the min of the primal and bound multiplier step sizes + - "max": use the max of the primal and bound multiplier step sizes + - "full": take a full step of size one + - "min-dual-infeas": choose the step size minimizing the new dual + infeasibility + - "safer-min-dual-infeas": like "min-dual-infeas", but safeguarded by "min" + and "max" + - "primal-and-full": use the primal step size, and a full step if + ``delta_x <= alpha_for_y_tol`` + - "dual-and-full": use the dual step size, and a full step if + ``delta_x <= alpha_for_y_tol`` + - "acceptor": call LSAcceptor to get the step size for y + + """ + alpha_for_y_tol: NonNegativeFloat = 10 + """Tolerance for switching to full equality multiplier steps. + + This is only relevant if ``alpha_for_y`` is "primal-and-full" or + "dual-and-full". The step size for the equality constraint multipliers is + taken to be one if the max-norm of the primal step is less than this + tolerance. + + """ + tiny_step_tol: NonNegativeFloat = 2.22045 * 1e-15 + """Advanced feature. Tolerance for detecting numerically insignificant steps. + + If the search direction in the primal variables (x and s) is, in relative + terms for each component, less than this value, the algorithm accepts the full + step without a line search. If this happens repeatedly, the algorithm + terminates with a corresponding exit message. The default value is 10 times + machine precision. + + """ + tiny_step_y_tol: NonNegativeFloat = 0.01 + """Advanced feature. Tolerance for quitting because of numerically + insignificant steps. + + If the search direction in the primal variables (x and s) is, in relative + terms for each component, repeatedly less than ``tiny_step_tol`` and the step + in the y variables is smaller than this threshold, the algorithm terminates. + + """ + watchdog_shortened_iter_trigger: NonNegativeInt = 10 + """Number of shortened iterations that trigger the watchdog procedure. + + If the number of successive iterations in which the backtracking line search + did not accept the first trial point exceeds this number, the watchdog + procedure is activated. Choosing 0 disables the watchdog procedure. + + """ + watchdog_trial_iter_max: PositiveInt = 3 + """Maximum number of watchdog iterations. + + This option determines the number of trial iterations allowed before the + watchdog procedure is aborted and the algorithm returns to the stored point. + + """ + theta_max_fact: PositiveFloat = 10_000 + """Advanced feature. Determines the upper bound for the constraint violation + in the filter. + + The algorithmic parameter theta_max is determined as this value times the + maximum of 1 and the constraint violation at the initial point. Any point with + a constraint violation larger than theta_max is unacceptable to the filter + (see Eqn. (21) in :cite:`Waechter2005b`). + + """ + theta_min_fact: PositiveFloat = 0.0001 + """Advanced feature. Determines the constraint violation threshold in the + switching rule. + + The algorithmic parameter theta_min is determined as this value times the + maximum of 1 and the constraint violation at the initial point. The switching + rule treats an iteration as an h-type iteration whenever the current + constraint violation is larger than theta_min (see paragraph before Eqn. (19) + in :cite:`Waechter2005b`). + + """ + # TODO: refine type to fix the range (0,0.5) eta_phi: PositiveFloat = 1e-8 + """Advanced feature. Relaxation factor in the Armijo condition. + + See Eqn. (20) in :cite:`Waechter2005b`. The valid range is (0, 0.5). + + """ + delta: PositiveFloat = 1 + """Advanced feature. Multiplier for the constraint violation in the switching + rule. + + See Eqn. (19) in :cite:`Waechter2005b`. + + """ + s_phi: GtOneFloat = 2.3 + """Advanced feature. Exponent for the linear barrier function model in the + switching rule. + + See Eqn. (19) in :cite:`Waechter2005b`. + + """ + s_theta: GtOneFloat = 1.1 + """Advanced feature. Exponent for the current constraint violation in the + switching rule. + + See Eqn. (19) in :cite:`Waechter2005b`. + + """ + # TODO: refine type to fix the range (0,1) gamma_phi: PositiveFloat = 1e-8 + """Advanced feature. Relaxation factor in the filter margin for the barrier + function. + + See Eqn. (18a) in :cite:`Waechter2005b`. The valid range is (0, 1). + + """ + # TODO: refine type to fix the range (0,1) gamma_theta: PositiveFloat = 1e-5 + """Advanced feature. Relaxation factor in the filter margin for the constraint + violation. + + See Eqn. (18b) in :cite:`Waechter2005b`. The valid range is (0, 1). + + """ + # TODO: refine type to fix the range (0,1) alpha_min_frac: PositiveFloat = 0.05 + """Advanced feature. Safety factor for the minimal step size (before switching + to the restoration phase). + + This is gamma_alpha in Eqn. (20) in :cite:`Waechter2005b`. The valid range is + (0, 1). + + """ + max_soc: NonNegativeInt = 4 + """Maximum number of second order correction trial steps at each iteration. + + Choosing 0 disables the second order corrections. This is p^max of Step A-5.9 + of Algorithm A in :cite:`Waechter2005b`. + + """ + kappa_soc: PositiveFloat = 0.99 + """Advanced feature. Factor in the sufficient reduction rule for the second + order correction. + + This option determines how much a second order correction step must reduce the + constraint violation so that further correction steps are attempted. See Step + A-5.9 of Algorithm A in :cite:`Waechter2005b`. + + """ + obj_max_inc: float = 5.0 + """Advanced feature. Upper bound on the acceptable increase of the barrier + objective function. + + Trial points are rejected if they lead to an increase in the barrier objective + function by more than this many orders of magnitude. The valid range is + (1, inf). + + """ + max_filter_resets: NonNegativeInt = 5 + """Advanced feature. Maximal allowed number of filter resets. + + A positive number enables a heuristic that resets the filter whenever in more + than ``filter_reset_trigger`` successive iterations the last rejected trial + step size was rejected because of the filter. This option determines the + maximal number of resets that are allowed to take place. + + """ + filter_reset_trigger: PositiveInt = 5 + """Advanced feature. Number of iterations that trigger the filter reset. + + If the filter reset heuristic is active and the last rejected trial step size + was rejected because of the filter for this many successive iterations, the + filter is reset. + + """ + corrector_type: ( Literal[ "none", @@ -254,69 +1202,486 @@ class Ipopt(Algorithm): ] | None ) = None + """Advanced feature. The type of corrector steps that should be taken. + + If ``mu_strategy`` is "adaptive", this option determines what kind of + corrector steps should be tried. Possible values are ``None`` (or "none", no + corrector), "affine" (corrector step towards mu=0) and "primal-dual" + (corrector step towards the current mu). Changing this option is experimental. + + """ + skip_corr_if_neg_curv: YesNoBool = True + """Advanced feature. Whether to skip the corrector step in negative curvature + iterations. + + The corrector step is not tried if negative curvature has been encountered + during the computation of the search direction in the current iteration. Only + used if ``mu_strategy`` is "adaptive". Changing this option is experimental. + + """ + skip_corr_in_monotone_mode: YesNoBool = True + """Advanced feature. Whether to skip the corrector step during the monotone + barrier parameter mode. + + The corrector step is not tried if the algorithm is currently in the monotone + mode. Only used if ``mu_strategy`` is "adaptive". Changing this option is + experimental. + + """ + corrector_compl_avrg_red_fact: PositiveFloat = 1 + """Advanced feature. Complementarity tolerance factor for accepting a + corrector step. + + This option determines the factor by which the complementarity is allowed to + increase for a corrector step to be accepted. Changing this option is + experimental. + + """ + soc_method: Literal[0, 1] = 0 + """Ways to apply the second order correction. + + 0 is the method described in :cite:`Waechter2005b`; 1 is a modified way which + adds alpha on the right hand side of the x and s rows. + + """ + nu_init: PositiveFloat = 1e-6 + """Advanced feature. + + Initial value of the penalty parameter in the penalty function based line + search. + + """ + nu_inc: PositiveFloat = 0.0001 + """Advanced feature. + + Increment of the penalty parameter in the penalty function based line search. + + """ + # TODO: refine type to fix the range (0,1) rho: PositiveFloat = 0.1 + """Advanced feature. Value in the penalty parameter update formula. + + The valid range is (0, 1). + + """ + kappa_sigma: PositiveFloat = 1e10 + """Advanced feature. Factor limiting the deviation of the dual variables from + their primal estimates. + + If the dual variables deviate from their primal estimates, a correction is + performed. See Eqn. (16) in :cite:`Waechter2005b`. Setting the value to less + than 1 disables the correction. + + """ + recalc_y: YesNoBool = False + """Whether to recalculate the equality and inequality multipliers as + least-squares estimates. + + This asks the algorithm to recompute the multipliers whenever the current + infeasibility is less than ``recalc_y_feas_tol``. Choosing True might be + helpful in the quasi-Newton option, but each recalculation requires an extra + factorization of the linear system. If a limited-memory quasi-Newton option is + chosen, this is used by default. + + """ + recalc_y_feas_tol: PositiveFloat = 1e-6 + """Feasibility threshold for the recomputation of multipliers. + + If ``recalc_y`` is chosen and the current infeasibility is less than this + value, the multipliers are recomputed. + + """ + slack_move: NonNegativeFloat = 1.81899 * 1e-12 + """Advanced feature. Correction size for very small slacks. + + Due to numerical issues or the lack of an interior, the slack variables might + become very small. If a slack becomes very small compared to machine + precision, the corresponding bound is moved slightly; this parameter + determines how large the move should be. The default is mach_eps^(3/4). See + also the end of Section 3.5 in :cite:`Waechter2005b` (the actual + implementation might be somewhat different). + + """ + constraint_violation_norm_type: Literal[ "1-norm", "2-norm", "max-norm", ] = "1-norm" + """Advanced feature. + + Norm to be used for the constraint violation in the line search. + + """ + # step calculation mehrotra_algorithm: YesNoBool = False + """Whether to do Mehrotra's predictor-corrector algorithm. + + If enabled, the line search is disabled and the (unglobalized) adaptive mu + strategy is chosen with the "probing" oracle, and "corrector_type=affine" is + used without any safeguards; you should not explicitly set any of those + options in addition. Also, unless otherwise specified, the values of + ``bound_push``, ``bound_frac`` and ``bound_mult_init_val`` are set more + aggressively, and "alpha_for_y=bound-mult" is used. Mehrotra's + predictor-corrector algorithm usually works very well for LPs and convex QPs. + + """ + fast_step_computation: YesNoBool = False + """Whether the linear system should be solved quickly. + + If enabled, the algorithm assumes that the linear system that is solved to + obtain the search direction is solved sufficiently well. In that case, no + residuals are computed to verify the solution and the computation of the + search direction is a little faster. + + """ + min_refinement_steps: NonNegativeInt = 1 + """Minimum number of iterative refinement steps per linear system solve. + + Iterative refinement (on the full unsymmetric system) is performed for each + right hand side; at least this many iterative refinement steps are enforced + per right hand side. + + """ + max_refinement_steps: NonNegativeInt = 10 + """Maximum number of iterative refinement steps per linear system solve. + + Iterative refinement (on the full unsymmetric system) is performed for each + right hand side. + + """ + residual_ratio_max: PositiveFloat = 1e-10 + """Advanced feature. Iterative refinement tolerance. + + Iterative refinement is performed until the residual test ratio is less than + this tolerance (or until ``max_refinement_steps`` refinement steps are + performed). + + """ + residual_ratio_singular: PositiveFloat = 1e-5 + """Advanced feature. Threshold for declaring the linear system singular after + failed iterative refinement. + + If the residual test ratio is larger than this value after failed iterative + refinement, the algorithm pretends that the linear system is singular. + + """ + residual_improvement_factor: PositiveFloat = 1 + """Advanced feature. Minimal required reduction of the residual test ratio in + iterative refinement. + + If the improvement of the residual test ratio made by one iterative refinement + step is not better than this factor, iterative refinement is aborted. + + """ + neg_curv_test_tol: NonNegativeFloat = 0 + """Tolerance for the heuristic to ignore wrong inertia. + + If nonzero, incorrect inertia in the augmented system is ignored and Ipopt + tests if the direction is a direction of positive curvature. This tolerance is + alpha_n in :cite:`Chiang2014` and it determines when the direction is + considered to be sufficiently positive. A value in the range [1e-12, 1e-11] is + recommended. + + """ + neg_curv_test_reg: YesNoBool = True + """Whether to do the curvature test with the primal regularization (see + :cite:`Chiang2014`). + + If False, the original Ipopt approach is used, in which the primal + regularization is ignored. + + """ + max_hessian_perturbation: PositiveFloat = 1e20 + """Maximum value of the regularization parameter for handling negative + curvature. + + To guarantee that the search directions are proper descent directions, Ipopt + requires that the inertia of the (augmented) linear system for the step + computation has the correct number of negative and positive eigenvalues. This + guides the algorithm away from maximizers and makes Ipopt more likely to + converge to first order optimal points that are minimizers. If the inertia is + not correct, a multiple of the identity matrix is added to the Hessian of the + Lagrangian in the augmented system. This parameter gives the maximum value of + the regularization parameter; if a regularization of that size is not enough, + the algorithm skips this iteration and goes to the restoration phase. This is + delta_w^max in :cite:`Waechter2005b`. + + """ + min_hessian_perturbation: NonNegativeFloat = 1e-20 + """Smallest perturbation of the Hessian block. + + The size of the perturbation of the Hessian block is never selected smaller + than this value, unless no perturbation is necessary. This is delta_w^min in + :cite:`Waechter2005b`. + + """ + perturb_inc_fact_first: GtOneFloat = 100 + """Increase factor for the x-s perturbation for the very first perturbation. + + The factor by which the perturbation is increased when a trial value was not + sufficient; this value is used for the computation of the very first + perturbation and allows a different value for the first perturbation than + that used for the remaining perturbations. This is bar_kappa_w^+ in + :cite:`Waechter2005b`. + + """ + perturb_inc_fact: GtOneFloat = 8 + """Increase factor for the x-s perturbation. + + The factor by which the perturbation is increased when a trial value was not + sufficient; this value is used for the computation of all perturbations except + for the first. This is kappa_w^+ in :cite:`Waechter2005b`. + + """ + # TODO: refine type to fix the range (0,1) perturb_dec_fact: PositiveFloat = 0.333333 + """Decrease factor for the x-s perturbation. + + The factor by which the perturbation is decreased when a trial value is + deduced from the size of the most recent successful perturbation. This is + kappa_w^- in :cite:`Waechter2005b`. The valid range is (0, 1). + + """ + first_hessian_perturbation: PositiveFloat = 0.0001 + """Size of the first x-s perturbation tried. + + This is the first value tried for the x-s perturbation in the inertia + correction scheme. It is delta_0 in :cite:`Waechter2005b`. + + """ + jacobian_regularization_value: NonNegativeFloat = 1e-8 + """Size of the regularization for rank-deficient constraint Jacobians. + + This is bar_delta_c in :cite:`Waechter2005b`. + + """ + jacobian_regularization_exponent: NonNegativeFloat = 0.25 + """Advanced feature. Exponent for mu in the regularization for rank-deficient + constraint Jacobians. + + This is kappa_c in :cite:`Waechter2005b`. + + """ + perturb_always_cd: YesNoBool = False + """Advanced feature. Activate the permanent perturbation of the constraint + linearization. + + Enabling this option leads to using the delta_c and delta_d perturbations for + the computation of every search direction. Usually, they are only used when + the iteration matrix is singular. + + """ + # restoration phase expect_infeasible_problem: YesNoBool = False + """Enable heuristics to quickly detect an infeasible problem. + + This option activates heuristics that may speed up the infeasibility + determination if you expect that there is a good chance for the problem to be + infeasible. In the filter line search procedure, the restoration phase is + called more quickly than usual, and more reduction in the constraint violation + is enforced before the restoration phase is left. If the problem is square, + this option is enabled automatically. + + """ + expect_infeasible_problem_ctol: NonNegativeFloat = 0.001 + """Threshold for disabling the ``expect_infeasible_problem`` option. + + If the constraint violation becomes smaller than this threshold, the + ``expect_infeasible_problem`` heuristics in the filter line search are + disabled. If the problem is square, this option is set to 0. + + """ + expect_infeasible_problem_ytol: PositiveFloat = 1e8 + """Multiplier threshold for activating the ``expect_infeasible_problem`` + option. + + If the max-norm of the constraint multipliers becomes larger than this value + and ``expect_infeasible_problem`` is chosen, then the restoration phase is + entered. + + """ + start_with_resto: YesNoBool = False + """Whether to switch to the restoration phase in the first iteration. + + Setting this option to True forces the algorithm to switch to the feasibility + restoration phase in the first iteration. If the initial point is feasible, + the algorithm will abort with a failure. + + """ + soft_resto_pderror_reduction_factor: NonNegativeFloat = 0.9999 + """Required reduction in the primal-dual error in the soft restoration phase. + + The soft restoration phase attempts to reduce the primal-dual error with + regular steps. If the damped primal-dual step (damped only to satisfy the + fraction-to-the-boundary rule) is not decreasing the primal-dual error by at + least this factor, then the regular restoration phase is called. Choosing 0 + here disables the soft restoration phase. + + """ + max_soft_resto_iters: NonNegativeInt = 10 + """Advanced feature. Maximum number of iterations performed successively in + the soft restoration phase. + + If the soft restoration phase is performed for more than this many iterations + in a row, the regular restoration phase is called. + + """ + # TODO: refine type to fix the range [0,1) required_infeasibility_reduction: NonNegativeFloat = 0.9 + """Required reduction of the infeasibility before leaving the restoration + phase. + + The restoration phase algorithm is performed until a point is found that is + acceptable to the filter and the infeasibility has been reduced by at least + the fraction given by this option. The valid range is [0, 1). + + """ + max_resto_iter: NonNegativeInt = 3_000_000 + """Advanced feature. Maximum number of successive iterations in the + restoration phase. + + The algorithm terminates with an error message if the number of iterations + successively taken in the restoration phase exceeds this number. + + """ + evaluate_orig_obj_at_resto_trial: YesNoBool = True + """Whether the original objective function should be evaluated at restoration + phase trial points. + + Enabling this option makes the restoration phase algorithm evaluate the + objective function of the original problem at every trial point encountered + during the restoration phase, even if this value is not required. This + guarantees that the original objective function can be evaluated without error + at all accepted iterates; otherwise the algorithm might fail at a point where + the restoration phase accepts an iterate that is good for the restoration + phase problem but not the original problem. On the other hand, if the + evaluation of the original objective is expensive, this might be costly. + + """ + resto_penalty_parameter: PositiveFloat = 1000 + """Advanced feature. Penalty parameter in the restoration phase objective + function. + + This is the parameter rho in equation (31a) in :cite:`Waechter2005b`. + + """ + resto_proximity_weight: NonNegativeFloat = 1 + """Advanced feature. Weighting factor for the proximity term in the + restoration phase objective. + + This determines how the parameter zeta in equation (29a) in + :cite:`Waechter2005b` is computed. zeta here is this value times the square + root of mu, where mu is the current barrier parameter. + + """ + bound_mult_reset_threshold: NonNegativeFloat = 1000 + """Threshold for resetting the bound multipliers after the restoration phase. + + After returning from the restoration phase, the bound multipliers are updated + with a Newton step for complementarity, where the change in the primal + variables during the entire restoration phase is taken to be the corresponding + primal Newton step. However, if after the update the largest bound multiplier + exceeds this threshold, the multipliers are all reset to 1. + + """ + constr_mult_reset_threshold: NonNegativeFloat = 0 + """Threshold for resetting the equality and inequality multipliers after the + restoration phase. + + After returning from the restoration phase, the constraint multipliers are + recomputed by a least-squares estimate. This option triggers when those + least-squares estimates should be ignored. + + """ + resto_failure_feasibility_threshold: NonNegativeFloat | None = None + """Advanced feature. Threshold for the primal infeasibility to declare failure + of the restoration phase. + + If the restoration phase is terminated because of the "acceptable" termination + criteria and the primal infeasibility is smaller than this value, the + restoration phase is declared to have failed. If ``None`` (the default), it is + set to ``100 * convergence_ftol_rel``. + + """ + # hessian approximation limited_memory_aug_solver: Literal[ "sherman-morrison", "extended", ] = "sherman-morrison" + """Advanced feature. Strategy for solving the augmented system for the + low-rank Hessian. + + "sherman-morrison" uses the Sherman-Morrison formula; "extended" uses an + extended augmented system. + + """ + limited_memory_max_history: NonNegativeInt = 6 + """Maximum size of the history for the limited-memory quasi-Newton Hessian + approximation. + + This option determines the number of most recent iterations that are taken + into account for the limited-memory quasi-Newton approximation. + + """ + limited_memory_update_type: Literal[ "bfgs", "sr1", ] = "bfgs" + """Quasi-Newton update formula for the limited-memory quasi-Newton + approximation. + + "bfgs" is the BFGS update (with skipping); "sr1" is the SR1 update (not + working well). + + """ + limited_memory_initialization: Literal[ "scalar1", "scalar2", @@ -324,24 +1689,105 @@ class Ipopt(Algorithm): "scalar4", "constant", ] = "scalar1" + """Initialization strategy for the limited-memory quasi-Newton approximation. + + Determines how the diagonal matrix B_0 as the first term in the limited-memory + approximation should be computed. Possible values: + + - "scalar1": sigma = s^T y / s^T s + - "scalar2": sigma = y^T y / s^T y + - "scalar3": arithmetic average of scalar1 and scalar2 + - "scalar4": geometric average of scalar1 and scalar2 + - "constant": sigma = ``limited_memory_init_val`` + + """ + limited_memory_init_val: PositiveFloat = 1 + """Value for B_0 in the low-rank update. + + The starting matrix in the low-rank update, B_0, is chosen to be this multiple + of the identity in the first iteration (when no updates have been performed + yet), and is constantly chosen as this value if + ``limited_memory_initialization`` is "constant". + + """ + limited_memory_init_val_max: PositiveFloat = 1e8 + """Upper bound on the value for B_0 in the low-rank update.""" + limited_memory_init_val_min: PositiveFloat = 1e-8 + """Lower bound on the value for B_0 in the low-rank update.""" + limited_memory_max_skipping: PositiveInt = 2 + """Threshold for successive iterations where the quasi-Newton update is + skipped. + + If the update is skipped more than this number of successive iterations, the + quasi-Newton approximation is reset. + + """ + limited_memory_special_for_resto: YesNoBool = False + """Whether the quasi-Newton updates should be special during the restoration + phase. + + Until Nov 2010, Ipopt used a special update during the restoration phase, but + it turned out that this does not work well. The new default uses the regular + update procedure, which improves results. If for some reason you want to get + back to the original update, set this option to True. + + """ + hessian_approximation: Literal[ "limited-memory", "exact", ] = "limited-memory" + """Indicates what Hessian information is to be used. + + "exact" uses second derivatives provided by the NLP; "limited-memory" performs + a limited-memory quasi-Newton approximation of the Hessian of the Lagrangian. + Since optimagic does not pass second derivatives to Ipopt, the default is + "limited-memory". + + """ + hessian_approximation_space: Literal[ "nonlinear-variables", "all-variables", ] = "nonlinear-variables" + """Advanced feature. Indicates in which subspace the Hessian information is to + be approximated. + + "nonlinear-variables" approximates only in the space of the nonlinear + variables; "all-variables" approximates in the space of all variables (without + slacks). + + """ + # linear solver linear_solver: Literal[ "mumps", "ma27", "ma57", "ma77", "ma86", "ma97", "pardiso", "custom" ] = "mumps" + """Linear solver used for the step computations. + + Determines which linear algebra package is used for the solution of the + augmented linear system (for obtaining the search directions). The default in + optimagic is "mumps" (the MUMPS package that ships with most cyipopt + installations). The Harwell routines "ma27", "ma57", "ma77", "ma86" and "ma97" + as well as "pardiso" are loaded from libraries at runtime and require these to + be installed; "custom" selects a custom linear solver (expert use). + + """ + linear_solver_options: dict[str, Any] | None = None + """Dictionary with the linear solver options, possibly including + ``linear_system_scaling``, ``hsllib`` and ``pardisolib``. + + See the `Ipopt documentation + `_ for details. The linear + solver options are not automatically converted to float at the moment. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/nag_optimizers.py b/src/optimagic/optimizers/nag_optimizers.py index 5546e847b..5c900e030 100644 --- a/src/optimagic/optimizers/nag_optimizers.py +++ b/src/optimagic/optimizers/nag_optimizers.py @@ -8,6 +8,8 @@ """ +from __future__ import annotations + import warnings from dataclasses import dataclass from typing import Any, Callable, Literal, cast @@ -345,54 +347,460 @@ ) @dataclass(frozen=True) class NagDFOLS(Algorithm): + r"""Minimize a function with least-squares structure using the DFO-LS algorithm. + + The DFO-LS algorithm :cite:`Cartis2018b` is a model-based derivative-free + trust-region algorithm developed by the Numerical Algorithms Group and researchers + at the University of Oxford. It is designed to solve the nonlinear least-squares + minimization problem (with optional bound constraints) + + .. math:: + + \min_{x\in\mathbb{R}^n} &\quad f(x) := \sum_{i=1}^{m}r_{i}(x)^2 \\ + \text{s.t.} &\quad \text{lower_bounds} \leq x \leq \text{upper_bounds} + + The :math:`r_{i}` are called root contributions in optimagic. + + DFO-LS is a derivative-free algorithm: it does not require the user to provide the + derivatives of :math:`f(x)` or :math:`r_{i}(x)`, nor does it attempt to estimate + them internally (by using finite differencing, for instance). Instead, it + constructs an interpolation-based model of the root contributions that exploits + the least-squares structure of the problem and minimizes this model within a + trust region. + + There are two main situations when using a derivative-free algorithm (such as + DFO-LS) is preferable to a derivative-based algorithm (which is the vast majority + of least-squares solvers): + + 1. If the root contributions are noisy, then calculating or even estimating their + derivatives may be impossible (or at least very inaccurate). By noisy we mean + that if we evaluate :math:`r_{i}(x)` multiple times at the same value of x, we + get different results. This may happen when a Monte Carlo simulation is used, + for instance. + + 2. If the root contributions are expensive to evaluate, then estimating + derivatives (which requires n evaluations of each :math:`r_{i}(x)` for every + point of interest x) may be prohibitively expensive. Derivative-free methods + are designed to solve the problem with the fewest number of evaluations of the + criterion as possible. + + DFO-LS was developed by the same group as Py-BOBYQA (``nag_pybobyqa``), its + general-purpose scalar counterpart. If your problem has least-squares structure, + DFO-LS typically needs far fewer criterion evaluations than Py-BOBYQA and other + scalar optimizers because it exploits this structure. + + There are four possible convergence criteria: + + 1. when the lower trust region radius is shrunk below a minimum + (``convergence_minimal_trustregion_radius_tolerance``). + + 2. when the improvements of iterations become very small + (``convergence_slow_progress``). This is similar to a relative criterion + tolerance, but more general, because you can specify not only the threshold + for convergence but also a period over which the improvements must have been + very small. + + 3. when a sufficient reduction of the criterion value relative to its value at + the start parameters has been reached, i.e. when + :math:`f(x_k)/f(x_0) \leq \textsf{convergence_ftol_scaled}`. + + 4. when all evaluations on the interpolation points fall within a scaled version + of the noise level of the criterion function. This is only applicable if the + criterion function is noisy. You can specify this criterion with + ``convergence_noise_corrected_criterion_tolerance``. + + DFO-LS supports resetting the optimization and doing a fast start by starting + with a smaller interpolation set and growing it dynamically. For more information + see `the detailed DFO-LS documentation + `_ and :cite:`Cartis2018b`. + + Remember to cite :cite:`Cartis2018b` when using DFO-LS in addition to optimagic. + + .. note:: + We recommend to install DFO-LS version 1.5.3 or higher. Versions 1.5.0 or + lower also work, but the versions 1.5.1 and 1.5.2 contain bugs that can lead + to errors being raised. + + .. note:: + The following arguments of ``dfols.solve`` are not supported by optimagic: + ``scaling_within_bounds``, ``init.run_in_parallel``, ``do_logging``, + ``print_progress`` and all their advanced options. + + """ + clip_criterion_if_overflowing: bool = CLIP_CRITERION_IF_OVERFLOWING + """Whether to clip the criterion if it would raise an ``OverflowError`` + otherwise.""" + convergence_minimal_trustregion_radius_tolerance: NonNegativeFloat = ( CONVERGENCE_MINIMAL_TRUSTREGION_RADIUS_TOLERANCE # noqa: E501 ) + """Stop when the lower trust-region radius falls below this value. + + This is approximately equivalent to an absolute parameter tolerance and + corresponds to ``rhoend`` in the DFO-LS documentation, from which the default + value is taken. + + """ + convergence_noise_corrected_criterion_tolerance: NonNegativeFloat = ( CONVERGENCE_NOISE_CORRECTED_FTOL # noqa: E501 ) + """Stop when the evaluations on the set of interpolation points all fall within + this factor of the noise level. + + The default is 1, i.e. when all evaluations are within the noise level. If you + want to not use this criterion but still flag your criterion function as noisy, + set this tolerance to 0.0. + + .. warning:: + Very small values, as in most other tolerances, don't make sense here. + + """ + convergence_ftol_scaled: NonNegativeFloat = 0.0 + r"""Stop if the criterion value falls below this fraction of its value at the + start parameters, i.e. terminate if + :math:`f(x_k)/f(x_0) \leq \textsf{convergence_ftol_scaled}` is reached. + + This is ``model.rel_tol`` in the DFO-LS documentation. With the default of 0.0 + this criterion is deactivated unless the lowest mathematically possible criterion + value (0.0) is actually achieved. + + """ + convergence_slow_progress: dict[str, Any] | None = None + """Specification of when to terminate (or reset) the optimization because of only + slow improvements. + + This is similar to a relative criterion tolerance, only that instead of a single + improvement the average improvement over several iterations must be small. + Possible entries are: + + - ``threshold_to_characterize_as_slow`` (float): Threshold whether an improvement + is insufficient. Note that the improvement is divided by the + ``comparison_period``, so this is the required average improvement per + iteration over the comparison period. Default is 1e-8. + - ``max_insufficient_improvements`` (int): Number of consecutive insufficient + improvements before termination (or reset). Default is ``20 * len(x)``. + - ``comparison_period`` (int): How many iterations to go back to calculate the + improvement. For example 5 would mean that each criterion evaluation is + compared to the criterion value from 5 iterations before. Default is 5. + + """ + initial_directions: Literal[ "coordinate", "random", ] = "coordinate" + """Whether to draw the initial directions used to build the first interpolation + set as coordinate directions ("coordinate") or random directions ("random").""" + interpolation_rounding_error: float = INTERPOLATION_ROUNDING_ERROR + r"""Scaling factor that controls when the interpolation base point is re-centered + to reduce roundoff errors. + + Internally, all the NAG algorithms store interpolation points with respect to a + base point :math:`x_b`; that is, they store :math:`\{y_t - x_b\}`, which reduces + the risk of roundoff errors. The base point :math:`x_b` is shifted to the current + iterate :math:`x_k` when + :math:`\text{proposed step} \leq \textsf{interpolation_rounding_error} \cdot + \|x_k - x_b\|`. This is ``general.rounding_error_constant`` in the DFO-LS + documentation, from which the default value is taken. + + """ + noise_additive_level: float | None = None + """Amount of additive noise in the criterion function. + + It is used for determining the presence of noise and for the convergence + criterion that all evaluations on the interpolation points are within the noise + level. 0 means no additive noise. Only additive or multiplicative noise can be + specified, not both. + + """ + noise_multiplicative_level: float | None = None + """Amount of multiplicative noise in the criterion function. + + It is used for determining the presence of noise and for the convergence + criterion that all evaluations on the interpolation points are within the noise + level. 0 means no multiplicative noise. Only additive or multiplicative noise can + be specified, not both. + + """ + noise_n_evals_per_point: NonNegativeInt | None = None + r"""How often to evaluate the criterion function at each point. + + This is only applicable for criterion functions with noise, when averaging + multiple evaluations at the same point produces a more accurate value. It must be + a function with the keyword arguments ``upper_trustregion_radius`` + (:math:`\Delta`), ``lower_trustregion_radius`` (:math:`\rho`), ``n_iterations`` + and ``n_resets`` that returns the number of evaluations as an integer. The + default is no averaging, i.e. to evaluate the criterion only once at each point. + + """ + random_directions_orthogonal: bool = RANDOM_DIRECTIONS_ORTHOGONAL + """Whether to make randomly drawn initial directions orthogonal. + + This is only relevant if ``initial_directions`` is "random". + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of criterion evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. + + """ + threshold_for_safety_step: NonNegativeFloat = THRESHOLD_FOR_SAFETY_STEP + r"""Threshold for when to call the safety step (:math:`\gamma_s`). + + A safety step is called when + :math:`\text{proposed step} \leq \textsf{threshold_for_safety_step} \cdot + \rho_k`, where :math:`\rho_k` is the current lower trust-region radius. The + default value is taken from DFO-LS. + + """ + trustregion_expansion_factor_successful: NonNegativeFloat = ( TRUSTREGION_EXPANSION_FACTOR_SUCCESSFUL ) + r"""Ratio by which to expand the upper trust-region radius :math:`\Delta_k` in + very successful iterations (:math:`\gamma_{inc}` in the notation of the paper). + + The default value is taken from DFO-LS. + + """ + trustregion_expansion_factor_very_successful: NonNegativeFloat = ( TRUSTREGION_EXPANSION_FACTOR_VERY_SUCCESSFUL # noqa: E501 ) + r"""Ratio of the proposed step (:math:`\|s_k\|`) by which to expand the upper + trust-region radius (:math:`\Delta_k`) in very successful iterations + (:math:`\overline{\gamma}_{inc}` in the notation of the paper). + + The default value is taken from DFO-LS. + + """ + trustregion_fast_start_options: dict[str, Any] | None = None + r"""Options to start the optimization while building the full trust-region model. + + To activate this, set the number of interpolation points at which to evaluate the + criterion before doing the first step (``min_inital_points``) to something + smaller than the number of parameters. Possible entries are: + + - ``min_inital_points`` (int): Number of initial interpolation points in addition + to the start point. This should only be changed to a value less than + ``len(x)``, and only if the default setup cost of ``len(x) + 1`` evaluations of + the criterion is impractical. If this is set to be less than the default, the + input value of ``trustregion_n_interpolation_points`` should be set to + ``len(x)``. If the default is used, all the other parameters have no effect. + Default is ``trustregion_n_interpolation_points - 1``. If the default setup + costs of the evaluations are very large, DFO-LS can start with less than + ``len(x)`` interpolation points and add points to the trust-region model with + every iteration. Note that the option name is indeed ``min_inital_points``, + i.e. it contains a spelling mistake. + - ``method`` ("jacobian", "trustregion" or "auto"): When there are less + interpolation points than ``len(x)``, the model is underdetermined. This can be + fixed in two ways: If "jacobian", the interpolated Jacobian is perturbed to + have full rank, allowing the trust-region step to include components in the + full search space. This is the default if the number of root contributions is + at least ``len(x)``. If "trustregion", the trust-region step is perturbed by an + orthogonal direction not yet searched. This is the default if the number of + root contributions is smaller than ``len(x)``. + - ``scale_of_trustregion_step_perturbation`` (float): When adding new search + directions, the length of the step is the trust-region radius multiplied by + this value. The default is 0.1 if ``method == "trustregion"`` else 1. + - ``scale_of_jacobian_components_perturbation`` (float): Magnitude of the extra + components added to the Jacobian. The default is 1e-2. + - ``floor_of_jacobian_singular_values`` (float): Floor the singular values of the + Jacobian at this factor of the last non-zero value. As of version 1.2.1 this + option is not yet supported by DFO-LS! + - ``jacobian_max_condition_number`` (float): Cap on the condition number of the + Jacobian after applying floors to singular values (effectively another floor on + the smallest singular value, since the largest singular value is fixed). + - ``geometry_improving_steps`` (bool): Whether to do geometry-improving steps in + the trust-region algorithm, as per the usual algorithm during the fast start. + - ``safety_steps`` (bool): Whether to perform safety steps. + - ``shrink_upper_radius_in_safety_steps`` (bool): During the fast start, whether + to shrink the upper trust-region radius in safety steps. + - ``full_geometry_improving_step`` (bool): During the fast start, whether to do a + full geometry-improving step within safety steps (the same as the post fast + start phase of the algorithm). Since this involves reducing the upper + trust-region radius, this can only be ``True`` if + ``shrink_upper_radius_in_safety_steps`` is ``False``. + - ``reset_trustregion_radius_after_fast_start`` (bool): Whether to reset the + upper trust-region radius to its initial value at the end of the fast start + phase. + - ``reset_min_trustregion_radius_after_fast_start`` (bool): Whether to reset the + minimum trust-region radius (:math:`\rho_k`) to its initial value at the end of + the fast start phase. + - ``shrinking_factor_not_successful`` (float): Ratio by which to shrink the + trust-region radius when the realized improvement does not match the + ``trustregion_threshold_successful`` during the fast start phase. By default it + is the same as ``trustregion_shrinking_factor_not_successful``. + - ``n_extra_search_directions_per_iteration`` (int): Number of new search + directions to add with each iteration where we do not have a full set of search + directions. This approach is not recommended! Default is 0. + + """ + trustregion_initial_radius: NonNegativeFloat | None = None + r"""Initial value of the trust-region radius. + + This is ``rhobeg`` in the DFO-LS documentation. By default it is set to + :math:`0.1 \max(\|x_0\|_{\infty}, 1)`, as in DFO-LS. + + """ + trustregion_method_to_replace_extra_points: ( Literal["geometry_improving", "momentum"] | None ) = "geometry_improving" + """If replacing extra points in successful iterations, whether to use geometry + improving steps ("geometry_improving") or the momentum method ("momentum"). + + This is only relevant if + ``trustregion_n_extra_points_to_replace_successful > 0``. + + """ + trustregion_n_extra_points_to_replace_successful: NonNegativeInt = 0 + """The number of extra points (other than accepting the trust-region step) to + replace in successful iterations. + + This is ``regression.num_extra_steps`` in the DFO-LS documentation. It is useful + when ``trustregion_n_interpolation_points > len(x) + 1``. + + """ + trustregion_n_interpolation_points: NonNegativeInt | None = None + """The number of interpolation points to use. + + This is ``npt`` in the DFO-LS documentation. The default is ``len(x) + 1``, as in + DFO-LS. If using resets, this is the number of points to use in the first run of + the solver, before any resets. + + """ + trustregion_precondition_interpolation: bool = ( TRUSTREGION_PRECONDITION_INTERPOLATION ) + """Whether to scale the interpolation linear system to improve conditioning. + + The default value is taken from DFO-LS. + + """ + trustregion_reset_options: dict[str, Any] | None = None + r"""Options for resetting the optimization. + + Possible entries are: + + - ``use_resets`` (bool): Whether to do resets when the lower trust-region radius + (:math:`\rho_k`) reaches the stopping criterion (:math:`\rho_{end}`), or + (optionally) when all interpolation points are within the noise level. The + default is ``True`` if the criterion is noisy. + - ``minimal_trustregion_radius_tolerance_scaling_at_reset`` (float): Factor with + which the trust-region stopping criterion is multiplied at each reset. + - ``reset_type`` (str): Whether to use "soft" or "hard" resets. The default is + "soft". + - ``move_center_at_soft_reset`` (bool): Whether to move the trust-region center + (:math:`x_k`) to the best new point evaluated instead of keeping it constant. + - ``points_to_replace_at_soft_reset`` (int): Number of interpolation points to + move at each soft reset. + - ``reuse_criterion_value_at_hard_reset`` (bool): Whether or not to recycle the + criterion value at the best iterate found when performing a hard reset. This + saves one criterion evaluation. + - ``max_iterations_without_new_best_after_soft_reset`` (int): The maximum number + of successful steps in a given run where the new criterion value is worse than + the best value found in previous runs before terminating. The default is + ``stopping_maxfun``. + - ``auto_detect`` (bool): Whether or not to automatically determine when to + reset. This is an additional condition and resets can still be triggered by a + small upper trust-region radius, etc. There are two criteria used: upper + trust-region radius shrinkage (no increases over the history, more decreases + than no changes) and changes in the model Jacobian (consistently increasing + trend as measured by the slope and correlation coefficient of the line of best + fit). + - ``auto_detect_history`` (int): How many iterations of model changes and trust + region radii to store. + - ``auto_detect_min_jacobian_increase`` (float): Minimum rate of increase of the + Jacobian over past iterations to cause a reset. + - ``auto_detect_min_correlations`` (float): Minimum correlation of the Jacobian + data set required to cause a reset. + - ``max_consecutive_unsuccessful_resets`` (int): Maximum number of consecutive + unsuccessful resets allowed (i.e. resets which did not outperform the best + known value from earlier runs). + - ``max_interpolation_points`` (int): Maximum allowed value of the number of + interpolation points. This is useful if the number of interpolation points + increases with each reset, e.g. when + ``n_extra_interpolation_points_per_soft_reset > 0``. The default is + ``trustregion_n_interpolation_points``. + - ``n_extra_interpolation_points_per_soft_reset`` (int): Number of points to add + to the interpolation set with each soft reset. + - ``n_extra_interpolation_points_per_hard_reset`` (int): Number of points to add + to the interpolation set with each hard reset. + - ``n_additional_extra_points_to_replace_per_reset`` (int): This parameter + modifies ``trustregion_n_extra_points_to_replace_successful``. With each reset + it is increased by this number. + + """ + trustregion_shrinking_factor_not_successful: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_NOT_SUCCESSFUL # noqa: E501 ) + """Ratio by which to shrink the upper trust-region radius when the realized + improvement does not match the ``trustregion_threshold_successful``. + + This is ``tr_radius.gamma_dec`` in the DFO-LS documentation. The default is 0.98 + if the criterion is noisy and 0.5 else, as in DFO-LS. + + """ + trustregion_shrinking_factor_lower_radius: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_LOWER_RADIUS ) + r"""Ratio by which to shrink the lower trust-region radius (:math:`\rho_k`) + (:math:`\alpha_1` in the notation of the paper). + + The default is 0.9 if the criterion is noisy and 0.1 else, as in DFO-LS. + + """ + trustregion_shrinking_factor_upper_radius: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_UPPER_RADIUS ) + r"""Ratio of the current lower trust-region radius (:math:`\rho_k`) by which to + shrink the upper trust-region radius (:math:`\Delta_k`) when the lower one is + shrunk (:math:`\alpha_2` in the notation of the paper). + + The default is 0.95 if the criterion is noisy and 0.5 else, as in DFO-LS. + + """ + trustregion_threshold_successful: float = TRUSTREGION_THRESHOLD_SUCCESSFUL + """Share of the predicted improvement that has to be achieved for a trust-region + iteration to count as successful. + + This is ``tr_radius.eta1`` in the DFO-LS documentation, from which the default + value is taken. + + """ + trustregion_threshold_very_successful: float = TRUSTREGION_THRESHOLD_VERY_SUCCESSFUL + """Share of the predicted improvement that has to be achieved for a trust-region + iteration to count as very successful. + + This is ``tr_radius.eta2`` in the DFO-LS documentation, from which the default + value is taken. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -650,51 +1058,376 @@ def nag_dfols_internal( ) @dataclass(frozen=True) class NagPyBOBYQA(Algorithm): + """Minimize a scalar function using the BOBYQA algorithm. + + BOBYQA (:cite:`Powell2009`, :cite:`Cartis2018`, :cite:`Cartis2018a`) is a + derivative-free trust-region method. It is designed to solve nonlinear local + minimization problems (with optional bound constraints). + + This wraps Py-BOBYQA, a flexible Python implementation of Powell's BOBYQA (Bound + Optimization BY Quadratic Approximation, :cite:`Powell2009`) developed by the + Numerical Algorithms Group and researchers at the University of Oxford + :cite:`Cartis2018`. In each iteration, the algorithm builds a quadratic model + that interpolates the criterion function at a set of points and minimizes this + model within a trust region. Compared to Powell's original implementation, + Py-BOBYQA adds features for noisy problems (averaging of multiple evaluations + and noise-aware termination), multiple restarts, and a heuristic for escaping + local minima (``seek_global_optimum``, :cite:`Cartis2018a`). + + Remember to cite :cite:`Powell2009` and :cite:`Cartis2018` when using Py-BOBYQA + in addition to optimagic. If you take advantage of the ``seek_global_optimum`` + option, cite :cite:`Cartis2018a` additionally. + + There are two main situations when using a derivative-free algorithm like BOBYQA + is preferable to derivative-based algorithms: + + 1. The criterion function is not deterministic, i.e. if we evaluate the criterion + function multiple times at the same parameter vector we get different results. + + 2. The criterion function is very expensive to evaluate and only finite + differences are available to calculate its derivative. + + Py-BOBYQA was developed by the same group as DFO-LS (``nag_dfols``). If your + criterion function has least-squares structure, use ``nag_dfols`` instead, which + exploits this structure and typically needs far fewer criterion evaluations. + + The detailed documentation of the algorithm can be found `in the Py-BOBYQA + documentation `_. + + There are four possible convergence criteria: + + 1. when the lower trust-region radius is shrunk below a minimum + (``convergence_minimal_trustregion_radius_tolerance``). This is approximately + equivalent to an absolute parameter tolerance. + + 2. when the criterion value falls below an absolute, user-specified value + (``convergence_criterion_value``), the optimization terminates successfully. + + 3. when insufficient improvements have been gained over a certain number of + iterations (``convergence_slow_progress``). The (absolute) threshold for what + constitutes an insufficient improvement, how many iterations have to be + insufficient and with which iteration to compare can all be specified by the + user. + + 4. when all evaluations on the interpolation points fall within a scaled version + of the noise level of the criterion function + (``convergence_noise_corrected_criterion_tolerance``). This is only applicable + if the criterion function is noisy. + + .. note:: + The following arguments of ``pybobyqa.solve`` are not supported by optimagic: + ``scaling_within_bounds``, ``init.run_in_parallel``, ``do_logging``, + ``print_progress`` and all their advanced options. + + """ + clip_criterion_if_overflowing: bool = CLIP_CRITERION_IF_OVERFLOWING + """Whether to clip the criterion if it would raise an ``OverflowError`` + otherwise.""" + convergence_minimal_trustregion_radius_tolerance: NonNegativeFloat = ( CONVERGENCE_MINIMAL_TRUSTREGION_RADIUS_TOLERANCE # noqa: E501 ) + """Stop when the lower trust-region radius falls below this value. + + This is approximately equivalent to an absolute parameter tolerance and + corresponds to ``rhoend`` in the Py-BOBYQA documentation, from which the default + value is taken. + + """ + convergence_noise_corrected_criterion_tolerance: NonNegativeFloat = ( CONVERGENCE_NOISE_CORRECTED_FTOL # noqa: E501 ) + """Stop when the evaluations on the set of interpolation points all fall within + this factor of the noise level. + + The default is 1, i.e. when all evaluations are within the noise level. If you + want to not use this criterion but still flag your criterion function as noisy, + set this tolerance to 0.0. + + .. warning:: + Very small values, as in most other tolerances, don't make sense here. + + """ + convergence_criterion_value: float | None = None + """Terminate successfully if the criterion value falls below this threshold. + + This is deactivated (i.e. set to -inf) by default. It is ``model.abs_tol`` in the + Py-BOBYQA documentation. + + """ + convergence_slow_progress: dict[str, Any] | None = None + """Specification of when to terminate (or reset) the optimization because of only + slow improvements. + + This is similar to a relative criterion tolerance, only that instead of a single + improvement the average improvement over several iterations must be small. + Possible entries are: + + - ``threshold_to_characterize_as_slow`` (float): Threshold whether an improvement + is insufficient. Note that the improvement is divided by the + ``comparison_period``, so this is the required average improvement per + iteration over the comparison period. Default is 1e-8. + - ``max_insufficient_improvements`` (int): Number of consecutive insufficient + improvements before termination (or reset). Default is ``20 * len(x)``. + - ``comparison_period`` (int): How many iterations to go back to calculate the + improvement. For example 5 would mean that each criterion evaluation is + compared to the criterion value from 5 iterations before. Default is 5. + + """ + initial_directions: Literal[ "coordinate", "random", ] = "coordinate" + """Whether to draw the initial directions used to build the first interpolation + set as coordinate directions ("coordinate") or random directions ("random").""" + interpolation_rounding_error: float = INTERPOLATION_ROUNDING_ERROR + r"""Scaling factor that controls when the interpolation base point is re-centered + to reduce roundoff errors. + + Internally, all the NAG algorithms store interpolation points with respect to a + base point :math:`x_b`; that is, they store :math:`\{y_t - x_b\}`, which reduces + the risk of roundoff errors. The base point :math:`x_b` is shifted to the current + iterate :math:`x_k` when + :math:`\text{proposed step} \leq \textsf{interpolation_rounding_error} \cdot + \|x_k - x_b\|`. This is ``general.rounding_error_constant`` in the Py-BOBYQA + documentation, from which the default value is taken. + + """ + noise_additive_level: float | None = None + """Amount of additive noise in the criterion function. + + It is used for determining the presence of noise and for the convergence + criterion that all evaluations on the interpolation points are within the noise + level. 0 means no additive noise. Only additive or multiplicative noise can be + specified, not both. + + """ + noise_multiplicative_level: float | None = None + """Amount of multiplicative noise in the criterion function. + + It is used for determining the presence of noise and for the convergence + criterion that all evaluations on the interpolation points are within the noise + level. 0 means no multiplicative noise. Only additive or multiplicative noise can + be specified, not both. + + """ + noise_n_evals_per_point: NonNegativeInt | None = None + r"""How often to evaluate the criterion function at each point. + + This is only applicable for criterion functions with noise, when averaging + multiple evaluations at the same point produces a more accurate value. It must be + a function with the keyword arguments ``upper_trustregion_radius`` + (:math:`\Delta`), ``lower_trustregion_radius`` (:math:`\rho`), ``n_iterations`` + and ``n_resets`` that returns the number of evaluations as an integer. The + default is no averaging, i.e. to evaluate the criterion only once at each point. + + """ + random_directions_orthogonal: bool = RANDOM_DIRECTIONS_ORTHOGONAL + """Whether to make randomly drawn initial directions orthogonal. + + This is only relevant if ``initial_directions`` is "random". + + """ + seek_global_optimum: bool = False + """Whether to apply the heuristic to escape local minima presented in + :cite:`Cartis2018a`. + + The heuristic repeatedly restarts the optimization from the best point found so + far with an enlarged trust-region radius. It is only a heuristic, so there is no + guarantee that the global optimum is found. To use it, finite lower and upper + bounds must be provided for all parameters. + + """ + stopping_max_criterion_evaluations: PositiveInt = STOPPING_MAXFUN + """Maximum number of criterion evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. + + """ + threshold_for_safety_step: NonNegativeFloat = THRESHOLD_FOR_SAFETY_STEP + r"""Threshold for when to call the safety step (:math:`\gamma_s`). + + A safety step is called when + :math:`\text{proposed step} \leq \textsf{threshold_for_safety_step} \cdot + \rho_k`, where :math:`\rho_k` is the current lower trust-region radius. The + default value is taken from Py-BOBYQA. + + """ + trustregion_expansion_factor_successful: NonNegativeFloat = ( TRUSTREGION_EXPANSION_FACTOR_SUCCESSFUL ) + r"""Ratio by which to expand the upper trust-region radius :math:`\Delta_k` in + very successful iterations (:math:`\gamma_{inc}` in the notation of the paper). + + The default value is taken from Py-BOBYQA. + + """ + trustregion_expansion_factor_very_successful: NonNegativeFloat = ( TRUSTREGION_EXPANSION_FACTOR_VERY_SUCCESSFUL # noqa: E501 ) + r"""Ratio of the proposed step (:math:`\|s_k\|`) by which to expand the upper + trust-region radius (:math:`\Delta_k`) in very successful iterations + (:math:`\overline{\gamma}_{inc}` in the notation of the paper). + + The default value is taken from Py-BOBYQA. + + """ + trustregion_initial_radius: NonNegativeFloat | None = None + r"""Initial value of the trust-region radius. + + This is ``rhobeg`` in the Py-BOBYQA documentation. By default it is set to + :math:`0.1 \max(\|x_0\|_{\infty}, 1)`, as in Py-BOBYQA. + + """ + trustregion_minimum_change_hession_for_underdetermined_interpolation: bool = True + """Whether to solve the underdetermined quadratic interpolation problem by + minimizing the Frobenius norm of the change in the Hessian. + + If True (the default, as in Py-BOBYQA and Powell's original BOBYQA), the + quadratic model is chosen such that the Frobenius norm of the change in its + Hessian relative to the previous iteration is minimal. If False, the Frobenius + norm of the Hessian itself is minimized. This is + ``interpolation.minimum_change_hessian`` in the Py-BOBYQA documentation. + + """ + trustregion_n_interpolation_points: NonNegativeInt | None = None + r"""The number of interpolation points to use. + + This is ``npt`` in the Py-BOBYQA documentation. With :math:`n = len(x)` the + default is :math:`2n+1` if the criterion is not noisy. Otherwise, it is set to + :math:`(n+1)(n+2)/2`. Larger values are particularly useful for noisy problems. + Py-BOBYQA requires + + .. math:: + n + 1 \leq \textsf{trustregion_n_interpolation_points} \leq (n+1)(n+2)/2. + + """ + trustregion_precondition_interpolation: bool = ( TRUSTREGION_PRECONDITION_INTERPOLATION ) + """Whether to scale the interpolation linear system to improve conditioning. + + The default value is taken from Py-BOBYQA. + + """ + trustregion_reset_options: dict[str, Any] | None = None + r"""Options for resetting the optimization. + + Possible entries are: + + - ``use_resets`` (bool): Whether to do resets when the lower trust-region radius + (:math:`\rho_k`) reaches the stopping criterion (:math:`\rho_{end}`), or + (optionally) when all interpolation points are within the noise level. The + default is ``True`` if the criterion is noisy. + - ``minimal_trustregion_radius_tolerance_scaling_at_reset`` (float): Factor with + which the trust-region stopping criterion is multiplied at each reset. + - ``reset_type`` (str): Whether to use "soft" or "hard" resets. The default is + "soft". + - ``move_center_at_soft_reset`` (bool): Whether to move the trust-region center + (:math:`x_k`) to the best new point evaluated instead of keeping it constant. + - ``points_to_replace_at_soft_reset`` (int): Number of interpolation points to + move at each soft reset. + - ``reuse_criterion_value_at_hard_reset`` (bool): Whether or not to recycle the + criterion value at the best iterate found when performing a hard reset. This + saves one criterion evaluation. + - ``max_iterations_without_new_best_after_soft_reset`` (int): The maximum number + of successful steps in a given run where the new criterion value is worse than + the best value found in previous runs before terminating. The default is + ``stopping_max_criterion_evaluations``. + - ``auto_detect`` (bool): Whether or not to automatically determine when to + reset. This is an additional condition and resets can still be triggered by a + small upper trust-region radius, etc. There are two criteria used: upper + trust-region radius shrinkage (no increases over the history, more decreases + than no changes) and changes in the model Jacobian (consistently increasing + trend as measured by the slope and correlation coefficient of the line of best + fit). + - ``auto_detect_history`` (int): How many iterations of model changes and trust + region radii to store. + - ``auto_detect_min_jacobian_increase`` (float): Minimum rate of increase of the + Jacobian over past iterations to cause a reset. + - ``auto_detect_min_correlations`` (float): Minimum correlation of the Jacobian + data set required to cause a reset. + - ``max_consecutive_unsuccessful_resets`` (int): Maximum number of consecutive + unsuccessful resets allowed (i.e. resets which did not outperform the best + known value from earlier runs). + - ``max_unsuccessful_resets`` (int): Number of total unsuccessful resets allowed. + The default is 20 if ``seek_global_optimum`` and else unrestricted. + - ``trust_region_scaling_at_unsuccessful_reset`` (float): Factor by which to + expand the initial lower trust-region radius (:math:`\rho_{beg}`) after + unsuccessful resets. The default is 1.1 if ``seek_global_optimum`` else 1. + + """ + trustregion_shrinking_factor_not_successful: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_NOT_SUCCESSFUL # noqa: E501 ) + """Ratio by which to shrink the upper trust-region radius when the realized + improvement does not match the ``trustregion_threshold_successful``. + + This is ``tr_radius.gamma_dec`` in the Py-BOBYQA documentation. The default is + 0.98 if the criterion is noisy and 0.5 else, as in Py-BOBYQA. + + """ + trustregion_shrinking_factor_lower_radius: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_LOWER_RADIUS ) + r"""Ratio by which to shrink the lower trust-region radius (:math:`\rho_k`) + (:math:`\alpha_1` in the notation of the paper). + + The default is 0.9 if the criterion is noisy and 0.1 else, as in Py-BOBYQA. + + """ + trustregion_shrinking_factor_upper_radius: NonNegativeFloat | None = ( TRUSTREGION_SHRINKING_FACTOR_UPPER_RADIUS ) + r"""Ratio of the current lower trust-region radius (:math:`\rho_k`) by which to + shrink the upper trust-region radius (:math:`\Delta_k`) when the lower one is + shrunk (:math:`\alpha_2` in the notation of the paper). + + The default is 0.95 if the criterion is noisy and 0.5 else, as in Py-BOBYQA. + + """ + trustregion_threshold_successful: float = TRUSTREGION_THRESHOLD_SUCCESSFUL + """Share of the predicted improvement that has to be achieved for a trust-region + iteration to count as successful. + + This is ``tr_radius.eta1`` in the Py-BOBYQA documentation, from which the default + value is taken. + + """ + trustregion_threshold_very_successful: float = TRUSTREGION_THRESHOLD_VERY_SUCCESSFUL + """Share of the predicted improvement that has to be achieved for a trust-region + iteration to count as very successful. + + This is ``tr_radius.eta2`` in the Py-BOBYQA documentation, from which the default + value is taken. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/neldermead.py b/src/optimagic/optimizers/neldermead.py index ab5ddce84..3a69fc251 100644 --- a/src/optimagic/optimizers/neldermead.py +++ b/src/optimagic/optimizers/neldermead.py @@ -1,5 +1,7 @@ """Implementation of parallelosation of Nelder-Mead algorithm.""" +from __future__ import annotations + from dataclasses import dataclass from typing import Callable, Literal, cast @@ -41,54 +43,88 @@ ) @dataclass(frozen=True) class NelderMeadParallel(Algorithm): - r"""Parallel Nelder-Mead algorithm following Lee D., Wiswall M., A parallel - implementation of the simplex function minimization routine, Computational - Economics, 2007. + """Minimize a scalar function using a parallel Nelder-Mead algorithm. - Parameters - ---------- - criterion (callable): A function that takes a Numpy array_like as - an argument and return scalar floating point. + This is a parallel Nelder-Mead algorithm following :cite:`LeeWiswall2007`, + which generalizes the classic simplex algorithm of Nelder and Mead + (:cite:`Nelder1965`) to parallel processors. In each iteration, instead of + reflecting only the worst vertex of the simplex, the p worst vertices are + reflected (and possibly expanded or contracted) simultaneously, so that up to + p criterion evaluations can be carried out in parallel. With ``n_cores=1``, + the algorithm behaves like a standard sequential Nelder-Mead algorithm. - x (array_like): 1-D array of initial value of parameters + As a direct search method, it does not require derivatives of the objective + function and can handle a moderate amount of noise or non-smoothness. It is a + local optimizer for scalar objective functions and does not support bounds or + constraints. Parallelization is most beneficial when a single criterion + evaluation is expensive and the number of parameters is large relative to the + number of available cores. - init_simplex_method (string or callable): Name of the method to create initial - simplex or callable which takes as an argument initial value of parameters - and returns initial simplex as j+1 x j array, where j is length of x. - The default is "gao_han". + The algorithm was implemented in optimagic by Jacek Barszczewski. - n_cores (int): Degrees of parallization. The default is 1 (no parallelization). + .. note:: + This is a pure-Python implementation within optimagic. It is currently + considered experimental. - adaptive (bool): Adjust parameters of Nelder-Mead algorithm to accounf - for simplex size. - The default is True. + """ - stopping_maxiter (int): Maximum number of algorithm iterations. - The default is STOPPING_MAX_ITERATIONS. + init_simplex_method: InitSimplexLiteral | InitSimplexCallable = "gao_han" + """Method to create the initial simplex. - convergence_ftol_abs (float): maximal difference between - function value evaluated on simplex points. + Either the name of one of the implemented methods, i.e. "pfeffer" (due to L. + Pfeffer at Stanford; the default in MATLAB's fminsearch and SciPy), "nash" + (:cite:`Nash1990`; the default in R's optim), "gao_han" (:cite:`Gao2012`) or + "varadhan_borchers" (:cite:`Varadhan2016`), or a callable that takes the + initial parameter vector as argument and returns the initial simplex as an + array of shape (j + 1, j), where j is the number of parameters. The default is + "gao_han". See :cite:`Wessing2019` for a discussion of the importance of the + choice of the initial simplex. - convergence_xtol_abs (float): maximal distance between points in - the simplex. + """ - batch_evaluator (string or callable): See :ref:`batch_evaluators` for - details. Default "joblib". + n_cores: PositiveInt = 1 + """Degree of parallelization. - Returns: - ------- - TYPE - DESCRIPTION. + The default is 1 (no parallelization). Values larger than or equal to the + number of parameters are reduced to the number of parameters minus 1. """ - init_simplex_method: InitSimplexLiteral | InitSimplexCallable = "gao_han" - n_cores: PositiveInt = 1 adaptive: bool = True + """Adjust the parameters of the Nelder-Mead algorithm to the dimension of the + problem, following :cite:`Gao2012`. + + The default is True. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """Maximum number of algorithm iterations.""" + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_FTOL_ABS + """Maximal absolute difference between the function value at the best vertex of + the simplex and the function values at the remaining vertices. + + Convergence requires that this criterion and ``convergence_xtol_abs`` are + satisfied at the same time. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_XTOL_ABS + """Maximal absolute distance between the best vertex of the simplex and the + remaining vertices. + + Convergence requires that this criterion and ``convergence_ftol_abs`` are + satisfied at the same time. + + """ + batch_evaluator: BatchEvaluator | BatchEvaluatorLiteral = "joblib" + """Batch evaluator used for parallel function evaluations. + + See :ref:`batch_evaluators` for details. The default is "joblib". + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/nlopt_optimizers.py b/src/optimagic/optimizers/nlopt_optimizers.py index ce716fb0f..99aa60e49 100644 --- a/src/optimagic/optimizers/nlopt_optimizers.py +++ b/src/optimagic/optimizers/nlopt_optimizers.py @@ -1,9 +1,12 @@ """Implement `nlopt` algorithms. -The documentation is heavily based on (nlopt documentation)[nlopt.readthedocs.io]. +The documentation is heavily based on the documentation of the `NLopt library +`_. """ +from __future__ import annotations + from dataclasses import dataclass import numpy as np @@ -53,11 +56,75 @@ ) @dataclass(frozen=True) class NloptBOBYQA(Algorithm): + """Minimize a scalar function using the BOBYQA algorithm. + + BOBYQA (Bound Optimization BY Quadratic Approximation) is a derivative-free local + optimizer for bound-constrained problems by M. J. D. Powell :cite:`Powell2009`. + It iteratively constructs a quadratic approximation of the objective function + from function values only and minimizes this model in a trust region. + + The algorithm is a good choice for smooth scalar problems with a moderate number + of parameters when derivatives are not available. Because it relies on a + quadratic model, it may perform poorly for objective functions that are not + twice-differentiable. + + BOBYQA is the successor of Powell's NEWUOA algorithm (``nlopt_newuoa``) and + largely supersedes it; in contrast to NEWUOA, it supports bound constraints + natively. For noisy or very non-smooth problems, simplex-based methods such as + ``nlopt_neldermead`` or ``nlopt_sbplx`` can be more robust. + + The implementation in NLopt :cite:`Johnson2007` is a C translation of Powell's + original Fortran subroutine. In addition, it supports all NLopt stopping + criteria and unequal initial step sizes in the different parameters, achieved + by internally rescaling the parameters proportional to the initial steps. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -94,10 +161,60 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptNelderMead(Algorithm): + """Minimize a scalar function using the Nelder-Mead simplex algorithm. + + The Nelder-Mead algorithm :cite:`Nelder1965` is a derivative-free local + optimizer that maintains a simplex of n+1 points in n dimensions. In each + iteration, the simplex is updated by reflecting, expanding or contracting it + away from the worst point. + + The algorithm is a slow but robust workhorse that only requires function + values. It can make progress on noisy or non-smooth problems where model-based + methods such as ``nlopt_bobyqa`` fail, but it typically needs many function + evaluations. The related ``nlopt_sbplx`` algorithm applies Nelder-Mead on a + sequence of subspaces and is claimed to be more efficient and robust. + + The implementation in NLopt :cite:`Johnson2007` differs from the original + algorithm in that it explicitly supports bound constraints: whenever a new + point would lie outside the bounds, it is moved exactly onto the boundary. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -133,11 +250,79 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptPRAXIS(Algorithm): + """Minimize a scalar function using the principal-axis method. + + PRAXIS is a derivative-free local optimizer by Richard Brent :cite:`Brent1972`. + It assumes an approximately quadratic form of the objective function and + repeatedly updates a set of conjugate search directions (the principal axes) + along which one-dimensional minimizations are performed. + + The algorithm is not invariant to scaling of the objective function and may + fail under certain rank-preserving transformations of the objective (e.g. + transformations that lead to a non-quadratic shape of the objective function). + + The algorithm is not deterministic (it uses random perturbations to avoid + stagnation) and determinism cannot be achieved via seed setting. + + The implementation in NLopt :cite:`Johnson2007` is based on a C translation of + Brent's original Fortran code. The original algorithm was designed for + unconstrained optimization; NLopt only emulates bound constraints, at the cost + of a significantly reduced speed of convergence. For bound-constrained + problems, this method is dominated by algorithms with native bound support + such as ``nlopt_bobyqa`` and ``nlopt_cobyla``. + + .. warning:: + The NLopt implementation failed on a simple benchmark function with finite + parameter bounds. Passing bounds has therefore been disabled for this + algorithm in optimagic. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -174,11 +359,87 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptCOBYLA(Algorithm): + """Minimize a scalar function using the COBYLA algorithm. + + COBYLA (Constrained Optimization BY Linear Approximations) is a derivative-free + local optimizer by M. J. D. Powell that supports nonlinear inequality and + equality constraints :cite:`Powell1994`. + + It constructs successive linear approximations of the objective function and + constraints via a simplex of n+1 points (in n dimensions) and optimizes these + approximations in a trust region at each step. + + Use it for scalar problems without derivatives, in particular when nonlinear + constraints are present: it is the only derivative-free NLopt optimizer in + optimagic that supports them. For unconstrained or bound-constrained smooth + problems, ``nlopt_bobyqa`` usually converges faster because it uses quadratic + instead of linear approximations. + + The implementation in NLopt :cite:`Johnson2007` is based on a C translation of + Powell's original Fortran code and differs from the original implementation in + a few ways: + + - It incorporates all of the NLopt termination criteria. + - It adds explicit support for bound constraints. + - It allows the trust-region radius to increase if the predicted improvement + was approximately right and the simplex is satisfactory, which can improve + convergence speed. + - It pseudo-randomizes the simplex steps, improving robustness by avoiding + accidentally taking steps that do not improve conditioning, while preserving + the deterministic nature of the algorithm (a deterministic seed is used). + - It supports unequal initial step sizes in the different parameters. + + Since the underlying code only supports inequality constraints, equality + constraints are automatically transformed into pairs of inequality constraints + by NLopt. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -215,11 +476,67 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptSbplx(Algorithm): + """Minimize a scalar function using the Subplex algorithm. + + Sbplx is a re-implementation of Tom Rowan's Subplex method :cite:`Rowan1990`. + Subplex is a variant of the Nelder-Mead algorithm (``nlopt_neldermead``) that + applies Nelder-Mead on a sequence of subspaces. It is claimed to be more + efficient and robust than the original Nelder-Mead algorithm, while retaining + its ability to make progress on noisy and non-smooth objective functions. + + The implementation in NLopt :cite:`Johnson2007` was rewritten from scratch + because Rowan's original code could not be relicensed. It differs from the + original algorithm in that it explicitly supports bound constraints, which + gives a big improvement in the case where the optimum lies against one of the + bounds. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -256,11 +573,75 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptNEWUOA(Algorithm): + """Minimize a scalar function using the NEWUOA algorithm. + + NEWUOA is a derivative-free local optimizer for unconstrained optimization by + M. J. D. Powell :cite:`Powell2004`. Like BOBYQA, it iteratively constructs a + quadratic approximation of the objective function from function values only + and minimizes this model in a trust region. + + Use it for smooth scalar problems when derivatives are not available. NEWUOA + is largely superseded by its successor BOBYQA (``nlopt_bobyqa``), which + supports bound constraints natively and is usually preferable, in particular + for bound-constrained problems. + + The version in NLopt :cite:`Johnson2007` has been extended to support bound + constraints: if no finite bounds are specified, optimagic calls the original + ``LN_NEWUOA`` algorithm for unconstrained optimization; otherwise, the + ``LN_NEWUOA_BOUND`` variant for bound-constrained problems is used. + + .. note:: + NEWUOA requires the dimension n of the parameter space to be at least 2, + i.e. the implementation does not handle one-dimensional optimization + problems. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -305,11 +686,70 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptTNewton(Algorithm): + """Minimize a scalar function using a truncated Newton method. + + Truncated Newton methods are designed for large-scale smooth optimization + problems. Instead of solving the Newton equations exactly, which would require + the Hessian matrix, they solve them approximately (hence "truncated" or + "inexact") with a conjugate-gradient iteration that only requires gradient + information. A detailed description is given in :cite:`Dembo1983`. + + Use it for differentiable scalar problems, in particular with many parameters. + The gradient of the objective function is required. + + The implementation in NLopt :cite:`Johnson2007` is based on a Fortran + implementation written by Prof. Ladislav Luksan. NLopt also provides variants + of the method that add preconditioning via a low-storage BFGS approximation + and steepest-descent restarting; optimagic wraps the plain variant + (``LD_TNEWTON``) without preconditioning or restarting. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -346,11 +786,72 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptLBFGSB(Algorithm): + """Minimize a scalar function using the low-storage BFGS algorithm. + + L-BFGS is a limited-memory quasi-Newton method. It approximates the inverse + Hessian of the objective function from a limited number of stored past gradient + and parameter updates and uses this approximation to compute search directions + for a line search. Detailed descriptions of the algorithm are given in + :cite:`Nocedal1980` and :cite:`Nocedal1989`. + + Use it for smooth, differentiable scalar problems, in particular with many + parameters: due to the limited-memory Hessian approximation, memory use grows + only linearly in the number of parameters. The gradient of the objective + function is required. + + The implementation in NLopt :cite:`Johnson2007` is based on a Fortran + implementation of the low-storage BFGS algorithm written by Prof. Ladislav + Luksan. It is a different implementation than the one wrapped by + ``scipy_lbfgsb``, which is based on the original Fortran code by Nocedal and + coworkers. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -387,11 +888,76 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptCCSAQ(Algorithm): + """Minimize a scalar function using the CCSAQ algorithm. + + CCSAQ uses the quadratic variant of the conservative convex separable + approximation (CCSA) framework described in :cite:`Svanberg2002`. It is a + gradient-based local optimizer: at each candidate point, a separable quadratic + approximation of the objective function is constructed from the gradient, plus + a penalty term that renders the approximation convex and conservative. The + approximation is then minimized to obtain the next candidate point. + + The algorithm is "globally convergent" in the sense that it is guaranteed to + converge to a local optimum from any feasible starting point; it does not find + the global optimum. + + CCSAQ is closely related to ``nlopt_mma``, which belongs to the same CCSA + family but approximates the objective with moving asymptotes instead of a + quadratic model. + + .. note:: + The underlying NLopt implementation :cite:`Johnson2007` supports nonlinear + inequality constraints, but nonlinear constraints are not supported for + this algorithm in optimagic. Use ``nlopt_mma`` or ``nlopt_slsqp`` for + nonlinearly constrained problems. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -428,11 +994,74 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptMMA(Algorithm): + """Minimize a scalar function using the method of moving asymptotes (MMA). + + MMA is a gradient-based local optimizer that was originally developed for + structural optimization :cite:`Svanberg1987`. The implementation in NLopt + :cite:`Johnson2007` is based on the globally convergent variant of the + algorithm described in :cite:`Svanberg2002`. + + At each candidate point, an approximation of the objective function and + constraints is constructed from the gradient and so-called moving asymptotes + that are updated across iterations. A penalty term renders the approximation + convex and conservative. The resulting convex separable subproblem is solved + to obtain the next candidate point. The algorithm is "globally convergent" in + the sense that it is guaranteed to converge to a local optimum from any + feasible starting point; it does not find the global optimum. + + Use it for smooth scalar problems with bound and nonlinear constraints when + gradients are available. The underlying implementation supports nonlinear + inequality constraints, but no equality constraints; optimagic automatically + converts equality constraints into pairs of inequality constraints, so both + types can be specified. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -474,12 +1103,76 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptVAR(Algorithm): + """Minimize a scalar function using a limited-memory variable-metric method. + + The algorithm is a shifted limited-memory variable-metric method by Vlcek and + Luksan; a detailed description, including the rank-1 and rank-2 variants, is + given in :cite:`Vlcek2006`. Like L-BFGS, it approximates the inverse Hessian + from a limited number of stored past updates of the gradient. The larger the + number of stored updates, the more memory is consumed. + + Use it for smooth, differentiable scalar problems, in particular with many + parameters. The gradient of the objective function is required. Depending on + the ``rank_1_update`` option, either a rank-1 (``LD_VAR1``) or a rank-2 + (``LD_VAR2``) update formula for the variable-metric approximation is used. + + The implementation in NLopt :cite:`Johnson2007` is based on a Fortran + implementation written by Prof. Ladislav Luksan. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ + rank_1_update: bool = True + """Whether the rank-1 update method (``LD_VAR1``) is used. + + If False, the rank-2 update method (``LD_VAR2``) is used instead. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -520,11 +1213,71 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptSLSQP(Algorithm): + """Minimize a scalar function using the SLSQP algorithm. + + SLSQP (Sequential Least Squares Quadratic Programming) is a gradient-based + local optimizer for nonlinearly constrained problems. It supports bound + constraints as well as nonlinear equality and inequality constraints. The + algorithm is a sequential quadratic programming (SQP) method that treats the + optimization problem as a sequence of constrained least-squares problems. + Despite its name, it is a general nonlinear optimizer for scalar objectives, + not a least-squares solver. + + The implementation in NLopt :cite:`Johnson2007` is based on the procedure + described in :cite:`Kraft1988` and :cite:`Kraft1994`. + + .. note:: + Because the underlying code uses dense-matrix methods, it requires O(n^2) + storage and O(n^3) time in n dimensions, which makes it less practical for + problems with more than a few thousand parameters. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -561,14 +1314,111 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptDirect(Algorithm): + """Minimize a scalar function using the DIRECT algorithm. + + DIRECT (DIviding RECTangles) is a deterministic derivative-free global + optimizer :cite:`Jones1993`. It systematically divides the bounded search + domain into smaller and smaller hyperrectangles and evaluates the objective + function at their centers, balancing global exploration and local refinement. + + Use it for global optimization of scalar problems with a small number of + parameters when no derivatives are available. Finite bounds on all parameters + are required. The basic algorithm is deterministic, so results are + reproducible without setting a seed. + + The implementation in NLopt :cite:`Johnson2007` provides several variants that + are selected via the ``locally_biased``, ``random_search`` and + ``unscaled_bounds`` options: + + - The locally biased variant DIRECT_L :cite:`Gablonsky2001` is more biased + towards local search and tends to be more efficient for objective functions + without too many local minima. + - The randomized variants (only available for the locally biased version) use + some randomization to decide which dimension to halve next in the case of + near-ties. + - By default, NLopt rescales the bounds to a unit hypercube. The unscaled + ("NOSCAL") variants skip this rescaling, which can be better if the + objective function varies on very different scales across dimensions. + + The available combinations are DIRECT, DIRECT_L, DIRECT_L_NOSCAL, + DIRECT_L_RAND, DIRECT_L_RAND_NOSCAL and DIRECT_NOSCAL. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. The + optimagic default for global optimizers is much lower than for local + optimizers to keep runtimes manageable. + + """ + locally_biased: bool = False + """Whether the locally biased variant DIRECT_L of the algorithm is used. + + The locally biased variant tends to be more efficient for objective functions + without too many local minima. See :cite:`Gablonsky2001` for details. + + """ + random_search: bool = False + """Whether the randomized variant of the locally biased algorithm is used. + + The randomized variant uses some randomization to decide which dimension to + halve next in the case of near-ties. It is only available for the locally + biased version of the algorithm, i.e. this option requires + ``locally_biased=True``. + + """ + unscaled_bounds: bool = False + """Whether the "NOSCAL" variant of the algorithm is used, which does not + rescale the bounds to a unit hypercube. + + Skipping the rescaling can be beneficial if the objective function varies on + very different scales across dimensions. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -624,11 +1474,70 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptESCH(Algorithm): + """Minimize a scalar function using the ESCH algorithm. + + ESCH is an evolutionary algorithm for global optimization developed by Carlos + Henrique da Silva Santos :cite:`DaSilva2010`, :cite:`DaSilva2010a`. It is a + modified evolution strategy :cite:`Beyer2002` that evolves a population of + candidate points via mutation and selection; see also :cite:`Vent1975`. + + Use it for global optimization of scalar problems when no derivatives are + available. Finite bounds on all parameters are required. The algorithm + supports bound constraints only; nonlinear constraints are not supported. + + Like most evolutionary algorithms, it typically needs many function + evaluations, and there is no guarantee that the global optimum is found within + a finite budget. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. The + optimagic default for global optimizers is much lower than for local + optimizers to keep runtimes manageable. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -665,11 +1574,74 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptISRES(Algorithm): + """Minimize a scalar function using the ISRES algorithm. + + ISRES (Improved Stochastic Ranking Evolution Strategy) is an evolutionary + algorithm for nonlinearly constrained global optimization + :cite:`PhilipRunarsson2005`. Its evolution strategy combines a mutation rule + (with a log-normal step-size update and exponential smoothing) with + differential variation (a Nelder-Mead-like update rule). For constrained + problems, individuals are ranked via the stochastic ranking procedure + described in :cite:`Thomas2000`. + + Use it for global optimization of scalar problems when no derivatives are + available; it is one of the few global optimizers that supports nonlinear + equality and inequality constraints. Finite bounds on all parameters are + required. The algorithm has heuristics to escape local optima, but no proof + of global convergence is available. + + In the implementation in NLopt :cite:`Johnson2007`, the population size + defaults to 20 * (number of parameters + 1). + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. The + optimagic default for global optimizers is much lower than for local + optimizers to keep runtimes manageable. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -706,12 +1678,78 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class NloptCRS2LM(Algorithm): + """Minimize a scalar function using the CRS2_LM algorithm. + + CRS2 (Controlled Random Search 2) with local mutation is a derivative-free + global optimizer. The CRS class of algorithms, originally described in + :cite:`Price1978` and :cite:`Price1983`, starts with a random population of + points and evolves these points by heuristic rules; the evolution somewhat + resembles a randomized Nelder-Mead algorithm. The implementation in NLopt + :cite:`Johnson2007` uses the CRS2 variant with the local mutation modification + described in :cite:`Kaelo2006`. + + Use it for global optimization of scalar problems when no derivatives are + available. Finite bounds on all parameters are required. Like all stochastic + global optimizers, it typically needs many function evaluations and offers no + guarantee of finding the global optimum. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this value multiplied by the absolute value of the + parameter. + + This corresponds to NLopt's ``xtol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes every + parameter by less than this absolute value. + + This corresponds to NLopt's ``xtol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this value multiplied by the absolute + value of the function value. + + This corresponds to NLopt's ``ftol_rel`` stopping criterion. A value of 0 means + that this criterion is disabled. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Stop when an optimization step (or an estimate of the optimum) changes the + objective function value by less than this absolute value. + + This corresponds to NLopt's ``ftol_abs`` stopping criterion. A value of 0 means + that this criterion is disabled, which is the optimagic default. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of objective function evaluations. + + If reached, the optimization stops, but this is not counted as successful + convergence. This corresponds to NLopt's ``maxeval`` stopping criterion. The + optimagic default for global optimizers is much lower than for local + optimizers to keep runtimes manageable. + + """ + population_size: PositiveInt | None = None + """Size of the random population of points that is evolved by the algorithm. + + If None, the population size is set to 10 * (number of parameters + 1), which + is the default of the NLopt implementation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/pounders.py b/src/optimagic/optimizers/pounders.py index 2fa2650b2..412460924 100644 --- a/src/optimagic/optimizers/pounders.py +++ b/src/optimagic/optimizers/pounders.py @@ -1,5 +1,7 @@ """Implement the POUNDERS algorithm.""" +from __future__ import annotations + import warnings from dataclasses import dataclass from typing import Any, Literal @@ -55,30 +57,179 @@ ) @dataclass(frozen=True) class Pounders(Algorithm): + r"""Minimize a nonlinear least-squares problem using the POUNDERS algorithm. + + POUNDERS (:cite:`Wild2015`, :cite:`Benson2017`) is a derivative-free + trust-region algorithm that is tailored to nonlinear least-squares problems. + This is a pure-Python implementation of the POUNDERS algorithm from the + Toolkit for Advanced Optimization (TAO), which is part of PETSc. In contrast + to the wrapped TAO version (``tao_pounders``), it runs on all platforms and + supports parallel function evaluations. + + POUNDERS can be a useful tool for economists who estimate structural models + using indirect inference, because unlike commonly used algorithms such as + Nelder-Mead, it exploits the least-squares structure of the objective + function by building and maintaining interpolation-based quadratic models of + the residuals. It may therefore require fewer criterion evaluations to arrive + at a local optimum than general-purpose derivative-free optimizers. + + Use pounders for smooth least-squares problems with a moderate number of + parameters when derivatives are not available. The objective function must be + a least-squares function, i.e. a function that is decorated with + ``om.mark.least_squares`` and returns the residuals rather than the sum of + their squares. Bound constraints are supported; other constraint types are + not. + + .. note:: + Scaling the problem is necessary such that bounds correspond to the unit + hypercube :math:`[0, 1]^n`. For unconstrained problems, scale each + parameter such that unit changes in parameters result in similar + order-of-magnitude changes in the criterion value(s). + + """ + convergence_gtol_abs: NonNegativeFloat = 1e-8 + """Convergence tolerance for the absolute gradient norm. + + Stop if the norm of the gradient is less than this value. + + """ + convergence_gtol_rel: NonNegativeFloat = 1e-8 + """Convergence tolerance for the relative gradient norm. + + Stop if the norm of the gradient relative to the criterion value is less than + this value. + + """ + # TODO: Why can this a bool convergence_gtol_scaled: NonNegativeFloat | bool = False + """Convergence tolerance for the scaled gradient norm. + + Stop if the norm of the gradient divided by the norm of the gradient at the + initial parameters is less than this value. Disabled, i.e. set to False, by + default. + + """ + max_interpolation_points: PositiveInt | None = None + """Maximum number of interpolation points. + + By default, 2 * n + 1 points are used, where n is the number of parameters. + + """ + # TODO: Why is this not higher? stopping_maxiter: PositiveInt = 2_000 + """Maximum number of iterations. + + If reached, terminate. + + """ + trustregion_initial_radius: PositiveFloat = 0.1 + """Initial trust-region radius.""" + trustregion_minimal_radius: PositiveFloat = 1e-6 + """Minimal trust-region radius.""" + trustregion_maximal_radius: PositiveFloat = 1e6 + """Maximal trust-region radius.""" + trustregion_shrinking_factor_not_successful: PositiveFloat = 0.5 + """Shrinking factor of the trust-region radius in case the solution vector of the + subproblem is not accepted, but the model is fully linear (i.e. "valid").""" + trustregion_expansion_factor_successful: PositiveFloat = 2 + """Expansion factor of the trust-region radius in case the solution vector of the + subproblem is accepted.""" + theta1: PositiveFloat = 1e-5 + """Threshold for adding the current candidate vector to the model. + + Used when searching for affine (i.e. model-improving) points. + + """ + theta2: PositiveFloat = 1e-4 + """Threshold for adding the current candidate vector to the model. + + Used when computing the feature matrices of the residual model. + + """ + trustregion_threshold_acceptance: NonNegativeFloat = 0 + """First threshold for accepting the solution vector of the trust-region + subproblem as the best candidate.""" + trustregion_threshold_successful: NonNegativeFloat = 0.1 + """Second threshold for accepting the solution vector of the trust-region + subproblem as the best candidate. + + If exceeded, the iteration counts as successful and the trust-region radius is + expanded. + + """ + c1: NonNegativeFloat | None = None + """Threshold for accepting the norm of the current candidate vector. + + Used when searching for affine points in case the set of model-improving + points is zero. By default, it is set to sqrt(n), where n is the number of + parameters. + + """ + c2: NonNegativeFloat = 10 + """Threshold for accepting the norm of the current candidate vector. + + Used when searching for affine points in case the set of model-improving + points is not zero. The default is 10. + + """ + trustregion_subproblem_solver: Literal[ "bntr", "gqtpar", ] = "bntr" + """Solver used for the trust-region subproblem. + + Two internal solvers are supported: "bntr" (Bounded Newton Trust Region; the + default, supports bound constraints) and "gqtpar" (based on :cite:`More1983`; + does not support bound constraints). + + """ + trustregion_subsolver_options: dict[str, Any] | None = None + """Options dictionary containing the stopping criteria for the trust-region + subproblem solver. + + It takes different keys depending on the subproblem solver used, with the + exception of the stopping criterion "maxiter" (default 50), which is always + included. + + If the subsolver "bntr" is used, the dictionary may additionally contain the + tolerance levels "gtol_abs" (default 1e-8), "gtol_rel" (default 1e-8), and + "gtol_scaled" (default 0), the maximum number of gradient descent iterations + "maxiter_gradient_descent" (default 5), and the "conjugate_gradient_method" + used to solve the subproblem. Available conjugate gradient methods are "cg" + (with the two additional stopping criteria "gtol_abs_cg", default 1e-8, and + "gtol_rel_cg", default 1e-6), "steihaug_toint", and "trsbox" (default). + + If the subsolver "gqtpar" is employed, the two stopping criteria are "k_easy" + (default 0.1) and "k_hard" (default 0.2). + + None of the dictionary keys need to be specified by default, but can be. + + """ + n_cores: PositiveInt = DEFAULT_N_CORES + """Number of processes used to parallelize the function evaluations. + + The default is 1 (no parallelization). + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/pygmo_optimizers.py b/src/optimagic/optimizers/pygmo_optimizers.py index b8e9b84ed..44e47a388 100644 --- a/src/optimagic/optimizers/pygmo_optimizers.py +++ b/src/optimagic/optimizers/pygmo_optimizers.py @@ -68,21 +68,122 @@ ) @dataclass(frozen=True) class PygmoGaco(Algorithm): + """Minimize a scalar function using the extended ant colony algorithm (GACO). + + The version available through pygmo is a generalized version of the original + ant colony algorithm proposed by :cite:`Schlueter2009`. + + Ant colony optimization is a class of optimization algorithms modeled on the + actions of an ant colony. Artificial "ants" (e.g. simulation agents) locate + optimal solutions by moving through a parameter space representing all + possible solutions. Real ants lay down pheromones directing each other to + resources while exploring their environment. The simulated "ants" similarly + record their positions and the quality of their solutions, so that in later + simulation iterations more ants locate better solutions. + + The extended ant colony algorithm generates future generations of ants by + using a multi-kernel gaussian distribution based on three parameters (i.e., + pheromone values) which are computed depending on the quality of each + previous solution. The solutions are ranked through an oracle penalty + method. + + Like all pygmo optimizers, this is a global, derivative-free, population + based algorithm. It is a good choice for multimodal problems where no + derivatives are available and a large budget of function evaluations is + affordable. In contrast to most other pygmo algorithms, it supports parallel + function evaluation via ``n_cores``. For smooth problems with only one + optimum, local optimizers are much more efficient. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + n_cores: int = 1 + """Number of cores used for parallel function evaluation.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + kernel_size: PositiveInt = 63 + """Number of solutions stored in the solution archive (the ``ker`` parameter + in pygmo). + + The default value is taken from pygmo. + + """ + speed_parameter_q: PositiveFloat = 1.0 + """This parameter manages the convergence speed towards the found minima (the + smaller the faster). + + In the pygmo documentation it is referred to as :math:`q`. It must be + positive and can be larger than 1. The default is 1.0 until ``threshold`` is + reached. Then it is set to 0.01 automatically. + + """ + oracle: float = 0.0 + """Oracle parameter used in the penalty method.""" + accuracy: PositiveFloat = 0.01 + """Accuracy parameter for maintaining a minimum penalty function's values + distances.""" + threshold: PositiveInt = 1 + """When the generation counter reaches the threshold, the convergence speed + ``speed_parameter_q`` is set to 0.01 automatically. + + It must lie between 1 and ``stopping_maxiter``. To deactivate this effect, + set the threshold to ``stopping_maxiter``, which is the largest allowed + value. + + """ + speed_of_std_values_convergence: int = 7 + """Parameter that determines the convergence speed of the standard + deviations. + + This must be an integer (the ``n_gen_mark`` parameter in pygmo and pagmo). + + """ + stopping_max_n_without_improvements: PositiveInt = 100000 + """If a positive integer is assigned here, the algorithm counts the runs + without improvements; if this number exceeds the given value, the algorithm + is stopped (the ``impstop`` parameter in pygmo).""" + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of function evaluations (the ``evalstop`` parameter in + pygmo).""" + focus: NonNegativeFloat = 0.0 + """This parameter makes the search for the optimum greedier and more focused + on local improvements (the higher the greedier). + + If the value is very high, the search is more focused around the current + best solutions. Values larger than 1 are allowed. + + """ + cache: bool = False + """If True, memory is activated in the algorithm for multiple calls (the + ``memory`` parameter in pygmo).""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -136,11 +237,53 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoBeeColony(Algorithm): + """Minimize a scalar function using the artificial bee colony algorithm. + + The artificial bee colony algorithm (ABC) was originally proposed by + :cite:`Karaboga2007`. The implemented version of the algorithm is the one + proposed in :cite:`Mernik2015`. + + The algorithm mimics the foraging behavior of a honey bee swarm: employed + bees exploit known food sources (candidate solutions), onlooker bees pick + promising sources based on their quality, and food sources that could not be + improved for ``max_n_trials`` trials are abandoned and replaced by random new + ones. + + It is a global, derivative-free, population based algorithm that is suited + for multimodal problems where no derivatives are available and many function + evaluations are affordable. For smooth problems with only one optimum, local + optimizers are much more efficient. + + The algorithm requires finite bounds on all parameters. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + max_n_trials: PositiveInt = 1 + """Maximum number of trials for abandoning a food source (the ``limit`` + parameter in pygmo). + + optimagic uses a default of 1, whereas pygmo's default is 20. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 20.""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -184,14 +327,59 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoDe(Algorithm): + """Minimize a scalar function using the differential evolution algorithm. + + Differential evolution (DE) is a heuristic global optimizer originally + presented in :cite:`Storn1997`. It maintains a population of candidate + solutions and creates new candidates by combining existing ones according to + a simple mutation and crossover recipe, keeping a new candidate whenever it + improves upon the population member it replaces. + + DE is derivative-free and copes well with multimodal and non-smooth scalar + problems. Like all population based algorithms, it needs many function + evaluations, so local optimizers should be preferred for smooth problems + with only one optimum. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 10.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,2] weight_coefficient: NonNegativeFloat = 0.8 + r"""Weight coefficient, denoted by :math:`F` in the differential evolution + literature. + + It controls the amplification of the differential variation + :math:`(x_{r_2, G} - x_{r_3, G})`. The original paper allows values in + [0, 2], but the pygmo implementation requires the value to lie in [0, 1]. + The default value is taken from pygmo. + + """ + # TODO: Probably refine type to fix range [0,1] crossover_probability: NonNegativeFloat = 0.9 + """Crossover probability. It must lie in [0, 1]. The default value is taken + from pygmo.""" + mutation_variant: Literal[ "best/1/exp", "rand/1/exp", @@ -204,8 +392,41 @@ class PygmoDe(Algorithm): "best/2/bin", "rand/2/bin", ] = "rand/1/exp" + """Mutation variant used to create a new candidate individual. + + The default is "rand/1/exp". The following variants are available (the pygmo + integer code given in parentheses is accepted as well): + + - "best/1/exp" (1) + - "rand/1/exp" (2) + - "rand-to-best/1/exp" (3) + - "best/2/exp" (4) + - "rand/2/exp" (5) + - "best/1/bin" (6) + - "rand/1/bin" (7) + - "rand-to-best/1/bin" (8) + - "best/2/bin" (9) + - "rand/2/bin" (10) + + """ + convergence_criterion_tolerance: NonNegativeFloat = 1e-6 + """Stop when the absolute difference between the best and the worst objective + value in the population is smaller than this value (the ``ftol`` parameter in + pygmo). + + The default value is taken from pygmo. + + """ + convergence_relative_params_tolerance: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when the sum of absolute differences between the best and the worst + parameter vector in the population is smaller than this value (the ``xtol`` + parameter in pygmo). + + In pygmo the default is 1e-6, but optimagic uses its default value of 1e-5. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -269,12 +490,49 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoSea(Algorithm): + r"""Minimize a scalar function using the (N+1)-ES simple evolutionary + algorithm. + + This algorithm represents the simplest evolutionary strategy: a population + of :math:`\lambda` individuals produces one offspring per generation by + mutating its best individual uniformly at random within the bounds. Should + the offspring be better than the worst individual in the population, it will + substitute it. See :cite:`Oliveto2007` for a time-complexity analysis of + this kind of evolutionary algorithm. + + Each generation computes the objective function only once, so progress is + slow but the algorithm is extremely simple and robust. It is mainly useful + as a baseline for other global, derivative-free optimizers on multimodal + problems. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 10.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = ( 10_000 # Each generation will compute the objective once ) + """Number of generations to evolve (the ``gen`` parameter in pygmo). + + Each generation computes the objective function once. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -317,32 +575,125 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoSga(Algorithm): + """Minimize a scalar function using a simple genetic algorithm. + + A simple genetic algorithm evolves a population of candidate solutions by + repeatedly applying selection, crossover, and mutation operators. The pygmo + implementation offers several strategies for each of these operators. A + detailed description of the algorithm can be found in the `pagmo2 + documentation `_. + See also :cite:`Oliveto2007` for a time-complexity analysis of evolutionary + algorithms. + + Genetic algorithms are global, derivative-free, population based optimizers + that are suited for multimodal problems where no derivatives are available + and many function evaluations are affordable. For smooth problems with only + one optimum, local optimizers are much more efficient. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,1] crossover_probability: NonNegativeFloat = 0.9 + """Crossover probability. It must lie in [0, 1]. The default value is taken + from pygmo.""" + crossover_strategy: Literal[ "exponential", "sbx", "single", "binomial", ] = "exponential" + """The crossover strategy. + + One of "exponential", "binomial", "single" or "sbx" (simulated binary + crossover). Default is "exponential". + + """ + # TODO: Refine type to fix range [1,100] eta_c: PositiveFloat | None = None + """Distribution index for "sbx" crossover. + + It must lie in [1, 100]. This parameter is ignored (and a warning is raised) + if another crossover strategy is selected. If None, pygmo's default of 1.0 + is used. + + """ + # TODO: Refine type to fix range [0,1] mutation_probability: NonNegativeFloat = 0.02 + """Mutation probability. It must lie in [0, 1]. The default value is taken + from pygmo.""" + mutation_strategy: Literal["uniform", "polynomial"] = "polynomial" + """Mutation strategy. Must be "uniform" or "polynomial". Default is + "polynomial".""" + # TODO: Refine type to fix range [0,1] mutation_polynomial_distribution_index: NonNegativeFloat | None = None + """Distribution index used by polynomial mutation (part of the ``param_m`` + parameter in pygmo). + + According to the pygmo documentation, it must lie in [1, 100]. It is only + used if ``mutation_strategy`` is "polynomial"; otherwise a warning is raised + and the value is ignored. If None, 1.0 is used. + + """ + # TODO: Refine type to fix range [0,1] mutation_gaussian_width: NonNegativeFloat | None = None + """Width of the gaussian used by gaussian mutation (part of the ``param_m`` + parameter in pygmo). + + It must lie in [0, 1]. It is only used if the mutation strategy is + "gaussian"; otherwise a warning is raised and the value is ignored. + + """ + selection_strategy: Literal["tournament", "truncated"] = "tournament" + """Selection strategy. Must be "tournament" or "truncated". Default is + "tournament".""" + # TODO: Check if should be NonNegativeInt selection_truncated_n_best: int | None = None + """Number of best individuals to use in the "truncated" selection mechanism. + + It is only used if ``selection_strategy`` is "truncated"; otherwise a + warning is raised and the value is ignored. If None, pygmo's default of 2 is + used. + + """ + # TODO Check if should be NonNegativeInt selection_tournament_size: int | None = None + """Size of the tournament in the "tournament" selection mechanism. + + It is only used if ``selection_strategy`` is "tournament"; otherwise a + warning is raised and the value is ignored. If None, pygmo's default of 2 is + used. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -463,11 +814,58 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoSade(Algorithm): + r"""Minimize a scalar function using self-adaptive differential evolution. + + The original differential evolution algorithm (pygmo_de) can be + significantly improved by introducing the idea of parameter + self-adaptation. Many different proposals have been made to self-adapt both + the crossover probability :math:`CR` and the weight coefficient :math:`F` + of the original differential evolution algorithm. pygmo's implementation + supports two different mechanisms. The first one, proposed by + :cite:`Brest2006`, does not make use of the differential evolution + operators to produce new values for :math:`F` and :math:`CR` and, strictly + speaking, is thus not self-adaptation, but rather parameter control. The + resulting differential evolution variant is often referred to as jDE. The + second variant is inspired by the ideas introduced by :cite:`Elsayed2011` + and uses a variation of the selected DE operator to produce new :math:`CR` + and :math:`F` parameters for each individual. This variant is referred to + as iDE. + + Like plain differential evolution, this is a global, derivative-free, + population based algorithm for multimodal problems. The self-adaptation + removes the need to tune :math:`F` and :math:`CR` by hand. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + jde: bool = True + r"""Whether to use the jDE self-adaptation variant (:cite:`Brest2006`) to + control the :math:`F` and :math:`CR` parameters. + + If True, jDE is used, otherwise iDE (:cite:`Elsayed2011`). + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + mutation_variant: Literal[ "best/1/exp", "rand/1/exp", @@ -488,9 +886,59 @@ class PygmoSade(Algorithm): "rand-to-best-and-current/2/exp", "rand-to-best-and-current/2/bin", ] = "rand/1/exp" + """Mutation variant used to create a new candidate individual. + + The default is "rand/1/exp". The first ten variants are the classical + mutation variants introduced in the original DE algorithm; the remaining + ones are considered in the work by :cite:`Elsayed2011`. The following + variants are available (the pygmo integer code given in parentheses is + accepted as well): + + - "best/1/exp" (1) + - "rand/1/exp" (2) + - "rand-to-best/1/exp" (3) + - "best/2/exp" (4) + - "rand/2/exp" (5) + - "best/1/bin" (6) + - "rand/1/bin" (7) + - "rand-to-best/1/bin" (8) + - "best/2/bin" (9) + - "rand/2/bin" (10) + - "rand/3/exp" (11) + - "rand/3/bin" (12) + - "best/3/exp" (13) + - "best/3/bin" (14) + - "rand-to-current/2/exp" (15) + - "rand-to-current/2/bin" (16) + - "rand-to-best-and-current/2/exp" (17) + - "rand-to-best-and-current/2/bin" (18) + + """ + keep_adapted_params: bool = False + r"""When True, the adapted parameters :math:`CR` and :math:`F` are not reset + between successive calls to the evolve method (the ``memory`` parameter in + pygmo). + + Default is False. + + """ + ftol: NonNegativeFloat = 1e-6 + """Stop when the absolute difference between the best and the worst objective + value in the population is smaller than this value. + + The default value is taken from pygmo. + + """ + xtol: NonNegativeFloat = 1e-6 + """Stop when the sum of absolute differences between the best and the worst + parameter vector in the population is smaller than this value. + + The default value is taken from pygmo. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -561,23 +1009,121 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoCmaes(Algorithm): + """Minimize a scalar function using the covariance matrix evolutionary + strategy. + + CMA-ES is one of the most successful algorithms, classified as an + evolutionary strategy, for derivative-free global optimization. The version + supported by optimagic is the version described in :cite:`Hansen2006`. + + CMA-ES samples new candidate solutions from a multivariate normal + distribution whose covariance matrix is adapted based on the success of + previous generations. It is a good default choice among the population based + pygmo optimizers for continuous problems that are multimodal, non-smooth, or + noisy and for which no derivatives are available. Note that the algorithm is + not elitist, i.e. the best solution found so far is not guaranteed to be + part of the current population. + + In contrast to the pygmo version, optimagic always sets ``force_bounds`` to + True. This avoids that ill defined parameter values are evaluated, at the + cost of a somewhat worse covariance matrix adaptation. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,1] backward_horizon: NonNegativeFloat | None = None + """Backward time horizon for the evolution path (``cc`` in the pygmo + documentation). + + It must lie between 0 and 1. If None, it is determined automatically by + pygmo. + + """ + # TODO: Refine type to fix range [0,1] variance_loss_compensation: NonNegativeFloat | None = None + """Makes partly up for the small variance loss in case the indicator is zero + (``cs`` in the pygmo documentation and in the MATLAB code of + :cite:`Hansen2006`). + + It must lie between 0 and 1. If None, it is determined automatically by + pygmo. + + """ + # TODO: Refine type to fix range [0,1] learning_rate_rank_one_update: NonNegativeFloat | None = None + """Learning rate for the rank-one update of the covariance matrix (``c1`` in + the pygmo and pagmo documentation). + + It must lie between 0 and 1. If None, it is determined automatically by + pygmo. + + """ + # TODO: Refine type to fix range [0,1] learning_rate_rank_mu_update: NonNegativeFloat | None = None + """Learning rate for the rank-mu update of the covariance matrix (``cmu`` in + the pygmo and pagmo documentation). + + It must lie between 0 and 1. If None, it is determined automatically by + pygmo. + + """ + # TODO: Check if should be NonNegativeFloat initial_step_size: float = 0.5 + r"""Initial step size, :math:`\sigma^0` in the original paper. + + The default value is taken from pygmo. + + """ + ftol: NonNegativeFloat = 1e-6 + """Stop when the absolute difference between the best and the worst objective + value in the population is smaller than this value. + + The default value is taken from pygmo. + + """ + xtol: NonNegativeFloat = 1e-6 + """Stopping tolerance in the parameter space. + + The algorithm stops when the sampling distribution has become so narrow that + the typical parameter change falls below this value. The default value is + taken from pygmo. + + """ + keep_adapted_params: bool = False + """When True, the adapted parameters are not reset between successive calls + to the evolve method (the ``memory`` parameter in pygmo). + + Default is False. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -635,20 +1181,73 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoSimulatedAnnealing(Algorithm): + """Minimize a scalar function with the simulated annealing algorithm. + + This version of the simulated annealing algorithm is, essentially, an + iterative random search procedure with adaptive moves along the coordinate + directions. It permits uphill moves under the control of a metropolis + criterion, in the hope to avoid the first local minima encountered. This + version is the one proposed in :cite:`Corana1987`. + + Simulated annealing is a derivative-free global optimizer that is suited + for multimodal problems where no derivatives are available. It should not + be used for stochastic (noisy) problems. + + .. note:: + When selecting the starting and final temperature values it helps to + think about the temperature as the deterioration in the objective + function value that still has a 37% chance of being accepted. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + start_temperature: PositiveFloat = 10.0 + """Starting temperature. Must be > 0. The default value is taken from + pygmo.""" + # TODO: Check if type should be same as start_temperature end_temperature: float = 0.01 + """Final temperature. + + It must be positive and smaller than ``start_temperature``. Our default + (0.01) is lower than the default in pygmo and pagmo (0.1). + + """ + # TODO: Check if type should be NonNegativeInt n_temp_adjustments: int = 10 + """Number of temperature adjustments in the annealing schedule.""" + # TODO: Check if type should be NonNegativeInt n_range_adjustments: int = 10 + """Number of adjustments of the search range performed at a constant + temperature.""" + # TODO: Check if type should be NonNegativeInt bin_size: int = 10 + """Number of mutations that are used to compute the acceptance rate.""" + # TODO: Refine type to fix range [0,1] start_range: NonNegativeFloat = 1.0 + """Starting range for mutating the decision vector. It must lie in (0, + 1].""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -696,18 +1295,88 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoPso(Algorithm): + r"""Minimize a scalar function using particle swarm optimization. + + Particle swarm optimization (PSO) is a population based algorithm inspired + by the foraging behaviour of swarms, originally proposed by + :cite:`Kennedy1995`. In PSO each point has memory of the position where it + achieved the best performance :math:`x^l_i` (local memory) and of the best + decision vector :math:`x^g` in a certain neighbourhood, and uses this + information to update its position. For a survey on particle swarm + optimization algorithms, see :cite:`Poli2007`. + + Each particle determines its future position :math:`x_{i+1} = x_i + v_i` + where + + .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - + x^{l}_i) + \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) + + PSO is a global, derivative-free optimizer suited for multimodal problems + where no derivatives are available and many function evaluations are + affordable. This version updates one particle at a time and should not be + used for stochastic problems; see pygmo_pso_gen for a generational variant + that supports them and allows parallelization. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 10.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,1] omega: NonNegativeFloat = 0.7298 + r"""Depending on the variant chosen, :math:`\omega` is the particles' + inertia weight or the constriction coefficient. + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,4] force_of_previous_best: NonNegativeFloat = 2.05 + r""":math:`\eta_1` in the equation above. + + It is the magnitude of the force, applied to the particle's velocity, in + the direction of its previous best position. It must lie between 0 and 4. + The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,4] force_of_best_in_neighborhood: NonNegativeFloat = 2.05 + r""":math:`\eta_2` in the equation above. + + It is the magnitude of the force, applied to the particle's velocity, in + the direction of the best position in its neighborhood. It must lie between + 0 and 4. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,1] max_velocity: NonNegativeFloat = 0.5 + """Maximum allowed particle velocity as fraction of the box bounds. + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + algo_variant: Literal[ "canonical_inertia", "social_and_cog_rand", @@ -716,14 +1385,49 @@ class PygmoPso(Algorithm): "canonical_constriction", "fips", ] = "canonical_constriction" + """Algorithm variant to be used: + + - "canonical_inertia" (1): canonical PSO with inertia weight + - "social_and_cog_rand" (2): same social and cognitive random vector + - "all_components_rand" (3): same random vector for all components + - "one_rand" (4): only one random number per velocity update + - "canonical_constriction" (5): canonical PSO with constriction factor + - "fips" (6): fully informed particle swarm + + The pygmo integer code given in parentheses is accepted as well. + + """ + neighbor_definition: Literal[ "gbest", "lbest", "Von Neumann", "Adaptive random", ] = "lbest" + """Swarm topology that defines each particle's neighbors. + + One of "gbest" (1), "lbest" (2), "Von Neumann" (3) or "Adaptive random" + (4). The pygmo integer code given in parentheses is accepted as well. + + """ + neighbor_param: int | None = None + """The neighbourhood parameter. + + If the "lbest" topology is selected, it represents each particle's indegree + (also outdegree) in the swarm topology. Particles have neighbours up to a + radius of ``neighbor_param / 2`` in the ring. If the "Adaptive random" + topology is selected, it represents each particle's maximum outdegree in + the swarm topology; the minimum outdegree is 1 (the particle always + connects back to itself). If the neighbor definition is "gbest" or "Von + Neumann", this parameter is ignored and a warning is raised if it is set. + If None, pygmo's default of 4 is used. + + """ + keep_velocities: bool = False + """When True, the particle velocities are not reset between successive calls + to the evolve method (the ``memory`` parameter in pygmo).""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -803,19 +1507,90 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoPsoGen(Algorithm): + r"""Minimize a scalar function with generational particle swarm + optimization. + + Particle swarm optimization (generational) is identical to pygmo_pso, but + it updates the velocities of all particles before the new particle + positions are computed (taking into consideration all updated particle + velocities). The whole population is thus evaluated in one batch per + generation, as opposed to the standard PSO, which evaluates a single + particle at a time. Consequently, the generational PSO algorithm is suited + for stochastic optimization problems and supports parallel function + evaluation via ``n_cores``. + + For a survey on particle swarm optimization algorithms, see + :cite:`Poli2007`. + + Each particle determines its future position :math:`x_{i+1} = x_i + v_i` + where + + .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - + x^{l}_i) + \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 10.""" + n_cores: PositiveInt = 1 + """Number of cores used for parallel function evaluation.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,1] omega: NonNegativeFloat = 0.7298 + r"""Depending on the variant chosen, :math:`\omega` is the particles' + inertia weight or the constriction coefficient. + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,4] force_of_previous_best: NonNegativeFloat = 2.05 + r""":math:`\eta_1` in the equation above. + + It is the magnitude of the force, applied to the particle's velocity, in + the direction of its previous best position. It must lie between 0 and 4. + The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,4] force_of_best_in_neighborhood: NonNegativeFloat = 2.05 + r""":math:`\eta_2` in the equation above. + + It is the magnitude of the force, applied to the particle's velocity, in + the direction of the best position in its neighborhood. It must lie between + 0 and 4. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,1] max_velocity: NonNegativeFloat = 0.5 + """Maximum allowed particle velocity as fraction of the box bounds. + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + algo_variant: Literal[ "canonical_inertia", "social_and_cog_rand", @@ -824,14 +1599,49 @@ class PygmoPsoGen(Algorithm): "canonical_constriction", "fips", ] = "canonical_constriction" + """Algorithm variant to be used: + + - "canonical_inertia" (1): canonical PSO with inertia weight + - "social_and_cog_rand" (2): same social and cognitive random vector + - "all_components_rand" (3): same random vector for all components + - "one_rand" (4): only one random number per velocity update + - "canonical_constriction" (5): canonical PSO with constriction factor + - "fips" (6): fully informed particle swarm + + The pygmo integer code given in parentheses is accepted as well. + + """ + neighbor_definition: Literal[ "gbest", "lbest", "Von Neumann", "Adaptive random", ] = "lbest" + """Swarm topology that defines each particle's neighbors. + + One of "gbest" (1), "lbest" (2), "Von Neumann" (3) or "Adaptive random" + (4). The pygmo integer code given in parentheses is accepted as well. + + """ + neighbor_param: int | None = None + """The neighbourhood parameter. + + If the "lbest" topology is selected, it represents each particle's indegree + (also outdegree) in the swarm topology. Particles have neighbours up to a + radius of ``neighbor_param / 2`` in the ring. If the "Adaptive random" + topology is selected, it represents each particle's maximum outdegree in + the swarm topology; the minimum outdegree is 1 (the particle always + connects back to itself). If the neighbor definition is "gbest" or "Von + Neumann", this parameter is ignored and a warning is raised if it is set. + If None, pygmo's default of 4 is used. + + """ + keep_velocities: bool = False + """When True, the particle velocities are not reset between successive calls + to the evolve method (the ``memory`` parameter in pygmo).""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -910,13 +1720,70 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoMbh(Algorithm): + r"""Minimize a scalar function using generalized monotonic basin hopping. + + Monotonic basin hopping, or simply basin hopping, is an algorithm rooted in + the idea of mapping the objective function :math:`f(x_0)` into the local + minimum found when starting a local search from :math:`x_0`. This simple + idea allows a substantial increase of efficiency in solving problems, such + as the Lennard-Jones cluster or the MGA-1DSM interplanetary trajectory + problem, that are conjectured to have a so-called funnel structure. See + :cite:`Wales1997` for the paper introducing the basin hopping idea for a + Lennard-Jones cluster optimization. + + pygmo provides an original generalization of this concept resulting in a + meta-algorithm that operates on a population: the population is randomly + perturbed, the inner algorithm is run on it, and the result is kept only if + it improves upon the best known solution. When a population containing a + single individual is used, the original method is recovered. + + Basin hopping is a good choice for derivative-free problems with many local + optima in which good local optima are close to each other (a funnel + structure). The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 250.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + inner_algorithm: pg.algorithm | None = None + """A pygmo algorithm or a user-defined pygmo algorithm, either C++ or Python, + that is used for the inner (local) searches. + + If None, the ``pygmo.compass_search`` algorithm is used. + + """ + # this is 30 instead of 5 in pygmo for our sum of squares test to pass stopping_max_inner_runs_without_improvement: PositiveInt = 30 + """Number of consecutive runs of the inner algorithm that need to result in + no improvement for the algorithm to stop (the ``stop`` parameter in pygmo). + + optimagic uses a default of 30, whereas pygmo's default is 5. + + """ + perturbation: float = 0.01 + """The perturbation to be applied to each component of the decision vector + when a new starting point is generated, expressed as a fraction of the width + of the bounds. + + It must lie in (0, 1]. The default value is taken from pygmo. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -963,21 +1830,117 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoXnes(Algorithm): + """Minimize a scalar function using exponential natural evolution + strategies. + + Exponential natural evolution strategies (xNES) is an algorithm closely + related to CMA-ES and based on the adaptation of a gaussian sampling + distribution via the so-called natural gradient. Like CMA-ES, it is based + on the idea of sampling new trial vectors from a multivariate distribution + and using the new sampled points to update the distribution parameters. + Naively, this could be done following the gradient of the expected fitness + as approximated by a finite number of sampled points. While this idea + offers a powerful lead on algorithmic construction, it has some major + drawbacks that are solved in the so-called natural evolution strategies + class of algorithms by adopting, instead, the natural gradient. xNES is one + of the most performing variants in this class. + + See :cite:`Glasmachers2010` and the `pagmo documentation on xNES + `_ for details. + + Like CMA-ES, xNES is a global, derivative-free, population based optimizer + for continuous problems that are multimodal, non-smooth, or noisy. The + algorithm is not elitist, i.e. the best solution found so far is not + guaranteed to be part of the current population. In contrast to the pygmo + version, optimagic always sets ``force_bounds`` to True, so the criterion + function is never evaluated outside of the bounds, at the cost of a + somewhat worse adaptation of the sampling distribution. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: float | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + # TODO: Refine type to fix range [0,1] learning_rate_mean_update: NonNegativeFloat | None = 1.0 + r"""Learning rate for the mean update (:math:`\eta_\mu` in the pygmo + documentation). + + It must lie between 0 and 1. If None, it is chosen automatically by pygmo. + + """ + # TODO: Refine type to fix range [0,1] learning_rate_step_size_update: NonNegativeFloat | None = None + """Learning rate for the step-size update (``eta_sigma`` in the pygmo + documentation). + + It must lie between 0 and 1. If None, it is chosen automatically by pygmo. + + """ + # TODO: Refine type to fix range [0,1] learning_rate_cov_matrix_update: NonNegativeFloat | None = None + """Learning rate for the covariance matrix update (``eta_b`` in the pygmo + documentation). + + It must lie between 0 and 1. If None, it is chosen automatically by pygmo. + + """ + # TODO: Refine type to fix range [0,1] initial_search_share: NonNegativeFloat | None = 1.0 + """Share of the given search space that will be initially searched + (``sigma0`` in the pygmo documentation). + + The width of the initial sampling distribution along dimension ``i`` is + ``initial_search_share * (upper_bound_i - lower_bound_i)``. It must lie + between 0 and 1. Default is 1. + + """ + ftol: NonNegativeFloat = 1e-6 + """Stop when the absolute difference between the best and the worst objective + value in the population is smaller than this value. + + The default value is taken from pygmo. + + """ + xtol: NonNegativeFloat = 1e-6 + """Stopping tolerance in the parameter space. + + The algorithm stops when the sampling distribution has become so narrow that + the typical parameter change falls below this value. The default value is + taken from pygmo. + + """ + keep_adapted_params: bool = False + """When True, the adapted parameters are not reset between successive calls + to the evolve method (the ``memory`` parameter in pygmo). + + Default is False. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1042,10 +2005,47 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoGwo(Algorithm): + """Minimize a scalar function using the grey wolf optimizer. + + The grey wolf optimizer was proposed by :cite:`Mirjalili2014`. The pygmo + implementation that is wrapped by optimagic is based on the pseudo-code + provided in that paper. The algorithm mimics the leadership hierarchy and + hunting mechanism of grey wolves, with the three best solutions (called + alpha, beta, and delta) guiding the movement of the remaining population. + + This algorithm is a classic example of a highly criticizable line of + research that led in the first decades of our millennium to the development + of an entire zoo of metaphors inspiring optimization heuristics. In the + opinion of the pagmo developers, they are often but small variations of + already existing heuristics rebranded with unnecessary and convoluted + biological metaphors. In the case of GWO this is particularly evident as + the position update rule is strikingly trivial and can also be easily seen + as a product of an evolutionary metaphor or a particle swarm one. Such an + update rule is also not particularly effective and results in a rather poor + performance most of the time. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1086,16 +2086,70 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoCompassSearch(Algorithm): + """Minimize a scalar function using compass search. + + Compass search is a direct (pattern) search method: at each step, the + objective function is evaluated at points obtained by changing one + parameter at a time by the current step size. If an improvement is found, + the search moves there; otherwise, the step size is reduced. The algorithm + is described in :cite:`Kolda2003`. + + In contrast to the other pygmo algorithms, compass search is a local, + derivative-free optimizer. It is considered slow but reliable and is a + reasonable choice for non-differentiable problems where only a local + optimum is needed. It should not be used for stochastic problems. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the initial population. + + Compass search itself is not population based, so this value only + determines how many random initial points are evaluated when the population + is created; the search then continues from the best of them. If a value is + specified, a warning is raised. If None, 100 is used. + + """ + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """Maximum number of function evaluations (the ``max_fevals`` parameter in + pygmo).""" + # TODO: Refine type to fix range (0,1] start_range: PositiveFloat = 0.1 + """The start range. Must lie in (0, 1]. The default value is taken from + pygmo.""" + # TODO?: mus be in (0,start_range] stop_range: PositiveFloat = 0.01 + """The stop range; the search stops when the range falls below this value. + + Must lie in (0, start_range]. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range (0,1) reduction_coeff: PositiveFloat = 0.5 + """The range reduction coefficient by which the range is multiplied when no + improvement is found. + + Must lie in (0, 1). The default value is taken from pygmo. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1148,18 +2202,96 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoIhs(Algorithm): + """Minimize a scalar function using the improved harmony search algorithm. + + Improved harmony search (IHS) was introduced by :cite:`Mahdavi2007`. + Harmony search mimics the improvisation process of a music orchestra: new + candidate solutions ("harmonies") are composed by recombining components of + the solutions stored in the harmony memory (the population), by adjusting + them ("pitch adjustment"), or by drawing random values. In the improved + variant, the pitch adjustment rate and the adjustment bandwidth vary over + the course of the optimization (linearly and exponentially, respectively). + + IHS is a derivative-free global optimizer for multimodal problems. Each + generation creates and evaluates exactly one new candidate solution, so + progress is slow but steady. IHS supports stochastic problems. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population (the harmony memory). + + If None, it is set to ``10 * (n_params + 1)``. If a value is specified, a + warning is raised because the population size has no effect on the + performance of IHS. + + """ + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo). + + Each generation computes the objective function once. + + """ + # TODO: Probably refine type to fix range [0,1] choose_from_memory_probability: NonNegativeFloat = 0.85 + """Probability of choosing from memory (similar to a crossover probability; + the ``phmcr`` parameter in pygmo). + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,1] min_pitch_adjustment_rate: NonNegativeFloat = 0.35 + """Minimum pitch adjustment rate (similar to a mutation rate; the + ``ppar_min`` parameter in pygmo). + + It must lie between 0 and 1. The default value is taken from pygmo. + + """ + # TODO: Refine type to fix range [0,1] max_pitch_adjustment_rate: NonNegativeFloat = 0.99 + """Maximum pitch adjustment rate (similar to a mutation rate; the + ``ppar_max`` parameter in pygmo). + + It must lie between 0 and 1 and be larger than + ``min_pitch_adjustment_rate``. The default value is taken from pygmo. + + """ + min_distance_bandwidth: PositiveFloat = 1e-5 + """Minimum distance bandwidth (similar to a mutation width; the ``bw_min`` + parameter in pygmo). + + It must be positive. The default value is taken from pygmo. + + """ + max_distance_bandwidth: PositiveFloat = 1.0 + """Maximum distance bandwidth (similar to a mutation width; the ``bw_max`` + parameter in pygmo). + + It must be larger than ``min_distance_bandwidth``. The default value is + taken from pygmo. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1209,15 +2341,110 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class PygmoDe1220(Algorithm): + r"""Minimize a scalar function using pygmo's self-adaptive differential + evolution. + + de1220, also known as pDE, is pygmo's own flavor of self-adaptive + differential evolution. In addition to self-adapting the crossover + probability :math:`CR` and the weight coefficient :math:`F` with either the + jDE (:cite:`Brest2006`) or the iDE (:cite:`Elsayed2011`) mechanism, each + individual also carries and adapts the mutation variant that is used to + create new candidates. See the `pagmo documentation on de1220 + `_ for + details. + + Like the other differential evolution variants, it is a global, + derivative-free, population based optimizer for multimodal problems. It is + a good choice if you do not want to tune the mutation variant and the + control parameters of differential evolution by hand. + + The algorithm requires finite bounds on all parameters. + + """ + population_size: int | None = None + """Size of the population. If None, it is set to ``10 * (n_params + 1)``, but + at least 64.""" + seed: int | None = None + """Seed used by the internal random number generator.""" + discard_start_params: bool = False + """If True, the start parameters are not guaranteed to be part of the initial + population. + + This saves one criterion function evaluation that cannot be done in parallel + with other evaluations. Default False. + + """ + jde: bool = True + r"""Whether to use the jDE self-adaptation variant (:cite:`Brest2006`) to + control the :math:`F` and :math:`CR` parameters. + + If True, jDE is used, otherwise iDE (:cite:`Elsayed2011`). + + """ + stopping_maxiter: PositiveInt = STOPPING_MAX_ITERATIONS_GENETIC + """Number of generations to evolve (the ``gen`` parameter in pygmo).""" + allowed_variants: List[str] | None = None + """Mutation variants the self-adaptation is allowed to choose from. + + Each variant can be given as a string or as the pygmo integer code (in + parentheses below). The first ten variants are the classical mutation + variants introduced in the original DE algorithm; the remaining ones are + considered in the work by :cite:`Elsayed2011`. If None, the default is + ["rand/1/exp", "rand-to-best/1/exp", "rand/1/bin", "rand/2/bin", + "best/3/exp", "best/3/bin", "rand-to-current/2/exp", + "rand-to-current/2/bin"]. The following variants are available: + + - "best/1/exp" (1) + - "rand/1/exp" (2) + - "rand-to-best/1/exp" (3) + - "best/2/exp" (4) + - "rand/2/exp" (5) + - "best/1/bin" (6) + - "rand/1/bin" (7) + - "rand-to-best/1/bin" (8) + - "best/2/bin" (9) + - "rand/2/bin" (10) + - "rand/3/exp" (11) + - "rand/3/bin" (12) + - "best/3/exp" (13) + - "best/3/bin" (14) + - "rand-to-current/2/exp" (15) + - "rand-to-current/2/bin" (16) + - "rand-to-best-and-current/2/exp" (17) + - "rand-to-best-and-current/2/bin" (18) + + """ + keep_adapted_params: bool = False + r"""When True, the adapted parameters :math:`CR` and :math:`F` and the + mutation variant are not reset between successive calls to the evolve + method (the ``memory`` parameter in pygmo). + + Default is False. + + """ + ftol: NonNegativeFloat = 1e-6 + """Stop when the absolute difference between the best and the worst objective + value in the population is smaller than this value. + + The default value is taken from pygmo. + + """ + xtol: NonNegativeFloat = 1e-6 + """Stop when the sum of absolute differences between the best and the worst + parameter vector in the population is smaller than this value. + + The default value is taken from pygmo. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/scipy_optimizers.py b/src/optimagic/optimizers/scipy_optimizers.py index b35aef29d..8ff158e68 100644 --- a/src/optimagic/optimizers/scipy_optimizers.py +++ b/src/optimagic/optimizers/scipy_optimizers.py @@ -206,9 +206,40 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipySLSQP(Algorithm): + """Minimize a scalar function of one or more variables using the SLSQP algorithm. + + SLSQP stands for Sequential Least Squares Programming. It is a sequential + quadratic programming (SQP) line search algorithm that solves a quadratic + approximation of the problem in each iteration. It uses first derivatives of + the objective function and the constraints. + + SLSQP is well suited for continuously differentiable scalar optimization + problems with up to several hundred parameters. It supports bounds as well as + nonlinear equality and inequality constraints. + + The optimizer is taken from SciPy, which wraps the SLSQP optimization + subroutine originally implemented by :cite:`Kraft1988`. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_FTOL_ABS + """Absolute precision goal for the value of the objective function in the + stopping criterion. + + optimagic's default is 1e-8, which is stricter than SciPy's default of 1e-6. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -248,12 +279,69 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyNelderMead(Algorithm): + """Minimize a scalar function using the Nelder-Mead simplex algorithm. + + The Nelder-Mead algorithm (:cite:`Nelder1965`) is a direct search method (based + on function comparison) and is often applied to nonlinear optimization problems + for which derivatives are not known. It maintains a simplex of ``n + 1`` points + (where ``n`` is the number of parameters) that is updated via reflection, + expansion, contraction and shrinkage operations. Bounds are supported. + + Unlike most modern optimization methods, the Nelder-Mead heuristic can converge + to a non-stationary point, unless the problem satisfies stronger conditions than + are necessary for modern methods. + + Nelder-Mead is never the best algorithm to solve a problem but rarely the worst. + Its popularity is likely due to historic reasons and much larger than its + properties warrant. + + .. note:: + The SciPy argument ``initial_simplex`` is not supported by optimagic because + it is not compatible with optimagic's handling of constraints. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_FTOL_ABS + """Absolute difference in the criterion value between iterations that is tolerated + to declare convergence. + + As no relative tolerances can be passed to Nelder-Mead, optimagic sets a non-zero + default of 1e-8, which is stricter than SciPy's default of 1e-4. + + """ + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_XTOL_ABS + """Absolute difference in parameters between iterations that is tolerated to + declare convergence. + + As no relative tolerances can be passed to Nelder-Mead, optimagic sets a non-zero + default of 1e-8, which is stricter than SciPy's default of 1e-4. + + """ + adaptive: bool = False + """Adapt the algorithm parameters (reflection, expansion, contraction and + shrinkage) to the dimensionality of the problem. + + This is useful for high-dimensional minimization (:cite:`Gao2012`). The default + value False is taken from SciPy. + + """ + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -295,11 +383,60 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyPowell(Algorithm): + """Minimize a scalar function using the modified Powell method. + + Powell's method (:cite:`Powell1964`) is a conjugate direction method that + minimizes the function by a sequence of one-dimensional searches: In each + iteration it performs a bidirectional line search along each vector of a + direction set and updates the direction set such that the search directions + become approximately conjugate. + + The algorithm is derivative-free: In contrast to gradient-based algorithms like + ``scipy_bfgs`` or ``scipy_conjugate_gradient``, the criterion function need not + be differentiable and no derivatives are evaluated. Bounds are supported. + + .. warning:: + In our benchmark using a quadratic objective function, the Powell algorithm + did not find the optimum very precisely (less than 4 decimal places). If you + require high precision, you should refine an optimum found with Powell with + another local optimizer. + + .. note:: + The SciPy argument ``direc``, which is the initial set of direction vectors, + is not supported by optimagic because it is incompatible with how optimagic + handles constraints. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when the relative movement between parameter vectors is smaller than + this.""" + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + r"""Stop when the relative improvement between two iterations is smaller than + this. More formally, this is expressed as + + .. math:: + + \frac{f^k - f^{k+1}}{\max\{{|f^k|, |f^{k+1}|, 1}\}} \leq + \textsf{convergence_ftol_rel}. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -339,13 +476,94 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyBFGS(Algorithm): + """Minimize a scalar differentiable function using the BFGS algorithm. + + BFGS stands for Broyden-Fletcher-Goldfarb-Shanno algorithm (see + :cite:`Nocedal2006`). It is a quasi-Newton line search algorithm for + unconstrained nonlinear optimization problems. At each iteration it computes a + search direction from the gradient and an approximation of the inverse Hessian + that is built up from the history of gradient evaluations, and then performs a + line search that satisfies the strong Wolfe conditions. + + BFGS is not guaranteed to converge unless the function has a quadratic Taylor + expansion near an optimum. However, BFGS can have acceptable performance even + for non-smooth optimization instances. + + Bounds and other constraints are not supported. For problems with bound + constraints or with a very large number of parameters, consider the + limited-memory variant ``scipy_lbfgsb`` instead. + + """ + convergence_gtol_abs: NonNegativeFloat = CONVERGENCE_GTOL_ABS + """Stop if the norm of the gradient (as defined by the ``norm`` option) is + smaller than this value. + + With the default infinity norm, this means that the optimization stops when all + elements of the gradient are smaller than this value. The default value is + taken from SciPy. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + norm: NonNegativeFloat = np.inf + """Order of the vector norm that is used to calculate the gradient's "score" that + is compared to the gradient tolerance to determine convergence. + + The default is infinity, which means that the largest entry of the gradient + vector is compared to the gradient tolerance. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Relative tolerance in the parameters. Terminate successfully if the step size + is less than ``xk * convergence_xtol_rel`` where ``xk`` is the current parameter + vector. This is the 'xrtol' parameter in the SciPy documentation. + + optimagic's default is 1e-5, while SciPy's default is 0. + + """ + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ + armijo_condition: NonNegativeFloat = 1e-4 + r"""Parameter for the Armijo condition rule of the line search. This is the 'c1' + parameter in the SciPy documentation. It ensures + + .. math:: + + f(x_k + \alpha p_k) \leq f(x_k) + \textsf{armijo_condition} \cdot \alpha + \nabla f(x_k)^\top p_k, + + so each step yields at least a fraction ``armijo_condition`` of the predicted + decrease. Smaller values allow more aggressive steps, larger values enforce more + conservative ones. The default value is taken from SciPy. + + """ + curvature_condition: NonNegativeFloat = 0.9 + r"""Parameter for the curvature condition rule of the line search. This is the + 'c2' parameter in the SciPy documentation. It ensures + + .. math:: + + \nabla f(x_k + \alpha p_k)^\top p_k \geq \textsf{curvature_condition} \cdot + \nabla f(x_k)^\top p_k, + + so the slope at the new point is not too negative. Smaller values enforce a + stricter curvature reduction (smaller steps), larger values are looser (bigger + steps). SciPy requires ``0 < armijo_condition < curvature_condition < 1``. The + default value is taken from SciPy. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -383,10 +601,55 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyConjugateGradient(Algorithm): + """Minimize a scalar function using a nonlinear conjugate gradient algorithm. + + The conjugate gradient method finds local optima of a differentiable function + using only first derivatives; no Hessian information is required, which keeps + the memory requirements low. SciPy's implementation is based on the + Polak-Ribiere variant of the algorithm, described in :cite:`Nocedal2006`, + pp. 120-122. Bounds and other constraints are not supported. + + Conjugate gradient methods tend to work better when: + + - the criterion has a unique global minimizing point, and no local minima or + other stationary points, + - the criterion is, at least locally, reasonably well approximated by a + quadratic function, + - the criterion is continuous and has a continuous gradient, + - the gradient is not too large, e.g., has a norm less than 1000, + - the initial guess is reasonably close to the criterion's global minimizer. + + """ + convergence_gtol_abs: NonNegativeFloat = CONVERGENCE_GTOL_ABS + """Stop if the norm of the gradient (as defined by the ``norm`` option) is + smaller than this value. + + With the default infinity norm, this means that the optimization stops when all + elements of the gradient are smaller than this value. The default value is + taken from SciPy. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + norm: NonNegativeFloat = np.inf + """Order of the vector norm that is used to calculate the gradient's "score" that + is compared to the gradient tolerance to determine convergence. + + The default is infinity, which means that the largest entry of the gradient + vector is compared to the gradient tolerance. + + """ + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -421,9 +684,58 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyNewtonCG(Algorithm): + """Minimize a scalar function using Newton's conjugate gradient algorithm. + + Newton's conjugate gradient algorithm computes the search direction in each + iteration by approximately solving the Newton equations with a conjugate + gradient method that only requires an approximation of the Hessian. It is + practical for both small and large problems (see :cite:`Nocedal2006`, p. 140). + + .. warning:: + In our benchmark using a quadratic objective function, the newton_cg + algorithm did not find the optimum very precisely (less than 4 decimal + places). If you require high precision, you should refine an optimum found + with newton_cg with another local optimizer. + + Newton-CG methods are also called truncated Newton methods. + ``scipy_newton_cg`` differs from ``scipy_truncated_newton`` because + + - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy + and SciPy, while ``scipy_truncated_newton``'s algorithm calls a C function, + - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization, + while ``scipy_truncated_newton``'s algorithm supports bounds. + + Conjugate gradient methods tend to work better when: + + - the criterion has a unique global minimizing point, and no local minima or + other stationary points, + - the criterion is, at least locally, reasonably well approximated by a + quadratic function, + - the criterion is continuous and has a continuous gradient, + - the gradient is not too large, e.g., has a norm less than 1000, + - the initial guess is reasonably close to the criterion's global minimizer. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when the relative movement between parameter vectors is smaller than + this. + + Newton-CG uses the average relative change in the parameters for determining + convergence. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -461,10 +773,58 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyCOBYLA(Algorithm): + """Minimize a scalar function of one or more variables using the COBYLA + algorithm. + + COBYLA stands for Constrained Optimization By Linear Approximation. It is a + derivative-free trust-region method that models the objective and constraint + functions by linear interpolation at the vertices of a simplex. For more + information on COBYLA see :cite:`Powell1994`, :cite:`Powell1998` and + :cite:`Powell2007`. + + COBYLA supports nonlinear constraints. Natively, it can only handle inequality + constraints; optimagic converts each equality constraint into two inequality + constraints before passing the problem to SciPy. Bounds are not supported by + this optimizer in optimagic. + + SciPy's implementation wraps Powell's original Fortran implementation of the + algorithm. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Stop when the relative movement between parameter vectors is smaller than + this. + + In case of COBYLA this is a lower bound on the size of the trust region and can + be seen as the required accuracy in the variables, but this accuracy is not + guaranteed. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + trustregion_initial_radius: PositiveFloat | None = None + """Initial value of the trust region radius. + + Since a linear approximation is likely only good near the current simplex, the + linear program is given the further requirement that the solution, which will + become the next evaluation point, must be within a radius RHO_j from x_j. RHO_j + only decreases, never increases. The initial RHO_j is the + ``trustregion_initial_radius``. In this way COBYLA's iterations behave like a + trust region algorithm. If None (the default), the radius is determined from + the magnitude of the start parameters. + + """ + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -515,12 +875,84 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyLSTRF(Algorithm): + """Minimize a nonlinear least-squares problem using the Trust Region Reflective + method. + + The Trust Region Reflective (TRF) algorithm (:cite:`Branch1999`, + :cite:`Coleman1996`) is an interior trust-region method for bound-constrained + minimization. In each iteration it solves a trust-region subproblem augmented + by a special diagonal quadratic term, where the shape of the trust region is + determined by the distance from the bounds and the direction of the gradient. + + The algorithm can only be used for least-squares problems, i.e. the objective + function must be marked with the ``mark.least_squares`` decorator. It is + particularly suitable for large problems with bounds and is a generally robust + method; it is SciPy's default least-squares method. For small problems with + bounds, ``scipy_ls_dogbox`` can be more efficient, and for small unconstrained + problems ``scipy_ls_lm`` is usually the best choice. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when the relative improvement of the cost function between two iterations + is below this value. + + More precisely, the optimization stops when ``dF < convergence_ftol_rel * F`` + and there was an adequate agreement between a local quadratic model and the + true model in the last step. + + """ + convergence_gtol_rel: NonNegativeFloat = CONVERGENCE_GTOL_REL + """Stop when the uniform norm of the gradient, scaled to account for the presence + of bounds, is below this value. + + The default value is taken from SciPy. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + relative_step_size_diff_approx: NonNegativeFloat | None = None + """Relative step size for SciPy's finite difference approximation of the + Jacobian. This is the 'diff_step' parameter in the SciPy documentation. + + The actual step is computed as ``x * relative_step_size_diff_approx``. If None + (default), it is chosen automatically. Note that optimagic always passes a + Jacobian function (either the user-provided derivative or optimagic's numerical + derivative) to SciPy, in which case this option has no effect. + + """ + tr_solver: Literal["exact", "lsmr"] | None = None + """Method for solving the trust-region subproblems. + + - 'exact' is suitable for not very large problems with dense Jacobian matrices. + The computational complexity per iteration is comparable to a singular value + decomposition of the Jacobian matrix. + - 'lsmr' is suitable for problems with sparse and large Jacobian matrices. It + uses the iterative procedure ``scipy.sparse.linalg.lsmr`` for finding a + solution of a linear least-squares problem and only requires matrix-vector + product evaluations. + + If None (default), the solver is chosen based on the type of Jacobian returned + on the first iteration. + + """ + tr_solver_options: dict[str, Any] | None = None + """Keyword options passed to the trust-region solver. This is the 'tr_options' + parameter in the SciPy documentation. + + - If ``tr_solver='exact'``, the options are ignored. + - If ``tr_solver='lsmr'``, the options are passed to + ``scipy.sparse.linalg.lsmr``. Additionally, the 'regularize' option (bool, + default True) adds a regularization term to the normal equation, which + improves convergence if the Jacobian is rank-deficient. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -568,12 +1000,81 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyLSDogbox(Algorithm): + """Minimize a nonlinear least-squares problem using a rectangular trust region + method. + + The dogbox algorithm (:cite:`Voglis2004`) operates in a trust-region framework, + but uses rectangular trust regions as opposed to conventional ellipsoids. The + intersection of the trust region and the bounds is again rectangular, so in + each iteration a bound-constrained quadratic subproblem is solved approximately + by Powell's dogleg method. + + The algorithm can only be used for least-squares problems, i.e. the objective + function must be marked with the ``mark.least_squares`` decorator. The typical + use case is small problems with bounds, where it often outperforms + ``scipy_ls_trf``. It is not recommended for problems with a rank-deficient + Jacobian. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when the relative improvement of the cost function between two iterations + is below this value. + + More precisely, the optimization stops when ``dF < convergence_ftol_rel * F`` + and there was an adequate agreement between a local quadratic model and the + true model in the last step. + + """ + convergence_gtol_rel: NonNegativeFloat = CONVERGENCE_GTOL_REL + """Stop when the uniform norm of the gradient with respect to the variables that + are not at their bounds is below this value. + + The default value is taken from SciPy. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + relative_step_size_diff_approx: NonNegativeFloat | None = None + """Relative step size for SciPy's finite difference approximation of the + Jacobian. This is the 'diff_step' parameter in the SciPy documentation. + + The actual step is computed as ``x * relative_step_size_diff_approx``. If None + (default), it is chosen automatically. Note that optimagic always passes a + Jacobian function (either the user-provided derivative or optimagic's numerical + derivative) to SciPy, in which case this option has no effect. + + """ + tr_solver: Literal["exact", "lsmr"] | None = None + """Method for solving the trust-region subproblems. + + - 'exact' is suitable for not very large problems with dense Jacobian matrices. + The computational complexity per iteration is comparable to a singular value + decomposition of the Jacobian matrix. + - 'lsmr' is suitable for problems with sparse and large Jacobian matrices. It + uses the iterative procedure ``scipy.sparse.linalg.lsmr`` for finding a + solution of a linear least-squares problem and only requires matrix-vector + product evaluations. + + If None (default), the solver is chosen based on the type of Jacobian returned + on the first iteration. + + """ + tr_solver_options: dict[str, Any] | None = None + """Keyword options passed to the trust-region solver. This is the 'tr_options' + parameter in the SciPy documentation. + + - If ``tr_solver='exact'``, the options are ignored. + - If ``tr_solver='lsmr'``, the options are passed to + ``scipy.sparse.linalg.lsmr``. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -621,10 +1122,53 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyLSLM(Algorithm): + """Minimize a nonlinear least-squares problem using the Levenberg-Marquardt + method. + + SciPy's implementation wraps the classic Levenberg-Marquardt algorithm as + implemented in MINPACK (:cite:`More1978`). It is usually the most efficient + method for small, unconstrained least-squares problems. + + The algorithm can only be used for least-squares problems, i.e. the objective + function must be marked with the ``mark.least_squares`` decorator. It does not + handle bounds or sparse Jacobians and requires that the number of residuals is + at least as large as the number of parameters. For problems with bounds, use + ``scipy_ls_trf`` or ``scipy_ls_dogbox`` instead. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """Stop when the relative improvement of the cost function between two iterations + is below this value. + + More precisely, the optimization stops when ``dF < convergence_ftol_rel * F`` + and there was an adequate agreement between a local quadratic model and the + true model in the last step. + + """ + convergence_gtol_rel: NonNegativeFloat = CONVERGENCE_GTOL_REL + """Stop when the maximum absolute value of the cosine of the angles between the + columns of the Jacobian and the residual vector is below this value. + + The default value is taken from SciPy. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + relative_step_size_diff_approx: NonNegativeFloat | None = None + """Relative step size for SciPy's finite difference approximation of the + Jacobian. This is the 'diff_step' parameter in the SciPy documentation. + + The actual step is computed as ``x * relative_step_size_diff_approx``. If None + (default), it is chosen automatically. Note that optimagic always passes a + Jacobian function (either the user-provided derivative or optimagic's numerical + derivative) to SciPy, in which case this option has no effect. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -661,20 +1205,123 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyTruncatedNewton(Algorithm): + """Minimize a scalar function using the truncated Newton (TNC) algorithm. + + A truncated Newton method computes the search direction in each iteration by + approximately solving the Newton equations with a conjugate gradient iteration + that is truncated after a limited number of steps (:cite:`Dembo1983`, + :cite:`Nash1984`). This makes the algorithm suitable for differentiable scalar + problems with many parameters. Bounds are supported. + + ``scipy_truncated_newton`` differs from ``scipy_newton_cg`` because + + - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy + and SciPy, while ``scipy_truncated_newton`` wraps a C implementation, + - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization, + while ``scipy_truncated_newton``'s algorithm supports bounds. + + Conjugate gradient methods tend to work better when: + + - the criterion has a unique global minimizing point, and no local minima or + other stationary points, + - the criterion is, at least locally, reasonably well approximated by a + quadratic function, + - the criterion is continuous and has a continuous gradient, + - the gradient is not too large, e.g., has a norm less than 1000, + - the initial guess is reasonably close to the criterion's global minimizer. + + .. note:: + optimagic does not support the SciPy arguments ``scale`` and ``offset`` + because they are not compatible with the way optimagic handles constraints. + It also does not support ``messg_num``, which is an additional way to + control the verbosity of the optimizer. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_FTOL_ABS + """Absolute difference in the criterion value between iterations after scaling + that is tolerated to declare convergence.""" + convergence_xtol_abs: NonNegativeFloat = CONVERGENCE_XTOL_ABS + """Absolute difference in parameters between iterations after scaling that is + tolerated to declare convergence.""" + convergence_gtol_abs: NonNegativeFloat = CONVERGENCE_GTOL_ABS + """Stop if the value of the projected gradient (after applying x scaling factors) + is smaller than this value. + + In SciPy, a negative value means that the tolerance is set to + ``1e-2 * sqrt(finite_difference_precision)``. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """If the maximum number of function evaluations is reached, the optimization + stops, but we do not count this as convergence.""" + max_hess_evaluations_per_iteration: int = -1 + """Maximum number of hessian*vector evaluations per main iteration. This is the + 'maxCGit' parameter in the SciPy documentation. + + If ``max_hess_evaluations_per_iteration == 0``, the direction chosen is + ``-gradient``. If ``max_hess_evaluations_per_iteration < 0``, it is set to + ``max(1, min(50, n/2))`` where n is the length of the parameter vector. This is + also the default. + + """ + max_step_for_line_search: NonNegativeFloat = 0 + """Maximum step for the line search. This is the 'stepmx' parameter in the SciPy + documentation. + + It may be increased during the optimization. If too small, it is set to 10.0. + The default value is taken from SciPy. + + """ + line_search_severity: float = -1 + """Severity of the line search. This is the 'eta' parameter in the SciPy + documentation. + + If < 0 or > 1, it is set to 0.25. The default value is taken from SciPy. + + """ + finite_difference_precision: NonNegativeFloat = 0 + """Relative precision for finite difference calculations. This is the 'accuracy' + parameter in the SciPy documentation. + + If it is smaller than the machine precision, it is set to the square root of + the machine precision. The default value is taken from SciPy. + + """ + criterion_rescale_factor: float = -1 + """Scaling factor (in log10) used to trigger rescaling of the criterion function. + This is the 'rescale' parameter in the SciPy documentation. + + If 0, rescale at each iteration. If a large value, never rescale. If < 0, it is + set to 1.3. The default value is taken from SciPy. + + """ + # TODO: Check type hint for `func_min_estimate` func_min_estimate: float = 0 + """Minimum function value estimate. This is the 'minfev' parameter in the SciPy + documentation. + + The default value of 0 is taken from SciPy. + + """ + display: bool = False + """Set to True to print convergence messages. - def _solve_internal_problem( + This is the 'disp' parameter in the SciPy documentation. + + """ + + def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] ) -> InternalOptimizeResult: options = { @@ -720,12 +1367,81 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyTrustConstr(Algorithm): + """Minimize a scalar function of one or more variables subject to constraints. + + ``scipy_trust_constr`` is a trust-region algorithm for constrained + optimization. It is the most versatile constrained minimization algorithm + implemented in SciPy and the most appropriate for large-scale problems. It + supports bounds as well as nonlinear constraints. Natively, it can handle both + equality and inequality constraints; optimagic converts each equality + constraint into two inequality constraints before passing the problem to SciPy. + + It switches between two implementations depending on the problem definition. + For equality-constrained problems it is an implementation of the + Byrd-Omojokun trust-region SQP method described in :cite:`Lalee1998` and in + :cite:`Conn2000`, p. 549. When inequality constraints are imposed as well, it + switches to the trust-region interior point method described in + :cite:`Byrd1999`. This interior point algorithm, in turn, solves inequality + constraints by introducing slack variables and solving a sequence of + equality-constrained barrier problems for progressively smaller values of the + barrier parameter. The previously described equality-constrained SQP method is + used to solve the subproblems with increasing levels of accuracy as the iterate + gets closer to a solution. + + It approximates the Hessian using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) + Hessian update strategy, so no Hessian needs to be provided. + + .. warning:: + In our benchmark using a quadratic objective function, the trust_constr + algorithm did not find the optimum very precisely (less than 4 decimal + places). If you require high precision, you should refine an optimum found + with trust_constr with another local optimizer. + + """ + # TODO: Check if can be set to CONVERGENCE_GTOL_ABS convergence_gtol_abs: NonNegativeFloat = 1e-08 + """Tolerance for termination by the norm of the Lagrangian gradient. + + The algorithm will terminate when both the infinity norm (i.e., max abs value) + of the Lagrangian gradient and the constraint violation are smaller than this + value. The default value of 1e-8 is taken from SciPy. + + """ + convergence_xtol_rel: NonNegativeFloat = CONVERGENCE_XTOL_REL + """Tolerance for termination by the change of the independent variable. + + The algorithm will terminate when the radius of the trust region used in the + algorithm is smaller than this value. optimagic's default is 1e-5, while + SciPy's default is 1e-8. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """If the maximum number of iterations is reached, the optimization stops, but we + do not count this as convergence.""" + trustregion_initial_radius: PositiveFloat | None = None + """Initial value of the trust region radius. + + The trust radius gives the maximum distance between solution points in + consecutive iterations. It reflects the trust the algorithm puts in the local + approximation of the optimization problem. For an accurate local approximation + the trust region should be large and for an approximation valid only close to + the current point it should be a small one. The trust radius is automatically + updated throughout the optimization process, with ``trustregion_initial_radius`` + being its initial value. If None (the default), the initial radius is + determined from the magnitude of the start parameters. + + """ + display: bool = False + """Set to True to print convergence messages. + + This is the 'disp' parameter in the SciPy documentation. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -826,6 +1542,24 @@ def _internal_to_scipy_constraint(c): ) @dataclass(frozen=True) class ScipyBasinhopping(Algorithm): + """Find the global minimum of a scalar function using the basin-hopping + algorithm. + + Basin-hopping (:cite:`Wales1997`) is a two-phase method that combines a global + stepping algorithm with local minimization at each step. Designed to mimic the + natural process of energy minimization of clusters of atoms, it works well for + similar problems with "funnel-like, but rugged" energy landscapes. + + This algorithm is mainly supported for completeness. Consider using optimagic's + built-in multistart optimization for a similar approach that can run multiple + optimizations in parallel, supports all local algorithms in optimagic (as + opposed to just those from SciPy) and allows for a better visualization of the + multistart history. + + When a derivative is provided, it is passed to the local minimization method. + + """ + local_algorithm: ( Literal[ "Nelder-Mead", @@ -845,17 +1579,97 @@ class ScipyBasinhopping(Algorithm): ] | Callable ) = "L-BFGS-B" + """The local optimization algorithm to be used. + + Can be the name of any SciPy local minimizer or a custom function for local + minimization. The default is "L-BFGS-B", as in SciPy. + + """ + n_local_optimizations: PositiveInt = 100 + """The number of local optimizations that are run. + + The default is 100, as in SciPy. + + """ + temperature: NonNegativeFloat = 1.0 + """The "temperature" parameter for the acceptance or rejection criterion. + + Higher temperatures mean that larger jumps in the function value will be + accepted. For best results, it should be comparable to the separation (in + function value) between local minima. The default is 1.0, as in SciPy. + + """ + stepsize: NonNegativeFloat = 0.5 + """Maximum step size for use in the random displacement. + + The default is 0.5, as in SciPy. + + """ + local_algo_options: dict[str, Any] | None = None + """Additional keyword arguments for the local minimizer. + + Check the documentation of the local SciPy algorithms for details on what is + supported. + + """ + take_step: Callable | None = None + """Replace the default step-taking routine with this callable. + + The default is None, which uses SciPy's default step-taking routine. + + """ + accept_test: Callable | None = None + """Define a test to judge the acceptance of steps. + + The default is None, which uses SciPy's default acceptance test. + + """ + interval: PositiveInt = 50 + """Determines how often the step size is updated. + + The default is 50, as in SciPy. + + """ + convergence_n_unchanged_iterations: PositiveInt | None = None + """Stop the run if the global minimum candidate remains the same for this number + of iterations. This is the 'niter_success' parameter in the SciPy + documentation. + + The default is None, as in SciPy. + + """ + seed: int | np.random.Generator | np.random.RandomState | None = None + """Seed or random number generator that makes the stochastic parts of the + algorithm reproducible. + + The default is None, as in SciPy. + + """ + target_accept_rate: NonNegativeFloat = 0.5 + """The target acceptance rate that is used to adjust the step size. + + If the current acceptance rate is greater than the target, the step size is + increased; otherwise, it is decreased. The range is (0, 1) and the default is + 0.5, as in SciPy. + + """ + stepwise_factor: NonNegativeFloat = 0.9 + """The step size is multiplied or divided by this factor upon each update. + + The range is (0, 1) and the default is 0.9, as in SciPy. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -908,10 +1722,55 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyBrute(Algorithm): + """Find the global minimum of a scalar function over a given range by brute + force. + + Brute force evaluates the criterion function at each point of a rectangular + grid defined by the bounds. Since the number of grid points grows exponentially + with the number of parameters, the algorithm is only suited for problems with + very few parameters. Finite bounds are required for all parameters. + + The start values are not actually used because the grid is defined solely by + the bounds. They are still necessary for optimagic to infer the number and + format of the parameters. + + The function evaluations can be parallelized over multiple cores. Due to the + parallelization, this algorithm cannot collect a history of parameters and + criterion evaluations. + + """ + n_grid_points: PositiveInt = 20 + """The number of grid points per parameter dimension used for the brute force + search. This is the 'Ns' parameter in the SciPy documentation. + + The default is 20, as in SciPy. + + """ + polishing_function: Callable | None = None + """Function to seek a more precise minimum near the best grid point, taking the + best grid point as initial guess as a positional argument. This is the 'finish' + parameter in the SciPy documentation. + + The default is None, which means that no polishing is performed. + + """ + n_cores: PositiveInt = 1 + """The number of cores on which the function is evaluated in parallel. + + The default is 1. + + """ + batch_evaluator: BatchEvaluatorLiteral | BatchEvaluator = "joblib" + """An optimagic batch evaluator that is used for the parallel function + evaluations. + + The default is "joblib". + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -959,6 +1818,28 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyDifferentialEvolution(Algorithm): + """Find the global minimum of a scalar function using differential evolution. + + Differential evolution (:cite:`Storn1997`) is a stochastic, population-based + evolutionary algorithm. It does not use gradient information, so the criterion + function does not need to be differentiable. In each generation, candidate + solutions are mutated by mixing them with other candidate solutions of the + population and are kept if they improve the criterion value. + + The algorithm can search large areas of the candidate space, but this often + comes at the cost of requiring more criterion evaluations than gradient-based + methods. It is a good choice for global optimization of non-smooth or noisy + problems with a moderate number of parameters. Finite bounds are required for + all parameters. Nonlinear constraints are supported. + + .. note:: + Due to optimagic's general parameter format, the SciPy arguments + ``integrality`` and ``vectorized`` are not supported. The SciPy argument + ``updating`` is always set to "deferred" so that the population can be + evaluated in parallel. + + """ + strategy: ( Literal[ "best1bin", @@ -976,24 +1857,120 @@ class ScipyDifferentialEvolution(Algorithm): ] | Callable ) = "best1bin" + """The differential evolution strategy that is used to create trial candidates. + + The default is "best1bin", as in SciPy. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXFUN_GLOBAL + """The maximum number of generations over which the entire population is evolved. + + The maximum number of criterion evaluations (without polishing) is + ``(stopping_maxiter + 1) * population_size_multiplier * number of parameters``. + + """ + population_size_multiplier: NonNegativeInt = 15 + """A multiplier for setting the total population size. This is the 'popsize' + parameter in the SciPy documentation. + + The number of individuals in the population is + ``population_size_multiplier * number of parameters``. The default is 15, as in + SciPy. + + """ + convergence_ftol_rel: NonNegativeFloat = 0.01 + """Relative tolerance for convergence. This is the 'tol' parameter in the SciPy + documentation. + + The optimization stops when the standard deviation of the criterion values in + the population is smaller than + ``convergence_ftol_abs + convergence_ftol_rel * |mean criterion value|``. The + default is 0.01, as in SciPy. + + """ + # TODO: Refine type to add ranges [0,2] if float. mutation_constant: NonNegativeFloat | Tuple[NonNegativeFloat, NonNegativeFloat] = ( 0.5, 1, ) + """The differential weight, denoted by F in the literature. This is the + 'mutation' parameter in the SciPy documentation. + + Should be within [0, 2]. If a tuple ``(min, max)`` is given, dithering is used: + the mutation constant is randomly changed on a generation by generation basis, + which can help speed up convergence significantly. The default is (0.5, 1), as + in SciPy. + + """ + # TODO: Refine type to add ranges [0,1]. recombination_constant: NonNegativeFloat = 0.7 + """The crossover probability, denoted by CR in the literature. This is the + 'recombination' parameter in the SciPy documentation. + + It determines the probability that two solution vectors are combined to produce + a new solution vector. Should be between 0 and 1. Increasing this value allows + a larger number of mutants to progress into the next generation, but at the + risk of population stability. The default is 0.7, as in SciPy. + + """ + seed: int | np.random.Generator | np.random.RandomState | None = None + """Seed or random number generator that makes the stochastic parts of the + algorithm reproducible. + + The default is None, as in SciPy. + + """ + polish: bool = True + """If True, the best population member is polished at the end. + + SciPy uses L-BFGS-B for unconstrained problems and trust-constr for constrained + problems to slightly improve the minimum. The default is True, as in SciPy. + + """ + sampling_method: ( Literal["latinhypercube", "random", "sobol", "halton"] | NDArray[np.float64] ) = "latinhypercube" + """Method used to generate the initial population. This is the 'init' parameter + in the SciPy documentation. + + Can be "latinhypercube", "sobol", "halton", "random", or an array specifying + the initial population of shape (total population size, number of parameters). + The initial population is clipped to the bounds before use. The default is + "latinhypercube", as in SciPy. + + """ + convergence_ftol_abs: NonNegativeFloat = CONVERGENCE_SECOND_BEST_FTOL_ABS + """Absolute tolerance for convergence. This is the 'atol' parameter in the SciPy + documentation. + + See ``convergence_ftol_rel`` for the exact convergence criterion. optimagic's + default is 1e-8, while SciPy's default is 0. + + """ + n_cores: PositiveInt = 1 + """The number of cores on which the population is evaluated in parallel. + + The default is 1. + + """ + batch_evaluator: BatchEvaluatorLiteral | BatchEvaluator = "joblib" + """An optimagic batch evaluator that is used for the parallel function + evaluations. + + The default is "joblib". + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1037,6 +2014,27 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipySHGO(Algorithm): + """Find the global minimum of a scalar function using simplicial homology global + optimization (SHGO). + + SHGO (:cite:`Endres2018`) is appropriate for solving general purpose nonlinear + programming and black-box optimization problems to global optimality + (low-dimensional problems). It samples the search space, builds a simplicial + complex on the sampled points from which promising candidate minimizers are + derived, and refines the candidates with a local optimization algorithm. Bounds + and nonlinear constraints are supported. + + With the default "simplicial" sampling method, the algorithm has theoretical + guarantees of convergence to the global minimum in finite time for Lipschitz + smooth functions. + + .. note:: + If nonlinear constraints are used, only the local algorithms "COBYLA" and + "SLSQP" take the constraints into account during the local refinement; the + other local algorithms ignore them. + + """ + local_algorithm: ( Literal[ "Nelder-Mead", @@ -1056,21 +2054,124 @@ class ScipySHGO(Algorithm): ] | Callable ) = "L-BFGS-B" + """The local optimization algorithm to be used. + + Can be the name of any SciPy local minimizer or a custom function for local + minimization. Only "COBYLA" and "SLSQP" support constraints. The default is + "L-BFGS-B", as in SciPy. + + """ + local_algo_options: dict[str, Any] | None = None + """Additional keyword arguments for the local minimizer. + + Check the documentation of the local SciPy algorithms for details on what is + supported. + + """ + n_sampling_points: PositiveInt = 128 + """The number of sampling points used in the construction of the simplicial + complex. This is the 'n' parameter in the SciPy documentation. + + optimagic's default is 128. + + """ + n_simplex_iterations: PositiveInt = 1 + """The number of iterations used in the construction of the simplicial complex. + This is the 'iters' parameter in the SciPy documentation. + + The default is 1, as in SciPy. + + """ + sampling_method: Literal["simplicial", "halton", "sobol"] | Callable = "simplicial" + """The method used for sampling the search space. + + The default "simplicial" provides the theoretical guarantee of convergence to + the global minimum in finite time. The "halton" and "sobol" methods are faster + in terms of sampling point generation at the cost of losing this guarantee. A + custom sampling function can also be passed. + + """ + max_sampling_evaluations: PositiveInt | None = None + """The maximum number of criterion evaluations in the feasible domain during the + sampling phase. This is the 'maxfev' option in the SciPy documentation.""" + convergence_minimum_criterion_value: float | None = None + """Specify the global minimum of the criterion function when it is known. This is + the 'f_min' option in the SciPy documentation. + + For maximization problems, flip the sign. + + """ + convergence_minimum_criterion_tolerance: NonNegativeFloat = 1e-4 + """Specify the relative error between the current best minimum and the supplied + global criterion minimum that is allowed. This is the 'f_tol' option in the + SciPy documentation. + + The default is 1e-4, as in SciPy. + + """ + stopping_maxiter: PositiveInt | None = None + """The maximum number of iterations. + + If reached, the optimization stops, but we do not count this as convergence. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN_GLOBAL + """The maximum number of sampling evaluations, including points outside the + feasible domain. This is the 'maxev' option in the SciPy documentation.""" + stopping_max_processing_time: PositiveFloat | None = None + """The maximum time allowed for the optimization. This is the 'maxtime' option in + the SciPy documentation.""" + minimum_homology_group_rank_differential: PositiveInt | None = None + """The minimum difference in the rank of the homology group between iterations. + This is the 'minhgrd' option in the SciPy documentation. + + The rank of the homology group approximately corresponds to the number of + locally convex subdomains (and thus local minima) found so far. The algorithm + terminates when the rank has not grown for this many iterations. + + """ + symmetry: List | bool = False + """Specify whether the criterion function contains symmetric variables. + + The search space (and therefore performance) can be improved by exploiting + symmetries. + + """ + minimize_every_iteration: bool = True + """Specify whether promising global sampling points are passed to the local + algorithm in every iteration.""" + max_local_minimizations_per_iteration: PositiveInt | bool = False + """The maximum number of local optimizations per iteration. This is the + 'local_iter' option in the SciPy documentation. + + If False (the default), all promising candidates from the minimizer pool are + refined, i.e. there is no limit. + + """ + infinity_constraints: bool = True + """Specify whether sampling points outside the feasible domain are saved. This is + the 'infty_constraints' option in the SciPy documentation. + + If True, the infeasible points are stored with a criterion value of infinity, + which requires more memory but fewer criterion evaluations. The default is + True, as in SciPy. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1145,7 +2246,29 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyDualAnnealing(Algorithm): + """Find the global minimum of a scalar function using dual annealing. + + Dual annealing is a stochastic global optimization algorithm that implements + the generalized simulated annealing (GSA) algorithm (:cite:`Tsallis1988`, + :cite:`TsallisStariolo1996`, :cite:`Xiang1997`), which couples classical and + fast simulated annealing via a distorted Cauchy-Lorentz visiting distribution, + and combines it with a local search that is applied to accepted candidate + solutions. + + The algorithm does not require derivatives of the criterion function for the + annealing phase (derivatives are passed to the local search algorithm) and is + suited for noisy or multi-modal problems. Finite bounds are required for all + parameters. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXFUN_GLOBAL + """The maximum number of global search iterations. + + The default is 1000, as in SciPy. + + """ + local_algorithm: ( Literal[ "Nelder-Mead", @@ -1165,18 +2288,83 @@ class ScipyDualAnnealing(Algorithm): ] | Callable ) = "L-BFGS-B" + """The local optimization algorithm to be used. + + Can be the name of any SciPy local minimizer or a custom function for local + minimization. The default is "L-BFGS-B", as in SciPy. + + """ + local_algo_options: dict[str, Any] | None = None + """Additional keyword arguments for the local minimizer. + + Check the documentation of the local SciPy algorithms for details on what is + supported. + + """ + # TODO: Refine type to add ranges (0.01, 5e4] initial_temperature: PositiveFloat = 5230.0 + """The temperature the algorithm starts with. + + Higher values facilitate a wider search of the energy landscape and allow the + algorithm to escape local minima that it is trapped in. The range is + (0.01, 5e4] and the default is 5230.0, as in SciPy. + + """ + # TODO: Refine type to add ranges (0,1) restart_temperature_ratio: PositiveFloat = 2e-05 + """Reannealing starts when the temperature has decreased to + ``initial_temperature * restart_temperature_ratio``. + + The range is (0, 1) and the default is 2e-5, as in SciPy. + + """ + # TODO: Refine type to add ranges (1, 3] visit: PositiveFloat = 2.62 + """Specify the thickness of the visiting distribution's tails. + + Higher values give the visiting distribution a heavier tail, which makes the + algorithm jump to more distant regions. The range is (1, 3] and the default is + 2.62, as in SciPy. + + """ + # TODO: Refine type to add ranges (-1e4, -5] accept: NegativeFloat = -5.0 + """Controls the probability of acceptance. + + Lower values lead to a smaller acceptance probability. The range is (-1e4, -5] + and the default is -5.0, as in SciPy. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Soft limit for the number of criterion evaluations. + + If the algorithm is in the middle of a local search when this limit is reached, + the limit is exceeded and the algorithm stops after the local search is done. + optimagic's default is 1,000,000, while SciPy's default is 1e7. + + """ + seed: int | np.random.Generator | np.random.RandomState | None = None + """Seed or random number generator that makes the stochastic parts of the + algorithm reproducible. + + The default is None, as in SciPy. + + """ + no_local_search: bool = False + """If set to True, a traditional Generalized Simulated Annealing is performed + without applying any local search strategy. + + The default is False, as in SciPy. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -1227,17 +2415,104 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class ScipyDirect(Algorithm): + """Find the global minimum of a scalar function using the DIRECT algorithm. + + DIRECT stands for DIviding RECTangles (:cite:`Jones1993`). It is a + deterministic, derivative-free global optimization algorithm based on + Lipschitzian optimization without knowledge of the Lipschitz constant. The + algorithm normalizes the search space to a unit hypercube, samples the + criterion function at the center of the hypercube and then iteratively divides + the most promising hyperrectangles. + + By default, the locally biased variant DIRECT_L (:cite:`Gablonsky2001`) is + used, which is often faster on problems without too many local minima. + + Finite bounds are required for all parameters. The start values are not used by + the algorithm, but they are still necessary for optimagic to infer the number + and format of the parameters. + + """ + convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL + """The minimum required difference of the criterion values between the current + best hyperrectangle and the next potentially optimal hyperrectangle to be + divided. This is the 'eps' parameter in the SciPy documentation. + + It determines the trade-off between local and global search: the larger the + value, the more the search is biased towards local exploration. optimagic's + default is 2e-9, while SciPy's default is 1e-4. + + """ + stopping_maxfun: PositiveInt = STOPPING_MAXFUN + """Approximate upper bound on the number of criterion evaluations. + + If reached, the optimization stops, but we do not count this as convergence. + optimagic's default is 1,000,000. Note that SciPy caps this value if necessary + to limit DIRECT's memory usage to approximately 1 GiB. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXFUN_GLOBAL + """The maximum number of iterations. + + If reached, the optimization stops, but we do not count this as convergence. + The default is 1000, as in SciPy. + + """ + locally_biased: bool = True + """If True (default), use the locally biased variant of the algorithm known as + DIRECT_L (:cite:`Gablonsky2001`). + + Set it to False to use the original, unbiased DIRECT algorithm, which is + recommended for very difficult problems with many local minima. + + """ + convergence_minimum_criterion_value: float = -np.inf + """Specify the global minimum of the criterion function when it is known. This is + the 'f_min' parameter in the SciPy documentation. + + The default is minus infinity, meaning that the global minimum is unknown. For + maximization problems, flip the sign. + + """ + # TODO: must be between 0 and 1 convergence_minimum_criterion_tolerance: NonNegativeFloat = 1e-4 + """Specify the relative error between the current best minimum and the supplied + global minimum that is allowed. This is the 'f_min_rtol' parameter in the SciPy + documentation. + + It is only used if ``convergence_minimum_criterion_value`` is set. The default + is 1e-4, as in SciPy. + + """ + # TODO: must be between 0 and 1 volume_hyperrectangle_tolerance: NonNegativeFloat = 1e-16 + """Stop when the volume of the hyperrectangle containing the lowest criterion + value is smaller than this fraction of the complete search space. This is the + 'vol_tol' parameter in the SciPy documentation. + + The range is (0, 1) and the default is 1e-16, as in SciPy. + + """ + # TODO: must be between 0 and 1 length_hyperrectangle_tolerance: NonNegativeFloat = 1e-6 + """Stopping criterion based on the length of the hyperrectangle containing the + lowest criterion value. This is the 'len_tol' parameter in the SciPy + documentation. + + If ``locally_biased`` is True, terminate when half of the normalized maximal + side length of this hyperrectangle is smaller than this value. If + ``locally_biased`` is False, terminate when half of the normalized diagonal of + this hyperrectangle is smaller than this value. The range is (0, 1) and the + default is 1e-6, as in SciPy. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/tao_optimizers.py b/src/optimagic/optimizers/tao_optimizers.py index b36e71778..8b8af6136 100644 --- a/src/optimagic/optimizers/tao_optimizers.py +++ b/src/optimagic/optimizers/tao_optimizers.py @@ -1,5 +1,7 @@ """This module implements the POUNDERs algorithm.""" +from __future__ import annotations + import functools from dataclasses import dataclass @@ -40,13 +42,105 @@ ) @dataclass(frozen=True) class TAOPounders(Algorithm): - """Implement the POUNDERs algorithm.""" + r"""Minimize a nonlinear least-squares problem using the POUNDERS algorithm. + + POUNDERS (Practical Optimization Using No Derivatives for sums of Squares, + :cite:`Wild2015`) is a derivative-free trust-region algorithm that is tailored + to nonlinear least-squares problems. It is part of the Toolkit for Advanced + Optimization (TAO, :cite:`Benson2017`), which is distributed together with + PETSc, and is wrapped in optimagic via + `petsc4py `_. + + Instead of working only with the scalar sum of squares, POUNDERS exploits the + availability of the full vector of residuals: it maintains an interpolation-based + quadratic model of each residual around the current iterate (building on the + minimal norm Hessian models of :cite:`Wild2008`) and combines these models + according to the known least-squares structure of the objective. Candidate steps + are obtained by minimizing the combined model inside a dynamically resized trust + region. + + Use POUNDERS if your criterion function is a nonlinear sum of squares, evaluating + the residuals is expensive (e.g., each evaluation requires a simulation), + derivatives are not available, and the residuals are possibly slightly noisy. + This makes it a useful tool for economists who estimate structural models using + indirect inference or the method of simulated moments. Compared to + Levenberg-Marquardt style solvers, which need the Jacobian of the residuals + (either analytically or via finite differences, which costs many extra + evaluations per iteration and is unreliable for noisy functions), POUNDERS gets + by without any derivative information. Compared to derivative-free algorithms + that ignore the least-squares structure, such as Nelder-Mead, it typically + requires far fewer criterion evaluations to arrive at a local optimum. If the + residuals are smooth, cheap to evaluate, and derivatives are available, a + derivative-based least-squares solver is usually the better choice. + + Scale the problem such that the bounds correspond to the unit hypercube + :math:`[0, 1]^n`. For unconstrained problems, scale each parameter such that + unit changes in the parameters result in similar order-of-magnitude changes in + the criterion value(s). + + .. note:: + This algorithm requires the petsc4py package, which is not available on + Windows. Windows users can use optimagic's own ``pounders`` algorithm + instead. + + """ convergence_gtol_abs: NonNegativeFloat = CONVERGENCE_GTOL_ABS + r"""Stop if the norm of the gradient falls below this value. + + .. math:: + + \lVert g(X) \rVert \leq \textsf{convergence_gtol_abs} + + This is the absolute gradient tolerance ``gatol`` in the TAO documentation. + Set it to zero to disable this criterion. + + """ + convergence_gtol_rel: NonNegativeFloat = CONVERGENCE_GTOL_REL + r"""Stop if the norm of the gradient relative to the criterion value falls below + this value. + + .. math:: + + \frac{\lVert g(X) \rVert}{|f(X)|} \leq \textsf{convergence_gtol_rel} + + This is the relative gradient tolerance ``grtol`` in the TAO documentation. + Set it to zero to disable this criterion. + + """ + convergence_gtol_scaled: NonNegativeFloat = CONVERGENCE_GTOL_SCALED + r"""Stop if the norm of the gradient falls below this fraction of the norm of the + gradient at the start parameters :math:`X_0`. + + .. math:: + + \frac{\lVert g(X) \rVert}{\lVert g(X_0) \rVert} \leq + \textsf{convergence_gtol_scaled} + + This is the gradient reduction tolerance ``gttol`` in the TAO documentation. + Set it to zero to disable this criterion. + + """ + trustregion_initial_radius: NonNegativeFloat | None = None + r"""Initial value of the trust region radius. It must be larger than zero. + + If None (the default), the radius is set to + :math:`0.1 \max(\lVert x_0 \rVert_\infty, 1)`, where :math:`x_0` are the start + parameters. + + """ + stopping_maxiter: PositiveInt = STOPPING_MAXITER + """Alternative stopping criterion. + + If set, the routine stops after the specified number of iterations or after the + step size is sufficiently small. If the variable is set, the default convergence + criteria are all ignored. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] diff --git a/src/optimagic/optimizers/tranquilo.py b/src/optimagic/optimizers/tranquilo.py index 38c7bda47..bd4b27181 100644 --- a/src/optimagic/optimizers/tranquilo.py +++ b/src/optimagic/optimizers/tranquilo.py @@ -67,31 +67,225 @@ ) @dataclass(frozen=True) class Tranquilo(Algorithm): + """Minimize a scalar function using the tranquilo algorithm. + + Tranquilo, short for "TrustRegion Adaptive Noise robust QuadratIc or Linear + approximation Optimizer", is a trust-region optimizer for derivative-free + optimization that was developed by the optimagic developers + (:cite:`Gabler2024`). It is designed for black-box problems whose objective + function is computationally expensive, noisy, or both, as they arise, for + example, when structural econometric models are estimated via the method of + simulated moments. + + This is the scalar version of tranquilo. If your objective function has a + least-squares structure, use ``tranquilo_ls`` instead, which exploits that + structure and typically needs far fewer function evaluations. + + tranquilo is a local optimizer that does not require derivatives. It supports + bound constraints, but no linear or nonlinear constraints. In each iteration + it fits a quadratic surrogate model to a sample of evaluated points, minimizes + the surrogate model within the current trust region to obtain a candidate + point, and accepts or rejects the candidate by comparing the actual with the + predicted improvement. For background on derivative-free trust-region methods, + see :cite:`Conn2009`. + + Two features distinguish tranquilo from most other derivative-free + trust-region optimizers: + + - Parallelization: If ``n_cores`` is larger than one, the objective function + is evaluated in parallel batches. A parallel line search and speculative + sampling exploit all available cores even in steps of the algorithm that are + inherently sequential. tranquilo thereby minimizes the number of evaluation + batches rather than the number of function evaluations. + - Noise handling: If ``noisy=True``, tranquilo adaptively determines how often + the objective function must be evaluated (and averaged) at each point in + order to keep making progress. The user does not have to provide any + information about the amount or type of noise in the objective function. + + All components of the algorithm (sampling of new points, filtering of existing + points, fitting of surrogate models, solving of the trust-region subproblem, + and the acceptance decision) are exchangeable and configurable via options. + The default values were selected based on extensive benchmarking and rarely + need to be adjusted. In benchmarks on noise-free problems without + parallelization, the scalar version of tranquilo is competitive with, but + slightly slower than, established derivative-free optimizers such as + ``nlopt_bobyqa``. Its strengths come into play for noisy objective functions + and parallel function evaluation. + + .. note:: + This algorithm requires the tranquilo package (version 0.1.0 or newer) to + be installed. You can install it with ``pip install tranquilo`` or + ``conda install -c conda-forge tranquilo``. + + """ + # function type functype: Literal["scalar"] = "scalar" + """The type of the objective function. + + For this algorithm it is always "scalar". Use ``tranquilo_ls`` for problems with + least-squares structure. + + """ + # basic options noisy: bool = False + """Whether the objective function is noisy, i.e. whether repeated evaluations at + the same parameter vector return different values. + + If True, tranquilo activates its adaptive noise handling: the objective function + is evaluated multiple times at each sample point, and the number of evaluations + per point is adjusted over the course of the optimization such that just enough + noise is averaged out to keep making progress. Moreover, the acceptance decision + for candidate points is then based on a statistical power analysis, and the + number of evaluations at the start parameters defaults to 5. + + """ + # convergence options disable_convergence: bool = False + """If True, the optimization is terminated only by the stopping criteria + (``stopping_maxfun``, ``stopping_maxiter``, ``stopping_maxtime``), never because + a convergence criterion is satisfied. + + This is mostly useful for benchmarking. + + """ + convergence_ftol_abs: NonNegativeFloat = 0.0 + r"""Converge if the absolute change in the objective function value between two + accepted iterations is less than this value. More formally, this is expressed as + + .. math:: + + |f^k - f^{k+1}| \leq \textsf{convergence_ftol_abs}. + + This criterion is disabled by default (0.0). + + """ + convergence_gtol_abs: NonNegativeFloat = 0.0 + """Converge if the Euclidean norm of the gradient of the surrogate model at the + current trust-region center is less than this value. + + Since tranquilo never evaluates derivatives of the objective function, the + gradient of the fitted surrogate model is used instead of the true gradient. + This criterion is disabled by default (0.0). + + """ + convergence_xtol_abs: NonNegativeFloat = 0.0 + r"""Converge if the Euclidean distance between the accepted parameter vectors of + two consecutive iterations is less than this value. More formally, this is + expressed as + + .. math:: + + \lVert x^k - x^{k+1} \rVert \leq \textsf{convergence_xtol_abs}. + + This criterion is disabled by default (0.0). + + """ + convergence_ftol_rel: NonNegativeFloat = 2e-9 + r"""Converge if the relative change in the objective function value between two + accepted iterations is less than this value. More formally, this is expressed as + + .. math:: + + \frac{|f^k - f^{k+1}|}{\max\{|f^k|, 1\}} \leq \textsf{convergence_ftol_rel}. + + """ + convergence_gtol_rel: NonNegativeFloat = 1e-8 + r"""Converge if the Euclidean norm of the gradient of the surrogate model at the + current trust-region center, divided by its maximum with 1, is less than this + value. Denoting the gradient of the surrogate model by :math:`g^k`, this is + expressed as + + .. math:: + + \frac{\lVert g^k \rVert}{\max\{\lVert g^k \rVert, 1\}} \leq + \textsf{convergence_gtol_rel}. + + """ + convergence_xtol_rel: NonNegativeFloat = 1e-8 + r"""Converge if the relative change of the accepted parameter vectors between two + consecutive iterations is less than this value. More formally, this is expressed + as + + .. math:: + + \left\lVert \frac{x^k - x^{k+1}}{\max\{|x^k|, 1\}} \right\rVert \leq + \textsf{convergence_xtol_rel}, + + where the division and the maximum are taken element-wise. + + """ + convergence_min_trust_region_radius: NonNegativeFloat = 0.0 + """Consider the optimization converged if the trust-region radius shrinks below + this value. + + This criterion is disabled by default (0.0). + + """ + # stopping options stopping_maxfun: PositiveInt = 2_000 + """Maximum number of objective function evaluations before termination. + + Repeated evaluations at the same point (as used for noisy problems) count + towards this limit. + + """ + stopping_maxiter: PositiveInt = 200 + """Maximum number of iterations (trust-region steps) before termination.""" + stopping_maxtime: NonNegativeFloat = np.inf + """Maximum running time in seconds before termination.""" + # single advanced options batch_evaluator: Literal[ "joblib", "pathos", ] = "joblib" + """Batch evaluator that is used to parallelize evaluations of the objective + function. + + See :ref:`batch_evaluators` for details. + + """ + n_cores: PositiveInt = 1 + """Number of cores used to evaluate the objective function in parallel. + + If larger than one, tranquilo evaluates batches of points in parallel and uses a + parallel line search as well as speculative sampling to keep all cores busy. + + """ + batch_size: PositiveInt | None = None + """Number of points that are evaluated in each batch of parallel function + evaluations. + + Must be at least as large as ``n_cores``, to which it defaults. + + """ + sample_size: PositiveInt | None = None + """Target number of evaluated points in the trust region on which the surrogate + model is fitted. + + By default, the sample size is determined from the model type and the dimension + :math:`n` of the problem: :math:`2 n + 1` for quadratic models, and :math:`n + + 1` (in the noisy or parallel case) or :math:`n + 2` (otherwise) for linear + models. + + """ + model_type: ( Literal[ "quadratic", @@ -99,14 +293,80 @@ class Tranquilo(Algorithm): ] | None ) = None + """Type of surrogate model that is fitted in each iteration. + + Defaults to "quadratic" for the scalar version of tranquilo. Linear models + require fewer function evaluations per iteration but contain no curvature + information. + + """ + search_radius_factor: PositiveFloat | None = None + """Factor by which the trust-region radius is multiplied to obtain the region in + which existing evaluation points are searched for reuse in the surrogate model. + + Reusing existing points saves function evaluations. The default is 4.25 for the + scalar version and 5.0 for the least-squares version of tranquilo. + + """ + n_evals_per_point: NonNegativeInt | None = None + """Initial number of times the objective function is evaluated (and averaged) at + each sample point. + + This is only relevant for noisy problems, where the default is derived from + ``noise_adaptation_options`` and the number of evaluations per point is + adaptively adjusted over the course of the optimization. For non-noisy problems, + the default is 1. + + """ + n_evals_at_start: NonNegativeInt | None = None + """Number of objective function evaluations at the start parameters. + + The default is 5 for noisy problems and 1 otherwise. Multiple evaluations at the + start parameters guarantee that enough evaluations are available to estimate the + variance of the noise in the objective function. + + """ + seed: int | None = 925408 + """Seed for the random number generators used to sample new points and to + simulate noise during the noise adaptation. + + Pass None for non-reproducible randomness. + + """ + # bundled advanced options radius_options: RadiusOptions | None = None + """Advanced options for the management of the trust-region radius, e.g. the + initial, minimal and maximal radius as well as expansion and shrinking factors. + + Pass an instance of ``tranquilo.options.RadiusOptions``; see the tranquilo + package for details. + + """ + stagnation_options: StagnationOptions | None = None + """Advanced options that determine how the algorithm reacts if it stagnates, + i.e. if a proposed step is very small relative to the trust-region radius. + + Pass an instance of ``tranquilo.options.StagnationOptions``; see the tranquilo + package for details. + + """ + noise_adaptation_options: NoiseAdaptationOptions | None = None + """Advanced options that govern how the number of evaluations per point is + adapted for noisy problems, e.g. the minimal and maximal number of evaluations + per point. + + Pass an instance of ``tranquilo.options.NoiseAdaptationOptions``; see the + tranquilo package for details. + + """ + # component names and related options sampler: ( Literal[ @@ -116,7 +376,24 @@ class Tranquilo(Algorithm): ] | Callable ) = "optimal_hull" + """Method used to sample new points within the trust region. + + "optimal_hull" (the default) places new points on the hull of the trust region + such that the minimal distance between all points (including existing ones) is + approximately maximized. "random_hull" and "random_interior" sample random + points on the hull or in the interior of the trust region. A custom sampling + function can be passed as a callable. + + """ + sampler_options: SamplerOptions | None = None + """Advanced options for the sampler. + + Pass an instance of ``tranquilo.options.SamplerOptions``; see the tranquilo + package for details. + + """ + sample_filter: ( Literal[ "discard_all", @@ -127,7 +404,26 @@ class Tranquilo(Algorithm): | Callable | None ) = None + """Method used to select which of the existing evaluation points inside the + search region are used to fit the surrogate model. + + "keep_all" keeps all points, "discard_all" keeps only the trust-region center, + "drop_excess" drops points if there are more than needed to fit the surrogate + model, preferring points that are spread out within the trust region, and + "clustering" keeps only one point per cluster of nearby points. The default is + "drop_excess" if ``batch_size`` is larger than one and "keep_all" otherwise. A + custom filter function can be passed as a callable. + + """ + sample_filter_options: FilterOptions | None = None + """Advanced options for the sample filter. + + Pass an instance of ``tranquilo.options.FilterOptions``; see the tranquilo + package for details. + + """ + model_fitter: ( Literal[ "ols", @@ -138,7 +434,28 @@ class Tranquilo(Algorithm): | Callable | None ) = None + """Method used to fit the surrogate model to the sample of evaluated points. + + "ols" uses ordinary least-squares regression, "ridge" uses ridge regression, + "powell" switches between ordinary least squares and a minimal-Frobenius-norm- + of-Hessian fit depending on the sample size, following ideas from + :cite:`Powell2004`, and "tranquilo" is a variant of "ols" that penalizes the + linear terms of the model less strongly when the fitting problem is + underdetermined. By default, "ols" is used if the sample size is large enough to + identify all model parameters and "tranquilo" otherwise. A custom fitting + function can be passed as a callable. + + """ + model_fitter_options: FitterOptions | None = None + """Advanced options for the model fitter, e.g. penalty terms for ridge + regression. + + Pass an instance of ``tranquilo.options.FitterOptions``; see the tranquilo + package for details. + + """ + cube_subsolver: ( Literal[ "bntr", @@ -148,6 +465,17 @@ class Tranquilo(Algorithm): ] | Callable ) = "bntr_fast" + """Solver for the trust-region subproblem if the trust region is a cube, which + is the case whenever bounds are binding. + + "bntr" and "bntr_fast" implement a bounded Newton trust-region algorithm, where + "bntr_fast" (the default) is a numba-accelerated version of "bntr". + "fallback_cube" and "fallback_multistart" are robust fallback solvers based on + SciPy's L-BFGS-B, where the multistart version restarts the solver from multiple + starting points. A custom subsolver can be passed as a callable. + + """ + sphere_subsolver: ( Literal[ "gqtpar", @@ -158,8 +486,30 @@ class Tranquilo(Algorithm): ] | Callable ) = "gqtpar_fast" + """Solver for the trust-region subproblem if the trust region is a sphere, which + is the case when no bounds are binding. + + "gqtpar" and "gqtpar_fast" solve the subproblem almost exactly with the method + of :cite:`More1983`, where "gqtpar_fast" (the default) is a numba-accelerated + version of "gqtpar". The "fallback_*" solvers are robust but less precise + alternatives based on general-purpose optimizers. A custom subsolver can be + passed as a callable. + + """ + retry_subproblem_with_fallback: bool = True + """Whether to retry solving the trust-region subproblem with a fallback solver + if the main subsolver raises an exception.""" + subsolver_options: SubsolverOptions | None = None + """Advanced options for the trust-region subproblem solvers, e.g. the maximum + number of iterations and gradient tolerances. + + Pass an instance of ``tranquilo.options.SubsolverOptions``; see the tranquilo + package for details. + + """ + acceptance_decider: ( Literal[ "classic", @@ -170,11 +520,67 @@ class Tranquilo(Algorithm): | Callable | None ) = None + """Method used to decide whether a candidate point is accepted as the new + trust-region center. + + "classic" performs the standard acceptance step of trust-region algorithms, + which compares the actual with the predicted improvement. + "classic_line_search" additionally evaluates speculative points along the + search direction in parallel. "naive_noisy" averages a fixed number of + evaluations at the candidate point. "noisy" uses a statistical power analysis + to determine how many evaluations at the candidate and the current point are + needed to decide which one is better. The default is "noisy" for noisy problems + and "classic" otherwise. A custom acceptance function can be passed as a + callable. + + """ + acceptance_decider_options: AcceptanceOptions | None = None + """Advanced options for the acceptance decider, e.g. the confidence and power + levels of the power analysis for noisy problems. + + Pass an instance of ``tranquilo.options.AcceptanceOptions``; see the tranquilo + package for details. + + """ + variance_estimator: Literal["classic"] | Callable = "classic" + """Method used to estimate the variance of the noise in the objective function + for noisy problems. + + The "classic" estimator uses existing repeated function evaluations at points in + a neighborhood of the current trust region and treats the noise variance as + locally constant. A custom estimation function can be passed as a callable. + + """ + variance_estimator_options: VarianceEstimatorOptions | None = None + """Advanced options for the variance estimator, e.g. the minimal number of + evaluations per point used in the estimation. + + Pass an instance of ``tranquilo.options.VarianceEstimatorOptions``; see the + tranquilo package for details. + + """ + infinity_handler: Literal["relative"] | Callable = "relative" + """Method used to clip infinite objective function values before fitting the + surrogate model. + + The "relative" method clips infinite values at a penalty value that is derived + from the range of the finite values in the sample. A custom function can be + passed as a callable. + + """ + residualize: bool | None = None + """Whether the surrogate model is fitted to the deviations of the function + values from the predictions of the previous surrogate model instead of the + function values themselves. + + Defaults to True if the "tranquilo" model fitter is used and False otherwise. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64] @@ -256,29 +662,224 @@ def _solve_internal_problem( ) @dataclass(frozen=True) class TranquiloLS(Algorithm): + r"""Minimize a least-squares problem using the tranquilo algorithm. + + Tranquilo, short for "TrustRegion Adaptive Noise robust QuadratIc or Linear + approximation Optimizer", is a trust-region optimizer for derivative-free + optimization that was developed by the optimagic developers + (:cite:`Gabler2024`). It is designed for black-box problems whose objective + function is computationally expensive, noisy, or both, as they arise, for + example, when structural econometric models are estimated via the method of + simulated moments. + + This is the least-squares version of tranquilo. It minimizes objective + functions of the form :math:`f(x) = \sum_j r_j(x)^2`, where the user provides + the vector of residuals :math:`r(x)`. Instead of fitting one surrogate model + for the objective function, it fits a surrogate model for each residual and + aggregates them into a model of the objective function. Exploiting the + least-squares structure in this way is very efficient: as few as :math:`n + 1` + function evaluations (where :math:`n` is the number of parameters) can be + enough to build a useful surrogate model. Use this version whenever your + objective function can be expressed as a sum of squares, e.g. in nonlinear + regression or method of simulated moments estimation; use the scalar version + (``tranquilo``) otherwise. + + tranquilo_ls is a local optimizer that does not require derivatives. It + supports bound constraints, but no linear or nonlinear constraints. In each + iteration it fits surrogate models to a sample of evaluated points, minimizes + the aggregated surrogate model within the current trust region to obtain a + candidate point, and accepts or rejects the candidate by comparing the actual + with the predicted improvement. For background on derivative-free trust-region + methods, see :cite:`Conn2009`. + + Two features distinguish tranquilo from most other derivative-free + trust-region optimizers: + + - Parallelization: If ``n_cores`` is larger than one, the objective function + is evaluated in parallel batches. A parallel line search and speculative + sampling exploit all available cores even in steps of the algorithm that are + inherently sequential. tranquilo thereby minimizes the number of evaluation + batches rather than the number of function evaluations. + - Noise handling: If ``noisy=True``, tranquilo adaptively determines how often + the objective function must be evaluated (and averaged) at each point in + order to keep making progress. The user does not have to provide any + information about the amount or type of noise in the objective function. + + All components of the algorithm (sampling of new points, filtering of existing + points, fitting of surrogate models, solving of the trust-region subproblem, + and the acceptance decision) are exchangeable and configurable via options. + The default values were selected based on extensive benchmarking and rarely + need to be adjusted. In benchmarks on noise-free problems without + parallelization, tranquilo_ls is slightly slower than DFO-LS + (:cite:`Cartis2018`) and faster than POUNDERS (:cite:`Wild2015`). With two or + more cores, it is considerably faster than DFO-LS, and in noisy settings it + outperforms all benchmarked configurations of DFO-LS. + + .. note:: + This algorithm requires the tranquilo package (version 0.1.0 or newer) to + be installed. You can install it with ``pip install tranquilo`` or + ``conda install -c conda-forge tranquilo``. + + """ + # basic options noisy: bool = False + """Whether the residual functions are noisy, i.e. whether repeated evaluations + at the same parameter vector return different values. + + If True, tranquilo activates its adaptive noise handling: the objective function + is evaluated multiple times at each sample point, and the number of evaluations + per point is adjusted over the course of the optimization such that just enough + noise is averaged out to keep making progress. Moreover, the acceptance decision + for candidate points is then based on a statistical power analysis, and the + number of evaluations at the start parameters defaults to 5. + + """ + # convergence options disable_convergence: bool = False + """If True, the optimization is terminated only by the stopping criteria + (``stopping_maxfun``, ``stopping_maxiter``, ``stopping_maxtime``), never because + a convergence criterion is satisfied. + + This is mostly useful for benchmarking. + + """ + convergence_ftol_abs: NonNegativeFloat = 0.0 + r"""Converge if the absolute change in the objective function value between two + accepted iterations is less than this value. More formally, this is expressed as + + .. math:: + + |f^k - f^{k+1}| \leq \textsf{convergence_ftol_abs}. + + This criterion is disabled by default (0.0). + + """ + convergence_gtol_abs: NonNegativeFloat = 0.0 + """Converge if the Euclidean norm of the gradient of the surrogate model at the + current trust-region center is less than this value. + + Since tranquilo never evaluates derivatives of the objective function, the + gradient of the fitted surrogate model is used instead of the true gradient. + This criterion is disabled by default (0.0). + + """ + convergence_xtol_abs: NonNegativeFloat = 0.0 + r"""Converge if the Euclidean distance between the accepted parameter vectors of + two consecutive iterations is less than this value. More formally, this is + expressed as + + .. math:: + + \lVert x^k - x^{k+1} \rVert \leq \textsf{convergence_xtol_abs}. + + This criterion is disabled by default (0.0). + + """ + convergence_ftol_rel: NonNegativeFloat = 2e-9 + r"""Converge if the relative change in the objective function value between two + accepted iterations is less than this value. More formally, this is expressed as + + .. math:: + + \frac{|f^k - f^{k+1}|}{\max\{|f^k|, 1\}} \leq \textsf{convergence_ftol_rel}. + + """ + convergence_gtol_rel: NonNegativeFloat = 1e-8 + r"""Converge if the Euclidean norm of the gradient of the surrogate model at the + current trust-region center, divided by its maximum with 1, is less than this + value. Denoting the gradient of the surrogate model by :math:`g^k`, this is + expressed as + + .. math:: + + \frac{\lVert g^k \rVert}{\max\{\lVert g^k \rVert, 1\}} \leq + \textsf{convergence_gtol_rel}. + + """ + convergence_xtol_rel: NonNegativeFloat = 1e-8 + r"""Converge if the relative change of the accepted parameter vectors between two + consecutive iterations is less than this value. More formally, this is expressed + as + + .. math:: + + \left\lVert \frac{x^k - x^{k+1}}{\max\{|x^k|, 1\}} \right\rVert \leq + \textsf{convergence_xtol_rel}, + + where the division and the maximum are taken element-wise. + + """ + convergence_min_trust_region_radius: NonNegativeFloat = 0.0 + """Consider the optimization converged if the trust-region radius shrinks below + this value. + + This criterion is disabled by default (0.0). + + """ + # stopping options stopping_maxfun: PositiveInt = 2_000 + """Maximum number of objective function evaluations before termination. + + Repeated evaluations at the same point (as used for noisy problems) count + towards this limit. + + """ + stopping_maxiter: PositiveInt = 200 + """Maximum number of iterations (trust-region steps) before termination.""" + stopping_maxtime: NonNegativeFloat = np.inf + """Maximum running time in seconds before termination.""" + # single advanced options batch_evaluator: Literal[ "joblib", "pathos", ] = "joblib" + """Batch evaluator that is used to parallelize evaluations of the objective + function. + + See :ref:`batch_evaluators` for details. + + """ + n_cores: PositiveInt = 1 + """Number of cores used to evaluate the objective function in parallel. + + If larger than one, tranquilo evaluates batches of points in parallel and uses a + parallel line search as well as speculative sampling to keep all cores busy. + + """ + batch_size: PositiveInt | None = None + """Number of points that are evaluated in each batch of parallel function + evaluations. + + Must be at least as large as ``n_cores``, to which it defaults. + + """ + sample_size: PositiveInt | None = None + """Target number of evaluated points in the trust region on which the surrogate + models are fitted. + + By default, the sample size is determined from the model type and the dimension + :math:`n` of the problem: :math:`n + 1` (in the noisy or parallel case) or + :math:`n + 2` (otherwise) for linear models, and :math:`2 n + 1` for quadratic + models. + + """ + model_type: ( Literal[ "quadratic", @@ -286,14 +887,80 @@ class TranquiloLS(Algorithm): ] | None ) = None + """Type of surrogate model that is fitted to each residual in every iteration. + + Defaults to "linear" for the least-squares version of tranquilo. The residual + models are aggregated into a quadratic model of the objective function, so even + with linear residual models the algorithm has curvature information. + + """ + search_radius_factor: PositiveFloat | None = None + """Factor by which the trust-region radius is multiplied to obtain the region in + which existing evaluation points are searched for reuse in the surrogate model. + + Reusing existing points saves function evaluations. The default is 5.0 for the + least-squares version and 4.25 for the scalar version of tranquilo. + + """ + n_evals_per_point: NonNegativeInt | None = None + """Initial number of times the objective function is evaluated (and averaged) at + each sample point. + + This is only relevant for noisy problems, where the default is derived from + ``noise_adaptation_options`` and the number of evaluations per point is + adaptively adjusted over the course of the optimization. For non-noisy problems, + the default is 1. + + """ + n_evals_at_start: NonNegativeInt | None = None + """Number of objective function evaluations at the start parameters. + + The default is 5 for noisy problems and 1 otherwise. Multiple evaluations at the + start parameters guarantee that enough evaluations are available to estimate the + variance of the noise in the objective function. + + """ + seed: int | None = 925408 + """Seed for the random number generators used to sample new points and to + simulate noise during the noise adaptation. + + Pass None for non-reproducible randomness. + + """ + # bundled advanced options radius_options: RadiusOptions | None = None + """Advanced options for the management of the trust-region radius, e.g. the + initial, minimal and maximal radius as well as expansion and shrinking factors. + + Pass an instance of ``tranquilo.options.RadiusOptions``; see the tranquilo + package for details. + + """ + stagnation_options: StagnationOptions | None = None + """Advanced options that determine how the algorithm reacts if it stagnates, + i.e. if a proposed step is very small relative to the trust-region radius. + + Pass an instance of ``tranquilo.options.StagnationOptions``; see the tranquilo + package for details. + + """ + noise_adaptation_options: NoiseAdaptationOptions | None = None + """Advanced options that govern how the number of evaluations per point is + adapted for noisy problems, e.g. the minimal and maximal number of evaluations + per point. + + Pass an instance of ``tranquilo.options.NoiseAdaptationOptions``; see the + tranquilo package for details. + + """ + # component names and related options sampler: ( Literal[ @@ -303,7 +970,24 @@ class TranquiloLS(Algorithm): ] | Callable ) = "optimal_hull" + """Method used to sample new points within the trust region. + + "optimal_hull" (the default) places new points on the hull of the trust region + such that the minimal distance between all points (including existing ones) is + approximately maximized. "random_hull" and "random_interior" sample random + points on the hull or in the interior of the trust region. A custom sampling + function can be passed as a callable. + + """ + sampler_options: SamplerOptions | None = None + """Advanced options for the sampler. + + Pass an instance of ``tranquilo.options.SamplerOptions``; see the tranquilo + package for details. + + """ + sample_filter: ( Literal[ "discard_all", @@ -314,7 +998,26 @@ class TranquiloLS(Algorithm): | Callable | None ) = None + """Method used to select which of the existing evaluation points inside the + search region are used to fit the surrogate models. + + "keep_all" keeps all points, "discard_all" keeps only the trust-region center, + "drop_excess" drops points if there are more than needed to fit the surrogate + model, preferring points that are spread out within the trust region, and + "clustering" keeps only one point per cluster of nearby points. The default is + "drop_excess" if ``batch_size`` is larger than one and "keep_all" otherwise. A + custom filter function can be passed as a callable. + + """ + sample_filter_options: FilterOptions | None = None + """Advanced options for the sample filter. + + Pass an instance of ``tranquilo.options.FilterOptions``; see the tranquilo + package for details. + + """ + model_fitter: ( Literal[ "ols", @@ -325,7 +1028,28 @@ class TranquiloLS(Algorithm): | Callable | None ) = None + """Method used to fit the surrogate models to the sample of evaluated points. + + "ols" uses ordinary least-squares regression, "ridge" uses ridge regression, + "powell" switches between ordinary least squares and a minimal-Frobenius-norm- + of-Hessian fit depending on the sample size, following ideas from + :cite:`Powell2004`, and "tranquilo" is a variant of "ols" that penalizes the + linear terms of the model less strongly when the fitting problem is + underdetermined. By default, "ols" is used if the sample size is large enough to + identify all model parameters and "tranquilo" otherwise. A custom fitting + function can be passed as a callable. + + """ + model_fitter_options: FitterOptions | None = None + """Advanced options for the model fitter, e.g. penalty terms for ridge + regression. + + Pass an instance of ``tranquilo.options.FitterOptions``; see the tranquilo + package for details. + + """ + cube_subsolver: ( Literal[ "bntr", @@ -335,6 +1059,17 @@ class TranquiloLS(Algorithm): ] | Callable ) = "bntr_fast" + """Solver for the trust-region subproblem if the trust region is a cube, which + is the case whenever bounds are binding. + + "bntr" and "bntr_fast" implement a bounded Newton trust-region algorithm, where + "bntr_fast" (the default) is a numba-accelerated version of "bntr". + "fallback_cube" and "fallback_multistart" are robust fallback solvers based on + SciPy's L-BFGS-B, where the multistart version restarts the solver from multiple + starting points. A custom subsolver can be passed as a callable. + + """ + sphere_subsolver: ( Literal[ "gqtpar", @@ -345,8 +1080,30 @@ class TranquiloLS(Algorithm): ] | Callable ) = "gqtpar_fast" + """Solver for the trust-region subproblem if the trust region is a sphere, which + is the case when no bounds are binding. + + "gqtpar" and "gqtpar_fast" solve the subproblem almost exactly with the method + of :cite:`More1983`, where "gqtpar_fast" (the default) is a numba-accelerated + version of "gqtpar". The "fallback_*" solvers are robust but less precise + alternatives based on general-purpose optimizers. A custom subsolver can be + passed as a callable. + + """ + retry_subproblem_with_fallback: bool = True + """Whether to retry solving the trust-region subproblem with a fallback solver + if the main subsolver raises an exception.""" + subsolver_options: SubsolverOptions | None = None + """Advanced options for the trust-region subproblem solvers, e.g. the maximum + number of iterations and gradient tolerances. + + Pass an instance of ``tranquilo.options.SubsolverOptions``; see the tranquilo + package for details. + + """ + acceptance_decider: ( Literal[ "classic", @@ -357,11 +1114,67 @@ class TranquiloLS(Algorithm): | Callable | None ) = None + """Method used to decide whether a candidate point is accepted as the new + trust-region center. + + "classic" performs the standard acceptance step of trust-region algorithms, + which compares the actual with the predicted improvement. + "classic_line_search" additionally evaluates speculative points along the + search direction in parallel. "naive_noisy" averages a fixed number of + evaluations at the candidate point. "noisy" uses a statistical power analysis + to determine how many evaluations at the candidate and the current point are + needed to decide which one is better. The default is "noisy" for noisy problems + and "classic" otherwise. A custom acceptance function can be passed as a + callable. + + """ + acceptance_decider_options: AcceptanceOptions | None = None + """Advanced options for the acceptance decider, e.g. the confidence and power + levels of the power analysis for noisy problems. + + Pass an instance of ``tranquilo.options.AcceptanceOptions``; see the tranquilo + package for details. + + """ + variance_estimator: Literal["classic"] | Callable = "classic" + """Method used to estimate the variance of the noise in the objective function + for noisy problems. + + The "classic" estimator uses existing repeated function evaluations at points in + a neighborhood of the current trust region and treats the noise variance as + locally constant. A custom estimation function can be passed as a callable. + + """ + variance_estimator_options: VarianceEstimatorOptions | None = None + """Advanced options for the variance estimator, e.g. the minimal number of + evaluations per point used in the estimation. + + Pass an instance of ``tranquilo.options.VarianceEstimatorOptions``; see the + tranquilo package for details. + + """ + infinity_handler: Literal["relative"] | Callable = "relative" + """Method used to clip infinite objective function values before fitting the + surrogate models. + + The "relative" method clips infinite values at a penalty value that is derived + from the range of the finite values in the sample. A custom function can be + passed as a callable. + + """ + residualize: bool | None = None + """Whether the surrogate models are fitted to the deviations of the function + values from the predictions of the previous surrogate model instead of the + function values themselves. + + Defaults to True if the "tranquilo" model fitter is used and False otherwise. + + """ def _solve_internal_problem( self, problem: InternalOptimizationProblem, x0: NDArray[np.float64]