Repository navigation
Nested Conditions result in incorrect deactivation #253
Description
Activity
I'm running into the same issue, however already for a bit simpler (I think) configuration space:
- 3 categorical params a = (A, B), b = (C, D), c = (E, F)
- b is active if a == A
- c is active if b == C (and then of course inactive if b is inactive)
The second condition (for activation of c) can be implemented in two ways:
- Using an
EqualsConditionon b == C - Using an
AndConjuctioncombining the above with the condition a == A
To me, the first seems more intuitive, however:
- The first approach does not work with
get_active_hyperparameters: c is incorrectly reported as active if b is inactive - The second approach does not work with
generate_grid: it fails to evaluate the AndConjuction when b is not active
Code to reproduce (v0.6.1):
import itertools import numpy as np from ConfigSpace import ConfigurationSpace, Configuration, Categorical, AndConjunction, EqualsCondition from ConfigSpace.util import generate_grid def check_sampling(cs): seen = set() for config in cs.sample_configuration(100): config.is_valid_configuration() x_seen = config.get_array().copy() x_seen[np.isnan(x_seen)] = -1 # Otherwise the set doesn't work correctly seen.add(tuple(x_seen)) assert len(seen) == 4 def check_active_params(cs): assert cs.get_hyperparameter_names() == ['a', 'b', 'c'] for x in itertools.product([0, 1], [0, 1], [0, 1]): x_active = cs.get_active_hyperparameters(Configuration(cs, vector=np.array(x), allow_inactive_with_values=True)) x_active_should_be = {'a'} if x[0] == 1 else ({'a', 'b'} if x[1] == 1 else {'a', 'b', 'c'}) try: assert x_active == x_active_should_be except AssertionError: print(f'{x} ({cs.name}): x_active = {x_active}, whereas it should be {x_active_should_be}') def check_generate_grid(cs): assert cs.get_hyperparameter_names() == ['a', 'b', 'c'] try: configs = generate_grid(cs) except ValueError as e: print(f'Encountered ValueError when generating grid for {cs.name}: {e!s}') return assert len(configs) == 4 for config in configs: config.is_valid_configuration() # First way of specifying nested conditions (preferred way): # Child conditions only include their immediate parents (assuming that if the # parent is not active the condition fails, and therefore the child is also not active) cs1 = ConfigurationSpace(name='cs1', space={ 'a': Categorical('a', ['A', 'B']), 'b': Categorical('b', ['C', 'D']), 'c': Categorical('c', ['E', 'F']), }) cs1.add_conditions([ EqualsCondition(cs1['b'], cs1['a'], 'A'), # b is active if a == A EqualsCondition(cs1['c'], cs1['b'], 'C'), # c is active if b == C (and b is active) ]) check_sampling(cs1) check_active_params(cs1) # Fails! check_generate_grid(cs1) # Second way of specifying nested conditions: # Child conditions include all ancestors in their condition cs2 = ConfigurationSpace(name='cs2', space={ 'a': Categorical('a', ['A', 'B']), 'b': Categorical('b', ['C', 'D']), 'c': Categorical('c', ['E', 'F']), }) cs2.add_conditions([ EqualsCondition(cs2['b'], cs2['a'], 'A'), # b is active if a == A # c is active if b == C (and b is active) AndConjunction(EqualsCondition(cs2['c'], cs2['a'], 'A'), EqualsCondition(cs2['c'], cs2['b'], 'C')), ]) check_sampling(cs2) check_active_params(cs2) check_generate_grid(cs2) # Fails!
Output:
(1, 0, 0) (cs1): x_active = {'a', 'c'}, whereas it should be {'a'} (1, 0, 1) (cs1): x_active = {'a', 'c'}, whereas it should be {'a'} Encountered ValueError when generating grid for cs2: Evaluate must be called with all instanstatiated parent hyperparameters in the conjunction; you are (at least) missing 'b'PS: there's also a typo in "instanstatiated" in conditions.pyx
Thank you very much for reporting. @filipbartek do you still plan to pick up your work on #197 again?
I currently estimate that I will not continue my work on #197. If I continue, I will probably do that before the end of October 2023.
This problem still exists in #346. Will add it as a TODO there
Reacted by Jasper BussemakerTried to reproduce the issue in all provided examples. Did not find any unexpected behaviour. Added in each case as a unit test, and created a PR for it in #422
When a variable is conditioned by a nested condition (combined with AndConjunction and OrConjunction) while all its parents in the AndConjunction are inactivated, the variable will be incorrectly deactivated even if OrConjunction is satisfied