Activity

1. joecastiglione

   added

   meetingMeeting notes.Meeting notes.

   on Apr 2, 2026
2. joecastiglione

   changed the title [-]YYYY-MM-DD Which Team[/-] [+]2026-04-16 Engineering Team[/+] on Apr 2, 2026
3. jpn-- commented on Apr 16, 2026

   

   jpn--

   on Apr 16, 2026

   Member

   More actions

   ActivitySim Engineering Meeting Notes

   Meeting Overview

   Agenda

   • AI note-taker policy discussion (deferred to Tuesday partners meeting)
   • Poisson-based person sampling (PS): status, integration, and coordination
   • Explicit error terms (ET): ongoing work and next steps
   • Inconsistency investigation: scope, model selection, and resource allocation

   Participants

   • Joe Castiglione (@joecastiglione) — Zephyr Foundation
   • Jeff Newman (@jpn--) — Driftless
   • Jan Zill (@janzill) — Outer Loop
   • Sumit Bindra — TransLink
   • Nick Fournier (@nick-fournier) — MTC
   • Sijia Wang (@i-am-sijia) — WSP

   Duration

   ~60 minutes

   ---

   Executive Summary

   The meeting focused on three overlapping workstreams related to explicit error terms (ET) and destination choice sampling in ActivitySim: (1) Outer Loop's work making ET production-ready, (2) RSG-led random number generation improvements, and (3) a VLC-contributed Poisson-based person sampling (PS) method for destination choice. The group debated how to prioritize and coordinate the PS contribution, which was not originally in scope but offers significant runtime improvements. Key concerns included the need for proper testing, documentation, and clarification of who would fund the integration work. The team agreed that Jan and Jeff should draft a near-term plan to vet and integrate the PS contribution alongside the existing ET branch.

   ---

   Meeting Notes

   Poisson-based person sampling (PS): background and value

   Jan described the VLC-contributed PS method as a potential drop-in replacement for the current importance sampling method used in destination choice. The primary benefit is runtime: PS draws far fewer random error terms, which are slow to generate in ActivitySim. Unlike explicit error terms, which fully eliminate counterintuitive switching by fixing error terms across scenarios, PS reduces noise in sampling without eliminating it entirely. Jan clarified that PS is most applicable to destination choice sampling specifically — where the sampled choice set is used to approximate a full alternative set — and is less directly applicable to mode choice, where no sub-sampling of alternatives occurs.

   Sumit raised a concern about the growing optionality in the model (Monte Carlo, ET, PS) and whether users would be able to navigate these choices correctly. Jan responded that PS and ET address different things: PS is a sampling implementation for runtime, while ET addresses simulation consistency. Joe confirmed that the intent is for PS to be a toggleable option, not a replacement for existing methods.

   Theoretical validity and calibration considerations

   Joe asked whether switching sampling methods is theoretically sound given that the original models were estimated using a different approach. Jan explained that estimation and application sampling are theoretically independent: as long as a correction factor (McFadden's correction) is applied, sampled estimation yields consistent parameters regardless of the sampling method used in application. For calibrated models, Jan noted that PS should behave very similarly to the current importance sampling method given the models in use, and recalibration is unlikely to be necessary — particularly because the implementation uses pick count = 1, which preserves the existing sample adjustment factor expression without modification.

   Integration and testing requirements

   Jeff outlined the key steps needed before PS can be merged into the main ActivitySim codebase:

   • Confirm the code runs correctly (Jan had already identified and fixed merge errors from the pull request)
   • Review and validate the underlying mathematical derivation
   • Run the method at scale on at least one model to confirm statistically comparable results
   • Write new integration tests — existing tests will not catch regressions since PS is not currently an active option in any test model
   • Add unit tests on the sampling method itself, including statistical comparisons (e.g., distributional tests analogous to those used for Monte Carlo and ET)
   • Produce documentation

   Nick noted that because PS is a toggled option, aggregate-level smoke tests are a reasonable starting point, with more targeted disaggregate-level tests to follow.

   Joe emphasized that the code should not be merged to main until testing and documentation are complete, even though it would remain a toggled, optional feature.

   Prioritization and resourcing

   Joe raised the question of sequencing: which of the pending changes (zone indexing fix, shadow pricing reset fix, PS) should be folded into the working ET branch before the inconsistency investigation proceeds? Jan confirmed that the existing ET pull request (#1064) already includes the indexing and shadow pricing fixes, and that a PS branch based on that could be prepared relatively easily for a side-by-side comparison.

   On resourcing, Joe noted that unlike some other consultancies, neither Outer Loop nor VLC currently has a funded budget for general software support work. However, the consortium has unallocated funds that could potentially be used. No specific allocation was decided in this meeting.

   Inconsistency investigation: model selection

   Sijia raised the question of whether the inconsistency investigation should use the SANDAG model (as in the previous phase) or also include another model such as MTC's, which could reveal different issues related to one-zone vs. two-zone structures and mass pre-sampling. Jan acknowledged this would be valuable but noted that it was not in Outer Loop's current scope or budget, and would roughly double the investigation workload. The group did not reach a decision on this point.

   Random number generation work

   The team briefly noted that RSG's random number generation improvements are on a separate track and are not expected to produce results for another two to three weeks. Joe observed that this work is largely orthogonal to PS — PS addresses runtime in location choice models specifically, while random number generation improvements would benefit Monte Carlo simulation more broadly. Jan agreed, with the caveat that if the random number generation work does not deliver expected runtime improvements, PS becomes even more important for keeping model runtimes manageable.

   ---

   Action Items

   • Jan Zill — Draft a near-term plan for vetting and integrating the PS contribution (starting with confirming the code runs, reviewing the algorithm, and running at scale); share draft with Jeff Newman for review.
   • Jeff Newman — Review and provide feedback on Jan's draft plan.
   • Jan Zill — Prepare a PS branch based on the ET pull request (#1064) to enable side-by-side comparison runs for the inconsistency investigation.
   • Joe Castiglione — Add the AI note-taker policy question to the agenda for the Tuesday partners meeting.
   • Joe Castiglione / Jeff Newman — Identify and propose a funding mechanism from unallocated consortium funds to cover Outer Loop's integration and testing work for PS.
   • Jan Zill / Jeff Newman — Determine whether inconsistency investigation runs should include a second model beyond SANDAG, and assess the budget implications if so.
4. m-richards commented on Apr 17, 2026

   

   m-richards

   on Apr 17, 2026

   Last edited by m-richards

   More actions

   A small note to add on the discussion here & from the 14th, the EET method itself is not perfect from an unintutive switch perspective either - it guarantees that the sample set definition is consistent with random utility theory, but doesn't guarantee the combined sampling + final choice is consistent with utility theory.

   As an example, if there are three alternatives A, B and C and in the base case, the sample set is [A, B], with the EET sampling utility of A >B, but B is chosen in the choosing step due to the unobserved component. In the project case, the attractions for C are improved slightly, such that the sample set is now [A, C], it's now possible that the final choice in the choosing step is A.

   C was improved, leading to a switch from B to A. We think this is an inherent property of sampling itself, and as such is common to Poisson sampling and EET at the sampling step. Given neither method is theoretically perfect yet both show significant stability improvements we expect Poisson sampling would be the pragmatic default choice when using EET.

   Edit: Jan is quite right below, perhaps saying this is a limitation of importance sampling is closer to the mark.
5. janzill commented on Apr 17, 2026

   

   janzill

   on Apr 17, 2026

   More actions

   Thanks, Matt. I agree that this is the case for all three importance sampling methods discussed, but I do not think it is inherent to sampling in general. One of the reasons to look at simple random sampling was that choice sets would be stable and therefore choices with EET would not suffer from this.

   👍React with 👍1Reacted by Matt Richards
6. joecastiglione

   closed this as completedon May 5, 2026