From ed2d35564441f9d0f0c1022613a6927cdade53bb Mon Sep 17 00:00:00 2001 From: Julien Cornebise Date: Tue, 28 Jul 2026 01:55:32 +0200 Subject: [PATCH] =?UTF-8?q?Clojure->Python=20CUTOVER:=20Step=204=20?= =?UTF-8?q?=E2=80=94=20remove=20the=20Clojure=20tree;=20move=20the=20Pytho?= =?UTF-8?q?n=20math=20engine=20to=20math/?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The final tidy-up, and a pure file reorganization (no behavior change): the retired Clojure source is deleted, and the Python math engine moves out of `delphi/` into the now-free top-level `math/` directory as its own Python package. `delphi/` keeps the UMAP/narrative service and ALL the math tests and datasets, and uses the engine as a library — one shared environment, `cd delphi && uv sync` exactly as before, `import polismath` unchanged everywhere. ## What's in this PR - **Deleted:** the Clojure source (60 files) and its CI workflow. It stays in git history — and that matters, because it is the ORACLE: the reference implementation the Python engine was certified bit-for-bit against. `math/README.md` documents a safe way to restore it temporarily if the engine ever changes and needs re-certification (and warns against `git checkout` in this jj-managed repo). - **Moved:** `delphi/polismath` -> `math/polismath`, with its own `math/pyproject.toml` declaring only the ~10 scientific/database dependencies — no LLM/GPU stack. The poller service now starts as `python -m polismath.poller`. - **Paths:** eight modules used to locate the datasets/tests directory by counting parent directories up from their own file — after the move that arithmetic silently points at the wrong tree. All cross-tree path knowledge now lives in one module (`polismath/paths.py`); everything else refers to it, and tests can still substitute paths exactly as before. - **Certification** still works without the Clojure source present: previously recorded reference outputs are used as-is from cache; only actually RE-RUNNING the Clojure side asks for the restored tree, with a clear error explaining how. - **Docker:** the delphi image build now pulls the engine source from a second directory (an "additional build context") — this requires docker compose v2.17+ with BuildKit on every machine that builds; see the merge gate. The test image also asserts after installing that the LOCAL polismath is what got installed — there is an unrelated "polismath" name on PyPI that must never be picked up. - **CI:** the python workflow now watches `math/**` too; a dead copy-and-coverage step for `run_math_pipeline` removed (it is an ordinary module of the package now, already covered). ## Testing 1211 passed / 29 skipped (baseline 1171/22, plus 44 new tests from the flip PR and this one, minus 5 that now skip because the Clojure tree is absent — by design). ## Merge gate (after Step 3 has soaked) - [ ] every prod host's docker-compose is v2.17+ with BuildKit — the deploy script parses the compose file on EVERY host type, so one old binary would break all deploys, not just the math host - [ ] python CI green on this branch - [ ] one successful `docker compose build delphi math-python` on a BuildKit machine ## Known and accepted 10 pre-existing type-checker complaints in `certify.py` / `test_certify.py` (identical code existed before the move — left alone to keep this PR purely a move); some docs still say `delphi/polismath` in historical narrative (docs pass queued). ## Rollback Revert this PR. Nothing in the database or in the flip/decommission steps depends on it, and the built image contains the same installed code either way. Series: Step 0 = #2685. Nothing merges without Julien's explicit go. commit-id:45664936 --- .github/workflows/python-ci.yml | 11 +- .github/workflows/test-clojure.yml | 40 - delphi/Dockerfile | 30 +- delphi/pyproject.toml | 20 +- delphi/run_delphi.py | 4 +- delphi/scripts/prodclone_extract.py | 2 +- delphi/tests/replay_harness/test_certify.py | 75 +- .../replay_harness/test_poller_equiv_seed.py | 2 +- .../test_dynamodb_consensus_roundtrip.py | 2 +- delphi/tests/test_prodclone_extract.py | 2 +- docker-compose.test.yml | 6 +- docker-compose.yml | 8 +- math/.dockerignore | 46 +- math/.gitignore | 58 +- math/.nrepl.edn | 2 - math/CONTRIBUTING.md | 4 - math/Dockerfile | 9 - math/LICENSE | 777 -------------- math/README.md | 178 ++-- math/bin/run | 12 - math/bin/util-server | 7 - math/data/legalization_conv.json | 1 - math/deps.edn | 83 -- math/dev/density_plotting.clj | 54 - math/dev/plumbing_viz.clj | 86 -- math/dev/proj_probe.clj | 349 ------- math/dev/replay.clj | 614 ------------ math/dev/replay_smoke.sh | 72 -- math/dev/user.clj | 489 --------- math/doc/configuration.md | 48 - math/example.env | 12 - math/misc/silh_test.R | 40 - math/perf | 324 ------ {delphi => math}/polismath/__init__.py | 0 .../polismath/benchmarks/__init__.py | 0 .../polismath/benchmarks/bench_pca.py | 0 .../polismath/benchmarks/bench_repness.py | 0 .../benchmarks/bench_update_votes.py | 0 .../polismath/benchmarks/benchmark_utils.py | 0 .../polismath/components/__init__.py | 0 .../polismath/components/config.py | 0 .../polismath/conversation/__init__.py | 0 .../polismath/conversation/conversation.py | 0 .../polismath/conversation/manager.py | 0 .../polismath/database/__init__.py | 0 .../polismath/database/dynamodb.py | 0 .../polismath/database/postgres.py | 0 math/polismath/paths.py | 51 + .../polismath/pca_kmeans_rep/__init__.py | 0 .../polismath/pca_kmeans_rep/clusters.py | 0 .../polismath/pca_kmeans_rep/corr.py | 0 .../pca_kmeans_rep/group_k_smoother.py | 0 .../polismath/pca_kmeans_rep/legacy_kmeans.py | 0 .../polismath/pca_kmeans_rep/pca.py | 0 .../polismath/pca_kmeans_rep/repness.py | 0 {delphi => math}/polismath/poller/__init__.py | 6 +- .../polismath/poller/__main__.py | 6 +- .../polismath/poller/math_writer.py | 0 {delphi => math}/polismath/poller/service.py | 0 .../polismath/poller/worker_pool.py | 0 math/{tmp/.touchfile => polismath/py.typed} | 0 .../polismath/regression/__init__.py | 0 .../polismath/regression/clojure_comparer.py | 0 .../polismath/regression/comparer.py | 7 +- .../polismath/regression/datasets.py | 6 +- .../polismath/regression/recorder.py | 0 .../polismath/regression/utils.py | 6 +- {delphi => math}/polismath/replay/__init__.py | 0 {delphi => math}/polismath/replay/certify.py | 73 +- .../polismath/replay/crosslang.py | 0 {delphi => math}/polismath/replay/driver.py | 0 .../polismath/replay/poller_equiv.py | 26 +- .../polismath/replay/prodclone.py | 0 .../polismath/replay/real_data.py | 4 +- {delphi => math}/polismath/replay/schedule.py | 0 math/polismath/replay/shadow_compare.py | 373 +++++++ .../polismath/replay/shard_bench.py | 3 +- .../polismath/replay/stepcompare.py | 0 {delphi => math}/polismath/replay/store.py | 11 +- {delphi => math}/polismath/replay/types.py | 0 .../polismath/run_math_pipeline.py | 0 {delphi => math}/polismath/utils/__init__.py | 0 {delphi => math}/polismath/utils/clj_hash.py | 0 {delphi => math}/polismath/utils/env_flags.py | 0 {delphi => math}/polismath/utils/general.py | 0 .../polismath/utils/serialization.py | 0 math/pyproject.toml | 44 + math/python_conversion/.gitignore | 54 - math/resources/config.edn | 81 -- math/src/data_readers.clj | 7 - math/src/polismath/components/config.clj | 198 ---- .../polismath/components/core_matrix_boot.clj | 69 -- math/src/polismath/components/env.clj | 14 - math/src/polismath/components/logger.clj | 41 - math/src/polismath/components/postgres.clj | 478 --------- math/src/polismath/components/random.clj | 7 - math/src/polismath/components/server.clj | 72 -- math/src/polismath/conv_man.clj | 473 --------- math/src/polismath/darwin/core.clj | 242 ----- math/src/polismath/darwin/export.clj | 712 ------------- math/src/polismath/math/clusters.clj | 441 -------- math/src/polismath/math/conversation.clj | 945 ------------------ math/src/polismath/math/corr.clj | 247 ----- math/src/polismath/math/named_matrix.clj | 278 ------ math/src/polismath/math/pca.clj | 182 ---- math/src/polismath/math/repness.clj | 417 -------- math/src/polismath/math/stats.clj | 49 - math/src/polismath/meta/metrics.clj | 89 -- math/src/polismath/meta/microscope.clj | 88 -- math/src/polismath/meta/notify.clj | 59 -- math/src/polismath/meta/simulation.clj | 270 ----- math/src/polismath/poller.clj | 60 -- math/src/polismath/runner.clj | 265 ----- math/src/polismath/stormspec.clj | 142 --- math/src/polismath/system.clj | 78 -- math/src/polismath/tasks.clj | 103 -- math/src/polismath/util/pretty_printers.clj | 29 - math/src/polismath/utils.clj | 173 ---- math/system.properties | 1 - math/test/cluster_tests.clj | 296 ------ math/test/conv_edge_cases_test.clj | 218 ---- math/test/conv_man_tests.clj | 62 -- math/test/conversation_test.clj | 174 ---- math/test/index_hash_test.clj | 54 - math/test/named_matrix_test.clj | 146 --- math/test/pca_test.clj | 61 -- math/test/ptpt_stats_test.clj | 31 - math/test/silhouette_test.clj | 60 -- math/test/stats_test.clj | 27 - math/test/test_helpers.clj | 42 - math/test/test_runner.clj | 42 - math/test/utils_test.clj | 20 - 132 files changed, 798 insertions(+), 10882 deletions(-) delete mode 100644 .github/workflows/test-clojure.yml delete mode 100644 math/.nrepl.edn delete mode 100644 math/CONTRIBUTING.md delete mode 100644 math/Dockerfile delete mode 100644 math/LICENSE delete mode 100755 math/bin/run delete mode 100755 math/bin/util-server delete mode 100644 math/data/legalization_conv.json delete mode 100644 math/deps.edn delete mode 100644 math/dev/density_plotting.clj delete mode 100644 math/dev/plumbing_viz.clj delete mode 100644 math/dev/proj_probe.clj delete mode 100644 math/dev/replay.clj delete mode 100755 math/dev/replay_smoke.sh delete mode 100644 math/dev/user.clj delete mode 100644 math/doc/configuration.md delete mode 100644 math/example.env delete mode 100755 math/misc/silh_test.R delete mode 100644 math/perf rename {delphi => math}/polismath/__init__.py (100%) rename {delphi => math}/polismath/benchmarks/__init__.py (100%) rename {delphi => math}/polismath/benchmarks/bench_pca.py (100%) rename {delphi => math}/polismath/benchmarks/bench_repness.py (100%) rename {delphi => math}/polismath/benchmarks/bench_update_votes.py (100%) rename {delphi => math}/polismath/benchmarks/benchmark_utils.py (100%) rename {delphi => math}/polismath/components/__init__.py (100%) rename {delphi => math}/polismath/components/config.py (100%) rename {delphi => math}/polismath/conversation/__init__.py (100%) rename {delphi => math}/polismath/conversation/conversation.py (100%) rename {delphi => math}/polismath/conversation/manager.py (100%) rename {delphi => math}/polismath/database/__init__.py (100%) rename {delphi => math}/polismath/database/dynamodb.py (100%) rename {delphi => math}/polismath/database/postgres.py (100%) create mode 100644 math/polismath/paths.py rename {delphi => math}/polismath/pca_kmeans_rep/__init__.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/clusters.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/corr.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/group_k_smoother.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/legacy_kmeans.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/pca.py (100%) rename {delphi => math}/polismath/pca_kmeans_rep/repness.py (100%) rename {delphi => math}/polismath/poller/__init__.py (98%) rename delphi/scripts/math_poller.py => math/polismath/poller/__main__.py (91%) rename {delphi => math}/polismath/poller/math_writer.py (100%) rename {delphi => math}/polismath/poller/service.py (100%) rename {delphi => math}/polismath/poller/worker_pool.py (100%) rename math/{tmp/.touchfile => polismath/py.typed} (100%) rename {delphi => math}/polismath/regression/__init__.py (100%) rename {delphi => math}/polismath/regression/clojure_comparer.py (100%) rename {delphi => math}/polismath/regression/comparer.py (99%) rename {delphi => math}/polismath/regression/datasets.py (98%) rename {delphi => math}/polismath/regression/recorder.py (100%) rename {delphi => math}/polismath/regression/utils.py (98%) rename {delphi => math}/polismath/replay/__init__.py (100%) rename {delphi => math}/polismath/replay/certify.py (94%) rename {delphi => math}/polismath/replay/crosslang.py (100%) rename {delphi => math}/polismath/replay/driver.py (100%) rename {delphi => math}/polismath/replay/poller_equiv.py (99%) rename {delphi => math}/polismath/replay/prodclone.py (100%) rename {delphi => math}/polismath/replay/real_data.py (97%) rename {delphi => math}/polismath/replay/schedule.py (100%) create mode 100644 math/polismath/replay/shadow_compare.py rename {delphi => math}/polismath/replay/shard_bench.py (99%) rename {delphi => math}/polismath/replay/stepcompare.py (100%) rename {delphi => math}/polismath/replay/store.py (97%) rename {delphi => math}/polismath/replay/types.py (100%) rename {delphi => math}/polismath/run_math_pipeline.py (100%) rename {delphi => math}/polismath/utils/__init__.py (100%) rename {delphi => math}/polismath/utils/clj_hash.py (100%) rename {delphi => math}/polismath/utils/env_flags.py (100%) rename {delphi => math}/polismath/utils/general.py (100%) rename {delphi => math}/polismath/utils/serialization.py (100%) create mode 100644 math/pyproject.toml delete mode 100644 math/python_conversion/.gitignore delete mode 100644 math/resources/config.edn delete mode 100644 math/src/data_readers.clj delete mode 100644 math/src/polismath/components/config.clj delete mode 100644 math/src/polismath/components/core_matrix_boot.clj delete mode 100644 math/src/polismath/components/env.clj delete mode 100644 math/src/polismath/components/logger.clj delete mode 100644 math/src/polismath/components/postgres.clj delete mode 100644 math/src/polismath/components/random.clj delete mode 100644 math/src/polismath/components/server.clj delete mode 100644 math/src/polismath/conv_man.clj delete mode 100644 math/src/polismath/darwin/core.clj delete mode 100644 math/src/polismath/darwin/export.clj delete mode 100644 math/src/polismath/math/clusters.clj delete mode 100644 math/src/polismath/math/conversation.clj delete mode 100644 math/src/polismath/math/corr.clj delete mode 100644 math/src/polismath/math/named_matrix.clj delete mode 100644 math/src/polismath/math/pca.clj delete mode 100644 math/src/polismath/math/repness.clj delete mode 100644 math/src/polismath/math/stats.clj delete mode 100644 math/src/polismath/meta/metrics.clj delete mode 100644 math/src/polismath/meta/microscope.clj delete mode 100644 math/src/polismath/meta/notify.clj delete mode 100644 math/src/polismath/meta/simulation.clj delete mode 100644 math/src/polismath/poller.clj delete mode 100644 math/src/polismath/runner.clj delete mode 100644 math/src/polismath/stormspec.clj delete mode 100644 math/src/polismath/system.clj delete mode 100644 math/src/polismath/tasks.clj delete mode 100644 math/src/polismath/util/pretty_printers.clj delete mode 100644 math/src/polismath/utils.clj delete mode 100644 math/system.properties delete mode 100644 math/test/cluster_tests.clj delete mode 100644 math/test/conv_edge_cases_test.clj delete mode 100644 math/test/conv_man_tests.clj delete mode 100644 math/test/conversation_test.clj delete mode 100644 math/test/index_hash_test.clj delete mode 100644 math/test/named_matrix_test.clj delete mode 100644 math/test/pca_test.clj delete mode 100644 math/test/ptpt_stats_test.clj delete mode 100644 math/test/silhouette_test.clj delete mode 100644 math/test/stats_test.clj delete mode 100644 math/test/test_helpers.clj delete mode 100644 math/test/test_runner.clj delete mode 100644 math/test/utils_test.clj diff --git a/.github/workflows/python-ci.yml b/.github/workflows/python-ci.yml index 048cd0cc76..68b2b0a132 100644 --- a/.github/workflows/python-ci.yml +++ b/.github/workflows/python-ci.yml @@ -10,7 +10,11 @@ on: paths: &paths - 'delphi/**/*.py' - 'delphi/requirements*.txt' + - 'delphi/requirements.lock' + - 'delphi/pyproject.toml' - 'delphi/Dockerfile' + - 'math/**/*.py' + - 'math/pyproject.toml' - '.github/workflows/python-ci.yml' pull_request: branches: @@ -97,8 +101,9 @@ jobs: docker compose -f docker-compose.test.yml cp delphi/real_data delphi:/app/real_data echo "Copying coverage script into container..." docker compose -f docker-compose.test.yml cp delphi/generate_coverage_md.py delphi:/app/generate_coverage_md.py - echo "Copying script to be tested into container..." - docker compose -f docker-compose.test.yml cp delphi/polismath/run_math_pipeline.py delphi:/app/run_math_pipeline.py + # (no run_math_pipeline.py cp: since the math package move it is a + # polismath submodule — installed in the image, covered by + # --cov=polismath; nothing imports a top-level copy.) docker compose -f docker-compose.test.yml cp delphi/umap_narrative delphi:/app/umap_narrative echo "Running tests and generating coverage report..." @@ -120,7 +125,7 @@ jobs: echo '--- Running Pytest ---'; \ export PYTHONPATH=\$PYTHONPATH:/app; \ export POLIS_TEST_POSTGRES_URL=\"postgresql://\$DATABASE_USER:\$DATABASE_PASSWORD@\$DATABASE_HOST/\$DATABASE_NAME\"; \ - pytest --cov=polismath --cov=run_math_pipeline --cov=./umap_narrative --cov-report=xml:/app/coverage.xml /app/tests --ignore=/app/tests/test_pakistan_conversation.py + pytest --cov=polismath --cov=./umap_narrative --cov-report=xml:/app/coverage.xml /app/tests --ignore=/app/tests/test_pakistan_conversation.py echo '--- Generating Coverage Comment Text ---'; \ python /app/generate_coverage_md.py > /app/coverage-comment.md \ " diff --git a/.github/workflows/test-clojure.yml b/.github/workflows/test-clojure.yml deleted file mode 100644 index 8a57ebc640..0000000000 --- a/.github/workflows/test-clojure.yml +++ /dev/null @@ -1,40 +0,0 @@ -name: Test Math - -concurrency: - group: test-clojure-${{ github.ref }} - cancel-in-progress: true - -on: - push: - branches: ["dev", "504-clj-tests"] - # Note: Only configured for client-admin right now. - paths: - - .github/workflows/test-clojure.yml - - math/** - pull_request: - types: ["opened", "reopened", "synchronize"] - paths: - - .github/workflows/test-clojure.yml - - math/** - -jobs: - test-clj: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Prepare java - uses: actions/setup-java@v4 - with: - distribution: adopt - java-version: 16.0.2 - - - name: Install Clojure tooling - uses: DeLaGuardo/setup-clojure@12.5 - with: - cli: 1.10.1.693 - - - name: Run Clojure tests - working-directory: math - # Runs all except integration tests which require database and setup/teardown - run: "clojure -M:test" diff --git a/delphi/Dockerfile b/delphi/Dockerfile index e34bbef3e0..19cd3136df 100644 --- a/delphi/Dockerfile +++ b/delphi/Dockerfile @@ -54,15 +54,24 @@ RUN --mount=type=cache,target=/root/.cache/uv \ # ===== OPTIMIZATION: Copy source code LAST (busts cache on code changes) ===== # Copy source code - this layer rebuilds when code changes but reuses dependency layers above - COPY polismath/ ./polismath/ COPY umap_narrative/ ./umap_narrative/ COPY scripts/ ./scripts/ COPY *.py ./ - # Install the project package (without dependencies - they're already installed) - # This registers entry points and installs the package in development mode + # The math engine lives OUTSIDE this build context (/math, cutover + # Step #4) and arrives via a named additional build context — compose + # declares `additional_contexts: mathsrc: ./math` on every service that + # builds this Dockerfile. Its deps are a subset of the requirements.lock + # installed above, so --no-deps is safe. + COPY --from=mathsrc . /math-src + + # Install both project packages (without dependencies - they're already + # installed). This registers entry points; polismath installs from the + # sibling source tree, NEVER from an index (supply-chain: the PyPI name + # "polismath" is not ours). RUN --mount=type=cache,target=/root/.cache/uv \ - uv pip install --no-deps . + uv pip install --no-deps . && \ + uv pip install --no-deps /math-src RUN echo "--- PyTorch Check (after pyproject.toml installation) ---" && \ pip show torch torchvision torchaudio && \ @@ -90,8 +99,8 @@ WORKDIR /app COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY --from=builder /usr/local/bin /usr/local/bin -# Copy source code from builder stage (already built into the package) -COPY --from=builder /app/polismath/ ./polismath/ +# Copy source code from builder stage (already built into the package; +# polismath ships via site-packages above — no /app/polismath dir anymore) COPY --from=builder /app/scripts/ ./scripts/ COPY --from=builder /app/umap_narrative/ ./umap_narrative/ COPY --from=builder /app/*.py ./ @@ -144,9 +153,14 @@ COPY --from=builder /opt/uv/uv /usr/local/bin/uv ENV UV_SYSTEM_PYTHON=1 COPY pyproject.toml . -# Install dev dependencies (pytest, etc.) using caching +# Install dev dependencies (pytest, etc.) using caching. The bare +# `polismath` requirement is satisfied by the copy already installed in +# the `final` stage this extends — assert that survived so a future +# --upgrade/--reinstall can never silently pull the foreign PyPI +# "polismath" package instead. RUN --mount=type=cache,target=/root/.cache/uv \ - uv pip install ".[dev]" + uv pip install ".[dev]" && \ + python -c "import polismath.poller" # Default command for test container (can be overridden) CMD ["tail", "-f", "/dev/null"] diff --git a/delphi/pyproject.toml b/delphi/pyproject.toml index 2db8c3986c..9582240ebc 100644 --- a/delphi/pyproject.toml +++ b/delphi/pyproject.toml @@ -9,6 +9,9 @@ description = "Mathematical analytics pipeline for Polis conversations" requires-python = ">=3.12,<3.13" # Pin to Python 3.12.x dependencies = [ + # The math engine — its own package at /math since cutover Step #4, + # installed editable from the sibling tree (see [tool.uv.sources]). + "polismath", "numpy>=1.26.4,<2.0", "pandas>=2.1.4", "sqlalchemy>=2.0.36", @@ -101,8 +104,8 @@ notebook = [ [project.scripts] delphi = "scripts.delphi_cli:main" run-delphi = "run_delphi:main" -# Pipeline entry points -run-math-pipeline = "polismath.run_math_pipeline:main" +# Pipeline entry points (run-math-pipeline moved to math/pyproject.toml +# with the polismath package, cutover Step #4) run-umap-pipeline = "umap_narrative.run_pipeline:main" calculate-extremity = "umap_narrative.501_calculate_comment_extremity:main" calculate-priorities = "umap_narrative.502_calculate_priorities:main" @@ -111,13 +114,17 @@ reset-conversation = "umap_narrative.reset_conversation:main" create-datamapplot = "umap_narrative.700_datamapplot_for_layer:main" setup-minio = "setup_minio:setup_minio_bucket" +# uv: resolve the polismath dependency from the sibling math/ tree, +# editable — one shared environment (delphi/.venv) for both packages. +[tool.uv.sources] +polismath = { path = "../math", editable = true } + # Hatchling configuration [tool.hatch.build.targets.wheel] -packages = ["polismath", "umap_narrative"] +packages = ["umap_narrative"] [tool.hatch.build.targets.sdist] include = [ - "/polismath", "/umap_narrative", "/scripts", "/tests", @@ -161,7 +168,10 @@ relative_files = true # maintain this). The mirror `polis/.venv` (also real or symlink) is what lets # editors opening the workspace at the repo root auto-discover the interpreter. [tool.pyright] -include = ["polismath", "umap_narrative", "tests", "scripts"] +# polismath moved to /math (cutover Step #4) and has its own +# [tool.pyright] there; here we cover the delphi tree, which imports it +# from the shared .venv (editable install). +include = ["umap_narrative", "tests", "scripts"] exclude = [ "**/__pycache__", "**/.pytest_cache", diff --git a/delphi/run_delphi.py b/delphi/run_delphi.py index 76c156c78a..6ceacd887f 100644 --- a/delphi/run_delphi.py +++ b/delphi/run_delphi.py @@ -97,8 +97,10 @@ def main(): # Run the math pipeline print(f"{GREEN}Running math pipeline...{NC}") + # polismath is an installed package (its source moved to /math at + # cutover Step #4) — invoke the pipeline as a module, not by file path. math_command = [ - "python", f"{app_path}/polismath/run_math_pipeline.py", + "python", "-m", "polismath.run_math_pipeline", f"--zid={zid}", ] if max_votes_arg: diff --git a/delphi/scripts/prodclone_extract.py b/delphi/scripts/prodclone_extract.py index dd4823a864..b180417a4e 100644 --- a/delphi/scripts/prodclone_extract.py +++ b/delphi/scripts/prodclone_extract.py @@ -3,7 +3,7 @@ a "prodclone" Postgres database (a clone of the production polis DB) into the replay-dataset export format, for Clojure↔Python math parity certification. -See ``delphi/polismath/replay/prodclone.py`` for the pure building blocks +See ``math/polismath/replay/prodclone.py`` for the pure building blocks (SQL builders, feature classifiers, CSV formatters, slug minting, the path-safety guard). This script is a thin click CLI wiring those together with a live psycopg2 connection — mirrors the style of diff --git a/delphi/tests/replay_harness/test_certify.py b/delphi/tests/replay_harness/test_certify.py index 0823bba2bd..fcaed7b32a 100644 --- a/delphi/tests/replay_harness/test_certify.py +++ b/delphi/tests/replay_harness/test_certify.py @@ -207,7 +207,7 @@ def test_sha256_tree_exclude_file_and_dir_prefix(tmp_path): def test_engine_tree_exclude_entries_exist_and_keep_replay_shapers(): """Every exclusion names a real path under polismath/ (a rename must not turn it into a silent no-op), and the replay-shaping files stay hashed.""" - pm = cert._DELPHI_ROOT / "polismath" + pm = cert.paths.PACKAGE_ROOT for e in cert._ENGINE_TREE_EXCLUDE: p = pm / e.rstrip("/") if e.endswith("/"): @@ -219,6 +219,71 @@ def test_engine_tree_exclude_entries_exist_and_keep_replay_shapers(): assert not kept & set(cert._ENGINE_TREE_EXCLUDE) +def test_clj_source_hashes_missing_oracle_raises_certify_error(tmp_path, monkeypatch): + """Post-Step-4 the Clojure oracle tree is absent from the working tree: + the clj cache-key hash must fail with pointed restore instructions + (CertifyError, stage 'clj-oracle') — not a bare FileNotFoundError.""" + monkeypatch.setattr(cert, "_MATH_ROOT", tmp_path / "math") + cert._clj_source_hashes.cache_clear() + try: + with pytest.raises(cert.CertifyError, match="git history"): + cert._clj_source_hashes() + finally: + cert._clj_source_hashes.cache_clear() + + +class TestEnsureCljRecordingWithoutOracle: + """Cached pairs MUST survive the oracle tree's removal (Step 4): a + recording whose INPUT keys (votes/schedule) match its manifest is + trusted as-is; only an actual re-replay demands the restored tree.""" + + def _spec_and_entry(self): + entry = _make_entry() + spec = sched.preset_single_cut("vw", 100, schedule_id=entry.schedule_id) + return entry, spec + + def _write_cached_recording(self, root, entry, spec, votes_sha): + clj_dir = cert.st.recording_dir(entry.dataset, entry.schedule_id, root=root) / "clj" + clj_dir.mkdir(parents=True) + cert._write_manifest(clj_dir / "cache_manifest.json", { + "votes_sha256": votes_sha, + "schedule_hash": cert.canonical_schedule_hash(spec), + "replay_clj_sha256": "recorded-against-this-tree", + "math_src_sha256": "recorded-against-this-tree", + }) + return clj_dir + + def test_cached_recording_is_used_without_oracle(self, tmp_path, monkeypatch): + monkeypatch.setattr(cert, "_MATH_ROOT", tmp_path / "no-oracle") + entry, spec = self._spec_and_entry() + root = tmp_path / "root" + expected_dir = self._write_cached_recording(root, entry, spec, "sha") + clj_dir, cached = cert.ensure_clj_recording( + entry, spec, "sha", tmp_path / "votes.csv", root=root + ) + assert cached is True and clj_dir == expected_dir + + def test_cache_miss_without_oracle_raises_clj_oracle(self, tmp_path, monkeypatch): + monkeypatch.setattr(cert, "_MATH_ROOT", tmp_path / "no-oracle") + entry, spec = self._spec_and_entry() + root = tmp_path / "root" + self._write_cached_recording(root, entry, spec, "sha") + with pytest.raises(cert.CertifyError, match="git history"): + cert.ensure_clj_recording( + entry, spec, "DIFFERENT-sha", tmp_path / "votes.csv", root=root + ) + + def test_refresh_without_oracle_raises_clj_oracle(self, tmp_path, monkeypatch): + monkeypatch.setattr(cert, "_MATH_ROOT", tmp_path / "no-oracle") + entry, spec = self._spec_and_entry() + root = tmp_path / "root" + self._write_cached_recording(root, entry, spec, "sha") + with pytest.raises(cert.CertifyError, match="git history"): + cert.ensure_clj_recording( + entry, spec, "sha", tmp_path / "votes.csv", root=root, refresh=True + ) + + def test_engine_tree_hash_ignores_harness_edits_sees_engine_edits(tmp_path): pm = tmp_path / "polismath" for rel in ("replay/certify.py", "replay/prodclone.py", "replay/driver.py", @@ -256,7 +321,7 @@ def fake_run(cmd, *, cwd, env): (pm / "replay").mkdir(parents=True) (pm / "replay" / "certify.py").write_text("h = 1\n") (pm / "replay" / "driver.py").write_text("d = 1\n") - monkeypatch.setattr(cert, "_DELPHI_ROOT", fake_delphi) + monkeypatch.setattr(cert, "_ENGINE_TREE_ROOT", pm) cert._engine_tree_hash_cached.cache_clear() try: entry = _make_entry() @@ -815,6 +880,9 @@ def _seed_cached_pair(root: Path, ds: str, sid: str, *, divergent: bool) -> None def test_run_battery_parallel_matches_serial_report_and_ledger(tmp_path, monkeypatch): monkeypatch.setattr(cert, "_manifest_matches", lambda mp, exp: True) + # Oracle-absent working trees (post-Step-4) consult the input-keys + # seam before the full-manifest one — fake both cache hits. + monkeypatch.setattr(cert, "_manifest_matches_inputs", lambda mp, keys: True) monkeypatch.setattr(cert, "_clj_source_hashes", lambda: ("x", "y")) entries = [ @@ -843,6 +911,9 @@ def test_run_battery_parallel_matches_serial_report_and_ledger(tmp_path, monkeyp def test_run_battery_workers_one_is_default_and_identical(tmp_path, monkeypatch): monkeypatch.setattr(cert, "_manifest_matches", lambda mp, exp: True) + # Oracle-absent working trees (post-Step-4) consult the input-keys + # seam before the full-manifest one — fake both cache hits. + monkeypatch.setattr(cert, "_manifest_matches_inputs", lambda mp, keys: True) monkeypatch.setattr(cert, "_clj_source_hashes", lambda: ("x", "y")) entries = [_make_entry(schedule_id="w1-only")] diff --git a/delphi/tests/replay_harness/test_poller_equiv_seed.py b/delphi/tests/replay_harness/test_poller_equiv_seed.py index afcc25ce4e..a62aa26adf 100644 --- a/delphi/tests/replay_harness/test_poller_equiv_seed.py +++ b/delphi/tests/replay_harness/test_poller_equiv_seed.py @@ -558,7 +558,7 @@ def test_py_poller_runner_cmd_cwd_env(self): database_url="postgresql://x/polis_equiv", math_env="py-shadow", base_env={}, ) - assert runner.cmd == ["uv", "run", "python", "scripts/math_poller.py"] + assert runner.cmd == ["uv", "run", "python", "-m", "polismath.poller"] # Equality with _DELPHI_ROOT is the contract; asserting the directory # NAME was layout-fragile — CI mounts the delphi tree at /app, where # .name == "app" (python-ci run 30071088647, 2026-07-24). diff --git a/delphi/tests/test_dynamodb_consensus_roundtrip.py b/delphi/tests/test_dynamodb_consensus_roundtrip.py index 1d89039cff..9fad71e9b0 100644 --- a/delphi/tests/test_dynamodb_consensus_roundtrip.py +++ b/delphi/tests/test_dynamodb_consensus_roundtrip.py @@ -1,5 +1,5 @@ """ -Tests for D11 cascade fix in `delphi/polismath/database/dynamodb.py`. +Tests for D11 cascade fix in `math/polismath/database/dynamodb.py`. Investigation B (2026-06-11) found three sites in the DynamoDB writer/reader that either dropped the new D11 `consensus_comments` dict shape diff --git a/delphi/tests/test_prodclone_extract.py b/delphi/tests/test_prodclone_extract.py index fb7d4fc7b0..cee5865f55 100644 --- a/delphi/tests/test_prodclone_extract.py +++ b/delphi/tests/test_prodclone_extract.py @@ -1,5 +1,5 @@ """Tests for the prodclone extractor (delphi/scripts/prodclone_extract.py + -delphi/polismath/replay/prodclone.py). +math/polismath/replay/prodclone.py). Unit tests exercise the PURE building blocks (SQL builders, feature classifiers, CSV row formatters, slug minting, path-safety guard, map diff --git a/docker-compose.test.yml b/docker-compose.test.yml index f908cb4ed3..7ef272404d 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -62,12 +62,14 @@ services: build: context: ./delphi target: final + additional_contexts: + mathsrc: ./math # Share every layer with the delphi service's test build (CPU torch); # without this the default CUDA wheels build a second multi-GB image # in CI for no benefit — the poller needs no torch at runtime. args: USE_CPU_TORCH: "true" - command: ["python", "scripts/math_poller.py"] + command: ["python", "-m", "polismath.poller"] labels: polis_tag: test environment: @@ -196,6 +198,8 @@ services: build: context: ./delphi target: test + additional_contexts: + mathsrc: ./math args: USE_CPU_TORCH: "true" labels: diff --git a/docker-compose.yml b/docker-compose.yml index 90395eccd4..8e9ead04cd 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -81,6 +81,10 @@ services: build: context: ./delphi target: final + # The math engine source (installed into the image) lives outside the + # build context since cutover Step #4 — see delphi/Dockerfile. + additional_contexts: + mathsrc: ./math labels: polis_tag: ${TAG:-dev} environment: @@ -135,9 +139,11 @@ services: build: context: ./delphi target: final + additional_contexts: + mathsrc: ./math labels: polis_tag: ${TAG:-dev} - command: ["python", "scripts/math_poller.py"] + command: ["python", "-m", "polismath.poller"] environment: - DATABASE_URL=${DATABASE_URL} - DATABASE_SSL_MODE=${DATABASE_SSL_MODE:-disable} diff --git a/math/.dockerignore b/math/.dockerignore index e386810476..beadfd052e 100644 --- a/math/.dockerignore +++ b/math/.dockerignore @@ -1,26 +1,20 @@ -/target -/lib -/classes -/checkouts -pom.xml -pom.xml.asc -*.jar -*.class -.lein-deps-sum -.lein-failures -.lein-plugins -.lein-repl-history -*.swo -db/ -.lein-env -.nrepl-port -scratch -src/polismath/play.clj -old -.gorilla-port -.lein-git-deps/hiphip -wiki -.lein-git-deps/debug-repl -data -.cpcache -.git +# This directory ships as the `mathsrc` additional build context of +# delphi/Dockerfile — keep caches, a temporarily-restored Clojure oracle +# tree (see README.md), and build litter out of the context. +__pycache__/ +*.py[cod] +.pytest_cache/ +dist/ +build/ +*.egg-info/ +# Restored-oracle entries (present only during a certify re-replay): +src/ +test/ +dev/ +resources/ +bin/ +misc/ +doc/ +data/ +tmp/ +deps.edn diff --git a/math/.gitignore b/math/.gitignore index a8c560ea1d..17ef72b87e 100644 --- a/math/.gitignore +++ b/math/.gitignore @@ -1,58 +1,8 @@ -/target -/lib -/classes -/checkouts -pom.xml -pom.xml.asc -*.jar -*.class -.lein-deps-sum -.lein-failures -.lein-plugins -.lein-repl-history -*.swo -db/ -.lein-env -.nrepl-port -scratch -src/polismath/play.clj -old -.gorilla-port -.lein-git-deps/hiphip -wiki -.lein-git-deps/debug-repl -data -.cpcache -errorconv* - -# Python-specific entries +# Python build/test litter — this tree is unignored at repo level, and in a +# jj-colocated checkout unignored files get auto-snapshotted into @. __pycache__/ *.py[cod] -*$py.class -*.so -.Python -build/ -develop-eggs/ +.pytest_cache/ dist/ -downloads/ -eggs/ -.eggs/ -lib/ -lib64/ -parts/ -sdist/ -var/ -wheels/ +build/ *.egg-info/ -.installed.cfg -*.egg -.venv/ -venv/ -ENV/ -.env - -# Jupyter Notebook -.ipynb_checkpoints -*/.ipynb_checkpoints/* - -real_data \ No newline at end of file diff --git a/math/.nrepl.edn b/math/.nrepl.edn deleted file mode 100644 index b97fa14cec..0000000000 --- a/math/.nrepl.edn +++ /dev/null @@ -1,2 +0,0 @@ -{:port 18975 - :bind "0.0.0.0"} diff --git a/math/CONTRIBUTING.md b/math/CONTRIBUTING.md deleted file mode 100644 index c0f71ec241..0000000000 --- a/math/CONTRIBUTING.md +++ /dev/null @@ -1,4 +0,0 @@ - -# Submitting Pull Requests - -Before we can accept pull requests, you'll need you to sign the Polis Contributor Agreement. https://pol.is/contrib diff --git a/math/Dockerfile b/math/Dockerfile deleted file mode 100644 index 81f431152b..0000000000 --- a/math/Dockerfile +++ /dev/null @@ -1,9 +0,0 @@ -FROM docker.io/clojure:temurin-17-tools-deps - -WORKDIR /app -COPY . . - -# Install clojure and fetch dependencies -RUN clojure -A:dev -P - -CMD ["./bin/run"] diff --git a/math/LICENSE b/math/LICENSE deleted file mode 100644 index 6871a5e85c..0000000000 --- a/math/LICENSE +++ /dev/null @@ -1,777 +0,0 @@ - GNU AFFERO GENERAL PUBLIC LICENSE - Version 3, 19 November 2007 - - Copyright (C) 2007 Free Software Foundation, Inc. - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. - - Preamble - - The GNU Affero General Public License is a free, copyleft license for -software and other kinds of works, specifically designed to ensure -cooperation with the community in the case of network server software. - - The licenses for most software and other practical works are designed -to take away your freedom to share and change the works. By contrast, -our General Public Licenses are intended to guarantee your freedom to -share and change all versions of a program--to make sure it remains free -software for all its users. - - When we speak of free software, we are referring to freedom, not -price. Our General Public Licenses are designed to make sure that you -have the freedom to distribute copies of free software (and charge for -them if you wish), that you receive source code or can get it if you -want it, that you can change the software or use pieces of it in new -free programs, and that you know you can do these things. - - Developers that use our General Public Licenses protect your rights -with two steps: (1) assert copyright on the software, and (2) offer -you this License which gives you legal permission to copy, distribute -and/or modify the software. - - A secondary benefit of defending all users' freedom is that -improvements made in alternate versions of the program, if they -receive widespread use, become available for other developers to -incorporate. Many developers of free software are heartened and -encouraged by the resulting cooperation. However, in the case of -software used on network servers, this result may fail to come about. -The GNU General Public License permits making a modified version and -letting the public access it on a server without ever releasing its -source code to the public. - - The GNU Affero General Public License is designed specifically to -ensure that, in such cases, the modified source code becomes available -to the community. It requires the operator of a network server to -provide the source code of the modified version running there to the -users of that server. Therefore, public use of a modified version, on -a publicly accessible server, gives the public access to the source -code of the modified version. - - An older license, called the Affero General Public License and -published by Affero, was designed to accomplish similar goals. This is -a different license, not a version of the Affero GPL, but Affero has -released a new version of the Affero GPL which permits relicensing under -this license. - - The precise terms and conditions for copying, distribution and -modification follow. - - TERMS AND CONDITIONS - - 0. Definitions. - - "This License" refers to version 3 of the GNU Affero General Public License. - - "Copyright" also means copyright-like laws that apply to other kinds of -works, such as semiconductor masks. - - "The Program" refers to any copyrightable work licensed under this -License. Each licensee is addressed as "you". "Licensees" and -"recipients" may be individuals or organizations. - - To "modify" a work means to copy from or adapt all or part of the work -in a fashion requiring copyright permission, other than the making of an -exact copy. The resulting work is called a "modified version" of the -earlier work or a work "based on" the earlier work. - - A "covered work" means either the unmodified Program or a work based -on the Program. - - To "propagate" a work means to do anything with it that, without -permission, would make you directly or secondarily liable for -infringement under applicable copyright law, except executing it on a -computer or modifying a private copy. Propagation includes copying, -distribution (with or without modification), making available to the -public, and in some countries other activities as well. - - To "convey" a work means any kind of propagation that enables other -parties to make or receive copies. Mere interaction with a user through -a computer network, with no transfer of a copy, is not conveying. - - An interactive user interface displays "Appropriate Legal Notices" -to the extent that it includes a convenient and prominently visible -feature that (1) displays an appropriate copyright notice, and (2) -tells the user that there is no warranty for the work (except to the -extent that warranties are provided), that licensees may convey the -work under this License, and how to view a copy of this License. If -the interface presents a list of user commands or options, such as a -menu, a prominent item in the list meets this criterion. - - 1. Source Code. - - The "source code" for a work means the preferred form of the work -for making modifications to it. "Object code" means any non-source -form of a work. - - A "Standard Interface" means an interface that either is an official -standard defined by a recognized standards body, or, in the case of -interfaces specified for a particular programming language, one that -is widely used among developers working in that language. - - The "System Libraries" of an executable work include anything, other -than the work as a whole, that (a) is included in the normal form of -packaging a Major Component, but which is not part of that Major -Component, and (b) serves only to enable use of the work with that -Major Component, or to implement a Standard Interface for which an -implementation is available to the public in source code form. A -"Major Component", in this context, means a major essential component -(kernel, window system, and so on) of the specific operating system -(if any) on which the executable work runs, or a compiler used to -produce the work, or an object code interpreter used to run it. - - The "Corresponding Source" for a work in object code form means all -the source code needed to generate, install, and (for an executable -work) run the object code and to modify the work, including scripts to -control those activities. However, it does not include the work's -System Libraries, or general-purpose tools or generally available free -programs which are used unmodified in performing those activities but -which are not part of the work. For example, Corresponding Source -includes interface definition files associated with source files for -the work, and the source code for shared libraries and dynamically -linked subprograms that the work is specifically designed to require, -such as by intimate data communication or control flow between those -subprograms and other parts of the work. - - The Corresponding Source need not include anything that users -can regenerate automatically from other parts of the Corresponding -Source. - - The Corresponding Source for a work in source code form is that -same work. - - 2. Basic Permissions. - - All rights granted under this License are granted for the term of -copyright on the Program, and are irrevocable provided the stated -conditions are met. This License explicitly affirms your unlimited -permission to run the unmodified Program. The output from running a -covered work is covered by this License only if the output, given its -content, constitutes a covered work. This License acknowledges your -rights of fair use or other equivalent, as provided by copyright law. - - You may make, run and propagate covered works that you do not -convey, without conditions so long as your license otherwise remains -in force. You may convey covered works to others for the sole purpose -of having them make modifications exclusively for you, or provide you -with facilities for running those works, provided that you comply with -the terms of this License in conveying all material for which you do -not control copyright. Those thus making or running the covered works -for you must do so exclusively on your behalf, under your direction -and control, on terms that prohibit them from making any copies of -your copyrighted material outside their relationship with you. - - Conveying under any other circumstances is permitted solely under -the conditions stated below. Sublicensing is not allowed; section 10 -makes it unnecessary. - - 3. Protecting Users' Legal Rights From Anti-Circumvention Law. - - No covered work shall be deemed part of an effective technological -measure under any applicable law fulfilling obligations under article -11 of the WIPO copyright treaty adopted on 20 December 1996, or -similar laws prohibiting or restricting circumvention of such -measures. - - When you convey a covered work, you waive any legal power to forbid -circumvention of technological measures to the extent such circumvention -is effected by exercising rights under this License with respect to -the covered work, and you disclaim any intention to limit operation or -modification of the work as a means of enforcing, against the work's -users, your or third parties' legal rights to forbid circumvention of -technological measures. - - 4. Conveying Verbatim Copies. - - You may convey verbatim copies of the Program's source code as you -receive it, in any medium, provided that you conspicuously and -appropriately publish on each copy an appropriate copyright notice; -keep intact all notices stating that this License and any -non-permissive terms added in accord with section 7 apply to the code; -keep intact all notices of the absence of any warranty; and give all -recipients a copy of this License along with the Program. - - You may charge any price or no price for each copy that you convey, -and you may offer support or warranty protection for a fee. - - 5. Conveying Modified Source Versions. - - You may convey a work based on the Program, or the modifications to -produce it from the Program, in the form of source code under the -terms of section 4, provided that you also meet all of these conditions: - - a) The work must carry prominent notices stating that you modified - it, and giving a relevant date. - - b) The work must carry prominent notices stating that it is - released under this License and any conditions added under section - 7. This requirement modifies the requirement in section 4 to - "keep intact all notices". - - c) You must license the entire work, as a whole, under this - License to anyone who comes into possession of a copy. This - License will therefore apply, along with any applicable section 7 - additional terms, to the whole of the work, and all its parts, - regardless of how they are packaged. This License gives no - permission to license the work in any other way, but it does not - invalidate such permission if you have separately received it. - - d) If the work has interactive user interfaces, each must display - Appropriate Legal Notices; however, if the Program has interactive - interfaces that do not display Appropriate Legal Notices, your - work need not make them do so. - - A compilation of a covered work with other separate and independent -works, which are not by their nature extensions of the covered work, -and which are not combined with it such as to form a larger program, -in or on a volume of a storage or distribution medium, is called an -"aggregate" if the compilation and its resulting copyright are not -used to limit the access or legal rights of the compilation's users -beyond what the individual works permit. Inclusion of a covered work -in an aggregate does not cause this License to apply to the other -parts of the aggregate. - - 6. Conveying Non-Source Forms. - - You may convey a covered work in object code form under the terms -of sections 4 and 5, provided that you also convey the -machine-readable Corresponding Source under the terms of this License, -in one of these ways: - - a) Convey the object code in, or embodied in, a physical product - (including a physical distribution medium), accompanied by the - Corresponding Source fixed on a durable physical medium - customarily used for software interchange. - - b) Convey the object code in, or embodied in, a physical product - (including a physical distribution medium), accompanied by a - written offer, valid for at least three years and valid for as - long as you offer spare parts or customer support for that product - model, to give anyone who possesses the object code either (1) a - copy of the Corresponding Source for all the software in the - product that is covered by this License, on a durable physical - medium customarily used for software interchange, for a price no - more than your reasonable cost of physically performing this - conveying of source, or (2) access to copy the - Corresponding Source from a network server at no charge. - - c) Convey individual copies of the object code with a copy of the - written offer to provide the Corresponding Source. This - alternative is allowed only occasionally and noncommercially, and - only if you received the object code with such an offer, in accord - with subsection 6b. - - d) Convey the object code by offering access from a designated - place (gratis or for a charge), and offer equivalent access to the - Corresponding Source in the same way through the same place at no - further charge. You need not require recipients to copy the - Corresponding Source along with the object code. If the place to - copy the object code is a network server, the Corresponding Source - may be on a different server (operated by you or a third party) - that supports equivalent copying facilities, provided you maintain - clear directions next to the object code saying where to find the - Corresponding Source. Regardless of what server hosts the - Corresponding Source, you remain obligated to ensure that it is - available for as long as needed to satisfy these requirements. - - e) Convey the object code using peer-to-peer transmission, provided - you inform other peers where the object code and Corresponding - Source of the work are being offered to the general public at no - charge under subsection 6d. - - A separable portion of the object code, whose source code is excluded -from the Corresponding Source as a System Library, need not be -included in conveying the object code work. - - A "User Product" is either (1) a "consumer product", which means any -tangible personal property which is normally used for personal, family, -or household purposes, or (2) anything designed or sold for incorporation -into a dwelling. In determining whether a product is a consumer product, -doubtful cases shall be resolved in favor of coverage. For a particular -product received by a particular user, "normally used" refers to a -typical or common use of that class of product, regardless of the status -of the particular user or of the way in which the particular user -actually uses, or expects or is expected to use, the product. A product -is a consumer product regardless of whether the product has substantial -commercial, industrial or non-consumer uses, unless such uses represent -the only significant mode of use of the product. - - "Installation Information" for a User Product means any methods, -procedures, authorization keys, or other information required to install -and execute modified versions of a covered work in that User Product from -a modified version of its Corresponding Source. The information must -suffice to ensure that the continued functioning of the modified object -code is in no case prevented or interfered with solely because -modification has been made. - - If you convey an object code work under this section in, or with, or -specifically for use in, a User Product, and the conveying occurs as -part of a transaction in which the right of possession and use of the -User Product is transferred to the recipient in perpetuity or for a -fixed term (regardless of how the transaction is characterized), the -Corresponding Source conveyed under this section must be accompanied -by the Installation Information. But this requirement does not apply -if neither you nor any third party retains the ability to install -modified object code on the User Product (for example, the work has -been installed in ROM). - - The requirement to provide Installation Information does not include a -requirement to continue to provide support service, warranty, or updates -for a work that has been modified or installed by the recipient, or for -the User Product in which it has been modified or installed. Access to a -network may be denied when the modification itself materially and -adversely affects the operation of the network or violates the rules and -protocols for communication across the network. - - Corresponding Source conveyed, and Installation Information provided, -in accord with this section must be in a format that is publicly -documented (and with an implementation available to the public in -source code form), and must require no special password or key for -unpacking, reading or copying. - - 7. Additional Terms. - - "Additional permissions" are terms that supplement the terms of this -License by making exceptions from one or more of its conditions. -Additional permissions that are applicable to the entire Program shall -be treated as though they were included in this License, to the extent -that they are valid under applicable law. If additional permissions -apply only to part of the Program, that part may be used separately -under those permissions, but the entire Program remains governed by -this License without regard to the additional permissions. - - When you convey a copy of a covered work, you may at your option -remove any additional permissions from that copy, or from any part of -it. (Additional permissions may be written to require their own -removal in certain cases when you modify the work.) You may place -additional permissions on material, added by you to a covered work, -for which you have or can give appropriate copyright permission. - - Notwithstanding any other provision of this License, for material you -add to a covered work, you may (if authorized by the copyright holders of -that material) supplement the terms of this License with terms: - - a) Disclaiming warranty or limiting liability differently from the - terms of sections 15 and 16 of this License; or - - b) Requiring preservation of specified reasonable legal notices or - author attributions in that material or in the Appropriate Legal - Notices displayed by works containing it; or - - c) Prohibiting misrepresentation of the origin of that material, or - requiring that modified versions of such material be marked in - reasonable ways as different from the original version; or - - d) Limiting the use for publicity purposes of names of licensors or - authors of the material; or - - e) Declining to grant rights under trademark law for use of some - trade names, trademarks, or service marks; or - - f) Requiring indemnification of licensors and authors of that - material by anyone who conveys the material (or modified versions of - it) with contractual assumptions of liability to the recipient, for - any liability that these contractual assumptions directly impose on - those licensors and authors. - - All other non-permissive additional terms are considered "further -restrictions" within the meaning of section 10. If the Program as you -received it, or any part of it, contains a notice stating that it is -governed by this License along with a term that is a further -restriction, you may remove that term. If a license document contains -a further restriction but permits relicensing or conveying under this -License, you may add to a covered work material governed by the terms -of that license document, provided that the further restriction does -not survive such relicensing or conveying. - - If you add terms to a covered work in accord with this section, you -must place, in the relevant source files, a statement of the -additional terms that apply to those files, or a notice indicating -where to find the applicable terms. - - Additional terms, permissive or non-permissive, may be stated in the -form of a separately written license, or stated as exceptions; -the above requirements apply either way. - - 8. Termination. - - You may not propagate or modify a covered work except as expressly -provided under this License. Any attempt otherwise to propagate or -modify it is void, and will automatically terminate your rights under -this License (including any patent licenses granted under the third -paragraph of section 11). - - However, if you cease all violation of this License, then your -license from a particular copyright holder is reinstated (a) -provisionally, unless and until the copyright holder explicitly and -finally terminates your license, and (b) permanently, if the copyright -holder fails to notify you of the violation by some reasonable means -prior to 60 days after the cessation. - - Moreover, your license from a particular copyright holder is -reinstated permanently if the copyright holder notifies you of the -violation by some reasonable means, this is the first time you have -received notice of violation of this License (for any work) from that -copyright holder, and you cure the violation prior to 30 days after -your receipt of the notice. - - Termination of your rights under this section does not terminate the -licenses of parties who have received copies or rights from you under -this License. If your rights have been terminated and not permanently -reinstated, you do not qualify to receive new licenses for the same -material under section 10. - - 9. Acceptance Not Required for Having Copies. - - You are not required to accept this License in order to receive or -run a copy of the Program. Ancillary propagation of a covered work -occurring solely as a consequence of using peer-to-peer transmission -to receive a copy likewise does not require acceptance. However, -nothing other than this License grants you permission to propagate or -modify any covered work. These actions infringe copyright if you do -not accept this License. Therefore, by modifying or propagating a -covered work, you indicate your acceptance of this License to do so. - - 10. Automatic Licensing of Downstream Recipients. - - Each time you convey a covered work, the recipient automatically -receives a license from the original licensors, to run, modify and -propagate that work, subject to this License. You are not responsible -for enforcing compliance by third parties with this License. - - An "entity transaction" is a transaction transferring control of an -organization, or substantially all assets of one, or subdividing an -organization, or merging organizations. If propagation of a covered -work results from an entity transaction, each party to that -transaction who receives a copy of the work also receives whatever -licenses to the work the party's predecessor in interest had or could -give under the previous paragraph, plus a right to possession of the -Corresponding Source of the work from the predecessor in interest, if -the predecessor has it or can get it with reasonable efforts. - - You may not impose any further restrictions on the exercise of the -rights granted or affirmed under this License. For example, you may -not impose a license fee, royalty, or other charge for exercise of -rights granted under this License, and you may not initiate litigation -(including a cross-claim or counterclaim in a lawsuit) alleging that -any patent claim is infringed by making, using, selling, offering for -sale, or importing the Program or any portion of it. - - 11. Patents. - - A "contributor" is a copyright holder who authorizes use under this -License of the Program or a work on which the Program is based. The -work thus licensed is called the contributor's "contributor version". - - A contributor's "essential patent claims" are all patent claims -owned or controlled by the contributor, whether already acquired or -hereafter acquired, that would be infringed by some manner, permitted -by this License, of making, using, or selling its contributor version, -but do not include claims that would be infringed only as a -consequence of further modification of the contributor version. For -purposes of this definition, "control" includes the right to grant -patent sublicenses in a manner consistent with the requirements of -this License. - - Each contributor grants you a non-exclusive, worldwide, royalty-free -patent license under the contributor's essential patent claims, to -make, use, sell, offer for sale, import and otherwise run, modify and -propagate the contents of its contributor version. - - In the following three paragraphs, a "patent license" is any express -agreement or commitment, however denominated, not to enforce a patent -(such as an express permission to practice a patent or covenant not to -sue for patent infringement). To "grant" such a patent license to a -party means to make such an agreement or commitment not to enforce a -patent against the party. - - If you convey a covered work, knowingly relying on a patent license, -and the Corresponding Source of the work is not available for anyone -to copy, free of charge and under the terms of this License, through a -publicly available network server or other readily accessible means, -then you must either (1) cause the Corresponding Source to be so -available, or (2) arrange to deprive yourself of the benefit of the -patent license for this particular work, or (3) arrange, in a manner -consistent with the requirements of this License, to extend the patent -license to downstream recipients. "Knowingly relying" means you have -actual knowledge that, but for the patent license, your conveying the -covered work in a country, or your recipient's use of the covered work -in a country, would infringe one or more identifiable patents in that -country that you have reason to believe are valid. - - If, pursuant to or in connection with a single transaction or -arrangement, you convey, or propagate by procuring conveyance of, a -covered work, and grant a patent license to some of the parties -receiving the covered work authorizing them to use, propagate, modify -or convey a specific copy of the covered work, then the patent license -you grant is automatically extended to all recipients of the covered -work and works based on it. - - A patent license is "discriminatory" if it does not include within -the scope of its coverage, prohibits the exercise of, or is -conditioned on the non-exercise of one or more of the rights that are -specifically granted under this License. You may not convey a covered -work if you are a party to an arrangement with a third party that is -in the business of distributing software, under which you make payment -to the third party based on the extent of your activity of conveying -the work, and under which the third party grants, to any of the -parties who would receive the covered work from you, a discriminatory -patent license (a) in connection with copies of the covered work -conveyed by you (or copies made from those copies), or (b) primarily -for and in connection with specific products or compilations that -contain the covered work, unless you entered into that arrangement, -or that patent license was granted, prior to 28 March 2007. - - Nothing in this License shall be construed as excluding or limiting -any implied license or other defenses to infringement that may -otherwise be available to you under applicable patent law. - - 12. No Surrender of Others' Freedom. - - If conditions are imposed on you (whether by court order, agreement or -otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot convey a -covered work so as to satisfy simultaneously your obligations under this -License and any other pertinent obligations, then as a consequence you may -not convey it at all. For example, if you agree to terms that obligate you -to collect a royalty for further conveying from those to whom you convey -the Program, the only way you could satisfy both those terms and this -License would be to refrain entirely from conveying the Program. - - 13. Remote Network Interaction; Use with the GNU General Public License. - - Notwithstanding any other provision of this License, if you modify the -Program, your modified version must prominently offer all users -interacting with it remotely through a computer network (if your version -supports such interaction) an opportunity to receive the Corresponding -Source of your version by providing access to the Corresponding Source -from a network server at no charge, through some standard or customary -means of facilitating copying of software. This Corresponding Source -shall include the Corresponding Source for any work covered by version 3 -of the GNU General Public License that is incorporated pursuant to the -following paragraph. - - Notwithstanding any other provision of this License, you have -permission to link or combine any covered work with a work licensed -under version 3 of the GNU General Public License into a single -combined work, and to convey the resulting work. The terms of this -License will continue to apply to the part which is the covered work, -but the work with which it is combined will remain governed by version -3 of the GNU General Public License. - - 14. Revised Versions of this License. - - The Free Software Foundation may publish revised and/or new versions of -the GNU Affero General Public License from time to time. Such new versions -will be similar in spirit to the present version, but may differ in detail to -address new problems or concerns. - - Each version is given a distinguishing version number. If the -Program specifies that a certain numbered version of the GNU Affero General -Public License "or any later version" applies to it, you have the -option of following the terms and conditions either of that numbered -version or of any later version published by the Free Software -Foundation. If the Program does not specify a version number of the -GNU Affero General Public License, you may choose any version ever published -by the Free Software Foundation. - - If the Program specifies that a proxy can decide which future -versions of the GNU Affero General Public License can be used, that proxy's -public statement of acceptance of a version permanently authorizes you -to choose that version for the Program. - - Later license versions may give you additional or different -permissions. However, no additional obligations are imposed on any -author or copyright holder as a result of your choosing to follow a -later version. - - 15. Disclaimer of Warranty. - - THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY -APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT -HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY -OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, -THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR -PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM -IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF -ALL NECESSARY SERVICING, REPAIR OR CORRECTION. - - 16. Limitation of Liability. - - IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING -WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS -THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY -GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE -USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF -DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD -PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), -EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF -SUCH DAMAGES. - - 17. Interpretation of Sections 15 and 16. - - If the disclaimer of warranty and limitation of liability provided -above cannot be given local legal effect according to their terms, -reviewing courts shall apply local law that most closely approximates -an absolute waiver of all civil liability in connection with the -Program, unless a warranty or assumption of liability accompanies a -copy of the Program in return for a fee. - - END OF TERMS AND CONDITIONS - - How to Apply These Terms to Your New Programs - - If you develop a new program, and you want it to be of the greatest -possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these terms. - - To do so, attach the following notices to the program. It is safest -to attach them to the start of each source file to most effectively -state the exclusion of warranty; and each file should have at least -the "copyright" line and a pointer to where the full notice is found. - - - Copyright (C) - - This program is free software: you can redistribute it and/or modify - it under the terms of the GNU Affero General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU Affero General Public License for more details. - - You should have received a copy of the GNU Affero General Public License - along with this program. If not, see . - -Also add information on how to contact you by electronic and paper mail. - - If your software can interact with users remotely through a computer -network, you should also make sure that it provides a way for users to -get its source. For example, if your program is a web application, its -interface could display a "Source" link that leads users to an archive -of the code. There are many ways you could offer source, and different -solutions will be better for different programs; see section 13 for the -specific requirements. - - You should also get your employer (if you work as a programmer) or school, -if any, to sign a "copyright disclaimer" for the program, if necessary. -For more information on this, and how to apply and follow the GNU AGPL, see -. - -Additional permission under GNU AGPL version 3 section 7 - -If you modify this Program, or any covered work, by linking or combining it -with any of the below libraries (or a modified version of that library), -containing parts covered by the terms of any of the below libraries, the -licensors of this Program grant you additional permission to convey the -resulting work. Corresponding Source for a non-source form of such a -combination shall include the source code for the parts of any of the -below libraries used as well as that of the covered work. - -amazonica -beckon -cider/cider-nrepl -clj-commons/clj-yaml -clj-excel -clojure-complete -clojure-csv -clout -com.cemerick/clojurescript.test -com.cemerick/pomegranate -com.cemerick/url -com.clojure-goes-fast/lazy-require -com.stuartsierra/dependency -com.taoensso/encore -com.taoensso/nippy -com.taoensso/sente -com.taoensso/timbre -com.taoensso/truss -commonmark-hiccup -compojure -criterium -crypto-equality -crypto-random -environ -hawk -hiccup -hickory -honeysql -incanter -incanter/incanter-charts -incanter/incanter-core -incanter/incanter-excel -incanter/incanter-io -incanter/incanter-latex -incanter/incanter-mongodb -incanter/incanter-pdf -incanter/incanter-sql -incanter/incanter-svg -incanter/incanter-zoo -instaparse -io.forward/yaml -irresponsible/tentacles -korma -markdown-clj -medley -metasoarous/oz -mount -net.cgrand/parsley -net.cgrand/sjacket -net.cgrant/regex -net.mikera/core.matrix -net.mikera/core.matrix.stats -nrepl -nrepl/bencode -ns-tracker -org.clojure/algo.generic -org.clojure/clojure -org.clojure/clojurescript -org.clojure/core.async -org.clojure/core.cache -org.clojure/core.incubator -org.clojure/core.match -org.clojure/core.memoize -org.clojure/core.specs.alpha -org.clojure/data.codec -org.clojure/data.csv -org.clojure/data.finger-tree -org.clojure/data.json -org.clojure/data.priority-map -org.clojure/java.classpath -org.clojure/java.jdbc -org.clojure/math.combinatorics -org.clojure/math.numeric-tower -org.clojure/spec.alpha -org.clojure/test.check -org.clojure/tools.analyzer -org.clojure/tools.analyzer.jvm -org.clojure/tools.cli -org.clojure/tools.logging -org.clojure/tools.macro -org.clojure/tools.namespace -org.clojure/tools.nrepl -org.clojure/tools.reader -org.clojure/tools.trace -org.flatland/ordered -org.flatland/useful -org.sonatype.aether/aether-api -org.sonatype.aether/aether-connector-file -org.sonatype.aether/aether-connector-wagon -org.sonatype.aether/aether-impl -org.sonatype.aether/aether-spi -org.sonatype.aether/aether-util -org.sonatype.sisu/sisu-inject-plexus -org.tcrawley/dynapath -pathetic -prismatic/plumbing -prismatic/schema -quoin -ring-basic-authentication -robert/hooke -semantic-csv -slingshot -swingrepl -tigris -us.bpsm/edn-java diff --git a/math/README.md b/math/README.md index 928072afe9..46912ff5cb 100644 --- a/math/README.md +++ b/math/README.md @@ -1,129 +1,73 @@ -# polismath — ARCHIVED (decommissioned at Clojure->Python cutover Step #3) - -> **This Clojure service no longer runs anywhere.** The math engine is the -> Python poller (`math-python` service; code under `delphi/polismath/` — moving -> to top-level `math/` at cutover Step #4, which removes this tree). Its -> Clojure-exact behavior was certified against THIS tree (battery 20/20 + -> live poller equivalence — see `delphi/docs/CUTOVER_RUNBOOK.md`). After -> Step #4 this tree remains available in git history as the certification -> oracle (`git log -- math/`). - -The real-time machine learning system powering Polis. - -## Development environment - -This part of the codebase is implemented in [Clojure](https://clojure.org), a dynamic, data-driven, functional Lisp. -As with any Lisp, Clojure development typically revolves around the interactive REPL. -It's recommended that you use an editor plugin to connect to this REPL, letting you execute and experiment with code as you write it, display documentation, and debug all from the comfort of your favorite text editor. -(See [Calva](https://marketplace.visualstudio.com/items?itemName=betterthantomorrow.calva) for VS, [Fireplace](https://github.com/tpope/vim-fireplace) for Vim, [Cider](https://docs.cider.mx/cider/index.html) for Emacs, [Cursive](https://cursive-ide.com/) for IDEA, etc). - -The root directory of this codebase contains docker compose infrastructure for running a complete development environment. -In addition to running the rest of the system's components (database, server, client build processes, etc), it also runs the math component in tandem with an embedded nREPL. -This allows you to connect to and evaluate code from within the running math worker. -To use this infrastructure, run the following from the root of this repository: +# math — the Polis math engine (Python) + +The real-time opinion-matrix math powering Polis: PCA, base-cluster +k-means, group clustering, representativeness (repness), consensus, and +comment routing — plus the production **math poller** that tails votes and +moderation from Postgres and writes `math_main` / `math_bidtopid` / +`math_ptptstats` under one `math_env`. + +This package (`polismath`) replaced the original Clojure implementation +that lived in this directory (Clojure→Python cutover, 2026 — see +`delphi/docs/CUTOVER_RUNBOOK.md` and `delphi/docs/MATH_ALGORITHM_HISTORY.md`). +It is **certified Clojure-exact**: the replay battery holds 20/20 MATCH +dataset pairs against the Clojure reference, and live poller equivalence +runs matched the real Clojure container row-for-row +(`delphi/docs/MATH_POLLER_EQUIV_SPEC.md`). The retired Clojure tree — still +the certification oracle for any future engine change — lives in git +history: `git log -- math/` (any commit before cutover Step #4). + +**Restoring the certification oracle** (needed only to RE-REPLAY the +Clojure side after an engine change; cached battery pairs need no oracle): ```sh -docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile postgres up +# from the repo root; = any pre-Step-4 commit, e.g. the Step 3 merge +git archive math/ | tar -x --exclude='math/README.md' +# math/src, math/dev, math/deps.edn, ... reappear alongside polismath/ +# (the exclude keeps THIS file — every pre-cutover revision also had a +# math/README.md and would overwrite it). +# Run the battery (cd delphi && uv run python scripts/certify.py run), +# then remove the restored Clojure entries again — UNTRACKED entries only: +git status --porcelain math/ | awk '$1=="??" {print $2}' | xargs rm -r ``` -The first time you run this (or if you've edited any of the docker files, or certain other parts of the system), you may need to run this command with the `--build` flag. -In general, this shouldn't be necessary if you're just working on the Clojure code itself, however. - -A little while after running this, docker compose will log out a message telling you that the system has started, and that the nREPL server is running on port `18975`. -This message may get quickly swamped out by a stream of other logging messages relaying polling information (TODO: less verbose logging option). -Regardless, at this point, the system is running and can be connected to. -Most editor plugins will connect somewhat automatically if you try to evaluate code or display documentation, but you can refer to the documentation for your editor of choice if this is not the case. - -To get a sense for how various parts of these system can be used, take a look at the comment block at the [bottom of `dev/user.clj`](dev/user.clj#L328). - -If you're not familiar with Clojure and want a fun crash course, I highly recommend [Clojure for the Brave and True](https://www.braveclojure.com/), a delightful introduction to the language. - -### Without docker compose? - -If you're not using the docker compose infrastructure, you can run `clj -M:dev` to get nREPL going, but this will not start math worker's processing queue (or obviously any other parts of the system). -This may be preferable if you don't need the whole system running for whatever task you're working on. -You can always manually start the polling system by manually running `(runner/run!)`, as described below, as long as you have the `DATABASE_URL` environment variable pointing to a database (see [`doc/configuration.md`](doc/configuration.md)) - -## Starting and stopping the system - -This application uses Stuart Sierra's Component library for REPL-reloadability. -Sometimes, (e.g.) evaluating a new definition of an existing function will be picked up by the system immediately without any further work. -In other cases though, especially if something stateful is involved, it may be necessary to reload/restart the system. - -This can be performed using a set of utility functions in the `polismath.runner` namespace (generally assumed to be aliased to `runner`). -To stop the system, you can use `runner/stop!`, followed by `namespace.repl/refresh` to reload namespaces, and `runner/start!` to start the system back up. -The `runner/system-reset!` function will do all of this for you automatically, but offers less flexibility in specifying configuration details in how you start the system. - -While this setup is nice from the perspective of system reloadability, Stuart's Component library unfortunately requires that a lot of the core functions of the system end up having to explicitly accept an argument corresponding to their part of the system. -This ends up being somewhat annoying from the perspective of interactive development, as it requires grabbing the corresponding component out of the `runner/system` map, and passing that to the function in question. -We'll soon be switching to [Mount](https://github.com/tolitius/mount) over Component, for more automated reloadability, and less hassle passing around system/component objects (see [#1056](https://github.com/compdemocracy/polis/issues/1056)). - -## Running commands - -There are also a number of commands which can be run, either locally or with `docker compose run math ...`, from the root of the monorepo: - -* `clojure -M:run --help` - print run command help (noisy; sorry) -* `clojure -M:run export -f .zip` - export the conversation at `` to the given filename -* `clojure -M:run update -Z ` - update a particular conversation -* `clojure -M:run full` - run a full system (poller plus auxiliary task processing) -* etc. - -If you are exporting data, you will need to run `docker compose` here with the same `-f docker-compose.yml -f docker-compose.dev.yml --profile postgres` options as described above, so that the directory (volume) mounting in the dev configuration file will mirror the generated files over onto your local filesystem. +Notes: in this jj-colocated repo the restored files are snapshotted into +the working-copy commit on the next jj command — do the restore + battery ++ cleanup within one working-copy session (or in a scratch clone) and +check `jj st` afterwards. Do NOT use `git checkout -- math/` here — +checkout/restore clobbers the jj working copy state. -## Tests +## Layout -You can run tests by executing `clojure -M:test`. +- `polismath/` — the package. + - `conversation/`, `pca_kmeans_rep/` — the math itself (Clojure-exact + legacy semantics; quirk catalog in `delphi/docs/CLOJURE_QUIRKS.md`). + - `poller/` — the production service (`python -m polismath.poller`). + - `database/` — Postgres read/write path. + - `replay/` — replay harness, certify battery, poller-equivalence + + shadow comparers. + - `run_math_pipeline.py` — one-shot pipeline entry + (`run-math-pipeline` console script; also invoked by + `delphi/run_delphi.py`). -Since Clojure is slow to start though, you may find it easier to run the `test-runner/-main` function (located at [`test/test_runner.clj`](test/test_runner.clj)) from within your nREPL process. -There is an example of this in the [`dev/user.clj`](dev/user.clj) file mentioned above. -There are rough units tests for most of the basic math things, and one or two higher level integration tests (presently broken). -We're looking forward to setting up `clojure.spec` and some generative testing for more thorough coverage if this is something that excites you! +## Running -## Architecture +The poller service runs via the `math-python` compose service (the deploy +path), or directly: -The system is designed around a polling mechanism which queries the database at a regular interval for new votes and moderation status. -These data are then routed to a "conversation manager", an agent like thing that maintains the current state of the conversation, and orchestrates updates to the data. -The reason for this particular design is that when a conversation is very active, votes can come in at a very rapid rate. -Meanwhile, the time it takes to run an update increases. -We need to have a way of queueing up vote and moderation data updates, so that they're ready to be processed once the last conversation update has completed. - -You can see the conversation manager implementation at [`src/polismath/conv_man.clj`](src/polismath/conv_man.clj). - -## Logging Configuration - -The system's logging level defaults to `:warn` but can be configured in several ways: - -1. Set the `LOGGING_LEVEL` environment variable to change from the default `:warn` level: - - ```bash - MATH_LOG_LEVEL=info docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile postgres up - ``` - -2. Modify the logging level in `resources/config.edn` -3. Set the level at runtime via the REPL: - - ```clojure - (require '[taoensso.timbre :as timbre]) - (timbre/set-level! :info) ; Example of setting to a more verbose level - ``` - -Available log levels (from lowest to highest): `:trace`, `:debug`, `:info`, `:warn`, `:error`, `:fatal`, `:report` - -## Production setup - -The [`docker-compose.yml`](../docker-compose.yml) file in the root of this directory is provided as a basis for production deployment. -Outstanding issues which need to be resolved before it would be ready include ensuring only necessary ports are exposed, etc. -The individual `Dockerfile`s that make up this infrastructure can currently be used by themselves, separate from `docker compose`, for deployment. - -## Requirements - -Nonetheless, if you wish to run this part of the system directly on a machine (outside of docker), the only requirements are that you: - -* [install Clojure](https://clojure.org/guides/getting_started). -* Setup the postgresql client (`sudo apt-get install postgresql postgresql-client` on ubuntu). +```sh +cd delphi && uv run python -m polismath.poller # run forever +cd delphi && uv run python -m polismath.poller --once # one poll cycle +``` -If you go this route you will want to take a look at the (see [`doc/configuration.md`](doc/configuration.md)). +## Development environment -## Licensing +One shared environment with delphi: `polismath` is installed **editable** +into `delphi/.venv` (see `delphi/pyproject.toml` `[tool.uv.sources]`), so +`cd delphi && uv sync` is the only setup step. The test estate — unit and +parity tests, `real_data/` datasets, golden snapshots, replay stores — +deliberately stays under `delphi/` (`delphi/tests/`, `delphi/real_data/`); +run tests from there: -Please see LICENSE +```sh +cd delphi && uv run pytest tests/ -v +``` diff --git a/math/bin/run b/math/bin/run deleted file mode 100755 index 4d5f5f5cf5..0000000000 --- a/math/bin/run +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/sh -# Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - - -while [ 1 ] -do - echo " " - echo "XXXXXXXXXXXXXXXXXX REBOOTING MATH WORKER XXXXXXXXXXXXXXXXXX" - echo "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" - echo " " - timeout -s KILL 14400 clojure -M:run full -done diff --git a/math/bin/util-server b/math/bin/util-server deleted file mode 100755 index 7fdd45c538..0000000000 --- a/math/bin/util-server +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/sh -# Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -while [ 1 ] -do - lein with-profile production trampoline run -m polismath.server -done diff --git a/math/data/legalization_conv.json b/math/data/legalization_conv.json deleted file mode 100644 index 3d5b837836..0000000000 --- a/math/data/legalization_conv.json +++ /dev/null @@ -1 +0,0 @@ -{"reactions":{"comment_id":["5152aece814288b555000006","5152ae54814288b555000001","5152ae54814288b555000001","5152aece814288b555000006","5152aef6910cbdba55000001","5152af2a910cbdba5500000d","5152b6268f4698d75500000e","5152b6268f4698d75500000e","5152b68e8f4698d755000012","5152b68e8f4698d755000012","5152b77b8f4698d75500001f","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b85d8f4698d75500002e","5152b9a28f4698d75500003f","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b9a28f4698d75500003f","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152ba7a8f4698d755000052","5152bb078f4698d755000062","5152bb078f4698d755000062","5152bb078f4698d755000062","5152bb768f4698d75500006d","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bbda8f4698d75500007f","5152bb768f4698d75500006d","5152bbda8f4698d75500007f","5152bbda8f4698d75500007f","5152bc178f4698d755000088","5152bc5d8f4698d75500008c","5152bc5d8f4698d75500008c","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","5152bb768f4698d75500006d","5152bc178f4698d755000088","5152c0ebbceb379059000005","5152c0ebbceb379059000005","5152c0ebbceb379059000005","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b68e8f4698d755000012","5152b68e8f4698d755000012","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","51531a58c01d860500000050","51531970c01d86050000004d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","51531970c01d86050000004d","51531a58c01d860500000050","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b77b8f4698d75500001f","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","51531970c01d86050000004d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531a58c01d860500000050","515326fac01d86050000008a","51532756c01d86050000008d","51531a58c01d860500000050","515326fac01d86050000008a","51532756c01d86050000008d","51531970c01d86050000004d","5152b68e8f4698d755000012","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","51531970c01d86050000004d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bc178f4698d755000088","5152bb078f4698d755000062","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531a58c01d860500000050","515326fac01d86050000008a","51532756c01d86050000008d","515330d0c3104205000000b6","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b85d8f4698d75500002e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","51531a58c01d860500000050","515326fac01d86050000008a","51532756c01d86050000008d","51531970c01d86050000004d","51531a58c01d860500000050","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","515330d0c3104205000000b6","515330d0c3104205000000b6","51533c61e48367050000000f","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","515376dee7e0a3050000001a","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152ba7a8f4698d755000052","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531970c01d86050000004d","51531a58c01d860500000050","5152b68e8f4698d755000012","515326fac01d86050000008a","515376dee7e0a3050000001a","515376dee7e0a3050000001a","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","51531a58c01d860500000050","515326fac01d86050000008a","51532756c01d86050000008d","51533c61e48367050000000f","515326fac01d86050000008a","51533c61e48367050000000f","51532756c01d86050000008d","515376dee7e0a3050000001a","515330d0c3104205000000b6","51532756c01d86050000008d","515326fac01d86050000008a","51533c61e48367050000000f","515376dee7e0a3050000001a","515330d0c3104205000000b6","51533c61e48367050000000f","515376dee7e0a3050000001a","515326fac01d86050000008a","51532756c01d86050000008d","515330d0c3104205000000b6","5152bb768f4698d75500006d","51531a58c01d860500000050","51531970c01d86050000004d","5152bc5d8f4698d75500008c","5152bbda8f4698d75500007f","5152b68e8f4698d755000012","5152b77b8f4698d75500001f","51531a58c01d860500000050","51532756c01d86050000008d","5152c0ebbceb379059000005","51531970c01d86050000004d","51531a58c01d860500000050","5152ba7a8f4698d755000052","515326fac01d86050000008a","5152bc5d8f4698d75500008c","515376dee7e0a3050000001a","5152b9a28f4698d75500003f","515330d0c3104205000000b6","515330d0c3104205000000b6","51531970c01d86050000004d","515376dee7e0a3050000001a","5152b6268f4698d75500000e","51531a58c01d860500000050","5152bc178f4698d755000088","5152bc5d8f4698d75500008c","5152b77b8f4698d75500001f","5152c0ebbceb379059000005","5152b85d8f4698d75500002e","5152ba7a8f4698d755000052","5152ba7a8f4698d755000052","5152bb078f4698d755000062","515326fac01d86050000008a","5152c0ebbceb379059000005","5152b6268f4698d75500000e","5152b85d8f4698d75500002e","51531970c01d86050000004d","51531a58c01d860500000050","5152bb078f4698d755000062","5152bbda8f4698d75500007f","5152b68e8f4698d755000012","51532756c01d86050000008d","51531a58c01d860500000050","51533c61e48367050000000f","51532756c01d86050000008d","5152b9a28f4698d75500003f","5152b68e8f4698d755000012","5152bb768f4698d75500006d","5152b68e8f4698d755000012","51531a58c01d860500000050","5152c0ebbceb379059000005","5152bb078f4698d755000062","5152bbda8f4698d75500007f","51531970c01d86050000004d","5154b210faa9130500000078","5152b77b8f4698d75500001f","5152ba7a8f4698d755000052","5154b210faa9130500000078","515330d0c3104205000000b6","5152b6268f4698d75500000e","5152bc5d8f4698d75500008c","5152b68e8f4698d755000012","515376dee7e0a3050000001a","51532756c01d86050000008d","51532756c01d86050000008d","5152b9a28f4698d75500003f","5152b9a28f4698d75500003f","5152bc178f4698d755000088","5152b6268f4698d75500000e","51533c61e48367050000000f","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","5152bc5d8f4698d75500008c","5152bb768f4698d75500006d","515326fac01d86050000008a","5152bbda8f4698d75500007f","515376dee7e0a3050000001a","515326fac01d86050000008a","5152b85d8f4698d75500002e","5152c0ebbceb379059000005","5152bc178f4698d755000088","5152b85d8f4698d75500002e","515330d0c3104205000000b6","5152bb078f4698d755000062","5152b77b8f4698d75500001f","51533c61e48367050000000f","5154b4a1804004050000008a","5154b4a1804004050000008a","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152bc178f4698d755000088","5152c0ebbceb379059000005","5152b85d8f4698d75500002e","5154b210faa9130500000078","5152bbda8f4698d75500007f","51531970c01d86050000004d","5154b4a1804004050000008a","5152bc5d8f4698d75500008c","5152ba7a8f4698d755000052","51533c61e48367050000000f","5152b9a28f4698d75500003f","5154b4a1804004050000008a","5152bc178f4698d755000088","515376dee7e0a3050000001a","515326fac01d86050000008a","5152c0ebbceb379059000005","5154b210faa9130500000078","5152bbda8f4698d75500007f","5152b68e8f4698d755000012","5154b210faa9130500000078","515330d0c3104205000000b6","51531a58c01d860500000050","5152bb078f4698d755000062","51532756c01d86050000008d","51532756c01d86050000008d","5152b85d8f4698d75500002e","51533c61e48367050000000f","5154b4a1804004050000008a","5152bb078f4698d755000062","5152b77b8f4698d75500001f","5152bc5d8f4698d75500008c","51531a58c01d860500000050","5152b77b8f4698d75500001f","5152bb768f4698d75500006d","515376dee7e0a3050000001a","5152b6268f4698d75500000e","51533c61e48367050000000f","5152bbda8f4698d75500007f","5152ba7a8f4698d755000052","5152b85d8f4698d75500002e","515330d0c3104205000000b6","5152b6268f4698d75500000e","5152bb768f4698d75500006d","5152c0ebbceb379059000005","5154b4a1804004050000008a","5152bc178f4698d755000088","5152b77b8f4698d75500001f","515376dee7e0a3050000001a","5154b210faa9130500000078","51533c61e48367050000000f","5152b9a28f4698d75500003f","515326fac01d86050000008a","5154b210faa9130500000078","515376dee7e0a3050000001a","51533c61e48367050000000f","5152c0ebbceb379059000005","5152bc178f4698d755000088","51531a58c01d860500000050","5152b6268f4698d75500000e","5152bbda8f4698d75500007f","5152bb078f4698d755000062","515326fac01d86050000008a","5152b9a28f4698d75500003f","5152b68e8f4698d755000012","51531970c01d86050000004d","5152bb768f4698d75500006d","5152bc5d8f4698d75500008c","51531a58c01d860500000050","51532756c01d86050000008d","5152b9a28f4698d75500003f","5152bbda8f4698d75500007f","5152bb768f4698d75500006d","5152b77b8f4698d75500001f","5154b210faa9130500000078","5152bc178f4698d755000088","515326fac01d86050000008a","5152ba7a8f4698d755000052","515330d0c3104205000000b6","5152b6268f4698d75500000e","5152b68e8f4698d755000012","5152b85d8f4698d75500002e","5154b4a1804004050000008a","515376dee7e0a3050000001a","51531970c01d86050000004d","5152c0ebbceb379059000005","51533c61e48367050000000f","5152bb078f4698d755000062","5152bc5d8f4698d75500008c","515326fac01d86050000008a","51532756c01d86050000008d","5154b4a1804004050000008a","5152b68e8f4698d755000012","5154e1f18040040500000104","5152bb078f4698d755000062","515330d0c3104205000000b6","5154e1f18040040500000104","5152bb768f4698d75500006d","5152b85d8f4698d75500002e","515376dee7e0a3050000001a","5152bb078f4698d755000062","5154b210faa9130500000078","5152bc5d8f4698d75500008c","5152bc178f4698d755000088","5154b4a1804004050000008a","5152c0ebbceb379059000005","5152b77b8f4698d75500001f","515376dee7e0a3050000001a","51531a58c01d860500000050","515326fac01d86050000008a","5154e1f18040040500000104","5154b4a1804004050000008a","5154b210faa9130500000078","5154b4a1804004050000008a","5152bc178f4698d755000088","5152bb768f4698d75500006d","5154b210faa9130500000078","51532756c01d86050000008d","515330d0c3104205000000b6","5152b77b8f4698d75500001f","51533c61e48367050000000f","5152ba7a8f4698d755000052","5152bbda8f4698d75500007f","5152c0ebbceb379059000005","51531970c01d86050000004d","5152b85d8f4698d75500002e","5152bb078f4698d755000062","515376dee7e0a3050000001a","515326fac01d86050000008a","51531a58c01d860500000050","5152b68e8f4698d755000012","5154e1f18040040500000104","5152b6268f4698d75500000e","5152bc5d8f4698d75500008c","5152b9a28f4698d75500003f","5155284e804004050000014d","5155284e804004050000014d","5152bc178f4698d755000088","5154b4a1804004050000008a","515326fac01d86050000008a","51532756c01d86050000008d","5152c0ebbceb379059000005","5152b68e8f4698d755000012","51531970c01d86050000004d","5152b77b8f4698d75500001f","51533c61e48367050000000f","5152b85d8f4698d75500002e","51531a58c01d860500000050","5152bc178f4698d755000088","515326fac01d86050000008a","5152bc178f4698d755000088","5152b9a28f4698d75500003f","5152b6268f4698d75500000e","5152b9a28f4698d75500003f","5154e1f18040040500000104","5155284e804004050000014d","5152bb768f4698d75500006d","5155284e804004050000014d","51531970c01d86050000004d","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152b6268f4698d75500000e","5155bd6799baee050000002d","5152bc5d8f4698d75500008c","5152ba7a8f4698d755000052","5154b210faa9130500000078","51533c61e48367050000000f","5155bdb999baee0500000036","51531970c01d86050000004d","5152bb768f4698d75500006d","5152b9a28f4698d75500003f","515376dee7e0a3050000001a","5152bc178f4698d755000088","5155bdb999baee0500000036","5155bd6799baee050000002d","5155bf0599baee0500000046","5154b4a1804004050000008a","5152bb078f4698d755000062","5154e1f18040040500000104","5155284e804004050000014d","51531970c01d86050000004d","5155284e804004050000014d","5155bdb999baee0500000036","5152bc5d8f4698d75500008c","5155bd6799baee050000002d","5155bf0599baee0500000046","5155bdb999baee0500000036","5154e1f18040040500000104","5155bdb999baee0500000036","5152bbda8f4698d75500007f","5152c0ebbceb379059000005","5155bd6799baee050000002d","5152bb768f4698d75500006d","5152b6268f4698d75500000e","51532756c01d86050000008d","5154b4a1804004050000008a","515330d0c3104205000000b6","515326fac01d86050000008a","5152bc5d8f4698d75500008c","51531a58c01d860500000050","5154b210faa9130500000078","5152b9a28f4698d75500003f","5152bb078f4698d755000062","51531970c01d86050000004d","5152ba7a8f4698d755000052","5155bf0599baee0500000046","5152b68e8f4698d755000012","5152b77b8f4698d75500001f","515376dee7e0a3050000001a","5154e1f18040040500000104","51531970c01d86050000004d","5152b6268f4698d75500000e","5155284e804004050000014d","5152b68e8f4698d755000012","5155bf0599baee0500000046","5155bdb999baee0500000036","5152bc5d8f4698d75500008c","515376dee7e0a3050000001a","51531a58c01d860500000050","51531970c01d86050000004d","5154b4a1804004050000008a","5152bbda8f4698d75500007f","5155bdb999baee0500000036","515376dee7e0a3050000001a","515330d0c3104205000000b6","515326fac01d86050000008a","5154b210faa9130500000078","5155bd6799baee050000002d","5152bc178f4698d755000088","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5152b77b8f4698d75500001f","5152bbda8f4698d75500007f","5154b4a1804004050000008a","51531a58c01d860500000050","5152bb768f4698d75500006d","5152bb078f4698d755000062","51531970c01d86050000004d","5152b68e8f4698d755000012","515330d0c3104205000000b6","5154e1f18040040500000104","51531a58c01d860500000050","5152bc178f4698d755000088","5152bb768f4698d75500006d","5152ba7a8f4698d755000052","5154b4a1804004050000008a","5152bbda8f4698d75500007f","51532756c01d86050000008d","515326fac01d86050000008a","5152b85d8f4698d75500002e","5152bb078f4698d755000062","515330d0c3104205000000b6","5154e1f18040040500000104","5152c0ebbceb379059000005","5155bd6799baee050000002d","51533c61e48367050000000f","5152b6268f4698d75500000e","51531970c01d86050000004d","5152b9a28f4698d75500003f","515326fac01d86050000008a","51531a58c01d860500000050","5152b68e8f4698d755000012","5152bc178f4698d755000088","5152bb768f4698d75500006d","5152b77b8f4698d75500001f","5152b6268f4698d75500000e","5152c0ebbceb379059000005","5152bb078f4698d755000062","5152b9a28f4698d75500003f","515326fac01d86050000008a","5152ba7a8f4698d755000052","5152bc178f4698d755000088","5155bd6799baee050000002d","51531a58c01d860500000050","5152bb768f4698d75500006d","5152b85d8f4698d75500002e","5152b68e8f4698d755000012","5154e1f18040040500000104","515330d0c3104205000000b6","5155bf0599baee0500000046","515376dee7e0a3050000001a","5152b6268f4698d75500000e","5154b210faa9130500000078","51532756c01d86050000008d","5155bdb999baee0500000036","5152bc5d8f4698d75500008c","5152bbda8f4698d75500007f","5154b4a1804004050000008a","51531970c01d86050000004d","51533c61e48367050000000f","5152b77b8f4698d75500001f","5155284e804004050000014d","5152c0ebbceb379059000005","51531970c01d86050000004d","515326fac01d86050000008a","5154b4a1804004050000008a","5152ba7a8f4698d755000052","5155fe1d99baee05000000fb","5152b9a28f4698d75500003f","5155284e804004050000014d","5152b68e8f4698d755000012","51531a58c01d860500000050","515330d0c3104205000000b6","5152b85d8f4698d75500002e","5155bf0599baee0500000046","5152bc5d8f4698d75500008c","5154b210faa9130500000078","5152c0ebbceb379059000005","51531970c01d86050000004d","51533c61e48367050000000f","515326fac01d86050000008a","5152b85d8f4698d75500002e","5152bc178f4698d755000088","51532756c01d86050000008d","5155fe1d99baee05000000fb","5154b4a1804004050000008a","515376dee7e0a3050000001a","5152b77b8f4698d75500001f","5155bdb999baee0500000036","5152b9a28f4698d75500003f","5152b68e8f4698d755000012","5152c0ebbceb379059000005","5152bbda8f4698d75500007f","515330d0c3104205000000b6","51533c61e48367050000000f","51531a58c01d860500000050","51531970c01d86050000004d","5154e1f18040040500000104","5155284e804004050000014d","5155bdb999baee0500000036","5154b4a1804004050000008a","5155bf0599baee0500000046","5152c0ebbceb379059000005","5152b77b8f4698d75500001f","51531970c01d86050000004d","5155bdb999baee0500000036","5154b4a1804004050000008a","5155bf0599baee0500000046","51533c61e48367050000000f","51532756c01d86050000008d","5154b210faa9130500000078","5152bc178f4698d755000088","515330d0c3104205000000b6","5155fe1d99baee05000000fb","5152b9a28f4698d75500003f","5154e1f18040040500000104","5152bbda8f4698d75500007f","5152b6268f4698d75500000e","51531a58c01d860500000050","5152b85d8f4698d75500002e","5152bc5d8f4698d75500008c","5152ba7a8f4698d755000052","5152bb078f4698d755000062","5152b68e8f4698d755000012","515376dee7e0a3050000001a","5155284e804004050000014d","5152bb768f4698d75500006d","515326fac01d86050000008a","51532756c01d86050000008d","5152bb768f4698d75500006d","5152b9a28f4698d75500003f","5155284e804004050000014d","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","51531970c01d86050000004d","5152b68e8f4698d755000012","51533c61e48367050000000f","5154e1f18040040500000104","5154b210faa9130500000078","5152b77b8f4698d75500001f","51533c61e48367050000000f","5152c0ebbceb379059000005","51532756c01d86050000008d","5152b9a28f4698d75500003f","5155284e804004050000014d","51531970c01d86050000004d","5152bb768f4698d75500006d","5152bb078f4698d755000062","5152c0ebbceb379059000005","5154b4a1804004050000008a","5152bbda8f4698d75500007f","51531a58c01d860500000050","5155284e804004050000014d","5155bd6799baee050000002d","515326fac01d86050000008a","5152b77b8f4698d75500001f","5152ba7a8f4698d755000052","5152b6268f4698d75500000e","5155fe1d99baee05000000fb","5152bc5d8f4698d75500008c","5155bf0599baee0500000046","515330d0c3104205000000b6","51532756c01d86050000008d","5152bb078f4698d755000062","5155bf0599baee0500000046","5155bdb999baee0500000036","515330d0c3104205000000b6","5152b68e8f4698d755000012","515326fac01d86050000008a","5152bc5d8f4698d75500008c","5154b210faa9130500000078","5152bbda8f4698d75500007f","5152b6268f4698d75500000e","51532756c01d86050000008d","5152b68e8f4698d755000012","515376dee7e0a3050000001a","5152b9a28f4698d75500003f","5155fe1d99baee05000000fb","5152bbda8f4698d75500007f","5152b77b8f4698d75500001f","5155bd6799baee050000002d","5154e1f18040040500000104","5152bb768f4698d75500006d","51532756c01d86050000008d","5154b4a1804004050000008a","5155284e804004050000014d","51531a58c01d860500000050","5155bdb999baee0500000036","515330d0c3104205000000b6","515326fac01d86050000008a","5152ba7a8f4698d755000052","5152b6268f4698d75500000e","5152bb078f4698d755000062","5152b68e8f4698d755000012","515376dee7e0a3050000001a","5152b85d8f4698d75500002e","5152bc178f4698d755000088","5154b210faa9130500000078","5152b9a28f4698d75500003f","5152b6268f4698d75500000e","5155fe1d99baee05000000fb","5152b77b8f4698d75500001f","5152bb768f4698d75500006d","5155284e804004050000014d","5155bdb999baee0500000036","5155bd6799baee050000002d","5152ba7a8f4698d755000052","5152bbda8f4698d75500007f","5152b85d8f4698d75500002e","51532756c01d86050000008d","5152b68e8f4698d755000012","5152c0ebbceb379059000005","51533c61e48367050000000f","515326fac01d86050000008a","5152bc5d8f4698d75500008c","5152bb078f4698d755000062","5154e1f18040040500000104","5154e1f18040040500000104","5155bdb999baee0500000036","5155bf0599baee0500000046","5155284e804004050000014d","5152bc178f4698d755000088","5155fe1d99baee05000000fb","515376dee7e0a3050000001a","51531a58c01d860500000050","515710e01398870500000056","515376dee7e0a3050000001a","5152b9a28f4698d75500003f","5152c0ebbceb379059000005","5155bf0599baee0500000046","51531a58c01d860500000050","5152b68e8f4698d755000012","51531970c01d86050000004d","5152bbda8f4698d75500007f","51532756c01d86050000008d","51531970c01d86050000004d","5152bc5d8f4698d75500008c","515710e01398870500000056","515710e01398870500000056","515330d0c3104205000000b6","5154b210faa9130500000078","5154b4a1804004050000008a","515711863efa030500000052","5155bd6799baee050000002d","5152b6268f4698d75500000e","51533c61e48367050000000f","515326fac01d86050000008a","515711863efa030500000052","5157125e1398870500000077","5154b4a1804004050000008a","5152b77b8f4698d75500001f","5152bb768f4698d75500006d","5152ba7a8f4698d755000052","5152bc178f4698d755000088","515330d0c3104205000000b6","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5157125e1398870500000077","5152bc5d8f4698d75500008c","5155bf0599baee0500000046","51532756c01d86050000008d","515330d0c3104205000000b6","5155fe1d99baee05000000fb","5152b68e8f4698d755000012","5155bd6799baee050000002d","5155284e804004050000014d","5154b4a1804004050000008a","515711863efa030500000052","51533c61e48367050000000f","5152b85d8f4698d75500002e","5152b85d8f4698d75500002e","5157125e1398870500000077","5155284e804004050000014d","5152b9a28f4698d75500003f","515711863efa030500000052","5157125e1398870500000077","5157225013988705000000a1","515726023efa030500000078","5157225013988705000000a1","5155284e804004050000014d","5152ba7a8f4698d755000052","5152bb768f4698d75500006d","5152bc5d8f4698d75500008c","5152bb078f4698d755000062","515376dee7e0a3050000001a","5155fe1d99baee05000000fb","5152b68e8f4698d755000012","5155bd6799baee050000002d","515710e01398870500000056","5152bbda8f4698d75500007f","5152c0ebbceb379059000005","51531a58c01d860500000050","51531970c01d86050000004d","5152b77b8f4698d75500001f","5154b210faa9130500000078","5155bf0599baee0500000046","5155bdb999baee0500000036","515326fac01d86050000008a","51533c61e48367050000000f","515711863efa030500000052","5155bdb999baee0500000036","51531970c01d86050000004d","5155bd6799baee050000002d","515726023efa030500000078","5152b9a28f4698d75500003f","5154b210faa9130500000078","515326fac01d86050000008a","5157225013988705000000a1","515330d0c3104205000000b6","5152c0ebbceb379059000005","5154b4a1804004050000008a","515710e01398870500000056","5152b68e8f4698d755000012","5152bb078f4698d755000062","51533c61e48367050000000f","5152bbda8f4698d75500007f","5152b6268f4698d75500000e","5152b77b8f4698d75500001f","5157125e1398870500000077","515376dee7e0a3050000001a","5152ba7a8f4698d755000052","51532756c01d86050000008d","5152b9a28f4698d75500003f","515726023efa030500000078","51531970c01d86050000004d","515326fac01d86050000008a","5154b210faa9130500000078","51531a58c01d860500000050","5152b77b8f4698d75500001f","5154b4a1804004050000008a","5152b68e8f4698d755000012","5157225013988705000000a1","5152b85d8f4698d75500002e","5152c0ebbceb379059000005","515330d0c3104205000000b6","515710e01398870500000056","5155bdb999baee0500000036","5155bd6799baee050000002d","515711863efa030500000052","5155bd6799baee050000002d","5155bd6799baee050000002d","5154e1f18040040500000104","51531a58c01d860500000050","5152ba7a8f4698d755000052","5152b77b8f4698d75500001f","5157125e1398870500000077","5152bb078f4698d755000062","515710e01398870500000056","515376dee7e0a3050000001a","5155284e804004050000014d","5154b4a1804004050000008a","51531970c01d86050000004d","515326fac01d86050000008a","5152bb768f4698d75500006d","5155bd6799baee050000002d","5152b85d8f4698d75500002e","5152bc5d8f4698d75500008c","5155fe1d99baee05000000fb","5157225013988705000000a1","5155bdb999baee0500000036","5152bc178f4698d755000088","515711863efa030500000052","5152bbda8f4698d75500007f","5155bf0599baee0500000046","515726023efa030500000078","515330d0c3104205000000b6","51532756c01d86050000008d","51533c61e48367050000000f","5154b210faa9130500000078","5152c0ebbceb379059000005","5152b9a28f4698d75500003f","5157660deea7730500000025","5152b6268f4698d75500000e","5152b68e8f4698d755000012","5152b77b8f4698d75500001f","5152b68e8f4698d755000012","51531970c01d86050000004d","5155284e804004050000014d","5152c0ebbceb379059000005","5152bc178f4698d755000088","5152b6268f4698d75500000e","5152b9a28f4698d75500003f","51533c61e48367050000000f","515330d0c3104205000000b6","515711863efa030500000052","515710e01398870500000056","515326fac01d86050000008a","5154b4a1804004050000008a","515376dee7e0a3050000001a","5152bb078f4698d755000062","5157660deea7730500000025","5155bd6799baee050000002d","51532756c01d86050000008d","51531a58c01d860500000050","5152b85d8f4698d75500002e","5152bc5d8f4698d75500008c","5157125e1398870500000077","5152bbda8f4698d75500007f","5154e1f18040040500000104","5155bf0599baee0500000046","5155bdb999baee0500000036","5157225013988705000000a1","5152ba7a8f4698d755000052","515726023efa030500000078","5154b210faa9130500000078","5155fe1d99baee05000000fb","5152bb768f4698d75500006d","5154b4a1804004050000008a","5155bf0599baee0500000046","5155fe1d99baee05000000fb","5155284e804004050000014d","515330d0c3104205000000b6","515376dee7e0a3050000001a","515711863efa030500000052","5157225013988705000000a1","515710e01398870500000056","5152bbda8f4698d75500007f","5154e1f18040040500000104","5155fe1d99baee05000000fb","515710e01398870500000056","5152b68e8f4698d755000012","5157225013988705000000a1","5152b77b8f4698d75500001f","51531970c01d86050000004d","51533c61e48367050000000f","515376dee7e0a3050000001a","51531a58c01d860500000050","5154b4a1804004050000008a","5152ba7a8f4698d755000052","5155284e804004050000014d","515330d0c3104205000000b6","515711863efa030500000052","5152bc178f4698d755000088","5152bc5d8f4698d75500008c","5152bb768f4698d75500006d","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5152bbda8f4698d75500007f","5155bf0599baee0500000046","5155bd6799baee050000002d","515326fac01d86050000008a","5152b9a28f4698d75500003f","5157125e1398870500000077","5152bb078f4698d755000062","5152c0ebbceb379059000005","5155bdb999baee0500000036","51531970c01d86050000004d","51532756c01d86050000008d","5157b6f27d39ed0500000078","5152b85d8f4698d75500002e","5154b210faa9130500000078","5157660deea7730500000025","515726023efa030500000078","5152c0ebbceb379059000005","515326fac01d86050000008a","5152bc178f4698d755000088","5152bb078f4698d755000062","5152bc5d8f4698d75500008c","5152ba7a8f4698d755000052","51531a58c01d860500000050","5155bd6799baee050000002d","5155bdb999baee0500000036","5152bb768f4698d75500006d","5152b9a28f4698d75500003f","5152b77b8f4698d75500001f","5157b6f27d39ed0500000078","51532756c01d86050000008d","5157660deea7730500000025","5152b85d8f4698d75500002e","5154e1f18040040500000104","5152b6268f4698d75500000e","51533c61e48367050000000f","5157125e1398870500000077","515726023efa030500000078","5154b210faa9130500000078","5155fe1d99baee05000000fb","5152c0ebbceb379059000005","515376dee7e0a3050000001a","5152b68e8f4698d755000012","5152b6268f4698d75500000e","5155bd6799baee050000002d","5154b4a1804004050000008a","5155bf0599baee0500000046","5154b210faa9130500000078","5152b85d8f4698d75500002e","515710e01398870500000056","5155bdb999baee0500000036","5152ba7a8f4698d755000052","5152b77b8f4698d75500001f","5157125e1398870500000077","515726023efa030500000078","515330d0c3104205000000b6","5152bc5d8f4698d75500008c","5152b9a28f4698d75500003f","51532756c01d86050000008d","5152bc178f4698d755000088","5152bb768f4698d75500006d","5157b6f27d39ed0500000078","51531970c01d86050000004d","515326fac01d86050000008a","5152bb078f4698d755000062","51533c61e48367050000000f","5155284e804004050000014d","5152bbda8f4698d75500007f","5154e1f18040040500000104","5157660deea7730500000025","51531a58c01d860500000050","5157225013988705000000a1","515711863efa030500000052","515726023efa030500000078","515710e01398870500000056","5157225013988705000000a1","515711863efa030500000052","5155284e804004050000014d","5157b6f27d39ed0500000078","5157660deea7730500000025","5155fe1d99baee05000000fb","5157125e1398870500000077","5154b4a1804004050000008a","515710e01398870500000056","5152ba7a8f4698d755000052","51588a977d39ed05000000dd","5155bd6799baee050000002d","51531a58c01d860500000050","515326fac01d86050000008a","5157b6f27d39ed0500000078","5155bf0599baee0500000046","5155284e804004050000014d","515711863efa030500000052","515376dee7e0a3050000001a","515726023efa030500000078","5154e1f18040040500000104","515330d0c3104205000000b6","51531a58c01d860500000050","5152bbda8f4698d75500007f","515710e01398870500000056","51588a977d39ed05000000dd","5155bd6799baee050000002d","51531970c01d86050000004d","5152b77b8f4698d75500001f","5152b6268f4698d75500000e","51588a977d39ed05000000dd","5157125e1398870500000077","51588a977d39ed05000000dd","5154e1f18040040500000104","5155bdb999baee0500000036","515726023efa030500000078","5155bd6799baee050000002d","5154b4a1804004050000008a","5155284e804004050000014d","515711863efa030500000052","5157b6f27d39ed0500000078","515710e01398870500000056","5152c0ebbceb379059000005","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5157b6f27d39ed0500000078","5157225013988705000000a1","5157660deea7730500000025","5155fe1d99baee05000000fb","5155bf0599baee0500000046","5155fe1d99baee05000000fb","5152b77b8f4698d75500001f","5154b210faa9130500000078","51531970c01d86050000004d","515376dee7e0a3050000001a","5154b210faa9130500000078","5155bd6799baee050000002d","5155bf0599baee0500000046","5152bc5d8f4698d75500008c","5157225013988705000000a1","5152b9a28f4698d75500003f","515726023efa030500000078","51533c61e48367050000000f","5152b68e8f4698d755000012","5152bbda8f4698d75500007f","515326fac01d86050000008a","5155fe1d99baee05000000fb","5154b210faa9130500000078","5155bdb999baee0500000036","5152bb078f4698d755000062","515710e01398870500000056","5152b85d8f4698d75500002e","5152bc178f4698d755000088","5152bb768f4698d75500006d","5152c0ebbceb379059000005","5155bd6799baee050000002d","5152b77b8f4698d75500001f","51531970c01d86050000004d","51532756c01d86050000008d","5154b4a1804004050000008a","5157660deea7730500000025","5152bc5d8f4698d75500008c","5154e1f18040040500000104","5152bc178f4698d755000088","51531970c01d86050000004d","5155fe1d99baee05000000fb","5154b210faa9130500000078","51532756c01d86050000008d","5152bbda8f4698d75500007f","51533c61e48367050000000f","5154b4a1804004050000008a","515726023efa030500000078","5155284e804004050000014d","5152ba7a8f4698d755000052","5152b77b8f4698d75500001f","5155bdb999baee0500000036","515710e01398870500000056","51531970c01d86050000004d","5155bd6799baee050000002d","51532756c01d86050000008d","5152b9a28f4698d75500003f","5152bc5d8f4698d75500008c","5155284e804004050000014d","5157225013988705000000a1","5152b9a28f4698d75500003f","5152ba7a8f4698d755000052","5154e1f18040040500000104","5157660deea7730500000025","515726023efa030500000078","51532756c01d86050000008d","515711863efa030500000052","5152b6268f4698d75500000e","5155fe1d99baee05000000fb","5152c0ebbceb379059000005","51588a977d39ed05000000dd","51533c61e48367050000000f","515726023efa030500000078","51531a58c01d860500000050","5157b6f27d39ed0500000078","5152bb768f4698d75500006d","515330d0c3104205000000b6","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5157125e1398870500000077","51531970c01d86050000004d","5157225013988705000000a1","515326fac01d86050000008a","515711863efa030500000052","5152b9a28f4698d75500003f","5155bf0599baee0500000046","5152c0ebbceb379059000005","5155bdb999baee0500000036","5152b85d8f4698d75500002e","5152bb768f4698d75500006d","5157125e1398870500000077","515376dee7e0a3050000001a","5159efa05b6c360500000022","5154b210faa9130500000078","5152ba7a8f4698d755000052","5155bd6799baee050000002d","515711863efa030500000052","51532756c01d86050000008d","5155fe1d99baee05000000fb","5152b68e8f4698d755000012","51588a977d39ed05000000dd","515726023efa030500000078","5155284e804004050000014d","5152bb078f4698d755000062","5157225013988705000000a1","5152bbda8f4698d75500007f","5152b6268f4698d75500000e","5154e1f18040040500000104","5152b77b8f4698d75500001f","5152b9a28f4698d75500003f","5157b6f27d39ed0500000078","515330d0c3104205000000b6","51531970c01d86050000004d","5152bc178f4698d755000088","5157660deea7730500000025","515710e01398870500000056","5154b4a1804004050000008a","5152bc5d8f4698d75500008c","5155bf0599baee0500000046","515326fac01d86050000008a","51531a58c01d860500000050","51533c61e48367050000000f","5159f113ef9382050000003f","515710e01398870500000056","515726023efa030500000078","5157b6f27d39ed0500000078","5152ba7a8f4698d755000052","5159f113ef9382050000003f","5155bd6799baee050000002d","5157b6f27d39ed0500000078","5154b4a1804004050000008a","5157660deea7730500000025","51588a977d39ed05000000dd","5154e1f18040040500000104","5152bc5d8f4698d75500008c","515330d0c3104205000000b6","515711863efa030500000052","5155284e804004050000014d","5152bbda8f4698d75500007f","515326fac01d86050000008a","5152b68e8f4698d755000012","5152b6268f4698d75500000e","515726023efa030500000078","5154b210faa9130500000078","51532756c01d86050000008d","51533c61e48367050000000f","5152b77b8f4698d75500001f","5155bdb999baee0500000036","5152b9a28f4698d75500003f","51531a58c01d860500000050","5152b85d8f4698d75500002e","5152bc178f4698d755000088","5157225013988705000000a1","5152c0ebbceb379059000005","51531970c01d86050000004d","515710e01398870500000056","515376dee7e0a3050000001a","5155fe1d99baee05000000fb","5157125e1398870500000077","5159efa05b6c360500000022","5152bb078f4698d755000062","5152bb768f4698d75500006d","5155bf0599baee0500000046","5157225013988705000000a1","5159efa05b6c360500000022","515726023efa030500000078","5157660deea7730500000025","5159f113ef9382050000003f","5157b6f27d39ed0500000078","51588a977d39ed05000000dd","5152bc5d8f4698d75500008c","5154b4a1804004050000008a","5152bb768f4698d75500006d","5155bd6799baee050000002d","515326fac01d86050000008a","5154e1f18040040500000104","515376dee7e0a3050000001a","5152ba7a8f4698d755000052","5159f113ef9382050000003f","5152bc5d8f4698d75500008c","51588a977d39ed05000000dd","5159f113ef9382050000003f","51532756c01d86050000008d","5155fe1d99baee05000000fb","5157125e1398870500000077","515710e01398870500000056","515726023efa030500000078","5157660deea7730500000025","5154e1f18040040500000104","51531a58c01d860500000050","5155fe1d99baee05000000fb","5157b6f27d39ed0500000078","515330d0c3104205000000b6","5152bbda8f4698d75500007f","51588a977d39ed05000000dd","51532756c01d86050000008d","5152b68e8f4698d755000012","51531970c01d86050000004d","5155bdb999baee0500000036","515726023efa030500000078","5152b6268f4698d75500000e","5152c0ebbceb379059000005","5159efa05b6c360500000022","5152bc5d8f4698d75500008c","5157125e1398870500000077","515710e01398870500000056","515711863efa030500000052","515376dee7e0a3050000001a","5155284e804004050000014d","5157225013988705000000a1","5152bb768f4698d75500006d","5155bd6799baee050000002d","5155bf0599baee0500000046","515326fac01d86050000008a","5152bc178f4698d755000088","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5152bb078f4698d755000062","5159f113ef9382050000003f","5152ba7a8f4698d755000052","5157660deea7730500000025","5152b9a28f4698d75500003f","5154b4a1804004050000008a","5154b210faa9130500000078","51533c61e48367050000000f","515b32927e4a740500000050","5152bc178f4698d755000088","5152c0ebbceb379059000005","51531a58c01d860500000050","5152b6268f4698d75500000e","515326fac01d86050000008a","5159f113ef9382050000003f","5152b77b8f4698d75500001f","5155bdb999baee0500000036","5155bd6799baee050000002d","5155284e804004050000014d","5155bf0599baee0500000046","51588a977d39ed05000000dd","51531970c01d86050000004d","515b32927e4a740500000050","5152bc5d8f4698d75500008c","515376dee7e0a3050000001a","515726023efa030500000078","5157225013988705000000a1","5152b68e8f4698d755000012","5159f113ef9382050000003f","5152b77b8f4698d75500001f","5155284e804004050000014d","5152bbda8f4698d75500007f","5152ba7a8f4698d755000052","5157b6f27d39ed0500000078","515710e01398870500000056","5154b210faa9130500000078","5157125e1398870500000077","5152b9a28f4698d75500003f","5154b4a1804004050000008a","5155fe1d99baee05000000fb","5152b85d8f4698d75500002e","51533c61e48367050000000f","515330d0c3104205000000b6","51533c61e48367050000000f","5152bb768f4698d75500006d","51532756c01d86050000008d","5152b68e8f4698d755000012","515711863efa030500000052","51532756c01d86050000008d","5152b68e8f4698d755000012","5154e1f18040040500000104","5159efa05b6c360500000022","5152bb078f4698d755000062","5157660deea7730500000025","51588a977d39ed05000000dd","5152c0ebbceb379059000005","5157225013988705000000a1","5152b77b8f4698d75500001f","5152ba7a8f4698d755000052","5152bc178f4698d755000088","5152b85d8f4698d75500002e","51533c61e48367050000000f","51531970c01d86050000004d","5152bc5d8f4698d75500008c","5159f113ef9382050000003f","515711863efa030500000052","5157660deea7730500000025","5155bf0599baee0500000046","5152b68e8f4698d755000012","5152bb078f4698d755000062","5155bdb999baee0500000036","51531a58c01d860500000050","515726023efa030500000078","5152bbda8f4698d75500007f","5159efa05b6c360500000022","515b32927e4a740500000050","5155fe1d99baee05000000fb","515b3c6a7e4a7405000000b8","5152b6268f4698d75500000e","51532756c01d86050000008d","5152b9a28f4698d75500003f","515376dee7e0a3050000001a","515710e01398870500000056","515326fac01d86050000008a","51532756c01d86050000008d","5154b210faa9130500000078","5152b77b8f4698d75500001f","5152b85d8f4698d75500002e","5155284e804004050000014d","51531970c01d86050000004d","5152b68e8f4698d755000012","51533c61e48367050000000f","5152bb078f4698d755000062","5155bdb999baee0500000036","5155fe1d99baee05000000fb","51531a58c01d860500000050","5152ba7a8f4698d755000052","5152bbda8f4698d75500007f","515330d0c3104205000000b6","5152bb078f4698d755000062","51531a58c01d860500000050","515711863efa030500000052","5152b68e8f4698d755000012","5157225013988705000000a1","515710e01398870500000056","515b3c6a7e4a7405000000b8","5152c0ebbceb379059000005","5154b4a1804004050000008a","5152b85d8f4698d75500002e","51588a977d39ed05000000dd","51532756c01d86050000008d","5152b77b8f4698d75500001f","5154e1f18040040500000104","5155bd6799baee050000002d","5159efa05b6c360500000022","515b32927e4a740500000050","5157125e1398870500000077","51531970c01d86050000004d","5157b6f27d39ed0500000078","5155fe1d99baee05000000fb","5152bc178f4698d755000088","51533c61e48367050000000f","5152b6268f4698d75500000e","5157660deea7730500000025","5155bf0599baee0500000046","5152bb768f4698d75500006d","515726023efa030500000078","5152b9a28f4698d75500003f","5154b210faa9130500000078","5159f113ef9382050000003f","5155bdb999baee0500000036","5155284e804004050000014d","515326fac01d86050000008a","515376dee7e0a3050000001a","5152bc5d8f4698d75500008c","515376dee7e0a3050000001a","5152bb768f4698d75500006d","5152b9a28f4698d75500003f","5154e1f18040040500000104","5155fe1d99baee05000000fb","5155284e804004050000014d","5152b6268f4698d75500000e","515711863efa030500000052","5152b6268f4698d75500000e","515710e01398870500000056","5152bc5d8f4698d75500008c","5155fe1d99baee05000000fb","51588a977d39ed05000000dd","51531970c01d86050000004d","51531a58c01d860500000050","5159efa05b6c360500000022","5155284e804004050000014d","5155bdb999baee0500000036","5152b9a28f4698d75500003f","5152bb768f4698d75500006d","515376dee7e0a3050000001a","5159f113ef9382050000003f","51533c61e48367050000000f","51532756c01d86050000008d","515b3c6a7e4a7405000000b8","5154b210faa9130500000078","515330d0c3104205000000b6","515726023efa030500000078","515b597cb03eae0500000118","5152bbda8f4698d75500007f","5155bf0599baee0500000046","515326fac01d86050000008a","5152c0ebbceb379059000005","5157660deea7730500000025","5157b6f27d39ed0500000078","5154e1f18040040500000104","5152b77b8f4698d75500001f","5155bd6799baee050000002d","515b32927e4a740500000050","515710e01398870500000056","515b597cb03eae0500000118","5152c0ebbceb379059000005","5159f113ef9382050000003f","5154e1f18040040500000104","515726023efa030500000078","5155bd6799baee050000002d","515330d0c3104205000000b6","5154b210faa9130500000078","515376dee7e0a3050000001a","5157b6f27d39ed0500000078","5157125e1398870500000077","5152bb078f4698d755000062","5155284e804004050000014d","5155bdb999baee0500000036","5155bf0599baee0500000046","515326fac01d86050000008a","5154b4a1804004050000008a","5152b85d8f4698d75500002e","5152bbda8f4698d75500007f","5152b9a28f4698d75500003f","5152b77b8f4698d75500001f","515b3c6a7e4a7405000000b8","51532756c01d86050000008d","5155fe1d99baee05000000fb","515711863efa030500000052","5152ba7a8f4698d755000052","5152b68e8f4698d755000012","5152bb768f4698d75500006d","5159efa05b6c360500000022","51588a977d39ed05000000dd","5157660deea7730500000025","51531970c01d86050000004d","5157225013988705000000a1","51531a58c01d860500000050","5152b6268f4698d75500000e","5152bc5d8f4698d75500008c","51533c61e48367050000000f","5152bc178f4698d755000088","515710e01398870500000056","5152bc178f4698d755000088","51531a58c01d860500000050","5152b85d8f4698d75500002e","5152c0ebbceb379059000005","5157660deea7730500000025","5155bf0599baee0500000046","5157225013988705000000a1","515376dee7e0a3050000001a","5152b6268f4698d75500000e","5155bdb999baee0500000036","5155fe1d99baee05000000fb","515b597cb03eae0500000118","5155284e804004050000014d","515726023efa030500000078","515330d0c3104205000000b6","5154b210faa9130500000078","5154b4a1804004050000008a","51531970c01d86050000004d","5155bd6799baee050000002d","5159f113ef9382050000003f","515326fac01d86050000008a","51532756c01d86050000008d","5152b77b8f4698d75500001f","5152bc5d8f4698d75500008c","5159efa05b6c360500000022","51588a977d39ed05000000dd","5152b9a28f4698d75500003f","5152bbda8f4698d75500007f","5157b6f27d39ed0500000078","515b3c6a7e4a7405000000b8","5152bb768f4698d75500006d","5152bb078f4698d755000062","5152ba7a8f4698d755000052","5152b68e8f4698d755000012","51533c61e48367050000000f","515711863efa030500000052","515b32927e4a740500000050","5157125e1398870500000077","5154e1f18040040500000104","5157660deea7730500000025","5152ba7a8f4698d755000052","5159efa05b6c360500000022","5157b6f27d39ed0500000078","5157125e1398870500000077","5155bd6799baee050000002d","5155bf0599baee0500000046","5154e1f18040040500000104","51532756c01d86050000008d","5152bb078f4698d755000062","5157b6f27d39ed0500000078","51531970c01d86050000004d","5152b77b8f4698d75500001f","51588a977d39ed05000000dd","5154b210faa9130500000078","5152bc178f4698d755000088","5152ba7a8f4698d755000052","515710e01398870500000056","515b32927e4a740500000050","5155bdb999baee0500000036","515b597cb03eae0500000118","515326fac01d86050000008a","51531970c01d86050000004d","5152bc5d8f4698d75500008c","515b3c6a7e4a7405000000b8","5157125e1398870500000077","5152bc5d8f4698d75500008c","5152b6268f4698d75500000e","51533c61e48367050000000f","515326fac01d86050000008a","5155284e804004050000014d","5152b77b8f4698d75500001f","515376dee7e0a3050000001a","515b32927e4a740500000050","5155bdb999baee0500000036","51531970c01d86050000004d","51588a977d39ed05000000dd","5154e1f18040040500000104","5152b9a28f4698d75500003f","5155fe1d99baee05000000fb","51531a58c01d860500000050","515330d0c3104205000000b6","5152b68e8f4698d755000012","51532756c01d86050000008d","5152bb078f4698d755000062","5159f113ef9382050000003f","5154b4a1804004050000008a","5157225013988705000000a1","5152bc178f4698d755000088","515b3c6a7e4a7405000000b8","515710e01398870500000056","5159efa05b6c360500000022","5157125e1398870500000077","5152bbda8f4698d75500007f","5155bf0599baee0500000046","5157b6f27d39ed0500000078","515726023efa030500000078","5157660deea7730500000025","5152bb768f4698d75500006d","5152c0ebbceb379059000005","515711863efa030500000052","515b597cb03eae0500000118","5152ba7a8f4698d755000052","5155bd6799baee050000002d","5154b210faa9130500000078","5152b85d8f4698d75500002e","515726023efa030500000078","5157225013988705000000a1","5159f113ef9382050000003f","5155bf0599baee0500000046","5155284e804004050000014d","515b3c6a7e4a7405000000b8","5152bb768f4698d75500006d","515b9b92cd84280000000007","515b9bf360e4eb0000000005","51531970c01d86050000004d","5154b4a1804004050000008a","515b9bf360e4eb0000000005","515b32927e4a740500000050","5152bc178f4698d755000088","515711863efa030500000052","515726023efa030500000078","5152bbda8f4698d75500007f","5157660deea7730500000025","515710e01398870500000056","5152b85d8f4698d75500002e","5154e1f18040040500000104","5155fe1d99baee05000000fb","51533c61e48367050000000f","5152b68e8f4698d755000012","515b3c6a7e4a7405000000b8","5152bbda8f4698d75500007f","5155bf0599baee0500000046","5155fe1d99baee05000000fb","5159f113ef9382050000003f","515326fac01d86050000008a","5152b9a28f4698d75500003f","515b597cb03eae0500000118","5152b68e8f4698d755000012","5152bbda8f4698d75500007f","5152b77b8f4698d75500001f","515b32927e4a740500000050","5155bdb999baee0500000036","515b9b92cd84280000000007","5159efa05b6c360500000022","515b32927e4a740500000050","5155bdb999baee0500000036","5152b6268f4698d75500000e","51531a58c01d860500000050","5152bb078f4698d755000062","5155bf0599baee0500000046","51531970c01d86050000004d","5154e1f18040040500000104","515726023efa030500000078","5152c0ebbceb379059000005","515330d0c3104205000000b6","51533c61e48367050000000f","515b3c6a7e4a7405000000b8","5157b6f27d39ed0500000078","515b9b92cd84280000000007","5155fe1d99baee05000000fb","515726023efa030500000078","5159f113ef9382050000003f","51588a977d39ed05000000dd","5157225013988705000000a1","515b9bf360e4eb0000000005","515711863efa030500000052","5157660deea7730500000025","5159efa05b6c360500000022","515b597cb03eae0500000118","515710e01398870500000056","5157125e1398870500000077","515b32927e4a740500000050","51531970c01d86050000004d","5152b6268f4698d75500000e","5152bbda8f4698d75500007f","5152b77b8f4698d75500001f","5152b9a28f4698d75500003f","5152c0ebbceb379059000005","5154b210faa9130500000078","5152b85d8f4698d75500002e","51532756c01d86050000008d","51533c61e48367050000000f","5157b6f27d39ed0500000078","5154e1f18040040500000104","5152bc178f4698d755000088","5155bdb999baee0500000036","5152b68e8f4698d755000012","5157125e1398870500000077","515b32927e4a740500000050","515326fac01d86050000008a","515b9bf360e4eb0000000005","515b3c6a7e4a7405000000b8","515376dee7e0a3050000001a","5155284e804004050000014d","5159efa05b6c360500000022","515b597cb03eae0500000118","5157225013988705000000a1","515710e01398870500000056","5157660deea7730500000025","5155fe1d99baee05000000fb","515c42d57e4a74050000027a","515c43e57e4a74050000027d","515b9bf360e4eb0000000005","5154e1f18040040500000104","5154b4a1804004050000008a","5152c0ebbceb379059000005","5152ba7a8f4698d755000052","5157660deea7730500000025","515b3c6a7e4a7405000000b8","51531a58c01d860500000050","515b32927e4a740500000050","5152b77b8f4698d75500001f","5152bc5d8f4698d75500008c","515b597cb03eae0500000118","5157125e1398870500000077","5152b68e8f4698d755000012","515711863efa030500000052","5152bb078f4698d755000062","51531970c01d86050000004d","51532756c01d86050000008d","5152b85d8f4698d75500002e","5152bb768f4698d75500006d","5155bdb999baee0500000036","5152b6268f4698d75500000e","5159f113ef9382050000003f","515b9b92cd84280000000007","515c42d57e4a74050000027a","5155284e804004050000014d","5157b6f27d39ed0500000078","5154b210faa9130500000078","5152b9a28f4698d75500003f","515726023efa030500000078","51588a977d39ed05000000dd","5157b6f27d39ed0500000078","5152ba7a8f4698d755000052","5154e1f18040040500000104","5152b68e8f4698d755000012","5155bd6799baee050000002d","515b3c6a7e4a7405000000b8","515326fac01d86050000008a","5159f113ef9382050000003f","51531970c01d86050000004d","5152bc5d8f4698d75500008c","5152bb768f4698d75500006d","5154b4a1804004050000008a","5152bbda8f4698d75500007f","5155bdb999baee0500000036","5152b85d8f4698d75500002e","5152b68e8f4698d755000012","515b9b92cd84280000000007","51532756c01d86050000008d","515b3c6a7e4a7405000000b8","51532756c01d86050000008d","5152bb078f4698d755000062","5155bf0599baee0500000046","5152b68e8f4698d755000012","5152ba7a8f4698d755000052","5155bf0599baee0500000046","5152b6268f4698d75500000e","51532756c01d86050000008d","515330d0c3104205000000b6","5154b210faa9130500000078","515c43e57e4a74050000027d","515c42d57e4a74050000027a","515c42d57e4a74050000027a","515b597cb03eae0500000118","515b9bf360e4eb0000000005","515b3c6a7e4a7405000000b8","5159efa05b6c360500000022","5159f113ef9382050000003f","5155bf0599baee0500000046","515b597cb03eae0500000118","51532756c01d86050000008d","515c43e57e4a74050000027d","51588a977d39ed05000000dd","5157125e1398870500000077","5152b6268f4698d75500000e","5154b210faa9130500000078","515b9bf360e4eb0000000005","5155bd6799baee050000002d","515c42d57e4a74050000027a","5157b6f27d39ed0500000078","5152bb768f4698d75500006d","515b597cb03eae0500000118","5155bdb999baee0500000036","5152bb078f4698d755000062","5152b68e8f4698d755000012","51533c61e48367050000000f","51532756c01d86050000008d","5155bd6799baee050000002d","5157225013988705000000a1","5155fe1d99baee05000000fb","5157125e1398870500000077","515726023efa030500000078","515c42d57e4a74050000027a","51531a58c01d860500000050","5154b210faa9130500000078","515c43e57e4a74050000027d","5159f113ef9382050000003f","515330d0c3104205000000b6","5159efa05b6c360500000022","5157660deea7730500000025","515710e01398870500000056","5157b6f27d39ed0500000078","5157225013988705000000a1","5157225013988705000000a1","5154e1f18040040500000104","5152b77b8f4698d75500001f","515710e01398870500000056","515711863efa030500000052","5152bb768f4698d75500006d","5152ba7a8f4698d755000052","5155284e804004050000014d","5152c0ebbceb379059000005","5155bd6799baee050000002d","5152b68e8f4698d755000012","515b9b92cd84280000000007","5152b6268f4698d75500000e","515b9bf360e4eb0000000005","515b597cb03eae0500000118","515b9b92cd84280000000007","515b32927e4a740500000050","515c42d57e4a74050000027a","515b3c6a7e4a7405000000b8","515c43e57e4a74050000027d","5157660deea7730500000025","51531a58c01d860500000050","51533c61e48367050000000f","5152b68e8f4698d755000012","5155bdb999baee0500000036","5157660deea7730500000025","5152bbda8f4698d75500007f","5155fe1d99baee05000000fb","515326fac01d86050000008a","515c42d57e4a74050000027a","515710e01398870500000056","5152b9a28f4698d75500003f","51588a977d39ed05000000dd","5159f113ef9382050000003f","5152bc178f4698d755000088","51531a58c01d860500000050","5154b4a1804004050000008a","5152bc5d8f4698d75500008c","5152c0ebbceb379059000005","5157225013988705000000a1","5154e1f18040040500000104","515b32927e4a740500000050","515b3c6a7e4a7405000000b8","5157125e1398870500000077","515b597cb03eae0500000118","5155284e804004050000014d","51532756c01d86050000008d","515c43e57e4a74050000027d","515726023efa030500000078","5157b6f27d39ed0500000078","515711863efa030500000052","5159efa05b6c360500000022","51533c61e48367050000000f","5152b77b8f4698d75500001f","5152bb078f4698d755000062","5152ba7a8f4698d755000052","515b9b92cd84280000000007","5155bdb999baee0500000036","5159f113ef9382050000003f","5157b6f27d39ed0500000078","5152c0ebbceb379059000005","5155bf0599baee0500000046","515b597cb03eae0500000118","5152b9a28f4698d75500003f","515710e01398870500000056","51532756c01d86050000008d","5157b6f27d39ed0500000078","515c43e57e4a74050000027d","515b9b92cd84280000000007","515376dee7e0a3050000001a","5155fe1d99baee05000000fb","515b9bf360e4eb0000000005","51531970c01d86050000004d","5152b77b8f4698d75500001f","515726023efa030500000078","5157660deea7730500000025","515b3c6a7e4a7405000000b8","5152bb078f4698d755000062","5152ba7a8f4698d755000052","5155284e804004050000014d","5152b77b8f4698d75500001f","5155bdb999baee0500000036","5152b68e8f4698d755000012","515c43e57e4a74050000027d","5155bf0599baee0500000046","5155fe1d99baee05000000fb","5152bc5d8f4698d75500008c","5152b9a28f4698d75500003f","51532756c01d86050000008d","515326fac01d86050000008a","5152ba7a8f4698d755000052","5155284e804004050000014d","5152bb768f4698d75500006d","515726023efa030500000078","5165cd7c050e0e0500000014","5157225013988705000000a1","5154e1f18040040500000104","5155bdb999baee0500000036","5152c0ebbceb379059000005","5157b6f27d39ed0500000078","515b597cb03eae0500000118","515b9bf360e4eb0000000005","5159f113ef9382050000003f","5159efa05b6c360500000022","515b3c6a7e4a7405000000b8","5152bbda8f4698d75500007f","51533c61e48367050000000f","5152b85d8f4698d75500002e","5152b77b8f4698d75500001f","515330d0c3104205000000b6","515711863efa030500000052","5152b6268f4698d75500000e","5152b9a28f4698d75500003f","5152bc178f4698d755000088","515726023efa030500000078","5165cd7c050e0e0500000014","515326fac01d86050000008a","5157125e1398870500000077","51531a58c01d860500000050","5157225013988705000000a1","5157b6f27d39ed0500000078","5155bd6799baee050000002d","515710e01398870500000056","51588a977d39ed05000000dd","5154e1f18040040500000104","5154b4a1804004050000008a","5152bbda8f4698d75500007f","5165cd7c050e0e0500000014","5155bdb999baee0500000036","5152bbda8f4698d75500007f","5159efa05b6c360500000022","5155284e804004050000014d","5152b6268f4698d75500000e","515b3c6a7e4a7405000000b8","5159f113ef9382050000003f","5152b77b8f4698d75500001f","51532756c01d86050000008d","5152b85d8f4698d75500002e","515711863efa030500000052","51588a977d39ed05000000dd","5152c0ebbceb379059000005","515c42d57e4a74050000027a","5155bf0599baee0500000046","5152bb768f4698d75500006d","5157660deea7730500000025","5154b4a1804004050000008a","515b3c6a7e4a7405000000b8","5152b85d8f4698d75500002e","5152b6268f4698d75500000e","5165cd7c050e0e0500000014","51531970c01d86050000004d","51531a58c01d860500000050","5152ba7a8f4698d755000052","5157660deea7730500000025","515b9b92cd84280000000007","51532756c01d86050000008d","515330d0c3104205000000b6","5155fe1d99baee05000000fb","515c42d57e4a74050000027a","5154e1f18040040500000104","515b3c6a7e4a7405000000b8","515b9bf360e4eb0000000005","5152b6268f4698d75500000e","5152ba7a8f4698d755000052","5154b210faa9130500000078","515c43e57e4a74050000027d","5154b4a1804004050000008a","515c43e57e4a74050000027d","515710e01398870500000056","5159f113ef9382050000003f","5152bc178f4698d755000088","5165cd7c050e0e0500000014","515330d0c3104205000000b6","515b597cb03eae0500000118","5155bf0599baee0500000046","51532756c01d86050000008d","5152bc178f4698d755000088","5159efa05b6c360500000022","515b9bf360e4eb0000000005","515b597cb03eae0500000118","5152c0ebbceb379059000005","5165cd7c050e0e0500000014","5152bc5d8f4698d75500008c","515376dee7e0a3050000001a","5152b77b8f4698d75500001f","5157225013988705000000a1","5152b6268f4698d75500000e","515b597cb03eae0500000118","5152b77b8f4698d75500001f","5152bbda8f4698d75500007f","51532756c01d86050000008d","5152ba7a8f4698d755000052","5155fe1d99baee05000000fb","515326fac01d86050000008a","51533c61e48367050000000f","515330d0c3104205000000b6","5152b68e8f4698d755000012","5152bbda8f4698d75500007f","51588a977d39ed05000000dd","5157225013988705000000a1","515b3c6a7e4a7405000000b8","515376dee7e0a3050000001a","5155fe1d99baee05000000fb","5155bdb999baee0500000036","5157125e1398870500000077","5154e1f18040040500000104","5154b4a1804004050000008a","515b9b92cd84280000000007","5159f113ef9382050000003f","5152c0ebbceb379059000005","5154b210faa9130500000078","5155bf0599baee0500000046","5157660deea7730500000025","51531970c01d86050000004d","515b9bf360e4eb0000000005","515726023efa030500000078","51531a58c01d860500000050","5152b6268f4698d75500000e","5154b4a1804004050000008a","515330d0c3104205000000b6","5157125e1398870500000077","5152bb078f4698d755000062","515710e01398870500000056","5152bc5d8f4698d75500008c","5152b6268f4698d75500000e","5155bd6799baee050000002d","51531a58c01d860500000050","515c43e57e4a74050000027d","515711863efa030500000052","5152bb768f4698d75500006d","51533c61e48367050000000f","5152c0ebbceb379059000005","51532756c01d86050000008d","5154e1f18040040500000104","51531a58c01d860500000050","5152bb078f4698d755000062","515b9b92cd84280000000007","515b3c6a7e4a7405000000b8","5157b6f27d39ed0500000078","515c42d57e4a74050000027a","51588a977d39ed05000000dd","5152b85d8f4698d75500002e","515b597cb03eae0500000118","5157b6f27d39ed0500000078","5152bb078f4698d755000062","5155284e804004050000014d","515376dee7e0a3050000001a","5155bd6799baee050000002d","5157225013988705000000a1","515726023efa030500000078","51588a977d39ed05000000dd","5159f113ef9382050000003f","515376dee7e0a3050000001a","5152bb078f4698d755000062","5154e1f18040040500000104","515b9bf360e4eb0000000005","51531970c01d86050000004d","515330d0c3104205000000b6","5152ba7a8f4698d755000052","51532756c01d86050000008d","5152c0ebbceb379059000005","5155284e804004050000014d","515b9bf360e4eb0000000005","5157225013988705000000a1","5155bdb999baee0500000036","51531970c01d86050000004d","5155bd6799baee050000002d","515c42d57e4a74050000027a","5152b85d8f4698d75500002e","5152b9a28f4698d75500003f","515b597cb03eae0500000118","5157660deea7730500000025","51588a977d39ed05000000dd","5152bb078f4698d755000062","515b3c6a7e4a7405000000b8","515c43e57e4a74050000027d","5152bbda8f4698d75500007f","5152bc5d8f4698d75500008c","5157125e1398870500000077","51532756c01d86050000008d","515711863efa030500000052","515b3c6a7e4a7405000000b8","5154b4a1804004050000008a","5154e1f18040040500000104","5152b68e8f4698d755000012","515c43e57e4a74050000027d","5152bb768f4698d75500006d","51532756c01d86050000008d","5152b68e8f4698d755000012","51588a977d39ed05000000dd","515b9bf360e4eb0000000005","5154e1f18040040500000104","5157b6f27d39ed0500000078","5159efa05b6c360500000022","5152b77b8f4698d75500001f","5152b6268f4698d75500000e","5152b85d8f4698d75500002e","5154b210faa9130500000078","5159efa05b6c360500000022","5155284e804004050000014d","5152bb078f4698d755000062","5157660deea7730500000025","5155bf0599baee0500000046","515376dee7e0a3050000001a","515330d0c3104205000000b6","5152b85d8f4698d75500002e","5157225013988705000000a1","515710e01398870500000056","5152c0ebbceb379059000005","5155fe1d99baee05000000fb","5152bb078f4698d755000062","5152ba7a8f4698d755000052"],"ptpt_id":["5152ad4c824d83a55500009f","5152ad4c824d83a55500009f","5152ad20824d83a55500009b","5152ad20824d83a55500009b","5152ad20824d83a55500009b","5152ad20824d83a55500009b","5152b5828f4698d75500000b","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5152b5828f4698d75500000b","5152b76b8f4698d75500001a","5152b76b8f4698d75500001a","5152b76b8f4698d75500001a","5152b5828f4698d75500000b","5152b57f8f4698d75500000a","5152b8408f4698d75500002a","5152b8408f4698d75500002a","5152b8408f4698d75500002a","5152b8408f4698d75500002a","5152b57f8f4698d75500000a","5152b76b8f4698d75500001a","5152b9948f4698d75500003a","5152b9948f4698d75500003a","5152b9948f4698d75500003a","5152b9948f4698d75500003a","5152b9948f4698d75500003a","5152b57f8f4698d75500000a","5152b8408f4698d75500002a","5152ba6f8f4698d75500004d","5152ba6f8f4698d75500004d","5152ba6f8f4698d75500004d","5152ba6f8f4698d75500004d","5152ba6f8f4698d75500004d","5152ba6f8f4698d75500004d","5152b8408f4698d75500002a","5152b57f8f4698d75500000a","5152b8408f4698d75500002a","5152ba6f8f4698d75500004d","5152b57f8f4698d75500000a","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152bb6f8f4698d755000068","5152b8408f4698d75500002a","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5152bb6f8f4698d755000068","5152b57f8f4698d75500000a","5152b8408f4698d75500002a","5152b57f8f4698d75500000a","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152bc698f4698d755000092","5152b8408f4698d75500002a","5152b8408f4698d75500002a","5152b8408f4698d75500002a","5152b57f8f4698d75500000a","5152bc698f4698d755000092","5152b5bb1145c605000000dc","5152b5bb1145c605000000dc","5152b5bb1145c605000000dc","5152c919c310420500000005","5152c919c310420500000005","5152c919c310420500000005","5150fb197d799505000000a7","5151221f1d23610500000040","51509498edeb93050000006b","51509498edeb93050000006b","51509498edeb93050000006b","51509498edeb93050000006b","5152fdf4c310420500000013","5152fdf4c310420500000013","5152fdf4c310420500000013","5152fdf4c310420500000013","5152fdf4c310420500000013","5152fdf4c310420500000013","51509498edeb93050000006b","51509498edeb93050000006b","51509498edeb93050000006b","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","5153129dc31042050000002d","5153129dc31042050000002d","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5153129dc31042050000002d","5151badaf2774c050000001e","5153129dc31042050000002d","5151badaf2774c050000001e","5151badaf2774c050000001e","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5151badaf2774c050000001e","5153129dc31042050000002d","5153129dc31042050000002d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","515120451d23610500000019","5151badaf2774c050000001e","515120451d23610500000019","515120451d23610500000019","51531ad7c01d86050000005d","515120451d23610500000019","515120451d23610500000019","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","5151221f1d23610500000040","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","515325c7c31042050000006f","5151221f1d23610500000040","5151221f1d23610500000040","5151221f1d23610500000040","51532a8ec01d860500000093","51532a8ec01d860500000093","51532c7dc31042050000008c","51532c7dc31042050000008c","51532c7dc31042050000008c","51532c7dc31042050000008c","51532c7dc31042050000008c","51532e70c01d8605000000a9","51532faec01d8605000000ac","51532e70c01d8605000000a9","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532e70c01d8605000000a9","51532faec01d8605000000ac","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532faec01d8605000000ac","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532e70c01d8605000000a9","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532e70c01d8605000000a9","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532fb3c01d8605000000af","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532e70c01d8605000000a9","515331aac01d8605000000e3","515331aac01d8605000000e3","515331aac01d8605000000e3","51532e70c01d8605000000a9","51532e70c01d8605000000a9","51532e70c01d8605000000a9","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5153382ac01d860500000109","5153382ac01d860500000109","5153382ac01d860500000109","5153382ac01d860500000109","5153382ac01d860500000109","5153382ac01d860500000109","5152b5bb1145c605000000dc","5152b5bb1145c605000000dc","5153382ac01d860500000109","5151221f1d23610500000040","51532faec01d8605000000ac","51532faec01d8605000000ac","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","51533d49d8e69b0500000011","5153439fd8e69b050000002f","5153439fd8e69b050000002f","5153439fd8e69b050000002f","5153439fd8e69b050000002f","5153439fd8e69b050000002f","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","51531ad7c01d86050000005d","514a1eec2878140500000015","514a1eec2878140500000015","514a1eec2878140500000015","514a1eec2878140500000015","51535ba62d7ddb050000000a","51535bc62d7ddb050000000e","51535c1ed3ccd30500000012","51535c35d3ccd30500000016","51535c3fd3ccd30500000018","51535c412d7ddb0500000012","51535c492d7ddb0500000016","51535caad3ccd30500000025","51535cc12d7ddb050000001c","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","515360ffd3ccd30500000035","51537165b3060b050000000c","51537168e7e0a30500000009","5153716cb3060b0500000012","5153717be7e0a3050000000d","514a485b28781405000000c2","514a485b28781405000000c2","514a485b28781405000000c2","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153759cb3060b0500000026","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","515382e7e7e0a30500000038","5153830ae7e0a3050000003a","51538390b3060b050000004f","515383a8b3060b0500000053","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","5153823bb3060b0500000042","515383dde7e0a30500000043","515383e3e7e0a30500000047","5153823bb3060b0500000042","515383e6b3060b050000005b","5153840ab3060b050000005f","51538452e7e0a3050000004c","5153846ae7e0a30500000050","515384c0e7e0a30500000054","515384efe7e0a30500000058","515384c0e7e0a30500000054","51538519e7e0a3050000005d","51532faec01d8605000000ac","51532faec01d8605000000ac","5153d982b3060b0500000098","5153d982b3060b0500000098","5153d982b3060b0500000098","5153d982b3060b0500000098","5153d982b3060b0500000098","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5153f791b71ab40500000002","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5152b57f8f4698d75500000a","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5151af8bf2774c0500000005","51549730804004050000001c","51549c6e8040040500000020","51549c6e8040040500000020","51549c6e8040040500000020","51549c6e8040040500000020","51549c6e8040040500000020","51549c6e8040040500000020","51549c6e8040040500000020","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5151af8bf2774c0500000005","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154b0138040040500000045","5154b0138040040500000045","5154ae78804004050000003d","5154b0138040040500000045","5154b0138040040500000045","5154b0138040040500000045","5154b038804004050000004a","5154b038804004050000004a","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154ae78804004050000003d","5154b0738040040500000052","5154ae78804004050000003d","5154b0738040040500000052","5154ae78804004050000003d","5154b0738040040500000052","5154ae78804004050000003d","5154b05b804004050000004e","5154b05b804004050000004e","5154b05b804004050000004e","5154b05b804004050000004e","5154b05b804004050000004e","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b038804004050000004a","5154b038804004050000004a","5154b05b804004050000004e","5154b05b804004050000004e","5154b05b804004050000004e","5154b038804004050000004a","5154b038804004050000004a","5154b05b804004050000004e","5154b05b804004050000004e","5154b038804004050000004a","5154b038804004050000004a","5154b038804004050000004a","5154b05b804004050000004e","5154b038804004050000004a","5154b05b804004050000004e","5154b038804004050000004a","5154b038804004050000004a","5154b038804004050000004a","5154b038804004050000004a","5154b038804004050000004a","5154ae78804004050000003d","5154b038804004050000004a","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154b05b804004050000004e","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b8e0804004050000009b","5154b94dfaa91305000000af","5154b8e0804004050000009b","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154b94dfaa91305000000af","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ba2980400405000000b6","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d3b980400405000000de","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154d63c80400405000000ec","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154b038804004050000004a","5154fe9dfaa913050000011a","5154fe9dfaa913050000011a","5154fe9dfaa913050000011a","5154fe9dfaa913050000011a","51532faec01d8605000000ac","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","51550edffaa9130500000127","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","51551ebffaa913050000013e","5154b038804004050000004a","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","51554db199baee0500000001","5155629999baee050000000e","5155629999baee050000000e","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","5155629999baee050000000e","5155629999baee050000000e","5155629999baee050000000e","5155629999baee050000000e","5155629999baee050000000e","5151badaf2774c050000001e","51533436c3104205000000dd","51533436c3104205000000dd","51533436c3104205000000dd","51533436c3104205000000dd","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5155bceee93dc30500000022","5151badaf2774c050000001e","5151badaf2774c050000001e","5151badaf2774c050000001e","5155bceee93dc30500000022","5154b0738040040500000052","5154b0738040040500000052","5155bceee93dc30500000022","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154b0738040040500000052","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155c8de99baee0500000059","5155ccede93dc30500000068","5155ccede93dc30500000068","5155ccede93dc30500000068","5155ccede93dc30500000068","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155ccede93dc30500000068","5155ccede93dc30500000068","5155ccede93dc30500000068","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cc6899baee0500000073","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155cda199baee0500000080","5155d4d0e93dc3050000009a","5155d4d0e93dc3050000009a","5155d4d0e93dc3050000009a","5155d4d0e93dc3050000009a","5155d4d0e93dc3050000009a","51554663804004050000015c","51554663804004050000015c","51554663804004050000015c","51554663804004050000015c","51554663804004050000015c","51554663804004050000015c","51554663804004050000015c","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155f7ec99baee05000000bb","5155faf399baee05000000d4","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fa0399baee05000000d2","5155fd9ce93dc305000000e7","5155fd9ce93dc305000000e7","5155fd9ce93dc305000000e7","5155fd9ce93dc305000000e7","5155fd9ce93dc305000000e7","5155fd9ce93dc305000000e7","51560250e93dc305000000ed","51560250e93dc305000000ed","51560250e93dc305000000ed","5156085799baee0500000104","5156085799baee0500000104","5156085799baee0500000104","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","5156169199baee0500000120","51561c1899baee0500000133","51561c1899baee0500000133","51561c1899baee0500000133","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51562466c2f75d0500000033","51564f5299baee050000015b","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","515660e0c2f75d0500000052","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","51565e7ec2f75d0500000050","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","5155414a8040040500000157","51564f5299baee050000015b","51564f5299baee050000015b","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156fdbecf788f0000000001","5156f9bb13bed40000000001","5156f9bb13bed40000000001","5156f9bb13bed40000000001","5156f9bb13bed40000000001","5156f9bb13bed40000000001","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51571015139887050000003b","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51560d4299baee0500000115","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51571015139887050000003b","51570ed5139887050000002d","51570ed5139887050000002d","51571015139887050000003b","51570ed5139887050000002d","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51560d4299baee0500000115","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51571015139887050000003b","51560d4299baee0500000115","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","515717431398870500000083","51571c5f3efa030500000070","51571c5f3efa030500000070","51571c5f3efa030500000070","51571c5f3efa030500000070","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","51570ed5139887050000002d","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","5157281213988705000000a3","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","515729f913988705000000ba","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","51573b653efa0305000000af","5152b57f8f4698d75500000a","51562466c2f75d0500000033","51562466c2f75d0500000033","5152b57f8f4698d75500000a","51554db199baee0500000001","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515756d013988705000000ea","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","515711003efa03050000004b","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157b1af7d39ed0500000054","5157ac53eea7730500000055","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157ac53eea7730500000055","5157ac53eea7730500000055","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157ac53eea7730500000055","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157b1af7d39ed0500000054","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157ac53eea7730500000055","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5157c4ddeea773050000009b","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","5154ad4dfaa9130500000037","51585334eea77305000000d0","51585334eea77305000000d0","515889707d39ed05000000d9","515889707d39ed05000000d9","515889707d39ed05000000d9","515889707d39ed05000000d9","515889707d39ed05000000d9","515889707d39ed05000000d9","515889707d39ed05000000d9","515468bc8040040500000008","515468bc8040040500000008","515468bc8040040500000008","515468bc8040040500000008","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","51589d968c0452050000000a","5157c4ddeea773050000009b","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5158ea73b19d1b05000000b6","5158ea73b19d1b05000000b6","5158ea73b19d1b05000000b6","5158ea73b19d1b05000000b6","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5153129dc31042050000002d","5158ea73b19d1b05000000b6","5158ea73b19d1b05000000b6","5153129dc31042050000002d","5158f3cf8c045205000000bf","5158f3cf8c045205000000bf","5158f3cf8c045205000000bf","5158f3cf8c045205000000bf","5158f3cf8c045205000000bf","5158f3cf8c045205000000bf","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","51539fdab3060b0500000074","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","5153759cb3060b0500000026","5159980db19d1b050000010b","5159980db19d1b050000010b","5159980db19d1b050000010b","5159980db19d1b050000010b","5159980db19d1b050000010b","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159b01fb19d1b0500000112","5159e0de5b6c360500000004","5159e0ef5b6c360500000007","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","515990168c04520500000102","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159edde5b6c36050000001b","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","5159901eb19d1b05000000fb","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","515a3551ef93820500000054","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","51560d4299baee0500000115","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b222f7e4a740500000007","515b222f7e4a740500000007","515b222f7e4a740500000007","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b222f7e4a740500000007","515b2268b03eae0500000010","515b2268b03eae0500000010","515b2268b03eae0500000010","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b27f77e4a74050000001d","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b38067e4a740500000074","515b38067e4a740500000074","515b38067e4a740500000074","515b38067e4a740500000074","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","5158ea73b19d1b05000000b6","515b37137e4a740500000056","5158ea73b19d1b05000000b6","5158ea73b19d1b05000000b6","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b37137e4a740500000056","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b39047e4a740500000099","515b40d67e4a7405000000bd","515b40d67e4a7405000000bd","515b40d67e4a7405000000bd","515b41197e4a7405000000c1","515b41197e4a7405000000c1","515b41197e4a7405000000c1","515b47e1b03eae05000000bc","515b47e1b03eae05000000bc","515b47e1b03eae05000000bc","515b47e1b03eae05000000bc","515b47e1b03eae05000000bc","515b47e1b03eae05000000bc","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b4a00b03eae05000000c5","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b535c7e4a7405000000f3","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b5435b03eae05000000fb","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b6340b03eae0500000129","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","515b66347e4a740500000149","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","515b86287e4a740500000185","515b86287e4a740500000185","515b8fb7b03eae0500000194","515b8fb7b03eae0500000194","515b8fb7b03eae0500000194","515b8fb7b03eae0500000194","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9112b03eae050000019b","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9326b03eae05000001b4","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9a314a60870000000001","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515b9f867e4a7405000001c9","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","51532de8c31042050000009a","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515ba4957e4a740500000208","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515bc5187e4a740500000225","515c399f7e4a740500000237","515c39b37e4a740500000239","515c39ebb03eae0500000249","515c39f2b03eae050000024b","515c3a2e7e4a74050000023d","515c3a587e4a740500000241","515c3a6eb03eae050000024f","515c3a7db03eae0500000251","515c3a81b03eae0500000253","515c3aa1b03eae0500000257","515c3ab3b03eae050000025b","515c3ada7e4a74050000024b","515c3aee7e4a74050000024d","515c3afb7e4a740500000251","515c3b02b03eae0500000261","515c3bdf7e4a740500000253","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c3f63b03eae0500000263","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c42f0b03eae0500000282","515c4fe47e4a7405000002a2","515c53e97e4a7405000002a7","515c53e97e4a7405000002a7","515c53e97e4a7405000002a7","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","515c76ed92ad6b0500000004","5154ad4dfaa9130500000037","515cb16fce4e46050000000f","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515dc1088a38a5050000001d","515dda99d9c1420500000001","515dda99d9c1420500000001","515dda99d9c1420500000001","515dda99d9c1420500000001","515e0612d9c142050000000d","515e061ed9c1420500000011","515ef22d64aff30500000024","515ef26b64aff30500000026","515ef2a664aff30500000028","515ef2a6d9c1420500000022","515ef2ccd9c1420500000026","515ef2f664aff3050000002e","515f00cfd9c1420500000028","515f00cfd9c1420500000028","515f00cfd9c1420500000028","515f00cfd9c1420500000028","515f11b864aff3050000003b","515f11b864aff3050000003b","515f11b864aff3050000003b","515f11b864aff3050000003b","515f1272d9c1420500000035","515f1272d9c1420500000035","515f1272d9c1420500000035","515f1272d9c1420500000035","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f20c964aff3050000004b","515f26d964aff30500000064","515f26ebff75d10500000005","515f2704ff75d10500000007","515f271164aff3050000006c","515f273664aff3050000006e","515f276564aff30500000072","515f2776ff75d1050000000d","515f2779ff75d1050000000f","515f277fff75d10500000013","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","5160780aa7779e0500000017","516081716d55ab0500000008","5160818f855e2b0500000003","516081936d55ab050000000c","516081d76d55ab0500000010","516081e06d55ab0500000014","5160820f6d55ab0500000016","51608216855e2b050000000b","5160ecda855e2b0500000011","5160f5306d55ab0500000023","5160f5456d55ab0500000027","5160f5586d55ab050000002b","5160f55b855e2b0500000023","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","515ce377ce4e460500000014","5161cdd16d55ab0500000058","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161d63df29cdf0500000001","5161ee4e17baf7050000001c","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","5162094517baf70500000024","51646bb3edca65050000000e","51646be7edca650500000010","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5165ca9ba85a980500000004","5166e70e84a6200500000008","5166e710892cba0500000008","5166e72a892cba050000000a","5166e742892cba050000000c","5166e745892cba050000000e","5166e74f892cba0500000012","5166e763892cba0500000014","5166e772892cba0500000018","5166e77c84a6200500000010","5166e78a84a6200500000012","5166e7a184a6200500000016","5166e7a8892cba0500000020","5166e7be84a620050000001a","5166e7c984a620050000001e","5166e7cf84a6200500000020","51674692892cba050000002f","5167469684a620050000002e","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676484892cba0500000033","51676930892cba0500000045","51676930892cba0500000045","51676a2584a6200500000046","51676a39892cba050000004d","51676a44892cba050000004f","51685979d13a390500000032","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5169fa1ee247c70500000004","5169fa35e247c70500000008","5169fa40e247c7050000000a","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","516a007ce247c70500000014","5164db5f0c2ca1da1c000001","516aeae6a994c90500000071","516af158f34ecd05000000b9","516af1a8a994c905000000a3","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","5164db5f0c2ca1da1c000001","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","516dcc8010bf980500000003","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517719e7d512840500000014","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517c88e0e5519a050000000d","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","517f61dc1e9e210500000004","5181a2068d5a8f0500000047","5181a2836c3ffb050000003f","5181a2836c3ffb050000003f","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","517a9ca39192920500000003","518a8ab32dc77f050000000f","518a8ab32dc77f050000000f","518a8ab32dc77f050000000f","518a8ab32dc77f050000000f","518a8ab32dc77f050000000f","518d768c1487570500000004","518d768c1487570500000004","518d768c1487570500000004","518d768c1487570500000004","518d768c1487570500000004","51909b64fd0b100500000005","51909b64fd0b100500000005","51909b64fd0b100500000005","51909b64fd0b100500000005","51909b64fd0b100500000005","51909b64fd0b100500000005","51909b64fd0b100500000005","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","5196648ba7b6da0500000004","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bff865723000500000009","519bff865723000500000009","519bff865723000500000009","519bff865723000500000009","517dc4e555919ccb0d00007e","51a6ba80e02c980500000001","51a6ba80e02c980500000001","51a6ba80e02c980500000001","51a6ba80e02c980500000001","51a6ba80e02c980500000001","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","519bfd535723000500000003","51abbf06dbb5350500000002","51abbf06dbb5350500000002","51abbf06dbb5350500000002","51abbf06dbb5350500000002","51abbf06dbb5350500000002","51abbf06dbb5350500000002","51abbf06dbb5350500000002","5181a2836c3ffb050000003f","5181a2836c3ffb050000003f"],"rating":[-1,0,1,1,1,1,-1,-1,-1,-1,-1,-1,1,1,1,-1,-1,0,1,-1,0,-1,-1,1,-1,1,1,1,-1,1,1,-1,1,-1,1,1,-1,1,-1,-1,-1,-1,1,-1,1,1,1,-1,-1,0,-1,-1,-1,-1,-1,-1,1,-1,1,1,1,-1,-1,-1,1,-1,-1,-1,1,0,0,-1,-1,0,-1,-1,-1,-1,0,1,1,-1,-1,-1,-1,-1,-1,-1,1,1,1,-1,-1,-1,1,-1,1,-1,1,-1,1,0,0,1,1,1,1,0,-1,-1,-1,-1,0,1,1,-1,-1,0,-1,-1,-1,-1,-1,-1,-1,1,-1,-1,1,0,1,-1,-1,1,-1,-1,-1,-1,-1,-1,1,-1,1,-1,-1,-1,-1,-1,0,1,-1,1,-1,1,-1,-1,1,1,1,1,1,-1,-1,-1,-1,-1,1,0,1,1,0,-1,-1,-1,0,1,-1,-1,0,-1,-1,-1,1,-1,1,-1,0,0,0,-1,-1,-1,1,1,1,0,-1,0,-1,-1,0,-1,1,0,0,-1,-1,-1,-1,1,1,-1,-1,1,1,-1,0,-1,0,-1,1,0,0,0,-1,-1,1,-1,1,-1,1,-1,-1,0,0,-1,1,0,-1,-1,1,-1,-1,-1,-1,1,1,1,-1,-1,-1,-1,1,0,0,1,1,1,-1,1,0,-1,0,1,-1,0,-1,-1,1,1,1,0,1,-1,1,1,1,-1,-1,-1,1,1,1,0,1,1,1,1,0,-1,-1,-1,-1,-1,1,-1,-1,-1,1,-1,-1,0,0,0,0,1,-1,1,-1,-1,1,1,1,1,-1,1,-1,-1,-1,0,-1,1,-1,-1,1,1,-1,1,-1,-1,-1,-1,1,-1,-1,-1,-1,1,-1,-1,0,-1,1,1,1,-1,-1,0,-1,-1,0,-1,-1,-1,-1,1,1,0,-1,-1,-1,-1,1,0,1,-1,-1,-1,-1,1,-1,1,1,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,1,1,1,1,-1,0,1,-1,0,0,0,-1,1,-1,-1,-1,-1,0,-1,1,0,1,0,1,-1,-1,-1,1,-1,1,-1,1,1,-1,-1,0,-1,-1,-1,0,0,1,1,-1,1,0,1,-1,-1,1,-1,-1,-1,-1,-1,0,-1,-1,-1,-1,1,0,-1,0,1,1,-1,1,-1,-1,0,1,-1,0,1,1,1,-1,0,-1,-1,1,-1,0,-1,-1,-1,-1,-1,0,0,-1,-1,-1,0,1,-1,-1,1,0,-1,0,-1,0,1,-1,-1,-1,-1,0,-1,-1,1,-1,-1,0,1,1,-1,-1,0,1,-1,-1,-1,-1,-1,-1,1,1,1,1,-1,-1,0,1,-1,1,1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,1,1,-1,-1,-1,-1,0,-1,0,-1,0,-1,-1,1,-1,-1,0,-1,1,1,0,1,-1,1,1,1,-1,-1,-1,-1,-1,1,-1,0,-1,1,1,1,1,0,-1,-1,-1,-1,-1,0,0,-1,0,1,1,-1,-1,-1,-1,0,0,1,0,1,1,-1,-1,-1,-1,1,1,1,0,1,-1,-1,-1,-1,0,-1,0,-1,-1,0,-1,-1,1,-1,0,1,1,1,0,1,-1,-1,0,0,-1,0,-1,-1,0,-1,0,1,0,0,0,1,1,0,-1,-1,0,-1,-1,1,-1,-1,0,1,-1,-1,1,1,0,1,-1,0,0,-1,-1,-1,1,0,1,-1,-1,1,0,-1,0,-1,-1,0,0,-1,-1,1,1,1,-1,-1,-1,-1,-1,0,-1,-1,-1,1,1,-1,1,-1,0,0,-1,-1,-1,-1,-1,-1,0,1,-1,1,-1,-1,-1,-1,-1,1,1,-1,1,1,1,-1,1,0,1,-1,-1,-1,-1,-1,0,0,-1,-1,-1,0,0,0,1,0,1,-1,0,-1,1,-1,1,-1,1,-1,1,-1,0,-1,-1,-1,1,-1,-1,1,1,-1,1,-1,-1,0,0,-1,-1,1,0,1,0,-1,-1,-1,-1,-1,1,-1,-1,-1,0,1,1,1,1,0,-1,1,0,0,-1,-1,-1,0,1,-1,-1,-1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,0,-1,-1,-1,1,1,1,-1,-1,-1,-1,1,-1,1,0,1,-1,-1,1,1,-1,-1,-1,0,-1,0,0,1,1,-1,1,1,1,-1,-1,-1,1,-1,-1,1,-1,-1,-1,-1,-1,1,1,1,1,-1,0,1,0,-1,-1,0,-1,0,-1,0,-1,-1,1,1,0,-1,-1,-1,-1,-1,1,0,0,1,-1,-1,0,0,1,0,-1,0,0,1,-1,0,1,-1,-1,-1,0,1,-1,1,0,-1,1,-1,-1,1,1,1,-1,-1,-1,-1,0,1,1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,-1,-1,-1,1,-1,1,1,-1,-1,-1,-1,1,1,0,-1,-1,-1,-1,-1,0,0,0,-1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,1,1,-1,1,-1,1,-1,0,-1,-1,1,-1,1,1,-1,1,0,1,-1,0,-1,1,-1,-1,1,-1,-1,-1,-1,1,-1,-1,0,-1,-1,-1,-1,1,-1,-1,1,0,1,-1,1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,0,0,-1,-1,-1,0,-1,-1,1,1,1,0,-1,-1,-1,-1,0,-1,1,-1,-1,1,1,1,1,1,1,-1,1,0,-1,0,-1,-1,0,1,-1,0,-1,1,-1,1,-1,0,-1,-1,1,-1,-1,-1,-1,-1,-1,-1,-1,1,0,0,-1,-1,0,-1,-1,-1,1,0,0,1,0,-1,0,0,-1,0,0,-1,-1,-1,1,0,-1,1,-1,0,-1,-1,-1,1,1,-1,-1,-1,-1,1,-1,0,-1,0,0,0,-1,0,0,1,0,-1,1,-1,-1,0,0,1,1,-1,-1,-1,-1,0,1,1,1,0,1,1,0,-1,0,1,1,1,0,0,1,-1,-1,1,-1,-1,-1,-1,1,1,1,1,1,0,1,1,1,1,1,0,-1,-1,0,1,1,0,-1,0,0,1,1,-1,1,1,1,1,1,0,0,1,1,1,0,-1,-1,-1,-1,-1,-1,1,-1,-1,-1,1,-1,0,-1,-1,-1,-1,0,-1,-1,1,-1,-1,-1,0,1,-1,1,-1,1,-1,-1,0,-1,-1,0,0,0,-1,-1,1,-1,-1,1,1,0,1,0,0,0,0,1,-1,0,1,-1,-1,-1,-1,-1,-1,1,-1,1,0,1,0,0,-1,0,-1,1,0,-1,1,-1,0,1,1,0,-1,0,0,-1,1,1,-1,0,0,-1,1,-1,0,-1,1,1,0,1,1,-1,0,1,-1,-1,-1,-1,-1,-1,-1,1,0,1,1,-1,1,1,1,-1,0,-1,-1,0,-1,1,-1,-1,1,1,1,-1,1,1,1,0,-1,-1,-1,-1,-1,1,1,0,-1,0,0,1,0,-1,-1,0,1,1,0,-1,1,1,-1,1,1,-1,-1,-1,-1,0,0,0,0,0,-1,1,-1,-1,-1,-1,-1,1,1,-1,1,-1,1,1,1,-1,-1,1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,1,1,1,-1,-1,1,1,-1,-1,-1,-1,-1,-1,1,-1,-1,-1,0,0,-1,-1,-1,-1,-1,1,-1,1,-1,-1,-1,-1,0,0,1,0,-1,0,-1,1,-1,-1,1,1,-1,-1,1,1,-1,1,-1,-1,1,1,-1,1,1,-1,1,-1,0,1,-1,-1,-1,1,1,-1,-1,1,0,-1,-1,1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,-1,-1,-1,-1,1,-1,-1,-1,0,-1,-1,-1,0,-1,1,0,-1,-1,-1,1,-1,1,1,1,1,1,-1,-1,1,-1,-1,-1,1,-1,-1,-1,-1,0,-1,1,1,-1,-1,1,-1,-1,0,-1,1,1,-1,-1,-1,1,-1,-1,1,0,1,1,0,1,1,1,-1,-1,-1,-1,0,0,-1,-1,1,1,-1,-1,-1,0,1,-1,0,-1,1,-1,1,-1,1,-1,0,1,0,1,-1,-1,1,1,0,0,-1,0,-1,-1,-1,-1,1,-1,1,-1,-1,-1,1,-1,-1,1,1,0,1,1,-1,0,1,-1,0,1,0,0,1,-1,0,0,-1,0,1,0,1,1,-1,1,1,0,1,-1,0,1,-1,-1,0,0,-1,-1,-1,-1,0,-1,-1,0,0,1,-1,-1,-1,1,0,0,0,1,0,0,-1,-1,0,0,-1,-1,0,-1,-1,0,0,0,0,-1,-1,1,0,1,1,-1,0,0,0,1,-1,-1,-1,0,-1,0,-1,0,0,-1,0,0,0,0,0,0,0,-1,-1,-1,0,0,0,0,-1,1,0,0,0,1,-1,1,-1,1,-1,-1,1,-1,-1,1,1,-1,-1,1,1,-1,1,0,0,-1,0,1,1,0,0,1,0,-1,-1,1,-1,1,0,1,-1,-1,1,1,-1,-1,-1,0,-1,0,-1,-1,1,1,-1,-1,-1,0,1,-1,-1,-1,-1,1,-1,-1,1,-1,-1,-1,0,0,1,1,0,-1,0,-1,-1,-1,-1,-1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,0,-1,1,0,1,-1,-1,-1,1,-1,-1,-1,-1,-1,1,-1,1,-1,1,-1,-1,-1,-1,-1,0,-1,-1,-1,1,1,-1,-1,-1,-1,-1,1,1,-1,1,-1,-1,1,-1,0,1,1,1,-1,-1,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,1,-1,-1,-1,-1,-1,-1,-1,1,-1,-1,1,-1,-1,1,1,-1,-1,-1,-1,1,-1,0,-1,-1,-1,-1,1,0,-1,1,0,-1,1,1,-1,-1,-1,-1,1,-1,-1,-1,0,0,0,0,-1,1,-1,-1,-1,1,1,1,0,1,-1,-1,0,-1,0,-1,1,0,1,0,0,0,0,0,0,0,-1,0,1,1,-1,0,1,0,0,-1,-1,1,-1,-1,1,-1,1,-1,-1,-1,-1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,-1,1,1,-1,-1,-1,0,-1,1,-1,-1,-1,1,-1,1,0,0,1,0,1,1,1,1,-1,1,-1,-1,-1,-1,0,0,0,1,-1,-1,-1,-1,1,-1,-1,-1,-1,0,-1,0,-1,-1,0,-1,-1,-1,-1,0,0,-1,-1,-1,0,-1,-1,1,1,1,-1,-1,1,1,1,-1,-1,-1,0,-1,-1,1,-1,0,0,1,1,-1,0,-1,-1,1,-1,0,0,0,0,1,0,-1,1,0,-1,1,-1,-1,-1,-1,-1,-1,0,0,-1,-1,0,0,0,0,0,-1,0,1,0,-1,0,1,-1,0,0,-1,-1,-1,0,-1,0,-1,-1,-1,-1,-1,0,0,0,0,-1,0,-1,0,0,1,1,0,0,-1,-1,0,0,0,0,-1,-1,0,0,0,0,1,0,-1,0,-1,0,-1,-1,-1,1,0,1,-1,-1,-1,-1,1,-1,-1,-1,-1,0,0,0,0,-1,-1,-1,-1,-1,1,1,-1,-1,-1,-1,1,-1,0,1,1,-1,1,1,-1,0,0,1,-1,1,1,0,0,0,1,-1,1,0,0,-1,0,-1,-1,1,-1,-1,1,-1,-1,1,1,-1,1,-1,-1,0,1,-1,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,-1,-1,-1,-1,1,-1,1,-1,-1,1,0,1,-1,1,1,-1,-1,-1,1,0,1,0,-1,1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,1,-1,1,1,0,-1,0,-1,1,1,-1,-1,-1,1,-1,-1,-1,1,1,-1,-1,0,-1,-1,-1,1,-1,-1,0,-1,1]}} diff --git a/math/deps.edn b/math/deps.edn deleted file mode 100644 index 44e634970c..0000000000 --- a/math/deps.edn +++ /dev/null @@ -1,83 +0,0 @@ -{:paths ["src"] - :deps {org.clojure/clojure {:mvn/version "1.11.1"} - org.clojure/spec.alpha {:mvn/version "0.3.218"} - org.clojure/core.async {:mvn/version "1.6.673"} - org.clojure/data.csv {:mvn/version "1.0.1"} - org.clojure/math.numeric-tower {:mvn/version "0.0.5"} - org.clojure/core.match {:mvn/version "1.0.1"} - org.clojure/tools.namespace {:mvn/version "1.3.0"} - org.clojure/tools.logging {:mvn/version "1.2.4"} - org.clojure/tools.trace {:mvn/version "0.7.11"} - org.clojure/tools.reader {:mvn/version "1.3.6"} - - org.flatland/ordered {:mvn/version "1.15.10"} - ;; Other stuff - commons-collections/commons-collections {:mvn/version "20040616"} - cheshire/cheshire {:mvn/version "5.11.0"} - com.taoensso/timbre {:mvn/version "5.2.1"} - ;; Troublesome carmine... was using this for simulation stuff - ;[com.taoensso/carmine "2.7.0" :exclusions [org.clojure/clojure]] - ;; Updates; requires fixing index conflict between named-matrix and core.matrix - net.mikera/core.matrix {:mvn/version "0.63.0"} - net.mikera/vectorz-clj {:mvn/version "0.48.0"} - ;[net.mikera/core.matrix "0.23.0"] - net.mikera/core.matrix.stats {:mvn/version "0.7.0"} - criterium/criterium {:mvn/version "0.4.6"} - clj-http/clj-http {:mvn/version "3.12.3"} - ;; We should be able to switch back to this now that we aren't using storm - org.clojure/tools.cli {:mvn/version "1.0.214"} - ;; implicitly requires jetty, component and ring - ring/ring-core {:mvn/version "1.9.6" :exclusions [clj-time/clj-time]} - ring-jetty-component/ring-jetty-component {:mvn/version "0.3.1" :exclusions [clj-time/clj-time]} - ring-basic-authentication/ring-basic-authentication {:mvn/version "1.1.1"} - ring/ring-ssl {:mvn/version "0.3.0"} - bidi/bidi {:mvn/version "2.1.6" :exclusions [prismatic/schema]} - ;; Taking out storm cause yeah... - ;[org.apache.storm/storm-core "0.9.2-incubating"] - ;; Can't find bouncycastle trying to use this - ;incanter {:mvn/version "1.9.3" :exclusions [org.clojure/clojure]} - bigml/sampling {:mvn/version "3.2"} - com.cognitect.aws/api {:mvn/version "0.8.641"} - com.cognitect.aws/endpoints {:mvn/version "1.1.12.398"} - com.cognitect.aws/s3 {:mvn/version "825.2.1250.0"} - - org.postgresql/postgresql {:mvn/version "42.5.2"} - korma/korma {:mvn/version "0.4.3"} - ;; Enhanced connection pooling - com.zaxxer/HikariCP {:mvn/version "5.0.1"} - clj-time/clj-time {:mvn/version "0.15.2"} - ;clj-excel {:mvn/version "0.0.1"} - semantic-csv/semantic-csv {:mvn/version "0.2.0"} - ;[dk.ative/docjure {:mvn/version "1.13.0"] - prismatic/plumbing {:mvn/version "0.6.0"} - environ/environ {:mvn/version "1.2.0"} - mount/mount {:mvn/version "0.1.17"} - honeysql/honeysql {:mvn/version "1.0.461"} - ;; Dev - org.clojure/test.check {:mvn/version "1.1.1"} - irresponsible/tentacles {:mvn/version "0.6.9"} - techascent/tech.ml.dataset {:mvn/version "6.104"}} - :aliases {:dev - {:extra-paths ["dev" "test"] - :extra-deps {cider/cider-nrepl {:mvn/version "0.30.0"} - metasoarous/oz {:mvn/version "2.0.0-alpha5"}} - :main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]} - :run - {:main-opts ["-m" "polismath.runner"]} - :dev-poller - {:extra-paths ["dev" "test"] - :extra-deps {cider/cider-nrepl {:mvn/version "0.30.0"}} - :exec-fn user/run-with-repl} - :test - {:extra-paths ["test"] - :main-opts ["-m" "test-runner"]} - ;; Replay harness Phase H-B — Clojure Mode A driver (dev/replay.clj). - ;; Loaded via -i (file path) rather than :extra-paths ["dev"] so the - ;; dev/user.clj namespace (which requires oz/cider) is NOT auto-loaded - ;; at startup — keeps the alias self-contained on the base :deps. - :replay - {:main-opts ["-i" "dev/replay.clj" "-m" "replay"]}} - ;:jvm-opts ^:replace [] - :jvm-opts ["-Xmx4g"]} - ;:mvn/repos {"twitter4j" {:url "https://twitter4j.org/maven2"}}} - diff --git a/math/dev/density_plotting.clj b/math/dev/density_plotting.clj deleted file mode 100644 index e9393fbde9..0000000000 --- a/math/dev/density_plotting.clj +++ /dev/null @@ -1,54 +0,0 @@ -(ns density-plotting - "# Density plotting test - - This doesn't currently work yet, but its an attempt at visualizing a point clouds as a contour map of its density distribution. - The example I'm working from for vega is: https://vega.github.io/vega/examples/contour-plot/. - Unfortunately, this is vega, not vega-lite, and the vizard lib at the moment only plots vega-lite. - I've added a couple of issues for this (see https://github.com/yieldbot/vizard/issues/12 and #13). - It seems like it should be possible to add this, and maybe if we end up caring enough I can do so. - For now, not super vital." - (:require - [oz.core :as oz])) - - -;(defonce start-viz - ;(viz/start-plot-server!)) - - -(def data-values - [{:x 3.4 :y 5.3} - {:x 2.2 :y 9.3} - {:x 4.4 :y 5.1} - {:x 1.7 :y 5.9} - {:x 2.4 :y 6.2} - {:x 7.4 :y 7.4} - {:x 2.8 :y 10.3} - {:x 3.9 :y 8.3}]) - -(def plot-spec-2 - {:width 1000 - :height 800 - :data {:values data-values} - :layer [{:mark {:type "point" :filled true} - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:value 500}}}]}) - -(def plot-spec - {:width 500 - :height 400 - :data [{:name "main" - :values data-values}] - ;{:name "contours" - ; :source "main" - ; :transform [{:type "contour"}]}] - :marks [{:name "dots" - :type "symbol" - :from {:data "main"} - :encode {:x {:field "x"} - :y {:field "y"}}}]}) - - -;(viz/to-json plot-spec-2) -;(oz/view! plot-spec-2) - diff --git a/math/dev/plumbing_viz.clj b/math/dev/plumbing_viz.clj deleted file mode 100644 index 9216c2f7d0..0000000000 --- a/math/dev/plumbing_viz.clj +++ /dev/null @@ -1,86 +0,0 @@ -;; Copyright Jason Wolfe and Prismatic, 2013. -;; Licensed under the EPL, same license as Clojure -(ns plumbing-viz - (:require [clojure.java.shell :as shell] - [clojure.string :as str] - [plumbing.core :as plumb])) -;(use 'plumbing.core) - -(import '[java.util HashSet] '[java.io File]) - -(defn double-quote [s] (str "\"" s "\"")) - -(defn- attribute-string [label-or-attribute-map] - (when label-or-attribute-map - (str "[" - (str/join "," - (map (fn [[k v]] (str (name k) "=" v)) - (if (map? label-or-attribute-map) - label-or-attribute-map - {:label (double-quote label-or-attribute-map)}))) - "]"))) - -(defn- walk-graph [root node-key-fn node-label-fn edge-child-pair-fn ^HashSet visited indexer] - (let [node-key (node-key-fn root) - node-map (node-label-fn root)] - (when-not (.contains visited node-key) - (.add visited node-key) - (apply str - (indexer node-key) (attribute-string node-map) ";\n" - (apply concat - (for [[edge-map child] (edge-child-pair-fn root)] - (cons (str (indexer node-key) " -> " (indexer (node-key-fn child)) - (attribute-string edge-map) - ";\n") - (walk-graph child node-key-fn node-label-fn edge-child-pair-fn visited indexer)))))))) - - -(defn write-graphviz [file-stem roots node-key-fn node-label-fn edge-child-pair-fn] - (let [dot-file (str file-stem ".dot") - svg-file (str file-stem ".svg") - indexer (memoize (fn [x] (double-quote (gensym)))) - vis (HashSet.)] - (spit dot-file - (str "strict digraph {\n" - " rankdir = TB;\n" - (apply str (for [root roots] (walk-graph root node-key-fn node-label-fn edge-child-pair-fn vis indexer))) - "}\n")) - (shell/sh "dot" "-Tsvg" "-o" svg-file dot-file) - svg-file)) - -(defn graphviz-el [file-stem edge-list] - (let [edge-map (plumb/map-vals #(map second %) (group-by first edge-list))] - (write-graphviz - file-stem - (set (apply concat edge-list)) - identity identity #(for [e (get edge-map %)] [nil e])))) - -(defn graph-edges [g] - (for [[k node] g - ;parent (keys (plumbing.fnk.pfnk/input-schema node)) - parent (keys (plumbing.fnk.schema/explicit-schema-key-map - (plumbing.fnk.pfnk/input-schema node)))] - [parent k])) - -(defn graphviz-graph - "Generate file-stem.dot and file-stem.svg representing the nodes and edges of Graph g" - [file-stem g] - (graphviz-el file-stem (graph-edges g))) - -(require '[polismath.math.conversation :as conv]) - -;(graphviz-graph "conv-comp-graph2" conv/base-conv-update-graph) -;(graphviz-graph "conv-comp-graph3" conv/small-conv-update-graph) -(graphviz-graph "conv-comp-graph6" (dissoc conv/small-conv-update-graph - :subgroup-ptpt-stats - :subgroup-k-smoother - :subgroup-clusters - :subgroup-votes - :subgroup-repness - :subgroup-clusterings-silhouettes - :subgroup-clusterings)) - -; zid, participant_count, context, parent_url -;(shell/sh "dot" "-Tsvg" "-o" "conv-comp-graph.svg" "/tmp/conv-comp-graph.dot") -;(graphviz-graph "/tmp/foobar" {:x (plumb/fnk [a]) :y (plumb/fnk [a x])}) -;; then check /tmp/foobar.svg diff --git a/math/dev/proj_probe.clj b/math/dev/proj_probe.clj deleted file mode 100644 index 620259fe3e..0000000000 --- a/math/dev/proj_probe.clj +++ /dev/null @@ -1,349 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns proj-probe - "Diagnostic probe (R1 goal, every-vote step-57 uniqify edge): replay the - first N votes of a dataset through conv-update and print the FULL-PRECISION - projection rows for selected pids at the final step. - - DELIBERATELY a separate file from replay.clj: the certify clj-recording - cache manifests hash dev/replay.clj, so touching that file forces a full - battery re-record. This probe reuses replay's own loaders via `load-file` - and changes nothing. - - Usage (from math/): - clojure -M dev/proj_probe.clj [ ...]" - (:require [polismath.math.conversation :as conv] - [polismath.math.clusters :as clusters] - [polismath.math.named-matrix :as nm] - [clojure.core.matrix :as matrix])) - -(load-file "dev/replay.clj") - -(defn -main [& args] - (let [[csv-path n-votes & pids] args - n-votes (Long/parseLong n-votes) - pids (set (map #(Long/parseLong %) pids)) - votes (->> (replay/read-votes-csv csv-path) - replay/build-dataset - (take n-votes)) - seed (-> (conv/new-conv) (assoc :zid 99999 :meta-tids #{})) - ;; Replay ONE VOTE PER UPDATE — the warm-start chain is path-dependent - ;; (PCA start-vectors thread tick to tick), so a cold single batch - ;; does NOT reproduce the every-vote recording's state. - conv' (reduce - (fn [c v] - (conv/conv-update c (replay/->conv-votes [v]) - replay/certify-conv-opts)) - seed - votes) - rownames (nm/rownames (:rating-mat conv')) - proj (:proj conv')] - (doseq [[pid row] (map vector rownames proj) - :when (contains? pids pid)] - (println (format "pid=%d proj=[%.17g %.17g]" - (long pid) - (double (first row)) - (double (second row))))) - (doseq [c (:base-clusters conv')] - (println (format "cluster id=%d members=%s center=[%.17g %.17g]" - (long (:id c)) - (pr-str (:members c)) - (double (first (:center c))) - (double (second (:center c)))))) - ;; Phase-by-phase clean-start replay: chain to n-1 votes (prev state), - ;; apply the final vote, then walk clean-start-clusters manually on the - ;; new projection nmat with the PREV clusters, printing each phase. - (let [prev (reduce - (fn [c v] - (conv/conv-update c (replay/->conv-votes [v]) - replay/certify-conv-opts)) - (-> (conv/new-conv) (assoc :zid 99999 :meta-tids #{})) - (butlast votes)) - cur (conv/conv-update prev (replay/->conv-votes [(last votes)]) - replay/certify-conv-opts) - pnmat (nm/named-matrix (nm/rownames (:rating-mat cur)) ["x" "y"] - (:proj cur)) - inmat (nm/rowname-subset pnmat (:in-conv cur)) - rec (clusters/safe-recenter-clusters inmat (:base-clusters prev)) - uniq (clusters/uniqify-clusters rec) - possible (min 100 (count (distinct (into [] (matrix/rows (nm/get-matrix inmat))))))] - (println "PHASE safe-recenter:") - (doseq [c rec] - (println (format " id=%d members=%s center=[%.17g %.17g]" - (long (:id c)) (pr-str (:members c)) - (double (first (:center c))) (double (second (:center c)))))) - (println (format "PHASE uniqify: %d clusters (ids %s)" - (count uniq) (pr-str (mapv :id uniq)))) - (println (format "PHASE possible-clusters: %d (rows %d)" - possible (count (nm/rownames inmat)))) - (let [km (clusters/kmeans inmat 100 - :last-clusters (:base-clusters prev) - :max-iters 100)] - (println "PHASE full-kmeans:") - (doseq [c (sort-by :id km)] - (println (format " id=%d members=%s" - (long (:id c)) (pr-str (:members c)))))) - (let [cs (clusters/clean-start-clusters inmat (:base-clusters prev) 100) - data-iter (map vector (nm/rownames inmat) - (matrix/rows (matrix/matrix (nm/get-matrix inmat)))) - s1 (clusters/cluster-step data-iter 100 cs)] - (println "PHASE clean-start-direct:" (pr-str (mapv (juxt :id :members) cs))) - (println "PHASE cluster-step-1:" (pr-str (mapv (juxt :id :members) s1))) - (let [c6 (first (filter #(= 6 (:id %)) cs)) - c8 (first (filter #(= 8 (:id %)) cs)) - row5 (nm/get-row-by-name inmat 5)] - (println (format "PHASE centers: c6=[%.17g %.17g] c8=[%.17g %.17g]" - (double (first (:center c6))) (double (second (:center c6))) - (double (first (:center c8))) (double (second (:center c8))))) - (println (format "PHASE dist: d(p5,c6)=%.20g d(p5,c8)=%.20g equal=%s" - (double (matrix/distance row5 (:center c6))) - (double (matrix/distance row5 (:center c8))) - (= (matrix/distance row5 (:center c6)) - (matrix/distance row5 (:center c8))))) - (let [cleared (clusters/cleared-clusters cs) - after (clusters/add-to-closest cleared [5 row5]) - winner (first (filter (fn [[_ c]] (seq (:members c))) after))] - (println "PHASE single-assign: pid 5 ->" (pr-str (key winner)) - "map-type:" (str (type cleared)) - "entry-order:" (pr-str (keys cleared))) - (doseq [[cid c] cleared] - (println (format " d(p5, c%s)=%.20g" - (str cid) - (double (matrix/distance row5 (:center c)))))) - (let [names (nm/rownames inmat) - rows (into [] (matrix/rows (matrix/matrix (nm/get-matrix inmat)))) - mism (for [[n r] (map vector names rows) - :when (not= (into [] r) - (into [] (nm/get-row-by-name inmat n)))] - n)] - (println "PHASE alignment: rownames=" (pr-str names) - "misaligned-names=" (pr-str (vec mism))) - (let [view5 (nth rows 4)] - (println (format "PHASE view-dist: d(view5,c6)=%.20g d(view5,c8)=%.20g types=%s/%s" - (double (matrix/distance view5 (:center c6))) - (double (matrix/distance view5 (:center c8))) - (str (type view5)) (str (type (:center c6))))) - (let [after (clusters/add-to-closest - (clusters/cleared-clusters cs) [5 view5]) - winner (first (filter (fn [[_ c]] (seq (:members c))) after))] - (println "PHASE view-assign: pid 5 ->" (pr-str (key winner)))))))))))) - -;; Auto-run only when invoked with args (clojure -M dev/proj_probe.clj ...); -;; library-style load-file (batch-probe callers) skips it. -(when (seq *command-line-args*) - (apply -main *command-line-args*)) - -;; Batch-mode probe (pc-revote-01 split-loop tie): replay TWO vote-count -;; batches, then print the in-conv subset ROW ORDER and the clean-start -;; split-loop extraction sequence with runner-up gaps. -(defn batch-probe [csv-path cut1 cut2] - (let [votes (->> (replay/read-votes-csv csv-path) replay/build-dataset) - b1 (subvec votes 0 cut1) - b2 (subvec votes cut1 cut2) - seed (-> (conv/new-conv) - (assoc :zid 99998 :meta-tids #{} - :pca replay/certify-cold-start-pca)) - prev (conv/conv-update seed (replay/->conv-votes b1) - replay/certify-conv-opts) - cur (conv/conv-update prev (replay/->conv-votes b2) - replay/certify-conv-opts) - pnmat (nm/named-matrix (nm/rownames (:rating-mat cur)) ["x" "y"] - (:proj cur)) - inmat (nm/rowname-subset pnmat (:in-conv cur))] - (println "BATCH rownames-head:" (pr-str (take 20 (nm/rownames inmat)))) - (println "BATCH rownames-tail:" (pr-str (take-last 8 (nm/rownames inmat)))) - (println "BATCH n-clusters cur:" (count (:base-clusters cur))) - (doseq [c (sort-by :id (:base-clusters cur)) - :when (> (count (:members c)) 1)] - (println " multi-member cluster id=" (:id c) "members=" (pr-str (:members c)))))) - -;; Mod-weaving distinct-rows probe (pc-modheavy-01 step-2 fork, journal -;; 2026-07-22 s4 What's Next #1): replay an N-cut prefix of a mod-interleave -;; schedule WITH woven moderation (replay's own read-mod-events + -;; slice-schedule; meta-tids empty — interleave schedules take meta via -;; mod-update only, as in replay/-main), then report, at the FINAL step, the -;; in-conv projection-row distinct count and every group of pids whose rows -;; are EQUAL in clj at %.17g — to diff against the python side (py: 92 -;; distinct of 105 at step 2; 12 row-pairs collide in clj only). -(defn mod-distinct-probe [votes-csv comments-csv zid & cuts] - (let [votes (->> (replay/read-votes-csv votes-csv) replay/build-dataset) - {mods :events} (replay/read-mod-events comments-csv) - slots (mapv long cuts) - steps (replay/slice-schedule votes slots mods) - results (replay/run-once zid #{} steps) - [_ conv'] (last results) - pnmat (nm/named-matrix (nm/rownames (:rating-mat conv')) ["x" "y"] - (:proj conv')) - inmat (nm/rowname-subset pnmat (:in-conv conv')) - names (nm/rownames inmat) - raw-rows (matrix/rows (nm/get-matrix inmat)) - rows (mapv #(into [] %) raw-rows)] - (println "MODPROBE final-step: in-conv rows=" (count rows) - "distinct(vectorz)=" (count (distinct (into [] raw-rows))) - "distinct(vec)=" (count (distinct rows))) - (doseq [[row prs] (->> (group-by second (map vector names rows)) - (filter (fn [[_ prs]] (> (count prs) 1))) - (sort-by (fn [[_ prs]] (long (ffirst prs)))))] - (println (format "COLLIDE pids=%s row=[%.17g %.17g]" - (pr-str (mapv first prs)) - (double (nth row 0)) (double (nth row 1))))))) - -;; Mod-weaving split-loop walk (pc-modheavy-01 step-2: clj records 80 base -;; clusters vs py 92 while BOTH see 92 distinct in-conv rows — so the clj -;; split loop stops early; this prints WHY). Replays an N-cut prefix with -;; woven mods, then at the FINAL step: runs the REAL clean-start-clusters -;; (count check), then mirrors clusters.clj:250-273 manually printing each -;; iteration's most-distal extraction (id/dist/clst-id at %.20g) up to the -;; stop, plus the remaining multi-member clusters at the stop. -(defn mod-split-probe [votes-csv comments-csv zid & cuts] - (let [votes (->> (replay/read-votes-csv votes-csv) replay/build-dataset) - {mods :events} (replay/read-mod-events comments-csv) - slots (mapv long cuts) - steps (replay/slice-schedule votes slots mods) - results (replay/run-once zid #{} steps) - [_ prev-conv] (nth results (- (count results) 2)) - [_ cur-conv] (last results) - pnmat (nm/named-matrix (nm/rownames (:rating-mat cur-conv)) ["x" "y"] - (:proj cur-conv)) - inmat (nm/rowname-subset pnmat (:in-conv cur-conv)) - prev-bc (:base-clusters prev-conv) - real-cs (clusters/clean-start-clusters inmat prev-bc 100) - rec (clusters/safe-recenter-clusters inmat prev-bc) - uniq (clusters/uniqify-clusters rec) - possible (min 100 (count (distinct (into [] (matrix/rows (nm/get-matrix inmat))))))] - (println "MODSPLIT prev-step clusters:" (count prev-bc) - "safe-recenter:" (count rec) "uniqify:" (count uniq) - "possible:" possible "rows:" (count (nm/rownames inmat)) - "REAL clean-start-clusters:" (count real-cs)) - (loop [clusters uniq, it 0] - (let [clusters (clusters/recenter-clusters inmat clusters)] - (if (> possible (count clusters)) - (let [outlier (clusters/most-distal inmat clusters)] - (println (format "MODSPLIT iter %d: n=%d extract pid=%s d=%.20g clst=%s" - it (count clusters) (str (:id outlier)) - (double (:dist outlier)) (str (:clst-id outlier)))) - (if (> (:dist outlier) 0) - (recur - (-> - (mapv - (fn [clst] - (assoc clst :members - (remove (set [(:id outlier)]) (:members clst)))) - clusters) - (conj {:id (inc (apply max (map :id clusters))) - :members [(:id outlier)] - :center (nm/get-row-by-name inmat (:id outlier))})) - (inc it)) - (do - (println "MODSPLIT STOPPED (zero-dist outlier) at n=" (count clusters)) - (doseq [c clusters - :when (> (count (:members c)) 1)] - (println " multi-member id=" (:id c) "members=" (pr-str (:members c))))))) - (println "MODSPLIT done (possible reached) n=" (count clusters))))))) - -;; Step-1 lineage probe (pc-modheavy-01 {1,3,8,11} id 2-vs-8): replay an -;; N-cut prefix with woven mods, then at the FINAL step print the REAL -;; clean-start seed clusters holding the tracked pids (centers %.17g), the -;; distances of each tracked row to the tracked cluster ids under -;; matrix/distance (the add-to-closest path), and the final kmeans outcome -;; for those pids — to pin WHERE clj's id survives vs the py port. -(defn lineage-probe [votes-csv comments-csv zid track-pids track-ids & cuts] - (let [votes (->> (replay/read-votes-csv votes-csv) replay/build-dataset) - {mods :events} (replay/read-mod-events comments-csv) - slots (mapv long cuts) - steps (replay/slice-schedule votes slots mods) - results (replay/run-once zid #{} steps) - [_ prev-conv] (nth results (- (count results) 2)) - [_ cur-conv] (last results) - pnmat (nm/named-matrix (nm/rownames (:rating-mat cur-conv)) ["x" "y"] - (:proj cur-conv)) - inmat (nm/rowname-subset pnmat (:in-conv cur-conv)) - prev-bc (:base-clusters prev-conv) - track-pids (set track-pids) - track-ids (set track-ids) - seed (clusters/clean-start-clusters inmat prev-bc 100)] - (println "LINEAGE prev-step ids holding tracked pids:") - (doseq [c prev-bc :when (seq (clojure.set/intersection track-pids (set (:members c))))] - (println (format " prev id=%d members=%s center=[%.17g %.17g]" - (long (:id c)) (pr-str (:members c)) - (double (first (:center c))) (double (second (:center c)))))) - (println "LINEAGE seed clusters holding tracked pids or ids:") - (doseq [c seed :when (or (seq (clojure.set/intersection track-pids (set (:members c)))) - (contains? track-ids (:id c)))] - (println (format " seed id=%d members=%s center=[%.17g %.17g]" - (long (:id c)) (pr-str (:members c)) - (double (first (:center c))) (double (second (:center c)))))) - (doseq [p track-pids] - (let [row (nm/get-row-by-name inmat p)] - (doseq [c seed :when (contains? track-ids (:id c))] - (println (format " d(row%s, c%d) = %.20g" - (str p) (long (:id c)) - (double (matrix/distance row (:center c)))))))) - (let [km (clusters/kmeans inmat 100 - :last-clusters prev-bc - :max-iters 100)] - (println "LINEAGE final kmeans clusters holding tracked pids:") - (doseq [c (sort-by :id km) - :when (seq (clojure.set/intersection track-pids (set (:members c))))] - (println (format " final id=%d members=%s" - (long (:id c)) (pr-str (:members c)))))))) - -;; Split-loop probe (pc-revote-01 step-1 extraction tie): replay two vote-count -;; batches like batch-probe, then walk clean-start-clusters' split loop -;; MANUALLY (mirroring clusters.clj:250-273 verbatim) printing, per iteration, -;; the ACTUAL most-distal extraction (id/dist/clst-id) plus the top-3 candidate -;; ranking with runner-up gaps, so the sequence can be diffed against the -;; python probe (delphi/scratch/probe_revote_split.py). -(defn split-probe [csv-path cut1 cut2] - (let [votes (->> (replay/read-votes-csv csv-path) replay/build-dataset) - b1 (subvec votes 0 cut1) - b2 (subvec votes cut1 cut2) - seed (-> (conv/new-conv) - (assoc :zid 99998 :meta-tids #{} - :pca replay/certify-cold-start-pca)) - prev (conv/conv-update seed (replay/->conv-votes b1) - replay/certify-conv-opts) - cur (conv/conv-update prev (replay/->conv-votes b2) - replay/certify-conv-opts) - pnmat (nm/named-matrix (nm/rownames (:rating-mat cur)) ["x" "y"] - (:proj cur)) - inmat (nm/rowname-subset pnmat (:in-conv cur)) - rec (clusters/safe-recenter-clusters inmat (:base-clusters prev)) - uniq (clusters/uniqify-clusters rec) - possible (min 100 (count (distinct (into [] (matrix/rows (nm/get-matrix inmat))))))] - (println "SPLIT start-clusters:" (count uniq) "possible:" possible - "rows:" (count (nm/rownames inmat))) - (loop [clusters uniq, it 0] - (let [clusters (clusters/recenter-clusters inmat clusters)] - (if (> possible (count clusters)) - (let [outlier (clusters/most-distal inmat clusters) - ranks (->> (nm/rownames inmat) - (map (fn [mem] - (let [row (nm/get-row-by-name inmat mem)] - [(apply min (map #(matrix/distance row (:center %)) - clusters)) - mem]))) - (sort-by first) - reverse - (take 3)) - [[d0 m0] [d1 m1] [d2 m2]] ranks] - (println (format "SPLIT iter %d: extract pid=%s d=%.20g clst=%s | top3 %s:%.20g %s:%.20g %s:%.20g | gap01=%.3e" - it (str (:id outlier)) (double (:dist outlier)) - (str (:clst-id outlier)) - (str m0) (double d0) (str m1) (double d1) - (str m2) (double d2) - (double (- d0 d1)))) - (if (> (:dist outlier) 0) - (recur - (-> - (mapv - (fn [clst] - (assoc clst :members - (remove (set [(:id outlier)]) (:members clst)))) - clusters) - (conj {:id (inc (apply max (map :id clusters))) - :members [(:id outlier)] - :center (nm/get-row-by-name inmat (:id outlier))})) - (inc it)) - (println "SPLIT done (zero-dist outlier) after iter" it))) - (println "SPLIT done (possible reached) n=" (count clusters))))))) diff --git a/math/dev/replay.clj b/math/dev/replay.clj deleted file mode 100644 index d9a950f30d..0000000000 --- a/math/dev/replay.clj +++ /dev/null @@ -1,614 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns replay - "Replay harness Phase H-B — the CLOJURE Mode A driver (REPLAY_HARNESS_DESIGN.md - §5). Pure in-process: no Postgres, no poller, no Docker. - - Given a schedule JSON (§4) and a votes CSV, it seeds a conversation exactly as - export.clj:624-632 (`get-export-data-at-time`) does — `(-> (conv/new-conv) - (assoc :zid zid :meta-tids meta-tids))` — then reduces `conv/conv-update` over - the schedule's vote batches (the pure pattern shown at dev/user.clj:416-428 and - test/conversation_test.clj:40-168), recording per step: - - clj/step-NNN.blob.json — the `prep-main` (conv_man.clj:43-74) key-whitelisted - production view, cheshire-encoded exactly as - postgres.clj pg-json does. This IS the `math_main` - blob shape and is the cross-language comparison - surface Python's `to_dict` targets. - clj/step-NNN.edn — (with --edn) full-fidelity conv state in the - `conv-update-dump` (conversation.clj:920) shape, - reloadable via `conv/load-conv-update`. - - BINDING SEMANTICS (verified against source, NOT guessed): - - * Vote signs — FLIP the CSV. Export CSVs (timestamp,datetime,comment-id, - voter-id,vote) carry EXPORT/Delphi signs (AGREE=+1); the flip is export-only - (export.clj:106-113). The Clojure math consumes RAW-DB signs (AGREE=-1), so - each CSV value v is fed as -v. Recorded as vote_sign_convention \"raw-db\". - (The Python driver feeds +v; each engine gets its own native convention, so - the OUTPUT blobs stay comparable.) - - * Slicing mirrors the H-A Python slicer (schedule.py `resolve_cut_slots` / - `slice_schedule`) EXACTLY: sort votes stably by (t_ms, input order), KEEP - revotes (no dedup — later-vote-wins is resolved inside conv-update), CSV - timestamps are SECONDS → ms via *1000 (mirror real_data.py). Cut modes - vote-count | timestamp | fraction | explicit-event-index resolve the same - way, including Python's round-half-to-even for `fraction`. - - * conv-update input shape (conversation_test.clj:18-21): a seq of maps - `{:pid :tid :vote :created }`. `:created` is REQUIRED - (the :last-vote-timestamp fnk maxes over it, conversation.clj:161-165); - `:zid` is taken from the seeded conv (`(or (:zid conv) (:zid (first votes)))`, - conversation.clj:157-159) so votes need not carry it. - - * Chained warm start is IMPLICIT: the reduce threads the conv object, whose - `:pca :comps` become the next step's `:start-vectors` (conversation.clj:381-387). - That IS `warm_start: chain`. Step 0 has no prior comps → cold-start PCA uses - unseeded `(rand)` (pca.clj:79-82), the §9 self-jitter source measured by - --repeats. - - * meta-tids seed from the comments CSV `is-meta`/`is_meta` column when present; - the public vw export has no such column, so the set is empty (documented in - provenance meta_tids_source). - - Moderation: vw has none. `moderation` other than \"none\" raises a clear - not-implemented error (Mode A mod-update interleaving is deferred, design §5). - - Run: cd math && clojure -M:replay --schedule --votes \\ - --out [--repeats N] [--edn] [--zid Z] [--comments c.csv]" - (:require [clojure.data.csv :as csv] - [clojure.java.io :as io] - [clojure.java.shell :as shell] - [clojure.string :as str] - [clojure.tools.cli :as cli] - [cheshire.core :as json] - [com.stuartsierra.component :as component] - [clojure.core.matrix :as matrix] - [polismath.math.conversation :as conv] - [polismath.math.named-matrix :as nm] - [polismath.conv-man :as cm] - [polismath.components.core-matrix-boot :as cmb]) - (:import [java.security MessageDigest] - [java.math BigDecimal RoundingMode] - [java.time Instant])) - -;; --------------------------------------------------------------------------- -;; CSV → sorted vote stream (mirror ReplayDataset.build / real_data.py). -;; --------------------------------------------------------------------------- - -(defn read-votes-csv - "Read an export votes CSV into vote maps in FILE order. Columns - timestamp,datetime,comment-id,voter-id,vote; timestamps are SECONDS → ms." - [path] - (with-open [rdr (io/reader path)] - (let [rows (csv/read-csv rdr) - header (first rows) - idx (zipmap header (range)) - ti (idx "timestamp") ci (idx "comment-id") - vi (idx "voter-id") si (idx "vote")] - (when (some nil? [ti ci vi si]) - (throw (ex-info "votes CSV missing a required column" - {:header header :need ["timestamp" "comment-id" "voter-id" "vote"]}))) - (mapv (fn [r] - {:t-ms (* 1000 (Long/parseLong (str/trim (nth r ti)))) - :pid (Long/parseLong (str/trim (nth r vi))) - :tid (Long/parseLong (str/trim (nth r ci))) - :sign (Long/parseLong (str/trim (nth r si)))}) - (rest rows))))) - -(defn build-dataset - "Sort raw file-order vote maps stably by (t_ms, input index) and 1-index them. - Revotes are KEPT — dedup is never applied (design §5). Returns a vector." - [raw-rows] - (->> raw-rows - (map-indexed (fn [i row] (assoc row :file-idx i))) - (sort-by (juxt :t-ms :file-idx)) - (map-indexed (fn [i row] (assoc row :k (inc i)))) - vec)) - -;; --------------------------------------------------------------------------- -;; Cut-mode resolution (mirror schedule.py resolve_cut_slots EXACTLY). -;; --------------------------------------------------------------------------- - -(def ^:private valid-modes #{"vote-count" "explicit-event-index" "timestamp" "fraction"}) - -(defn ^long py-round - "Round-half-to-even to a long — matches Python's built-in round() (banker's)." - [^double x] - (.longValueExact (.setScale (BigDecimal/valueOf x) 0 RoundingMode/HALF_EVEN))) - -(defn count-votes-up-to - "#votes with t_ms <= T in a time-sorted vector (linear; n is small)." - [votes t-ms] - (count (take-while #(<= (long (:t-ms %)) (long t-ms)) votes))) - -(defn validate-slots - "Mirror ReplayDataset.validate_schedule: 1<=s<=n, strictly increasing." - [slots n] - (loop [prev 0 [s & more] slots] - (when s - (when-not (<= 1 s n) - (throw (ex-info (str "cut slot " s " outside 1.." n) {:slot s :n n}))) - (when-not (> s prev) - (throw (ex-info (str "schedule not strictly increasing at slot " s) {:slot s}))) - (recur s more)))) - -(defn resolve-cut-slots - "Resolve a §4 `cuts` map into strictly-increasing 1-based slots in 1..n. - 0-slots dropped as degenerate, duplicates collapsed (== schedule.py)." - [votes cuts] - (let [mode (get cuts "mode") - at (get cuts "at" []) - n (count votes)] - (when-not (valid-modes mode) - (throw (ex-info (str "unknown cut mode " (pr-str mode) - "; expected one of " (sort valid-modes)) {:mode mode}))) - (let [raw (for [a at] - (cond - (= a "end") - n - (#{"vote-count" "explicit-event-index"} mode) - (long a) - (= mode "fraction") - (let [f (double a)] - (when-not (and (< 0.0 f) (<= f 1.0)) - (throw (ex-info (str "fraction cut " f " outside (0, 1]") {:f f}))) - (py-round (* f n))) - (= mode "timestamp") - (count-votes-up-to votes (long a)))) - slots (->> raw (filter pos?) (into (sorted-set)) vec)] - (validate-slots slots n) - slots))) - -;; --------------------------------------------------------------------------- -;; Slicer: schedule + dataset → ordered steps (mirror slice_schedule; the tail -;; after the last cut is intentionally NOT a step — include "end" to close it). -;; --------------------------------------------------------------------------- - -(defn slice-schedule - "Mods weave per schedule.py:204-210: a mod event attaches to the FIRST cut - whose cut-time reaches its :modified (and which is past the previous cut's - time); events after the last cut are dropped, like tail votes." - ([votes slots] (slice-schedule votes slots [])) - ([votes slots mod-events] - (loop [prev 0 [cut & more] slots i 0 acc []] - (if (nil? cut) - acc - (let [cut-time (:t-ms (nth votes (dec cut))) - prev-time (when (pos? prev) (:t-ms (nth votes (dec prev)))) - mods (filterv #(and (<= (long (:modified %)) (long cut-time)) - (or (nil? prev-time) - (> (long (:modified %)) (long prev-time)))) - mod-events)] - (recur cut more (inc i) - (conj acc {:index i - :prev-slot prev - :cut-slot cut - :votes (subvec votes prev cut) ; (prev, cut] 0-based - :mods mods - :cut-time-ms cut-time}))))))) - -;; --------------------------------------------------------------------------- -;; Feeding conv-update: FLIP the export sign to raw-DB (design §5). -;; --------------------------------------------------------------------------- - -(defn ->conv-votes - [batch] - (mapv (fn [{:keys [pid tid sign t-ms]}] - {:pid pid :tid tid :vote (- (long sign)) :created t-ms}) - batch)) - -;; --------------------------------------------------------------------------- -;; One full replay pass: seed → reduce conv-update, keeping conv per step. -;; --------------------------------------------------------------------------- - -;; Q10 carve-out (CLOJURE_QUIRKS.md): conv-update's large-conv dispatch -;; (n-ptpts > 10000 OR n-cmts > 5000, conversation.clj:784-815) runs -;; mini-batch partial-pca on an UNSEEDED random row sample — no -;; deterministic reference exists on that path, even between two Clojure -;; runs. Certification pins both cutoffs huge so every step takes the -;; deterministic full-PCA (small-conv) path at any size. Logged per run -;; here and by certify.py's acceptance notice. -(def certify-conv-opts - {:ptpt-cutoff 1000000000 - :cmt-cutoff 1000000000}) - -;; Q12 carve-out (CLOJURE_QUIRKS.md): the COLD-tick PCA start vector is -;; unseeded-random in production (rand-starting-vec, pca.clj:79-82 — the -;; original author's own "should really throw a [seeded] random number -;; generator in here... XXX" comment) — with a small eigengap, 100 power -;; iterations don't fully converge and the residual start-dependence makes -;; even two Clojure runs differ. Certification pins the cold start to the -;; ONES vector — the same value power-iteration pads new-comment columns -;; with (pca.clj:46-49) — by seeding the conv with single-element [1.0] -;; comps that the padding expands to all-ones at any width. Warm ticks are -;; untouched (real previous comps take over from tick 2). The Python replay -;; driver pins the same start. -(def certify-cold-start-pca - {:comps [[1.0] [1.0]]}) - -(defn parse-blob-json - "EXACTLY db/load-conv's key-fn (postgres.clj:419-433): numeric-string keys - become longs, everything else keywords — including the keyword/long - hash-map-key mismatches its own docstring warns about (e.g. :repness), - which are part of production restart semantics." - [s] - (json/parse-string s (fn [x] (try (Long/parseLong x) - (catch Exception _ (keyword x)))))) - -(defn restart-conv - "Replicate conv-man's load-or-init restart (conv_man.clj:188-207) - mid-schedule: rebuild the conv from its OWN just-computed math_main blob - (prep-main → JSON round-trip → restructure-json-conv), :recompute :reboot, - raw-rating-mat from the FULL vote log so far ([pid tid raw-db-vote] in - dataset order — conv-poll's created-order equivalent), then mod-update with - the FULL mod history so far (called even when empty, as load-or-init does). - Everything restructure-json-conv drops (rating-mat, per-k - :group-clusterings smoother memory, …) is LOST, exactly as in production." - [conv steps-so-far] - (let [votes-so-far (mapcat :votes steps-so-far) - mods-so-far (mapcat :mods steps-so-far)] - (-> (cm/prep-main conv) - json/generate-string - parse-blob-json - cm/restructure-json-conv - (assoc :recompute :reboot) - (assoc :raw-rating-mat - (nm/update-nmat (nm/named-matrix) - (mapv (fn [{:keys [pid tid sign]}] - [pid tid (- (long sign))]) - votes-so-far))) - (conv/mod-update (vec mods-so-far))))) - -(defn run-once - "Returns a vector of [step conv-after-update] pairs, one per cut slot. - The reduce threading the conv IS the implicit warm-start chain. - conv-update runs with certify-conv-opts (Q10 full-PCA carve-out) and the - seed conv carries certify-cold-start-pca (Q12 pinned cold start). - Step semantics mirror conv-man's per-batch [:votes :moderation] order - (conv_man.clj:361-371): votes → conv-update (recompute), then mods → - conv/mod-update (sets+watermark ONLY, no recompute — the mods take effect - at the NEXT votes recompute); ONE blob per step, recorded post-mods. - After recording step `restart-after`, the chain continues from - `restart-conv` (the production worker-restart seam)." - ([zid meta-tids steps] (run-once zid meta-tids steps nil)) - ([zid meta-tids steps restart-after] - (binding [*out* *err*] - (println "Q10 carve-out: large-conv mini-batch PCA disabled" - "(ptpt/cmt cutoffs pinned to 10^9; full PCA at every size)") - (println "Q12 carve-out: cold-tick PCA start pinned to ones" - "(production start is unseeded-random)")) - (let [seed (-> (conv/new-conv) - (assoc :zid zid - :meta-tids (set meta-tids) - :pca certify-cold-start-pca))] - (loop [conv seed [s & more] steps acc []] - (if (nil? s) - acc - (let [conv' (conv/conv-update conv (->conv-votes (:votes s)) - certify-conv-opts) - conv' (if (seq (:mods s)) - (conv/mod-update conv' (vec (:mods s))) - conv') - acc' (conj acc [s conv']) - conv'' (if (and restart-after (= (long (:index s)) (long restart-after))) - (do (binding [*out* *err*] - (println (format "restart seam after step %d (load-or-init replay)" - (long (:index s))))) - (restart-conv conv' (map first acc'))) - conv')] - (recur conv'' more acc'))))))) - -;; --------------------------------------------------------------------------- -;; Recording. -;; --------------------------------------------------------------------------- - -(defn write-blob! - "Write the prep-main math_main view for one step, cheshire-encoded exactly as - postgres.clj pg-json does (the DB blob's own serialization)." - [dir step conv] - (spit (io/file dir (format "step-%03d.blob.json" (:index step))) - (json/generate-string (cm/prep-main conv)))) - -(defn write-edn! - "Write full-fidelity conv state in the `conv-update-dump` shape - (conversation.clj:920), reloadable via `conv/load-conv-update`. The - core.matrix print-methods (conversation.clj:882-899) are already installed." - [dir step conv fed-votes] - (spit (io/file dir (format "step-%03d.edn" (:index step))) - (prn-str - {:conv (into {} - (assoc-in conv [:pca :center] - (matrix/matrix (into [] (:center (:pca conv)))))) - :votes fed-votes - :opts {} - :error nil}))) - -(defn write-results! - [dir results edn?] - (.mkdirs ^java.io.File dir) - (doseq [[s conv] results] - (write-blob! dir s conv) - (when edn? (write-edn! dir s conv (->conv-votes (:votes s)))))) - -;; --------------------------------------------------------------------------- -;; Provenance. -;; --------------------------------------------------------------------------- - -(defn sha256-file - [path] - (let [md (MessageDigest/getInstance "SHA-256") - buf (byte-array 65536)] - (with-open [in (io/input-stream path)] - (loop [] - (let [n (.read in buf)] - (when (pos? n) (.update md buf 0 n) (recur))))) - (->> (.digest md) (map #(format "%02x" (bit-and % 0xff))) (apply str)))) - -(defn git-commit - [dir] - (try - (let [{:keys [exit out]} (shell/sh "git" "-C" (str dir) "rev-parse" "HEAD")] - (if (zero? exit) (str/trim out) "unknown")) - (catch Exception _ "unknown"))) - -(defn build-provenance - [{:keys [schedule schedule-id source votes-path comments-path zid meta-tids - meta-tids-source warm-start repeats n-steps edn? - moderation n-mod-events n-mod-skipped restart-after]}] - {:engine "clj" - :moderation (or moderation "none") - :n_mod_events (or n-mod-events 0) - :n_mod_skipped_no_modified (or n-mod-skipped 0) - :restart_after restart-after - :mode "A" - :schedule_id schedule-id - :source source - :dataset (get schedule "dataset") - :zid zid - :n_steps n-steps - :repeats repeats - :edn edn? - :warm_start warm-start - :warm_start_note "chain is implicit: the reduce threads conv; :pca :comps seed the next step's start-vectors (conversation.clj:381-387)" - :vote_sign_convention "raw-db" - :vote_sign_note "export CSV signs (AGREE=+1) are FLIPPED to raw-DB (AGREE=-1) before conv-update; the flip is export-only (export.clj:106-113)" - :meta_tids (vec (sort meta-tids)) - :meta_tids_source meta-tids-source - ;; Run from math/ (user.dir), which is inside the repo → HEAD is the math/ - ;; commit (delphi and math share one repo in this checkout). - :math_git_commit (git-commit (System/getProperty "user.dir")) - :clojure_version (clojure-version) - :jvm_version (System/getProperty "java.version") - :jvm_runtime_version (System/getProperty "java.runtime.version") - :jvm_vm_name (System/getProperty "java.vm.name") - :matrix_implementation "vectorz" - :votes_file (.getName (io/file votes-path)) - :votes_sha256 (sha256-file votes-path) - :comments_file (when comments-path (.getName (io/file comments-path))) - :comments_sha256 (when comments-path (sha256-file comments-path)) - :created_at (str (Instant/now))}) - -;; --------------------------------------------------------------------------- -;; meta-tids from comments CSV (is-meta / is_meta column; else empty). -;; --------------------------------------------------------------------------- - -(defn read-meta-tids - "Returns [meta-tid-set source-description]. Empty when no comments CSV or no - is-meta column (as for the public vw export)." - [comments-path] - (if (nil? comments-path) - [#{} "empty (no comments CSV supplied)"] - (with-open [rdr (io/reader comments-path)] - (let [rows (csv/read-csv rdr) - header (first rows) - idx (zipmap header (range)) - ci (or (idx "comment-id") (idx "tid")) - mi (or (idx "is-meta") (idx "is_meta"))] - (if (or (nil? ci) (nil? mi)) - [#{} (str "empty (comments CSV has no is-meta column; header=" (vec header) ")")] - [(->> (rest rows) - (keep (fn [r] - (let [v (str/lower-case (str/trim (str (nth r mi ""))))] - (when (#{"1" "true" "t" "yes"} v) - (Long/parseLong (str/trim (nth r ci))))))) - (into #{})) - "comments CSV is-meta column"]))))) - -;; --------------------------------------------------------------------------- -;; Moderation rows from the comments CSV (interleave-by-timestamp schedules). -;; --------------------------------------------------------------------------- - -(defn read-mod-events - "Raw moderation rows {:tid :is_meta :mod :modified} from the comments CSV, - sorted by (modified, file order) — the conv-mod-poll stream equivalent. - `modified` is the DB value in MILLISECONDS, compared directly against vote - :t-ms at weave time (the py loader reads the same column identically). - Rows with an empty `modified` cannot be woven and are SKIPPED (counted in - :n-skipped for provenance). Columns: comment-id/tid, is-meta/is_meta, - mod/moderated, modified." - [comments-path] - (with-open [rdr (io/reader comments-path)] - (let [rows (doall (csv/read-csv rdr)) - header (first rows) - idx (zipmap header (range)) - ci (or (idx "comment-id") (idx "tid")) - mi (or (idx "is-meta") (idx "is_meta")) - modi (or (idx "mod") (idx "moderated")) - tsi (idx "modified")] - (when (some nil? [ci modi tsi]) - (throw (ex-info (str "comments CSV lacks moderation columns " - "(need comment-id, mod/moderated, modified); header=" - (vec header)) - {:header header}))) - (let [parsed (->> (rest rows) - (keep-indexed - (fn [i r] - (let [modified-raw (str/trim (str (nth r tsi "")))] - (when (seq modified-raw) - {:tid (Long/parseLong (str/trim (nth r ci))) - :is_meta (boolean - (when mi - (#{"1" "true" "t" "yes"} - (str/lower-case (str/trim (str (nth r mi ""))))))) - :mod (Long/parseLong (str/trim (nth r modi))) - :modified (Long/parseLong modified-raw) - :file-idx i}))))) - events (->> parsed - (sort-by (juxt :modified :file-idx)) - (mapv #(dissoc % :file-idx)))] - {:events events - :n-skipped (- (count (rest rows)) (count events))})))) - -;; --------------------------------------------------------------------------- -;; CLI. -;; --------------------------------------------------------------------------- - -(def cli-options - [["-s" "--schedule PATH" "Path to the schedule JSON (§4)."] - ["-v" "--votes PATH" "Path to the export votes CSV."] - ["-o" "--out DIR" "Recording dir (…//); clj/ is written under it."] - [nil "--comments PATH" "Optional comments CSV (meta-tids via is-meta column)."] - [nil "--zid ZID" "Conversation id to seed (default: schedule dataset name)."] - ["-r" "--repeats N" "Full-replay repeats for §9 self-jitter (default 1)." - :default 1 :parse-fn #(Integer/parseInt %)] - [nil "--edn" "Also write per-step full-state EDN (conv-update-dump shape)."] - ["-h" "--help"]]) - -(defn -main [& args] - (let [{:keys [options errors summary]} (cli/parse-opts args cli-options)] - (cond - (:help options) - (do (println "Replay harness — Clojure Mode A driver (Phase H-B)") - (println summary) - (System/exit 0)) - - errors - (do (binding [*out* *err*] (doseq [e errors] (println e)) (println summary)) - (System/exit 1)) - - (some nil? [(:schedule options) (:votes options) (:out options)]) - (do (binding [*out* *err*] - (println "ERROR: --schedule, --votes and --out are all required.") - (println summary)) - (System/exit 1)) - - :else - (let [schedule (json/parse-string (slurp (:schedule options))) - dataset (get schedule "dataset") - schedule-id (get schedule "schedule_id") - source (get schedule "source" "votes-csv") - cuts (get schedule "cuts") - moderation (get schedule "moderation" "none") - warm-start (get-in schedule ["clojure" "warm_start"] "chain") - zid (or (:zid options) dataset) - repeats (:repeats options) - edn? (boolean (:edn options)) - out (io/file (:out options)) - clj-dir (io/file out "clj")] - - (when-not (contains? #{"none" "interleave-by-timestamp" nil} moderation) - (throw (ex-info - (str "Unknown moderation mode " (pr-str moderation) - ". Use \"none\" or \"interleave-by-timestamp\" " - "(mod rows from --comments, woven by modified timestamp).") - {:moderation moderation}))) - (when (and (= moderation "interleave-by-timestamp") - (nil? (:comments options))) - (throw (ex-info "moderation=interleave-by-timestamp requires --comments" - {:moderation moderation}))) - - ;; Only "chain" warm-start is implemented (it is IMPLICIT: the reduce - ;; threads the conv, whose :pca :comps seed the next step's start-vectors, - ;; conversation.clj:381-387). Reject any other value loudly rather than - ;; silently ignoring it — mirrors the moderation guard above. - (when-not (contains? #{"chain" nil} warm-start) - (throw (ex-info - (str "Only warm_start=\"chain\" is implemented in the Mode A " - "driver (chained warm start is implicit). Got warm_start=" - (pr-str warm-start) ". Use \"chain\" (or omit it).") - {:warm_start warm-start}))) - - ;; Register cheshire encoders for core.matrix (mikera.*) — WITHOUT this, - ;; prep-main's :pca vectors fail to JSON-encode (postgres.clj relies on - ;; the same CoreMatrixBooter at system start). - (component/start - (cmb/create-core-matrix-booter {:config {:math {:matrix-implementation :vectorz}}})) - - (let [raw (read-votes-csv (:votes options)) - votes (build-dataset raw) - slots (resolve-cut-slots votes cuts) - restart-after (get schedule "restart_after") - {mod-events :events n-mod-skipped :n-skipped} - (if (= moderation "interleave-by-timestamp") - (read-mod-events (:comments options)) - {:events [] :n-skipped 0}) - steps (slice-schedule votes slots mod-events) - ;; Under interleave moderation, meta-tids enter EXCLUSIVELY via - ;; the woven mod-update rows (the production-reachable route) — - ;; seeding them at conv creation as well would front-load every - ;; is-meta comment into step 0's compute, which no production - ;; state can produce (found on pc-meta-01 step 0, 2026-07-22 s4: - ;; clj meta-tids = seed ∪ woven vs py's woven-only). The - ;; creation-time seed remains for moderation="none" runs with - ;; --comments (the original vw-compat path). - [meta-tids meta-src] - (if (= moderation "interleave-by-timestamp") - [#{} "empty (interleave moderation: meta-tids via mod-update only)"] - (read-meta-tids (:comments options)))] - - (when restart-after - (when-not (and (integer? restart-after) - (<= 0 (long restart-after) (- (count steps) 2))) - (throw (ex-info (str "restart_after must be a step index with at " - "least one step after it; got " - (pr-str restart-after) " for " (count steps) - " steps") - {:restart_after restart-after :n-steps (count steps)})))) - - (binding [*out* *err*] - (println (format "dataset=%s n_votes=%d schedule=%s cuts=%s" - dataset (count votes) schedule-id (pr-str slots))) - (println (format "steps=%d repeats=%d edn=%s zid=%s meta-tids=%d" - (count steps) repeats edn? (pr-str zid) (count meta-tids))) - (when (= moderation "interleave-by-timestamp") - (println (format "moderation=interleave-by-timestamp mod-events=%d skipped-no-modified=%d woven=%d" - (count mod-events) (long n-mod-skipped) - (reduce + (map (comp count :mods) steps))))) - (when restart-after - (println (format "restart_after=%d (load-or-init seam)" (long restart-after))))) - - (.mkdirs clj-dir) - ;; schedule.json verbatim (byte-faithful copy of the §4 input). - (io/copy (io/file (:schedule options)) (io/file out "schedule.json")) - - ;; Run repeats. rep 0 is also written flat to clj/ (the canonical - ;; cross-language surface); rep i>0 (and rep 0) go to clj/rep-i/. - (dotimes [rep repeats] - (let [results (run-once zid meta-tids steps restart-after) - rep-dir (if (> repeats 1) (io/file clj-dir (str "rep-" rep)) clj-dir)] - (write-results! rep-dir results edn?) - (when (and (> repeats 1) (zero? rep)) - (write-results! clj-dir results edn?)) - (binding [*out* *err*] - (println (format " rep %d/%d written → %s" (inc rep) repeats (str rep-dir)))))) - - ;; Provenance (recording dir + a clj/ mirror so a later Python run's - ;; provenance.json cannot clobber ours). - (let [prov (build-provenance - {:schedule schedule :schedule-id schedule-id :source source - :votes-path (:votes options) :comments-path (:comments options) - :zid zid :meta-tids meta-tids :meta-tids-source meta-src - :warm-start warm-start :repeats repeats - :n-steps (count steps) :edn? edn? - :moderation moderation - :n-mod-events (count mod-events) - :n-mod-skipped n-mod-skipped - :restart-after restart-after}) - prov-json (json/generate-string prov {:pretty true})] - (spit (io/file out "provenance.json") prov-json) - (spit (io/file clj-dir "provenance.json") prov-json)) - - (println (format "wrote %d steps (x%d reps) → %s" - (count steps) repeats (str clj-dir))) - (System/exit 0)))))) diff --git a/math/dev/replay_smoke.sh b/math/dev/replay_smoke.sh deleted file mode 100755 index 2d72aa616a..0000000000 --- a/math/dev/replay_smoke.sh +++ /dev/null @@ -1,72 +0,0 @@ -#!/usr/bin/env bash -# Replay harness Phase H-B — Clojure Mode A driver smoke test. -# -# Runs dev/replay.clj on the public vw dataset with a 3-cut vote-count schedule -# and asserts the final step has non-empty base-clusters and the math_main -# key shape. Optionally runs the Python driver on the SAME schedule into the -# same recording dir for a cross-language gap measurement (needs the delphi -# venv). No Docker, no Postgres. -# -# Usage (from math/): bash dev/replay_smoke.sh [--xlang] -set -euo pipefail - -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # math/ -DELPHI="$(cd "$HERE/../delphi" && pwd)" -VW_DIR="$(ls -d "$DELPHI"/real_data/*-vw 2>/dev/null | head -1)" -VOTES="$(ls "$VW_DIR"/*-votes.csv | head -1)" -OUT="$DELPHI/real_data/.local/replays/vw/hb-3cut" - -SCHED="$(mktemp -t hb-3cut-XXXX.json)" -cat > "$SCHED" <<'JSON' -{ - "dataset": "vw", - "schedule_id": "hb-3cut", - "source": "votes-csv", - "cuts": {"mode": "vote-count", "at": [1000, 2500, "end"]}, - "moderation": "none", - "clojure": {"warm_start": "chain"}, - "notes": "H-B smoke: 3-cut vote-count schedule on vw" -} -JSON - -echo ">> Clojure driver → $OUT/clj" -rm -rf "$OUT" -( cd "$HERE" && clojure -M:replay --schedule "$SCHED" --votes "$VOTES" --out "$OUT" --edn ) \ - 2>&1 | grep -Ev '^(WARNING|Warning)|DEBUG|INFO \[polismath' || true - -echo ">> Asserting final-step blob" -( cd "$DELPHI" && uv run python - "$OUT" ) <<'PY' -import json, sys, glob, os -out = sys.argv[1] -steps = sorted(glob.glob(os.path.join(out, "clj", "step-*.blob.json"))) -assert steps, "no clj step blobs written" -final = json.load(open(steps[-1])) -bc = final["base-clusters"]["id"] -assert len(bc) > 0, "final step has empty base-clusters" -expected = {"base-clusters","group-clusters","subgroup-clusters","in-conv","mod-out", - "mod-in","meta-tids","lastVoteTimestamp","lastModTimestamp","n","n-cmts", - "pca","repness","group-aware-consensus","consensus","zid","tids", - "user-vote-counts","votes-base","group-votes","subgroup-votes", - "subgroup-repness","comment-priorities"} -assert set(final.keys()) == expected, f"blob key shape drift: {set(final.keys()) ^ expected}" -print(f" OK: {len(steps)} steps, final base-clusters={len(bc)}, 23-key math_main shape") -PY - -if [[ "${1:-}" == "--xlang" ]]; then - echo ">> Python driver → $OUT/py (same schedule)" - ( cd "$DELPHI" && OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \ - uv run python scripts/replay_driver.py run --schedule "$SCHED" \ - --out real_data/.local/replays ) 2>&1 | tail -2 - echo ">> Cross-language compare (clj vs py)" - ( cd "$DELPHI" && OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 uv run python - "$OUT" <<'PY' -import sys, tempfile -from polismath.replay.crosslang import compare_clj_vs_py -from polismath.replay import stepcompare as sc -with tempfile.TemporaryDirectory() as shim: - print(sc.format_report(compare_clj_vs_py(sys.argv[1], shim_root=shim))) -PY - ) -fi - -rm -f "$SCHED" -echo ">> smoke OK" diff --git a/math/dev/user.clj b/math/dev/user.clj deleted file mode 100644 index 46081f5315..0000000000 --- a/math/dev/user.clj +++ /dev/null @@ -1,489 +0,0 @@ -(ns user - (:require [polismath.runner :as runner] - [polismath.system :as system] - [polismath.meta.microscope :as microscope] - [polismath.conv-man :as conv-man] - [polismath.math.conversation :as conv] - [polismath.components.postgres :as postgres] - [polismath.math.named-matrix :as nm] - [polismath.math.corr :as corr] - [polismath.tasks :as tasks] - [plumbing.core :as plmb] - [honeysql.core :as honey] - [clojure.core.matrix :as matrix] - [clojure.core.async :as async :refer [>! !! runner/system :conversation-manager :conversations deref (get zid) :conv))) - -(defn load-conv - [{:as args :keys [zid zinvite]}] - (let [zid (or zid (postgres/get-zid-from-zinvite (:postgres runner/system) zinvite))] - (conv-man/load-or-init (:conversation-manager runner/system) zid))) - - -;; Plotting utilities - -(defn cmnt-group-votes - [conv tid] - (into {} - (map - (fn [[gid data]] - [(str "g-"gid "-votes") (str (get-in data [:votes tid]))]) - (:group-votes conv)))) - - -(defn importance-metric - [A P S E] - (let [p (/ (+ P 1) (+ S 2)) - a (/ (+ A 1) (+ S 2))] - (* (- 1 p) (+ E 1) a))) - - -(defn cmnt-stats - [conv tid extremity] - (let [group-votes (:group-votes conv) - {:as total-votes :keys [A D S P]} - ;; reduce over votes per group, already aggregated - (reduce - (fn [votes [gid data]] - (let [{:as data :keys [A S D]} (get-in data [:votes tid]) - data (assoc data :P (+ (- S (+ A D))))] - ;; Add in each of the data's kv count pairs - (reduce - (fn [votes' [k v]] - (update votes' k + v)) - votes - data))) - {:A 0 :D 0 :S 0 :P 0} - group-votes) - importance - (importance-metric A P S extremity) - priority - (get-in conv [:comment-priorities tid])] - (log/spy - {:total-votes (str total-votes) - :priority priority - :importance importance - :fontSize (* priority 40)}))) - - -(defn comments-data - [conv] - (->> - (map - (fn [tid extremity [x y]] - (merge - {:tid tid - :label tid - :x x - :y y - :extremity extremity - :type "comment"} - (cmnt-group-votes conv tid) - (cmnt-stats conv tid extremity))) - (:tids conv) - (-> conv :pca :comment-extremity) - (-> conv :pca :comment-projection matrix/transpose)) - (remove (comp (:mod-out conv) :tid)))) - - -(defn groups-data - [conv] - (map - (fn [{:keys [id center members]}] - (let [[x y] (vec center) - pids (->> (:base-clusters conv) - (filter (comp (set members) :id)) - (mapcat :members))] - {:size (count pids) - :x x - :y y - :bids members - :pids pids - :type "group" - :label (str "g-" id)})) - (:group-clusters conv))) - -(defn base-clusters-data - [conv] - (map - (fn [{:keys [id center members]}] - (let [[x y] (vec center)] - {:size (count members) - :x x - :y y - :pids members - :type "base-cluster" - :label ""})) - (:base-clusters conv))) - - -(defn subgroup-clusters-data - [conv] - (mapcat - (fn [[gid subgroups]] - (map - (fn [{:keys [id center members parent-id]}] - (let [[x y] (vec center) - pids (->> (:base-clusters conv) - (filter (comp (set members) :id)) - (mapcat :members))] - {:size (count pids) - :x x - :y y - :bids members - :pids pids - :gid gid - :type "subgroup-cluster" - :parent parent-id - :label (str "g-" gid "-sg-" id)})) - subgroups)) - (:subgroup-clusters conv))) - - -(defn conv-data - [conv] - (concat - (comments-data conv) - (groups-data conv) - (subgroup-clusters-data conv) - (base-clusters-data conv))) - - -(def size-scale - {:bandSize 100 - :pointSizeRange [1 100]}) - -(def conv-plot - {:width 2000 - :height 1300 - :layer [ - {:mark "rule" - :encoding {:x {:value 0 :scale {:zero false}}}} - ;{:mark "rule" - ; :encoding {:x {:value 0.0}}} - {:mark {:type "point" :filled true} - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "size" :scale size-scale} - :color {:field "type"}}} - {:mark "point" - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "size" - :scale size-scale} - :opacity {:value 0.5} - :color {:value "#000"}}} - {:mark "text" - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "fontSize"} - :text {:field "label"}}}]}) - -(def conv-plot - {:width 2000 - :height 1300 - :layer [ - {:mark "rule" - :encoding {:x {:value 0 :scale {:zero false}}}} - ;{:mark "rule" - ; :encoding {:x {:value 0.0}}} - {:mark {:type "point" :filled true} - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "size" :scale size-scale} - :color {:field "type"}}} - {:mark "point" - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "size" - :scale size-scale} - :opacity {:value 0.5} - :color {:value "#000"}}} - {:mark "text" - :encoding {:x {:field "x"} - :y {:field "y"} - :size {:field "fontSize"} - :text {:field "label"}}}]}) - - - -(defn p! - ([conv] - (oz/v! conv-plot :data {:values (conv-data conv)}))) - -;(p! conv) - -;(oz/v! {:data {:values [{:x 1 :y 2} {:x 3 :y 4.3}]} - ;:layer [{:mark "point" - ;:encoding {:x {:field "x"} :y {:field "y"}}}]}) - -;; Test calls here -;(def plot-data -; (conv-data conv)) -;(def the-plot - ;(p! conv)) - - -(defn integrate - [coll] - (:result - (reduce - (fn [result x] - (-> result - (update :total + x) - (update :result conj (+ (:total result) x)))) - {:total 0 :result []} - coll))) - -(defn plot-priorities! - [conv & {:keys [strict-mod exclude-meta]}] - (let [values - (->> (:comment-priorities conv) - (remove (comp - (set/union - (if exclude-meta (:meta-tids conv) #{}) - (if strict-mod - (set/difference (set (:tids conv)) (:mod-in conv)) - (set/difference (:mod-out conv) (:meta-tids conv)))) - first)) - (sort-by second)) - max-integral - (apply max (integrate (vals values))) - comments - (into {} (map (fn [c] [(:tid c) c]) - (comments-data conv))) - entities - (map - (fn [i [tid x] X] - (merge - (get comments tid) - {:rank i - :rank-perc (/ i (count values)) - :tid tid - :priority x - :integral X - :prob (/ X max-integral) - :is-meta (boolean (get (:meta-tids conv) tid)) - :mod-out (boolean (get (:mod-out conv) tid))})) - (range) - values - (integrate (vals values)))] - (oz/v! - {:data {:values entities} - :title "Priority CDF" - :width 1400 - :height 900 - :mark "bar" - :encoding {:x {:field "rank-perc"} - :y {:field "prob"}}}))) - ;:y {:field "priority"}}}))) - - - -(defn prob-dist - "Messy function for investigating statistical distributions. Given a sample of comments selected - from the server's get comments fn, or from a page reload, this function returns the subset of those - comments which fall in the uppermost percentile of comments. By tracing back y values on a cdf plot to - the x values at which they intersect the cft, you can get a sense of what portion of sample should fall - within the specified percentile and then compare that to what you get here." - [conv sample percentile & {:keys [in-conv]}] - (let [comment-priorities - (->> (:comment-priorities conv) - (sort-by second) - (remove (comp (set/difference (:mod-out conv) (:meta-tids conv)) - first)) - (filter (comp - (set (or in-conv (:tids conv))) - first)))] - (log/info (count (vec comment-priorities))) - (set/intersection - (into #{} - (map first - (take-last (log/spy :info (int (* (count comment-priorities) percentile))) - comment-priorities))) - (set sample)))) - - -(defn run-with-repl - [_] - (runner/run! system/full-system) - (log/info "System running; starting nREPL on port 18975") - (nrepl-server/start-server :bind "0.0.0.0" :port 18975 :handler cider.nrepl/cider-nrepl-handler)) - - - -;; Can toggle between do and comment here for refiring entire file -;(do -(comment - (oz/start-plot-server!) - ;; Run one of these to interactively test out a particular system or subsystem - (runner/run! system/base-system {:math-env :preprod}) - ;(runner/run! system/poller-system {:math-env :dev :poll-from-days-ago 0.1}) - ;(runner/run! system/task-system {:math-env :preprod :poll-from-days-ago 3}) - ;(runner/run! system/full-system {:math-env :preprod :poll-from-days-ago 0.1}) - ;(runner/run! system/darwin-system) - - - ;; Execute this to run pure math/util tests - (require '[runner :as test-runner]) - ;; Rerun the tests once runner has been required by executing this - (test-runner/-main) - - - ;; Setting up load and interactive testing for a specific conversation - ; load conv and do a recompute - ;(def zid 17794) - ;(def zid 17175) - ;(def zid 16703) - ;(def zid 16906) - (def zid 17890) ; med-small - ;(def zid 18115) ; big cmnts; med ptpts - (def args {:zid zid}) - - (def conv - (-> (load-conv args) - (conv/conv-update []))) - - (def priority-plot (plot-priorities! conv :strict-mod true :exclude-meta true)) - (pp/pprint priority-plot) - (oz/publish-plot! priority-plot) - - - (->> (nm/get-matrix (:raw-rating-mat conv)) - matrix/shape) - (sort (keys conv)) - ;(let [matrix (nm/get-matrix (:raw-rating-mat conv))] - ;(->> matrix - ;matrix/columns - ;(map - ;(fn [col] - ;(let [seen (remove nil? col) - ;agree ()] - ;#(/ % (-> matrix matrix/shape first)) - ;count - ;double))) - ;sort)) - - ;; Look at profile output - (sort-by (comp - second) @(:profile-data conv)) - (reduce + (map second @(:profile-data conv))) - - ;; Plot the conversation - (p! conv) - - ;; Run correlation matrix - ;(corr/default-tids conv) - - ;; Let's try another conversation - - (def zid2 17023) - (def args2 {:zid zid2}) - (def conv2 (load-conv args2)) - - ;; queue votes through conv-man - (conv-man/queue-message-batch! (:conversation-manager runner/system) - :votes - zid2 - []) - - ;; test queuing report task - ;(conv-man/queue-message-batch! (:conversation-manager runner/system) - ; :generate_report_data - ; zid2 - ; ;; these are probably wrong... - ; [{:rid 6 :zid 15228 :math_tick 52}]) - - - ;; Testing out updates via the pure conv-update function - (let [updated-conv (conv/conv-update conv [{:zid 15228 :pid 0 :tid 0 :vote 2.0 :created (System/currentTimeMillis)}]) - _ (printn "colnames") - updated-conv (conv/mod-update conv [{:zid 15230 :tid 38 :is_meta true :modified (System/currentTimeMillis)}]) - _ (log/info "First update") - updated-conv' (conv/conv-update updated-conv [{:zid 15228 :pid 0 :tid 0 :vote -1 :created (System/currentTimeMillis)}]) - _ (log/info "Second update")] - (log/info "Conv keys:" (keys conv)) - (log/info "Updated conv keys:" (keys updated-conv)) - (log/info "Previous conv key:" (boolean (:conv updated-conv))) - (log/info "Previous conv key:" (boolean (:conv (:conv updated-conv')))) - (:subgroup-clusters updated-conv')) - ;(:repness updated-conv')) - - - ;; Postgres/db testbench - (postgres/query - (:postgres runner/system) - ["select * from votes - limit 10;"]) - - (postgres/query - (:postgres runner/system) - ["select * from votes - where zid = ?;" - 4]) - - (postgres/query - (:postgres runner/system) - ;; Builds a string like the above - (honey/format - {:select [:*] - :from [:votes] - :limit 10})) - - ;; This is the preferred way - (postgres/query - (:postgres runner/system) - ;; Processes maps automatically, so you can do either of the two - {:select [:*] - :from [:votes] - :limit 10}) - - - ;; Debugging issue - (postgres/query - (:postgres runner/system) - {:select [:*] - :from [:math_tasks] - :limit 10}) - - - ;; Getting config settings - (-> runner/system :config) - (-> runner/system :config :webserver-url) - - (-> runner/system :config :darwin) - (-> runner/system :config :darwin :webserver-url) - (-> runner/system :conversation-manager :conversations deref) - - - ;; Playing with core.async parallelism - (defn dowork [size] - (doseq [i (range size)] - (reduce + (range i)))) - - (let [kill-chan (async/promise-chan) - parallelism 4 - work-size 30000] - (doseq [_ (range parallelism)] - (thread - (dowork work-size)))) - - :end) diff --git a/math/doc/configuration.md b/math/doc/configuration.md deleted file mode 100644 index f9fe9958ab..0000000000 --- a/math/doc/configuration.md +++ /dev/null @@ -1,48 +0,0 @@ - -# Math worker configuration - -This set of documentation is currently incomplete, but describes a couple of the more important bits of configuration in the system. - -## Environment variables - -There are a number of variables for tuning and tweaking the system, many of which are exposed via environment variables. -Please see [`src/polismath/components/config.clj`](https://github.com/pol-is/polisMath/blob/master/src/polismath/components/config.clj#L51) for the complete listing of environment variables. - -The ones you're most frequently to need to tweak for one reason or another: - -* **`MATH_ENV`**: This defaults to `dev`, for local development environments. - Traditionally we've set this to `prod` and `preprod` for our production and pre-production deployments specifically. - This value is used in keying the math export json blobs as found in the `math_main` and other tables in the database. - This makes it possible to run multiple math environments (dev, testing, prod, preprod) all on the same database of votes. - This setting is something of a relic from an old architecture where prod and preprod environments ran off of the same database, and with the docker infrastructure is generally no longer needed. - Nevertheless, when you start the math server, you will need to run it with the same **`MATH_ENV`** setting as you ran the math worker with. -* **`POLL_FROM_DAYS_AGO`**: This defaults to 10 (at the time of this writing). - Conversations which have had vote or moderation activity in the specified range will be loaded into memory, and will be updated. - This prevents old inactive conversations from being loaded into memory every time the poller starts. - -You'll also need to pass database credentials. If using docker compose, this will be inherited from the `.env` file or process environment in which docker is being run. - - **`DATABASE_URL`**: url for the database: - - `postgres://:@:/` - -## Database Connection Pooling - -The math service now uses HikariCP for enhanced connection pooling, which significantly improves performance under concurrent workloads (such as during testing). - -* **`DATABASE_POOL_SIZE`**: Maximum number of connections in the pool (default: 10) - * For development: 5-10 connections - * For testing: 8-15 connections - * For production: 10-20 connections (adjust based on load) - -The connection pool automatically: - -* Maintains a minimum number of idle connections (25% of max pool size) -* Validates connections before use with `SELECT 1` -* Detects and prevents connection leaks -* Optimizes prepared statement caching -* Handles connection timeouts and retries gracefully - -### Connection Pool Monitoring - -The service logs connection pool configuration at startup and provides health check functionality. Connection failures are automatically retried up to 3 times with exponential backoff. diff --git a/math/example.env b/math/example.env deleted file mode 100644 index 6fae261e01..0000000000 --- a/math/example.env +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash - -# Useful if running math outside of docker. -# cp example.env .env, modify as needed - -export DATABASE_IGNORE_SSL=true -export DATABASE_URL=DATABASE_URL=postgres://postgres:oiPorg3Nrz0yqDLE@localhost:5432/polis-dev -export DD_ENABLED=false -export LOGGING_LEVEL=debug -export MATH_ENV=dev -export MATH_ZID_ALLOWLIST= -export RECOMPUTE=true diff --git a/math/misc/silh_test.R b/math/misc/silh_test.R deleted file mode 100755 index f511424fec..0000000000 --- a/math/misc/silh_test.R +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env Rscript -# Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -library(argparse) -library(cluster) - -n <- 3 -c <- 3 - -data <- matrix(sample(0:9, n * c, replace=T), nrow=n) -clusts <- kmeans(data, 2) -sil <- silhouette(clusts$cluster, dist(data)) - -print(data) -# [,1] [,2] [,3] -# [1,] 5 9 3 -# [2,] 3 2 7 -# [3,] 5 7 1 - -print(clusts) -# K-means clustering with 2 clusters of sizes 2, 1 -# Cluster means: -# [,1] [,2] [,3] -# 1 5 8 2 -# 2 3 2 7 -# -# Clustering vector: -# [1] 1 2 1 -# -# Within cluster sum of squares by cluster: -# [1] 4 0 -# (between_SS / total_SS = 91.5 %) - -print(sil) - -# cluster neighbor sil_width -# [1,] 1 2 0.6594974 -# [2,] 2 1 0.0000000 -# [3,] 1 2 0.6491768 - diff --git a/math/perf b/math/perf deleted file mode 100644 index f38ac1e1ff..0000000000 --- a/math/perf +++ /dev/null @@ -1,324 +0,0 @@ - -The following was computed on the a 5000x10 conversation; it seems that reflection is one of our problems here - - 2014-May-09 11:27:06 -0700 fullerine INFO [polismath.simulation] - Profiling: :user/clusters - Id Calls Min Max MAD Mean Time% Time - :user/clean-start-clusters 2 9.0ms 4.4s 2.2s 2.2s 30 4.4s - :user/safe-recenter-clusters 2 5.0ms 2.2s 1.1s 1.1s 15 2.2s - :user/row-subset 106 260.0μs 68.0ms 8.0ms 20.0ms 15 2.2s - :user/recenter-clusters 2 3.0ms 2.1s 1.1s 1.1s 14 2.1s - :user/rowname-subset 53 462.0μs 47.0ms 8.0ms 20.0ms 7 1.0s - :user/posible-clusters 2 306.0μs 23.0ms 12.0ms 12.0ms 0 24.0ms - :user/uniqify-clusters 2 20.0μs 1.0ms 737.0μs 757.0μs 0 2.0ms - Clock Time 100 14.8s - Accounted Time 81 12.0s - -Here are the reflection warnings that matter to our lib - - Auto-boxing loop arg: current-indent - exception.clj:44 recur arg for primitive local: i is not matching primitive, had: Object, needed: long - Auto-boxing loop arg: i - - # The main culprit? - Reflection warning, polismath/named_matrix.clj:21:41 - call to indexOf can't be resolved. - Reflection warning, polismath/named_matrix.clj:46:11 - call to indexOf can't be resolved. - Reflection warning, polismath/named_matrix.clj:55:11 - call to indexOf can't be resolved. - Reflection warning, polismath/named_matrix.clj:61:34 - call to indexOf can't be resolved. - Reflection warning, polismath/named_matrix.clj:67:24 - call to indexOf can't be resolved. - - pca.clj:51 recur arg for primitive local: last_eigval is not matching primitive, had: Object, needed: long - Auto-boxing loop arg: last-eigval - - Reflection warning, polismath/clusters.clj:136:27 - call to indexOf can't be resolved. - Reflection warning, polismath/clusters.clj:136:27 - call to indexOf can't be resolved. - Performance warning, polismath/clusters.clj:201:25 - case has int tests, but tested expression is not primitive. - Reflection warning, polismath/conversation.clj:22:13 - call to indexOf can't be resolved. - Reflection warning, polismath/simulation.clj:78:3 - reference to field nextElement can't be resolved. - - -Fuck... this doesn't seem to be working. I took away all of the reflection warnings with a type-indexof -function, but still no dice. What's left of the reflections: - - (defn typed-indexof [^java.util.List coll item] - (.indexOf coll item)) - - # Other libraries... - Reflection warning, /tmp/form-init1039726133741870246.clj:1:911 - call to invokeStaticMethod can't be resolved. - Reflection warning, alex_and_georges/debug_repl.clj:101:8 - reference to field getCause can't be resolved. - Reflection warning, alex_and_georges/debug_repl.clj:102:8 - reference to field getCause can't be resolved. - Reflection warning, plumbing/fnk/schema.clj:93:37 - call to java.lang.IllegalArgumentException ctor can't be resolved. - Reflection warning, clojure/tools/trace.clj:231:9 - call to setStackTrace can't be resolved. - Reflection warning, clojure/tools/trace.clj:242:11 - call to setStackTrace can't be resolved. - Reflection warning, clojure/tools/trace.clj:244:49 - call to java.lang.Exception ctor can't be resolved. - Reflection warning, clojure/tools/trace.clj:244:11 - call to setStackTrace can't be resolved. - Reflection warning, clojure/tools/trace.clj:276:9 - call to setStackTrace can't be resolved. - Reflection warning, clojure/tools/trace.clj:277:15 - call to setStackTrace can't be resolved. - Reflection warning, bigml/sampling/random.clj:11:11 - call to cern.jet.random.tdouble.engine.MersenneTwister64 vector can't be resolved. - Reflection warning, clj_time/core.clj:577:10 - reference to field getDayOfMonth can't be resolved. - - # Us???? - columns.clj:125 recur arg for primitive local: current_indent is not matching primitive, had: Object, needed: long - Auto-boxing loop arg: current-indent - exception.clj:44 recur arg for primitive local: i is not matching primitive, had: Object, needed: long - Auto-boxing loop arg: i - pca.clj:51 recur arg for primitive local: last_eigval is not matching primitive, had: Object, needed: long - Auto-boxing loop arg: last-eigval - - Performance warning, polismath/clusters.clj:201:25 - case has int tests, but tested expression is not primitive. - Reflection warning, polismath/conversation.clj:22:13 - call to indexOf can't be resolved. - Reflection warning, polismath/simulation.clj:78:3 - reference to field nextElement can't be resolved. - - -May 12, 2014 - -WOOT! It seems I've sorted out the biggest culrpit here. -It turned out to be the filter-by-index function. -It was creating a new set each time, which was absolutely killing performance. -I was able to cut down an order of magnitude by doing this alone. - -So now on to other things... - -For starters, I tried running on 100,000 ptpts 10 cmts. -There are the big and big-part update times - - N-ptpts: 99998 - Starting new conv update! - 1399963793428 rating-mat 1292120.671242 msecs #=> 1,292.120s #=> 20 minutes - 1399963793436 mat 7.876006 msecs - 1399963800685 pca 7249.400704 msecs # 7 sec - 1399963802045 proj 1359.328171 msecs # 1.3 sec - 1399963848873 base-clusters 46828.255953 msecs # 46 sec - 1399963848873 bid-to-pid 0.054336 msecs - 1399963849626 votes-base 752.377173 msecs - 1399963849626 counting-ptpts 0.003771 msecs - 1399963849628 group-clusters 2.392552 msecs - 1399963849628 CONVUP big 1353257.322353 msecs - - N-ptpts: 99998 - 1399963849843 rating-mat 140.33363 msecs - 1399963849849 mat 5.574799 msecs - 1399963855133 pca 5283.952589 msecs - 1399963856372 proj 1238.580412 msecs - 1399964442254 base-clusters 585882.249774 msecs # 585.882 # 10 min - 1399964442254 bid-to-pid 0.056502 msecs - 1399964442916 votes-base 661.268519 msecs - 1399964442916 counting-ptpts 0.003702 msecs - 1399964442917 group-clusters 1.644483 msecs - 1399964442918 CONVUP big-part 593284.9938 msecs - - 2014-May-13 00:00:42 -0700 noether INFO [polismath.simulation] - Profiling: :user/clusters - Id Calls Min Max MAD Mean Time% Time - :user/kmeans 2 2.0ms 585.9s 292.9s 292.9s 99 585.9s - :user/clean-start-clusters 2 1.0ms 546.2s 273.1s 273.1s 92 546.2s - :user/safe-recenter-clusters 2 604.0μs 276.2s 138.1s 138.1s 47 276.2s - :user/safe-recenter-map 2 552.0μs 276.2s 138.1s 138.1s 47 276.2s - :user/safe-rowname-subset 53 90.0μs 9.6s 1.7s 5.2s 46 274.5s - :user/recenter-clusters 2 485.0μs 269.9s 135.0s 135.0s 45 269.9s - :user/rowname-subset 53 82.0μs 8.7s 1.6s 5.1s 45 268.4s - :user/row-subset 106 46.0μs 47.0ms 3.0ms 28.0ms 1 3.0s - :user/filter-by-index 212 5.0μs 862.0μs 129.0μs 439.0μs 0 93.0ms - :user/uniqify-clusters 2 12.0μs 1.0ms 510.0μs 522.0μs 0 1.0ms - :user/safe-recenter-empty 2 3.0μs 301.0μs 149.0μs 152.0μs 0 303.0μs - :user/safe-recenter-nil 2 21.0μs 155.0μs 67.0μs 88.0μs 0 175.0μs - Clock Time 100 593.3s - Accounted Time 421 2500.3s - - -Hmm.... This is suggesting that performance on the rating-mat update may actually become a problem if we can't -keep everything else moving quickly enough. -Of course, this corresponds to 1,000,000 votes. -If we ever get 1,000,000 votes at once then (for a specific conv), God help us for now... XXX - -On the other hand, note that the base clustering takes QUITE a long time in this second one here (10 minutes). -Again, this seems to be due largely due to the clean-start-clusters. -We really have to figure out how to speed that up. XXX - -Another thought: initially I was concerned that something might not be quite right given that the second -(partial) updata would always take so much longer than the full matrix population. -But this actually makes sense: most of the time is in the clean-start-clusters, but the clean-start-clusters -is going to be a lot simpler if the last round didn't have that many members. - -Also, note that there are still some strange "unable to slice nil" errors cropping up every now and again. -A few of those leading to a backlog, and it might put us on ice while the conversation catches up. XXX - -One last thing: -Note that the PCA grows as a function of the number of comments, but that downstream analyses don't, because -they're just on projections. - - # 10 cmts - big - 1399967451528 pca 942.879957 msecs # 1 sec - 1399967451706 proj 177.306975 msecs # 0.2 sec - big-part - 1399967458208 pca 751.15802 msecs # < 1sec - 1399967458369 proj 161.097721 msecs - - # 100 cmts - big - 1399967669033 pca 6840.381594 msecs # 7 sec - 1399967669866 proj 833.318748 msecs # 0.8 sec - big-part - 1399967679124 pca 3850.089523 msecs # 4 sec - 1399967679911 proj 786.742437 msecs # 0.8 sec - -Something interesting here: Note the second times are a lot shorter in each case (1/2-ish). -This is likely because the power-iteration is working exactly as planned. -Which is good. -Still, if it's another 7x to increase both the cmts and ptpts 10x, then we could easily expect to see minute long -compute times for our PCA step. -So we'll either want to fix our Batch PCA or try to speed this bit up XXX - -Also note here that the time for update is much lower than it was when there were 10x more people (80sec). -Could this let us use a tranpose for this messy operation? XXX -In general this may also point to where some of the possible optimizations are with respect to this update -implementation. - - -May 18, 2014 - -Been a little while since I checked in... - -I've figured out what is causing the slow rating mtarix update. -Silly me... I was doing indexOf on the rows list to get to the matrix indices. -Bad idea... -I'm going to switch to hash-maos, through which I think we can cut the matrix-update code to seconds with 500k ptpts -This was what we had from that count: - - - 2014-May-17 22:27:26 -0700 noether INFO [polismath.simulation] - Profiling: :user/clusters - Id Calls Min Max MAD Mean Time% Time - :user/update-nmat 1 392.2s 392.2s 0ns 392.2s 92 392.2s - :user/MU-indices 600000 2.0μs 3.0ms 324.0μs 627.0μs 88 376.2s - :user/rowname-subset 54 47.0μs 15.7s 571.0ms 298.0ms 4 16.1s - :user/kmeans 2 2.0ms 12.1s 6.1s 6.1s 3 12.1s - :user/MU-rows-cols 600000 1.0μs 1.3s 4.0μs 4.0μs 1 2.6s - :user/clean-start-clusters 2 1.0ms 1.6s 818.0ms 819.0ms 0 1.6s - :user/MU-assoc-in 600000 1.0μs 4.0ms 278ns 2.0μs 0 1.4s - :user/MU-indices-2 600000 1.0μs 219.0μs 247ns 2.0μs 0 1.2s - :user/safe-recenter-clusters 2 576.0μs 820.0ms 410.0ms 410.0ms 0 820.0ms - :user/safe-recenter-map 2 564.0μs 819.0ms 409.0ms 410.0ms 0 820.0ms - :user/row-subset 107 29.0μs 45.0ms 988.0μs 7.0ms 0 771.0ms - :user/recenter-clusters 2 351.0μs 768.0ms 384.0ms 384.0ms 0 768.0ms - :user/MU-assoc 600000 698ns 1.0ms 226ns 1.0μs 0 626.0ms - :user/MU-row-add 600000 488ns 197.0μs 492ns 966ns 0 580.0ms - :user/MU-col-add 600000 488ns 293.0μs 101ns 911ns 0 547.0ms - :user/safe-rowname-subset 53 57.0μs 12.0ms 856.0μs 8.0ms 0 401.0ms - Clock Time 100 428.3s - Accounted Time 189 808.8s - -Will want to keep an eye on the other MU items on that list, as they are liable to be the next points for speeding things up. - - -May 21, 2014 - -Woot! I seem to have stamped this shit. -At least largely. -Before I go on though, I should make a correction. -I'm pretty sure the above was actually on 600,000 new _comments_, not ptpts. -The latter would not make sense with the call counts shown above. - -Anyway; it seems that using indexOf on matrices was basically just a terrible idea. -I thought lookup was constant time. -Which is silly in retrospect. -Anyway. -It's not. - -I've implemented a custom data type which maintains both a vector and a hash-map so we can quickly get (the -former equivalent of) indexOf. -Now look at the times: - - Starting new conv update! - N-ptpts: 60000 - count 720000 - 1400735231267 rating-mat 3003.265079 msecs - 1400735231267 counting-comments 0.0044 msecs - 1400735231336 user-vote-count 68.865603 msecs - 1400735231381 mat 5.540164 msecs - 1400735235871 pca 4489.99813 msecs - 1400735236737 proj 865.464804 msecs - 1400735256213 base-clusters 19475.856953 msecs - 1400735256213 bid-to-pid 0.054686 msecs - 1400735256715 votes-base 501.544857 msecs - 1400735256715 counting-ptpts 0.004539 msecs - 1400735256717 group-clusters 1.903313 msecs - 1400735256717 CONVUP big-part 31481.174735 msecs - 2014-May-21 22:07:36 -0700 noether INFO [polismath.simulation] - Profiling: :user/clusters - Id Calls Min Max MAD Mean Time% Time - :user/kmeans 2 2.0ms 19.3s 9.6s 9.6s 61 19.3s - :user/update-nmat 1 2.9s 2.9s 0ns 2.9s 9 2.9s - :user/clean-start-clusters 2 928.0μs 1.8s 921.0ms 921.0ms 6 1.8s - :user/safe-recenter-clusters 2 505.0μs 944.0ms 472.0ms 472.0ms 3 945.0ms - :user/safe-recenter-map 2 492.0μs 944.0ms 472.0ms 472.0ms 3 944.0ms - :user/recenter-clusters 2 268.0μs 837.0ms 418.0ms 418.0ms 3 837.0ms - :user/safe-rowname-subset 53 37.0μs 4.0ms 345.0μs 3.0ms 1 164.0ms - Clock Time 100 31.5s - Accounted Time 85 26.9s - - -Damn son. -Down to 30 seconds. -Still some room for improvment given I'm looking at some new reflection/performance warnings. -We'll see how things do after that. - - -Hmmm... I tried running on 1mil ptpts and 100 cmts and got this. - - Starting new conv update! - N-ptpts: 100000 - count 10200000 - 1400736772092 rating-mat 454258.957733 msecs - 1400736772093 counting-comments 0.005029 msecs - 1400736778787 user-vote-count 6694.438309 msecs - 1400736778866 mat 8.482992 msecs - 1400736853808 pca 74942.795939 msecs - 1400736863400 proj 9591.412637 msecs - 1400736914508 base-clusters 51108.178211 msecs - 1400736914509 bid-to-pid 0.065022 msecs - Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded - at clojure.lang.PersistentHashMap$ArrayNode.assoc(PersistentHashMap.java:364) - at clojure.lang.PersistentHashMap$ArrayNode.assoc(PersistentHashMap.java:361) - at clojure.lang.PersistentHashMap.assoc(PersistentHashMap.java:142) - at clojure.lang.PersistentHashMap.assoc(PersistentHashMap.java:28) - at clojure.lang.RT.assoc(RT.java:702) - at clojure.core$assoc.invoke(core.clj:187) - at clojure.core$zipmap.invoke(core.clj:2715) - at polismath.conversation$agg_bucket_votes_for_tid.invoke(conversation.clj:24) - at polismath.conversation$fn__4283$fnk4280_positional__4284$fn__4287.invoke(conversation.clj:168) - - -Can probably fix this with JVM settings, but for now we seem to have found an upper limit. -However, of interest is that the rating-mat update, while significant is operating on 100mil votes, which -we'll liekly never see in a single update. -If we have a milion people probably more like 10 per user minute, tops. -So if we keep everything else moving quickly enough, we might be fine. - -Here is some data for 30k x 20cmts: - - Starting new conv update! - N-ptpts: 30000 - count 660000 - 1400746217528 rating-mat 1503.576119 msecs - 1400746217529 counting-comments 0.0044 msecs - 1400746217583 user-vote-count 54.830368 msecs - 1400746217621 mat 4.400001 msecs - 1400746221937 pca 4316.419774 msecs - 1400746222599 proj 661.168579 msecs - 1400746234367 base-clusters 11768.231347 msecs - 1400746234367 bid-to-pid 0.054615 msecs - 1400746234788 votes-base 420.764263 msecs - 1400746234788 counting-ptpts 0.004191 msecs - 1400746234790 group-clusters 1.557323 msecs - 1400746234790 CONVUP big-part 21241.605796 msecs - 2014-May-22 01:10:34 -0700 noether INFO [polismath.simulation] - Profiling: :user/clusters - Id Calls Min Max MAD Mean Time% Time - :user/kmeans 2 1.0ms 11.7s 5.8s 5.8s 55 11.7s - :user/update-nmat 1 1.4s 1.4s 0ns 1.4s 6 1.4s - :user/clean-start-clusters 2 891.0μs 1.2s 588.0ms 589.0ms 6 1.2s - :user/safe-recenter-clusters 2 470.0μs 585.0ms 292.0ms 293.0ms 3 585.0ms - :user/safe-recenter-map 2 457.0μs 585.0ms 292.0ms 293.0ms 3 585.0ms - :user/recenter-clusters 2 260.0μs 520.0ms 260.0ms 260.0ms 2 521.0ms - :user/safe-rowname-subset 53 33.0μs 3.0ms 222.0μs 2.0ms 0 105.0ms - Clock Time 100 21.2s - Accounted Time 75 16.0s - - diff --git a/delphi/polismath/__init__.py b/math/polismath/__init__.py similarity index 100% rename from delphi/polismath/__init__.py rename to math/polismath/__init__.py diff --git a/delphi/polismath/benchmarks/__init__.py b/math/polismath/benchmarks/__init__.py similarity index 100% rename from delphi/polismath/benchmarks/__init__.py rename to math/polismath/benchmarks/__init__.py diff --git a/delphi/polismath/benchmarks/bench_pca.py b/math/polismath/benchmarks/bench_pca.py similarity index 100% rename from delphi/polismath/benchmarks/bench_pca.py rename to math/polismath/benchmarks/bench_pca.py diff --git a/delphi/polismath/benchmarks/bench_repness.py b/math/polismath/benchmarks/bench_repness.py similarity index 100% rename from delphi/polismath/benchmarks/bench_repness.py rename to math/polismath/benchmarks/bench_repness.py diff --git a/delphi/polismath/benchmarks/bench_update_votes.py b/math/polismath/benchmarks/bench_update_votes.py similarity index 100% rename from delphi/polismath/benchmarks/bench_update_votes.py rename to math/polismath/benchmarks/bench_update_votes.py diff --git a/delphi/polismath/benchmarks/benchmark_utils.py b/math/polismath/benchmarks/benchmark_utils.py similarity index 100% rename from delphi/polismath/benchmarks/benchmark_utils.py rename to math/polismath/benchmarks/benchmark_utils.py diff --git a/delphi/polismath/components/__init__.py b/math/polismath/components/__init__.py similarity index 100% rename from delphi/polismath/components/__init__.py rename to math/polismath/components/__init__.py diff --git a/delphi/polismath/components/config.py b/math/polismath/components/config.py similarity index 100% rename from delphi/polismath/components/config.py rename to math/polismath/components/config.py diff --git a/delphi/polismath/conversation/__init__.py b/math/polismath/conversation/__init__.py similarity index 100% rename from delphi/polismath/conversation/__init__.py rename to math/polismath/conversation/__init__.py diff --git a/delphi/polismath/conversation/conversation.py b/math/polismath/conversation/conversation.py similarity index 100% rename from delphi/polismath/conversation/conversation.py rename to math/polismath/conversation/conversation.py diff --git a/delphi/polismath/conversation/manager.py b/math/polismath/conversation/manager.py similarity index 100% rename from delphi/polismath/conversation/manager.py rename to math/polismath/conversation/manager.py diff --git a/delphi/polismath/database/__init__.py b/math/polismath/database/__init__.py similarity index 100% rename from delphi/polismath/database/__init__.py rename to math/polismath/database/__init__.py diff --git a/delphi/polismath/database/dynamodb.py b/math/polismath/database/dynamodb.py similarity index 100% rename from delphi/polismath/database/dynamodb.py rename to math/polismath/database/dynamodb.py diff --git a/delphi/polismath/database/postgres.py b/math/polismath/database/postgres.py similarity index 100% rename from delphi/polismath/database/postgres.py rename to math/polismath/database/postgres.py diff --git a/math/polismath/paths.py b/math/polismath/paths.py new file mode 100644 index 0000000000..487ebaa6f7 --- /dev/null +++ b/math/polismath/paths.py @@ -0,0 +1,51 @@ +"""Filesystem anchors for the polismath package. + +The package lives at ``/math/polismath`` (moved from +``delphi/polismath`` at cutover Step #4), while the test estate — +``real_data/`` datasets, golden snapshots, replay stores, ``.test_outputs`` +— deliberately stayed under ``/delphi/``. Before the move, modules +derived those locations from their own ``__file__`` (``parents[2]`` was the +delphi root); after it, the same arithmetic lands on ``math/`` — silently +wrong. Every cross-tree anchor now lives here, derived once. + +Modules keep their historical module-level names (``_DELPHI_ROOT``, +``REAL_DATA_ROOT``, ``_MATH_ROOT``, ...) assigned FROM these constants, so +tests that monkeypatch those attributes keep working unchanged. + +ASSUMPTION: these anchors are meaningful only under an EDITABLE / +repo-checkout install (delphi/.venv via [tool.uv.sources]). In the Docker +image polismath sits in site-packages, so REPO_ROOT resolves inside the +interpreter prefix and none of the cross-tree paths exist — which is fine +today because no production module (poller/, run_math_pipeline, +conversation/, pca_kmeans_rep/, database/) imports this module; only the +replay/regression harness does, and that runs from the repo checkout. +Keep it that way: production code must not import polismath.paths. +""" + +from __future__ import annotations + +from pathlib import Path + +# paths.py -> polismath -> math -> repo root. +REPO_ROOT = Path(__file__).resolve().parents[2] + +# Home of this package (and, pre-cutover, of the Clojure engine). +MATH_ROOT = REPO_ROOT / "math" + +# The polismath package directory itself (certify's engine-tree hash root). +PACKAGE_ROOT = Path(__file__).resolve().parent + +# The delphi tree: umap/narrative service AND the math test estate +# (tests, real_data datasets + goldens, replay stores, certify battery). +DELPHI_ROOT = REPO_ROOT / "delphi" + +# Datasets + golden snapshots (delphi/real_data; private under .local/). +REAL_DATA_ROOT = DELPHI_ROOT / "real_data" + +# The Clojure reference tree — the certification oracle. Removed from the +# working tree at cutover Step #4; it exists ONLY in git history +# (`git log -- math/`, any commit before Step #4). Code that replays the +# oracle (certify's ensure_clj_recording) must check this path exists and +# skip/fail with a message pointing at the history checkout otherwise — +# the requires_math_tree test guards rely on exactly that check. +CLJ_TREE_ROOT = REPO_ROOT / "math" diff --git a/delphi/polismath/pca_kmeans_rep/__init__.py b/math/polismath/pca_kmeans_rep/__init__.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/__init__.py rename to math/polismath/pca_kmeans_rep/__init__.py diff --git a/delphi/polismath/pca_kmeans_rep/clusters.py b/math/polismath/pca_kmeans_rep/clusters.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/clusters.py rename to math/polismath/pca_kmeans_rep/clusters.py diff --git a/delphi/polismath/pca_kmeans_rep/corr.py b/math/polismath/pca_kmeans_rep/corr.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/corr.py rename to math/polismath/pca_kmeans_rep/corr.py diff --git a/delphi/polismath/pca_kmeans_rep/group_k_smoother.py b/math/polismath/pca_kmeans_rep/group_k_smoother.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/group_k_smoother.py rename to math/polismath/pca_kmeans_rep/group_k_smoother.py diff --git a/delphi/polismath/pca_kmeans_rep/legacy_kmeans.py b/math/polismath/pca_kmeans_rep/legacy_kmeans.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/legacy_kmeans.py rename to math/polismath/pca_kmeans_rep/legacy_kmeans.py diff --git a/delphi/polismath/pca_kmeans_rep/pca.py b/math/polismath/pca_kmeans_rep/pca.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/pca.py rename to math/polismath/pca_kmeans_rep/pca.py diff --git a/delphi/polismath/pca_kmeans_rep/repness.py b/math/polismath/pca_kmeans_rep/repness.py similarity index 100% rename from delphi/polismath/pca_kmeans_rep/repness.py rename to math/polismath/pca_kmeans_rep/repness.py diff --git a/delphi/polismath/poller/__init__.py b/math/polismath/poller/__init__.py similarity index 98% rename from delphi/polismath/poller/__init__.py rename to math/polismath/poller/__init__.py index 77480263b3..c44300b913 100644 --- a/delphi/polismath/poller/__init__.py +++ b/math/polismath/poller/__init__.py @@ -9,7 +9,7 @@ ------------ :: - scripts/math_poller.py (CLI) + python -m polismath.poller (CLI, polismath/poller/__main__.py) └─ service.MathPollerService ├─ vote loop (thread): poll_votes_since(wm) cadence VOTE_POLLING_INTERVAL ├─ mod loop (thread): poll_moderation_since(wm) cadence MOD_POLLING_INTERVAL @@ -153,10 +153,10 @@ :: # Run the service (blocks; SIGTERM/SIGINT -> graceful stop) - uv run python scripts/math_poller.py + uv run python -m polismath.poller # Single poll cycle then exit (smoke test / cron-style) - uv run python scripts/math_poller.py --once + uv run python -m polismath.poller --once Shadow-mode deployment writes under a DISTINCT ``MATH_ENV`` (e.g. ``python``) next to the Clojure ``math`` container; ``UNIQUE(zid, math_env)`` keeps the rows diff --git a/delphi/scripts/math_poller.py b/math/polismath/poller/__main__.py similarity index 91% rename from delphi/scripts/math_poller.py rename to math/polismath/poller/__main__.py index ead4b0a7fa..8f24372656 100644 --- a/delphi/scripts/math_poller.py +++ b/math/polismath/poller/__main__.py @@ -6,9 +6,9 @@ math_env. See polismath.poller (package docstring) and delphi/docs/MATH_POLLER_DESIGN.md. -Usage: - uv run python scripts/math_poller.py # run forever (SIGTERM stops) - uv run python scripts/math_poller.py --once # one poll cycle then exit +Usage (was scripts/math_poller.py before cutover Step #4): + uv run python -m polismath.poller # run forever (SIGTERM stops) + uv run python -m polismath.poller --once # one poll cycle then exit """ import argparse diff --git a/delphi/polismath/poller/math_writer.py b/math/polismath/poller/math_writer.py similarity index 100% rename from delphi/polismath/poller/math_writer.py rename to math/polismath/poller/math_writer.py diff --git a/delphi/polismath/poller/service.py b/math/polismath/poller/service.py similarity index 100% rename from delphi/polismath/poller/service.py rename to math/polismath/poller/service.py diff --git a/delphi/polismath/poller/worker_pool.py b/math/polismath/poller/worker_pool.py similarity index 100% rename from delphi/polismath/poller/worker_pool.py rename to math/polismath/poller/worker_pool.py diff --git a/math/tmp/.touchfile b/math/polismath/py.typed similarity index 100% rename from math/tmp/.touchfile rename to math/polismath/py.typed diff --git a/delphi/polismath/regression/__init__.py b/math/polismath/regression/__init__.py similarity index 100% rename from delphi/polismath/regression/__init__.py rename to math/polismath/regression/__init__.py diff --git a/delphi/polismath/regression/clojure_comparer.py b/math/polismath/regression/clojure_comparer.py similarity index 100% rename from delphi/polismath/regression/clojure_comparer.py rename to math/polismath/regression/clojure_comparer.py diff --git a/delphi/polismath/regression/comparer.py b/math/polismath/regression/comparer.py similarity index 99% rename from delphi/polismath/regression/comparer.py rename to math/polismath/regression/comparer.py index 35969b0f90..f2cbca2f1b 100644 --- a/delphi/polismath/regression/comparer.py +++ b/math/polismath/regression/comparer.py @@ -7,6 +7,8 @@ import logging import re from pathlib import Path + +from polismath import paths from typing import Dict, Any, Optional from datetime import datetime from scipy import stats @@ -259,8 +261,9 @@ def compare_with_golden( # Write differences to log file if any were found diff_log_path = None timestamp = datetime.now().strftime("%Y%m%d-%H%M%S") - # Output to .test_outputs/regression directory - output_dir = Path(__file__).parent.parent.parent / ".test_outputs" / "regression" + # Output to delphi/.test_outputs/regression (the test estate stays + # under delphi/ — see polismath.paths). + output_dir = paths.DELPHI_ROOT / ".test_outputs" / "regression" output_dir.mkdir(parents=True, exist_ok=True) # Use dataset_name or fall back to report_id for the identifier diff --git a/delphi/polismath/regression/datasets.py b/math/polismath/regression/datasets.py similarity index 98% rename from delphi/polismath/regression/datasets.py rename to math/polismath/regression/datasets.py index 851e68d894..8b32cf6a31 100644 --- a/delphi/polismath/regression/datasets.py +++ b/math/polismath/regression/datasets.py @@ -15,6 +15,8 @@ import re from dataclasses import dataclass from pathlib import Path + +from polismath import paths from typing import Dict, List, Optional # Optional metadata for known datasets @@ -64,8 +66,8 @@ def description(self) -> str: def get_real_data_dir() -> Path: - """Path to delphi/real_data/""" - return (Path(__file__).parent.parent.parent / 'real_data').resolve() + """Path to delphi/real_data/ (stays under delphi/ — see polismath.paths).""" + return paths.REAL_DATA_ROOT def get_local_data_dir() -> Path: diff --git a/delphi/polismath/regression/recorder.py b/math/polismath/regression/recorder.py similarity index 100% rename from delphi/polismath/regression/recorder.py rename to math/polismath/regression/recorder.py diff --git a/delphi/polismath/regression/utils.py b/math/polismath/regression/utils.py similarity index 98% rename from delphi/polismath/regression/utils.py rename to math/polismath/regression/utils.py index 320cfafa08..61eeaa912c 100644 --- a/delphi/polismath/regression/utils.py +++ b/math/polismath/regression/utils.py @@ -15,6 +15,7 @@ from typing import Dict, Any, Tuple, Optional import pandas as pd +from polismath import paths from polismath.conversation.conversation import Conversation # Backward-compatible re-export: convert_numpy_types was defined here (nested in # save_golden_snapshot); it now lives in the shared serialization util so the @@ -117,8 +118,9 @@ def compute_all_stages( } debug_info["nan_info"] = nan_info - # Save debug info to .test_outputs/debug directory - debug_dir = Path(__file__).parent.parent / ".test_outputs" / "debug" + # Save debug info to delphi/.test_outputs/debug (the test + # estate stays under delphi/ — see polismath.paths). + debug_dir = paths.DELPHI_ROOT / ".test_outputs" / "debug" debug_dir.mkdir(parents=True, exist_ok=True) debug_path = debug_dir / f"pca_debug_{dataset_name}.json" with open(debug_path, "w") as f: diff --git a/delphi/polismath/replay/__init__.py b/math/polismath/replay/__init__.py similarity index 100% rename from delphi/polismath/replay/__init__.py rename to math/polismath/replay/__init__.py diff --git a/delphi/polismath/replay/certify.py b/math/polismath/replay/certify.py similarity index 94% rename from delphi/polismath/replay/certify.py rename to math/polismath/replay/certify.py index e2e1676982..a954761caa 100644 --- a/delphi/polismath/replay/certify.py +++ b/math/polismath/replay/certify.py @@ -47,6 +47,7 @@ from pathlib import Path from typing import Any +from polismath import paths from polismath.replay import real_data from polismath.replay import schedule as sched from polismath.replay import store as st @@ -69,10 +70,13 @@ # --------------------------------------------------------------------------- # Paths. # --------------------------------------------------------------------------- -# certify.py -> replay -> polismath -> delphi -> repo root (mirrors store.py). -_DELPHI_ROOT = Path(__file__).resolve().parents[2] -_REPO_ROOT = _DELPHI_ROOT.parents[0] -_MATH_ROOT = _REPO_ROOT / "math" +# Cross-tree anchors (see polismath.paths). _MATH_ROOT is the clj ORACLE +# tree — removed from the working tree at cutover Step #4; re-replaying the +# oracle needs a git-history checkout of math/ (the requires_math_tree test +# guards skip oracle-dependent paths when dev/replay.clj is absent). +_DELPHI_ROOT = paths.DELPHI_ROOT +_REPO_ROOT = paths.REPO_ROOT +_MATH_ROOT = paths.CLJ_TREE_ROOT DEFAULT_BATTERY_PATH = _DELPHI_ROOT / "scripts" / "certify_battery.json" @@ -530,6 +534,19 @@ def _manifest_matches(manifest_path: Path, expected: dict[str, Any]) -> bool: return existing == expected +def _manifest_matches_inputs(manifest_path: Path, input_keys: dict[str, Any]) -> bool: + """Subset match on the INPUT cache keys only (votes/schedule/comments) — + the oracle-absent path of :func:`ensure_clj_recording`, where the stored + clj-source hashes cannot be recomputed for comparison.""" + if not manifest_path.exists(): + return False + try: + existing = json.loads(manifest_path.read_text()) + except (OSError, json.JSONDecodeError): + return False + return all(existing.get(k) == v for k, v in input_keys.items()) + + def _write_manifest(manifest_path: Path, manifest: dict[str, Any]) -> None: manifest_path.parent.mkdir(parents=True, exist_ok=True) with open(manifest_path, "w") as fh: @@ -552,13 +569,19 @@ def _write_manifest(manifest_path: Path, manifest: dict[str, Any]) -> None: ) +#: Default root for :func:`engine_tree_hash` — the installed package tree. +#: Module-level so tests can monkeypatch the seam (pre-Step-#4 they patched +#: ``_DELPHI_ROOT`` and relied on the root being derived from it). +_ENGINE_TREE_ROOT = paths.PACKAGE_ROOT + + def engine_tree_hash(polismath_root: str | Path | None = None) -> str: """Tree hash of the ENGINE surface: every ``polismath/**/*.py`` except :data:`_ENGINE_TREE_EXCLUDE` — the py recording cache key. Harness-only edits therefore keep recordings cached (the ~36-min full py re-replay is reserved for actual engine changes). Uncached because tests mutate trees; the battery hot path goes through :func:`_engine_tree_hash_cached`.""" - root = Path(polismath_root) if polismath_root is not None else _DELPHI_ROOT / "polismath" + root = Path(polismath_root) if polismath_root is not None else _ENGINE_TREE_ROOT return sha256_tree(root, "**/*.py", exclude=_ENGINE_TREE_EXCLUDE) @@ -567,11 +590,28 @@ def _engine_tree_hash_cached() -> str: return engine_tree_hash() +def _raise_missing_oracle() -> None: + raise CertifyError( + "clj-oracle", + "Clojure oracle tree not present at math/ (removed at cutover " + "Step 4; it lives in git history). To RE-REPLAY the oracle, " + "restore it from a pre-Step-4 revision — recipe in math/README.md " + "('certification oracle'). Cached pairs need no oracle; only a " + "cache miss or --refresh forces a clj re-replay.", + ) + + @functools.lru_cache(maxsize=1) def _clj_source_hashes() -> tuple[str, str]: """(sha256 of dev/replay.clj, sha256 of the math/src tree) — cached since both are read-only per process and re-hashing the whole math/src tree on - every battery entry is wasted work.""" + every battery entry is wasted work. + + Post-cutover (Step 4) the Clojure oracle tree is absent from the + working tree — fail with instructions rather than a bare + FileNotFoundError (paths.py CLJ_TREE_ROOT's contract).""" + if not (_MATH_ROOT / "dev" / "replay.clj").exists(): + _raise_missing_oracle() return ( sha256_file(_MATH_ROOT / "dev" / "replay.clj"), sha256_tree(_MATH_ROOT / "src", "**/*"), @@ -630,15 +670,28 @@ def ensure_clj_recording( rec_dir = st.recording_dir(entry.dataset, entry.schedule_id, root=root) clj_dir = rec_dir / "clj" manifest_path = clj_dir / "cache_manifest.json" - replay_clj_sha256, math_src_sha256 = _clj_source_hashes() - expected = { + input_keys = { "votes_sha256": votes_sha, "schedule_hash": canonical_schedule_hash(spec), + } + if comments_csv is not None: + input_keys["comments_csv_sha256"] = sha256_file(comments_csv) + if not (_MATH_ROOT / "dev" / "replay.clj").exists(): + # Post-cutover-Step-4: the oracle tree lives only in git history. + # A cached recording whose INPUT keys (votes/schedule/comments) + # match is trusted as-is — its stored clj-source hashes describe + # the tree it was recorded against, which cannot be (and need not + # be) revalidated without the tree. Only an actual re-replay — + # cache miss or --refresh — demands the restored oracle. + if not refresh and _manifest_matches_inputs(manifest_path, input_keys): + return clj_dir, True + _raise_missing_oracle() + replay_clj_sha256, math_src_sha256 = _clj_source_hashes() + expected = { + **input_keys, "replay_clj_sha256": replay_clj_sha256, "math_src_sha256": math_src_sha256, } - if comments_csv is not None: - expected["comments_csv_sha256"] = sha256_file(comments_csv) if not refresh and _manifest_matches(manifest_path, expected): return clj_dir, True diff --git a/delphi/polismath/replay/crosslang.py b/math/polismath/replay/crosslang.py similarity index 100% rename from delphi/polismath/replay/crosslang.py rename to math/polismath/replay/crosslang.py diff --git a/delphi/polismath/replay/driver.py b/math/polismath/replay/driver.py similarity index 100% rename from delphi/polismath/replay/driver.py rename to math/polismath/replay/driver.py diff --git a/delphi/polismath/replay/poller_equiv.py b/math/polismath/replay/poller_equiv.py similarity index 99% rename from delphi/polismath/replay/poller_equiv.py rename to math/polismath/replay/poller_equiv.py index 9ba776d048..67c38f3aad 100644 --- a/delphi/polismath/replay/poller_equiv.py +++ b/math/polismath/replay/poller_equiv.py @@ -132,6 +132,7 @@ import sqlalchemy as sa +from polismath import paths from polismath.replay import real_data from polismath.replay import schedule as sched from polismath.replay.certify import _acceptance_projecting_comparer, normalize_path @@ -140,11 +141,12 @@ from polismath.replay.types import ModEvent, ReplayDataset from polismath.utils.general import delphi_vote_to_postgres -# poller_equiv.py -> replay -> polismath -> delphi -> repo root (mirrors -# certify.py / store.py). -_DELPHI_ROOT = Path(__file__).resolve().parents[2] -_REPO_ROOT = _DELPHI_ROOT.parents[0] -_MATH_ROOT = _REPO_ROOT / "math" +# Cross-tree anchors (see polismath.paths — the test estate stays under +# delphi/; the clj oracle tree exists only in git history post-cutover). +# Module-level names kept: tests monkeypatch/assert on them. +_DELPHI_ROOT = paths.DELPHI_ROOT +_REPO_ROOT = paths.REPO_ROOT +_MATH_ROOT = paths.CLJ_TREE_ROOT DEFAULT_DBNAME = "polis_equiv" DEFAULT_ZID = 1 @@ -820,7 +822,7 @@ def build_py_env( database_ssl_mode: str = "disable", base_env: Optional[dict[str, str]] = None, ) -> dict[str, str]: - """Env for ``scripts/math_poller.py``, keyed to ``PollerConfig.from_env`` + """Env for ``python -m polismath.poller``, keyed to ``PollerConfig.from_env`` (service.py:150-182): ``DATABASE_URL``, ``MATH_ENV``, ``POLL_FROM_DAYS_AGO`` (same names as the clj side — see :func:`build_clj_env`). @@ -833,7 +835,7 @@ def build_py_env( side must not leak that scheme here. ``DATABASE_SSL_MODE`` defaults to ``"disable"`` — root cause #3 - (2026-07-24 live debug): ``scripts/math_poller.py`` never loads a `.env` + (2026-07-24 live debug): the poller entry never loads a `.env` file, so ``PostgresClient``'s own fallback (``os.environ.get("DATABASE_SSL_MODE", "require")``, postgres.py:92) bites whenever this env var isn't already present in the CALLING shell — @@ -853,9 +855,11 @@ def build_py_env( class PyPollerRunner(_SubprocessRunner): - """The python math_poller CLI (``scripts/math_poller.py``), run-forever - mode — its only flag is ``--once`` (math_poller.py:43-48), which we do NOT - pass, so it runs until killed exactly like the clj container.""" + """The python math poller (``python -m polismath.poller`` — was + ``scripts/math_poller.py`` before cutover Step 4), run-forever mode — + its only flag is ``--once`` (polismath/poller/__main__.py), which we do + NOT pass, so it runs until killed exactly like the clj container. + cwd stays ``_DELPHI_ROOT``: that is where the uv project/venv lives.""" def __init__( self, @@ -875,7 +879,7 @@ def __init__( base_env=base_env, ) super().__init__( - ["uv", "run", "python", "scripts/math_poller.py"], cwd=_DELPHI_ROOT, env=env, + ["uv", "run", "python", "-m", "polismath.poller"], cwd=_DELPHI_ROOT, env=env, log_path=log_path, ) diff --git a/delphi/polismath/replay/prodclone.py b/math/polismath/replay/prodclone.py similarity index 100% rename from delphi/polismath/replay/prodclone.py rename to math/polismath/replay/prodclone.py diff --git a/delphi/polismath/replay/real_data.py b/math/polismath/replay/real_data.py similarity index 97% rename from delphi/polismath/replay/real_data.py rename to math/polismath/replay/real_data.py index cece77c727..3403e6b3c5 100644 --- a/delphi/polismath/replay/real_data.py +++ b/math/polismath/replay/real_data.py @@ -18,9 +18,11 @@ import re from pathlib import Path +from polismath import paths from polismath.replay.types import ModEvent, ReplayDataset -REAL_DATA_ROOT = Path(__file__).resolve().parents[2] / "real_data" +# delphi/real_data — the datasets stay under delphi/ (see polismath.paths). +REAL_DATA_ROOT = paths.REAL_DATA_ROOT # Slug allow-list — same precedent as prodclone.py's minted-slug regex # (``_SLUG_RE_TEMPLATE``): a slug flows unsanitized into a ``Path.glob()`` diff --git a/delphi/polismath/replay/schedule.py b/math/polismath/replay/schedule.py similarity index 100% rename from delphi/polismath/replay/schedule.py rename to math/polismath/replay/schedule.py diff --git a/math/polismath/replay/shadow_compare.py b/math/polismath/replay/shadow_compare.py new file mode 100644 index 0000000000..79460ab8c6 --- /dev/null +++ b/math/polismath/replay/shadow_compare.py @@ -0,0 +1,373 @@ +"""Live shadow-soak comparer for the Clojure→Python math cutover (Step #1). + +Compares math rows between two live math_envs in the SAME database — the +Clojure engine's rows (``math_env='prod'``) vs the Python poller's shadow +rows (``math_env='python'``) — using the same acceptance surface as the +certify battery and the poller-equivalence harness: + +- ``math_main`` → certify's acceptance-projecting comparer + (:func:`polismath.replay.certify._acceptance_projecting_comparer` — + subgroup-* excluded per Q7, structural identity + declared float + tolerances); +- ``math_bidtopid`` → EXACT (:func:`polismath.replay.poller_equiv.compare_bidtopid`, + modulo the documented pid int/str normalization); +- ``math_ptptstats`` → structural + tolerant + (:func:`polismath.replay.poller_equiv._ptptstats_comparer`). + +Unlike the replay harness there is NO batch gating: the two engines poll the +live vote stream independently, so at any instant a zid's row pair may +reflect different vote watermarks. Pairs are only judged when IN SYNC (equal +:func:`polismath.replay.poller_equiv.blob_total_votes`); out-of-sync pairs +are reported and retried on the next run, never counted as divergence. + +Large conversations — ``n`` > 10000 participants OR ``n-cmts`` > 5000 +comments, the Clojure large-conv dispatch cutoffs (conversation.clj:784-815) +— are EXPECTED to diverge: Clojure's mini-batch PCA path is unseeded-random +there (Q10, CLOJURE_QUIRKS.md — not even self-consistent between two Clojure +runs), while Python runs deterministic full PCA at every size (documented +improvement, same blob shape). Their divergences classify as +``large-conv-q10`` and never fail the run (CUTOVER_RUNBOOK.md risk #2). + +Exit-code semantics (this is the Step 2 gate — runbook "Shadow exit +checklist"): 0 = no unexpected divergence (and ``--min-matches`` satisfied); +1 = at least one UNEXPECTED divergence; 2 = clean but fewer full MATCH zids +than ``--min-matches`` demands (coverage not yet demonstrated — keep +soaking / re-run once engines sync). NOTE: click also exits 2 on a CLI +usage error — gate scripts must require exit 0, never distinguish 1 vs 2. + +CLI wrapper: ``delphi/scripts/shadow_compare.py``. +""" + +from __future__ import annotations + +from collections import Counter +from typing import Any, Mapping, Sequence + +import sqlalchemy as sa + +from polismath.replay import poller_equiv as pe +from polismath.replay.certify import _acceptance_projecting_comparer +from polismath.replay.stepcompare import StepComparer + +# Clojure's large-conv dispatch cutoffs — STRICTLY-GREATER comparisons +# (conversation.clj:784-815: n-ptpts > 10000 OR n-cmts > 5000). +LARGE_CONV_PTPT_CUTOFF = 10_000 +LARGE_CONV_CMT_CUTOFF = 5_000 + +TABLES = ("math_main", "math_bidtopid", "math_ptptstats") + +SYNC_IN_SYNC = "in-sync" +SYNC_OUT_OF_SYNC = "out-of-sync" +SYNC_NOT_READY = "not-ready" + +VERDICT_MATCH = "match" +VERDICT_DIVERGE = "diverge" +VERDICT_LARGE_CONV_Q10 = "large-conv-q10" +VERDICT_OUT_OF_SYNC = "out-of-sync" +VERDICT_NOT_READY = "not-ready" +VERDICT_MISSING = "missing" +VERDICT_PARTIAL = "partial" + +# Verdicts that MAY fail a run (see aggregate); everything else is +# expected/transient and only ever reported. +_FAILING_VERDICTS = (VERDICT_DIVERGE,) + +PairMap = Mapping[str, tuple[dict[str, Any] | None, dict[str, Any] | None]] + + +def is_large_conv(blob: Mapping[str, Any]) -> bool: + """Q10 classification from the blob's own ``n``/``n-cmts`` keys. + Missing/malformed keys count as small — a blob too degenerate to carry + its size cannot be excused as a large conv.""" + n = blob.get("n") + n_cmts = blob.get("n-cmts") + return (isinstance(n, (int, float)) and n > LARGE_CONV_PTPT_CUTOFF) or ( + isinstance(n_cmts, (int, float)) and n_cmts > LARGE_CONV_CMT_CUTOFF + ) + + +def sync_state(blob_a: Mapping[str, Any], blob_b: Mapping[str, Any]) -> str: + """Whether the two engines have processed the same cumulative vote set. + ``blob_total_votes`` returns None when a blob can't be judged yet — + treated as not-ready, never as zero (see its docstring). + + Equal totals alone can hide a REVOTE-only watermark delta (a revote + overwrites its (pid, tid) cell, never inflating the total), which would + surface as a false small-conv divergence. When BOTH blobs carry + ``lastVoteTimestamp`` it must match too; if either omits it, vote-total + equality remains the only (best-effort) signal.""" + va = pe.blob_total_votes(dict(blob_a)) + vb = pe.blob_total_votes(dict(blob_b)) + if va is None or vb is None: + return SYNC_NOT_READY + if va != vb: + return SYNC_OUT_OF_SYNC + la = blob_a.get("lastVoteTimestamp") + lb = blob_b.get("lastVoteTimestamp") + if la is not None and lb is not None and la != lb: + return SYNC_OUT_OF_SYNC + return SYNC_IN_SYNC + + +def _tick_summary(row: Mapping[str, Any]) -> dict[str, Any]: + return { + "caching_tick": row.get("caching_tick"), + "math_tick": row.get("math_tick"), + "last_vote_timestamp": row.get("last_vote_timestamp"), + "modified": row.get("modified"), + } + + +def classify_pair( + zid: int, + pair: PairMap, + math_envs: tuple[str, str], + *, + math_main_comparer: StepComparer | None = None, + ptptstats_comparer: StepComparer | None = None, +) -> dict[str, Any]: + """One zid's verdict over the three math tables (same table semantics as + :func:`polismath.replay.poller_equiv.compare_batch`, applied to live row + pairs instead of snapshot stores).""" + env_a, env_b = math_envs + main_a, main_b = pair.get("math_main", (None, None)) + if main_a is None or main_b is None: + return { + "zid": zid, + "verdict": VERDICT_MISSING, + "missing": [e for e, r in ((env_a, main_a), (env_b, main_b)) if r is None], + } + + blob_a, blob_b = main_a.get("data"), main_b.get("data") + if not isinstance(blob_a, Mapping) or not isinstance(blob_b, Mapping): + # Dirty live data (NULL / non-JSON-object column) must be survivable + # — report and retry, never crash the soak tool. + return { + "zid": zid, + "verdict": VERDICT_NOT_READY, + "reason": "malformed-data", + "malformed": [ + e for e, b in ((env_a, blob_a), (env_b, blob_b)) + if not isinstance(b, Mapping) + ], + } + + torn = _torn_tables(pair, math_envs) + if torn: + # Each engine mints ONE math_tick per write cycle shared across the + # three tables (math_writer.py write cycle; mirrors conv_man.clj) — + # a side table on a different tick than its own env's math_main is + # a torn read (the six SELECTs are not one snapshot): retry, don't + # false-diverge. + return { + "zid": zid, + "verdict": VERDICT_NOT_READY, + "reason": "torn-read", + "torn": torn, + } + + base = { + "zid": zid, + "ticks": {env_a: _tick_summary(main_a), env_b: _tick_summary(main_b)}, + "votes": { + env_a: pe.blob_total_votes(blob_a), + env_b: pe.blob_total_votes(blob_b), + }, + } + + state = sync_state(blob_a, blob_b) + if state == SYNC_NOT_READY: + return {**base, "verdict": VERDICT_NOT_READY} + if state == SYNC_OUT_OF_SYNC: + return {**base, "verdict": VERDICT_OUT_OF_SYNC} + + large = is_large_conv(blob_a) or is_large_conv(blob_b) + tables: dict[str, Any] = {} + + main_cmp = math_main_comparer or _acceptance_projecting_comparer() + step = main_cmp.compare_step(blob_a, blob_b, zid) + tables["math_main"] = { + "match": step["match"], + "n_divergences": step["n_divergences"], + "families": step["families"], + } + + bid_a, bid_b = pair.get("math_bidtopid", (None, None)) + if bid_a is None or bid_b is None: + tables["math_bidtopid"] = { + "match": None, + "reason": "missing-row", + "missing": [e for e, r in ((env_a, bid_a), (env_b, bid_b)) if r is None], + } + else: + cmp_bid = pe.compare_bidtopid(bid_a["data"], bid_b["data"]) + # Drop the (potentially huge) normalized payloads from the verdict; + # a diverging soak run re-fetches them for debugging anyway. + tables["math_bidtopid"] = {"match": cmp_bid["match"]} + + ppt_a, ppt_b = pair.get("math_ptptstats", (None, None)) + if ppt_a is None or ppt_b is None: + tables["math_ptptstats"] = { + "match": None, + "reason": "missing-row", + "missing": [e for e, r in ((env_a, ppt_a), (env_b, ppt_b)) if r is None], + } + else: + cmp_ppt = ptptstats_comparer or pe._ptptstats_comparer() + step_ppt = cmp_ppt.compare_step(ppt_a["data"], ppt_b["data"], zid) + tables["math_ptptstats"] = { + "match": step_ppt["match"], + "n_divergences": step_ppt["n_divergences"], + "families": step_ppt["families"], + } + + diverged = any(t.get("match") is False for t in tables.values()) + partial = any(t.get("match") is None for t in tables.values()) + if diverged: + # Q10 (unseeded-random clj mini-batch PCA) can only be the cause + # when math_main ITSELF diverges — a side-table-only divergence on + # a large conv is a genuine defect, never excused. + main_diverged = tables["math_main"].get("match") is False + verdict = VERDICT_LARGE_CONV_Q10 if (large and main_diverged) else VERDICT_DIVERGE + elif partial: + verdict = VERDICT_PARTIAL + else: + verdict = VERDICT_MATCH + return {**base, "verdict": verdict, "large_conv": large, "tables": tables} + + +def _torn_tables(pair: PairMap, math_envs: tuple[str, str]) -> list[dict[str, Any]]: + """Side tables whose ``math_tick`` disagrees with their own env's + math_main row (both rows present and both ticks known).""" + torn: list[dict[str, Any]] = [] + mains = pair.get("math_main", (None, None)) + for idx, env in enumerate(math_envs): + main_row = mains[idx] + main_tick = main_row.get("math_tick") if main_row is not None else None + if main_tick is None: + continue + for table in ("math_bidtopid", "math_ptptstats"): + row = pair.get(table, (None, None))[idx] + tick = row.get("math_tick") if row is not None else None + if tick is not None and tick != main_tick: + torn.append( + {"env": env, "table": table, "math_tick": tick, "main_tick": main_tick} + ) + return torn + + +def aggregate( + verdicts: Sequence[Mapping[str, Any]], *, min_matches: int = 0 +) -> dict[str, Any]: + """Run summary + exit code. Only UNEXPECTED divergence (``diverge``) + fails outright (exit 1). ``min_matches`` unmet — with no divergence — + exits 2: the soak hasn't yet DEMONSTRATED coverage, which is different + from having demonstrated a defect.""" + counts = Counter(v["verdict"] for v in verdicts) + n_matches = counts.get(VERDICT_MATCH, 0) + if any(counts.get(v) for v in _FAILING_VERDICTS): + exit_code = 1 + elif n_matches < min_matches: + exit_code = 2 + else: + exit_code = 0 + return { + "counts": dict(counts), + "n_zids": len(verdicts), + "n_matches": n_matches, + "min_matches": min_matches, + "exit_code": exit_code, + } + + +def discover_zids( + conn: Any, math_envs: tuple[str, str], *, limit: int | None = None +) -> list[int]: + """zids carrying a math_main row in EITHER env (a row missing on one + side is itself a finding — see ``classify_pair``'s missing verdict), + most recently modified first.""" + text = ( + "SELECT zid, MAX(modified) AS last_modified FROM math_main " + "WHERE math_env IN (:env_a, :env_b) " + "GROUP BY zid ORDER BY last_modified DESC" + ) + params: dict[str, Any] = {"env_a": math_envs[0], "env_b": math_envs[1]} + if limit is not None: + text += " LIMIT :limit" + params["limit"] = limit + result = conn.execute(sa.text(text), params) + return [int(row["zid"]) for row in result.mappings().all()] + + +def fetch_pair(conn: Any, zid: int, math_envs: tuple[str, str]) -> dict[str, Any]: + """The ``{table: (row_a, row_b)}`` mapping for one zid.""" + env_a, env_b = math_envs + return { + table: ( + pe.fetch_math_row(conn, table, zid, env_a), + pe.fetch_math_row(conn, table, zid, env_b), + ) + for table in TABLES + } + + +def run( + conn: Any, + *, + math_envs: tuple[str, str] = ("prod", "python"), + zids: Sequence[int] | None = None, + limit: int | None = 50, + min_matches: int = 0, +) -> dict[str, Any]: + """One soak-compare pass: discover (or take) zids, fetch each pair, + classify, aggregate. Read-only — never writes to the database.""" + if zids is None: + zids = discover_zids(conn, math_envs, limit=limit) + verdicts = [ + classify_pair(zid, fetch_pair(conn, zid, math_envs), math_envs) + for zid in zids + ] + return { + "math_envs": list(math_envs), + "verdicts": verdicts, + "summary": aggregate(verdicts, min_matches=min_matches), + } + + +def render_lines(report: Mapping[str, Any], *, max_lines: int = 200) -> list[str]: + """Human-readable soak report (one line per zid + a summary line).""" + env_a, env_b = report["math_envs"] + lines: list[str] = [] + for v in report["verdicts"][:max_lines]: + zid = v["zid"] + verdict = v["verdict"].upper() + if v["verdict"] == VERDICT_MISSING: + lines.append(f"zid {zid}: {verdict} (no row for: {', '.join(v['missing'])})") + continue + votes = v.get("votes", {}) + ticks = v.get("ticks", {}) + detail = ( + f"votes {env_a}={votes.get(env_a)} {env_b}={votes.get(env_b)}, " + f"caching_tick {env_a}={ticks.get(env_a, {}).get('caching_tick')} " + f"{env_b}={ticks.get(env_b, {}).get('caching_tick')}" + ) + if v["verdict"] in (VERDICT_DIVERGE, VERDICT_LARGE_CONV_Q10): + n_div = sum( + t.get("n_divergences", 0) or 0 for t in v.get("tables", {}).values() + ) + suffix = " — EXPECTED (Q10 unseeded-random clj large-conv path)" if ( + v["verdict"] == VERDICT_LARGE_CONV_Q10 + ) else "" + lines.append(f"zid {zid}: {verdict} ({n_div} divergences; {detail}){suffix}") + else: + lines.append(f"zid {zid}: {verdict} ({detail})") + n_hidden = len(report["verdicts"]) - min(len(report["verdicts"]), max_lines) + if n_hidden: + lines.append(f"... {n_hidden} more zids (see --json-out for the full report)") + s = report["summary"] + counts = ", ".join(f"{k}={n}" for k, n in sorted(s["counts"].items())) or "no zids" + lines.append( + f"summary: {counts}; matches {s['n_matches']}/{s['min_matches']} required" + f" -> exit {s['exit_code']}" + ) + return lines diff --git a/delphi/polismath/replay/shard_bench.py b/math/polismath/replay/shard_bench.py similarity index 99% rename from delphi/polismath/replay/shard_bench.py rename to math/polismath/replay/shard_bench.py index ca2fe89f4c..5da7f98d99 100644 --- a/delphi/polismath/replay/shard_bench.py +++ b/math/polismath/replay/shard_bench.py @@ -40,6 +40,7 @@ from pathlib import Path from typing import Any, Iterable, Sequence +from polismath import paths from polismath.poller.service import should_process_zid # Every knob a BLAS/OpenMP backend might read. Pinning only OMP_NUM_THREADS @@ -63,7 +64,7 @@ # scheduling are not), so "linear" cannot mean 1.00. DEFAULT_MIN_EFFICIENCY = 0.8 -_DELPHI_ROOT = Path(__file__).resolve().parents[2] +_DELPHI_ROOT = paths.DELPHI_ROOT CHILD_TIMEOUT_SEC = 1800.0 diff --git a/delphi/polismath/replay/stepcompare.py b/math/polismath/replay/stepcompare.py similarity index 100% rename from delphi/polismath/replay/stepcompare.py rename to math/polismath/replay/stepcompare.py diff --git a/delphi/polismath/replay/store.py b/math/polismath/replay/store.py similarity index 97% rename from delphi/polismath/replay/store.py rename to math/polismath/replay/store.py index b9a72b7ac5..b2ab24774c 100644 --- a/delphi/polismath/replay/store.py +++ b/math/polismath/replay/store.py @@ -27,6 +27,8 @@ import hashlib import json import os + +from polismath import paths import platform import subprocess from dataclasses import dataclass @@ -42,10 +44,9 @@ def replays_root() -> Path: - """Default store root: ``delphi/real_data/.local/replays`` (gitignored).""" - # store.py → replay → polismath → delphi - delphi = Path(__file__).resolve().parents[2] - return delphi / "real_data" / ".local" / "replays" + """Default store root: ``delphi/real_data/.local/replays`` (gitignored). + The test estate stays under delphi/ — see :mod:`polismath.paths`.""" + return paths.REAL_DATA_ROOT / ".local" / "replays" def _safe_path_component(name: str, *, label: str) -> str: @@ -202,7 +203,7 @@ def _sha256(path: str | Path) -> str | None: def _git_commit() -> str: - repo = Path(__file__).resolve().parents[3] # replay→polismath→delphi→repo + repo = paths.REPO_ROOT try: out = subprocess.run( ["git", "-C", str(repo), "rev-parse", "HEAD"], diff --git a/delphi/polismath/replay/types.py b/math/polismath/replay/types.py similarity index 100% rename from delphi/polismath/replay/types.py rename to math/polismath/replay/types.py diff --git a/delphi/polismath/run_math_pipeline.py b/math/polismath/run_math_pipeline.py similarity index 100% rename from delphi/polismath/run_math_pipeline.py rename to math/polismath/run_math_pipeline.py diff --git a/delphi/polismath/utils/__init__.py b/math/polismath/utils/__init__.py similarity index 100% rename from delphi/polismath/utils/__init__.py rename to math/polismath/utils/__init__.py diff --git a/delphi/polismath/utils/clj_hash.py b/math/polismath/utils/clj_hash.py similarity index 100% rename from delphi/polismath/utils/clj_hash.py rename to math/polismath/utils/clj_hash.py diff --git a/delphi/polismath/utils/env_flags.py b/math/polismath/utils/env_flags.py similarity index 100% rename from delphi/polismath/utils/env_flags.py rename to math/polismath/utils/env_flags.py diff --git a/delphi/polismath/utils/general.py b/math/polismath/utils/general.py similarity index 100% rename from delphi/polismath/utils/general.py rename to math/polismath/utils/general.py diff --git a/delphi/polismath/utils/serialization.py b/math/polismath/utils/serialization.py similarity index 100% rename from delphi/polismath/utils/serialization.py rename to math/polismath/utils/serialization.py diff --git a/math/pyproject.toml b/math/pyproject.toml new file mode 100644 index 0000000000..00bd5cb59d --- /dev/null +++ b/math/pyproject.toml @@ -0,0 +1,44 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "polismath" +version = "0.1.0" +description = "Polis math engine — certified Clojure-exact PCA / k-means / repness, with the production math poller" +requires-python = ">=3.12,<3.13" # Pin to Python 3.12.x (matches delphi) + +# Version pins mirror delphi/pyproject.toml (the two packages share one +# environment: delphi depends on polismath and installs both into +# delphi/.venv — see delphi/pyproject.toml [tool.uv.sources]). +dependencies = [ + "numpy>=1.26.4,<2.0", + "pandas>=2.1.4", + "scipy>=1.12.0", + "scikit-learn>=1.4.2", + "sqlalchemy>=2.0.36", + "psycopg2-binary==2.9.10", + "boto3==1.34.70", + "click>=8.3.1", + "natsort>=8.4.0", + "pyyaml>=6.0", +] + +[project.scripts] +run-math-pipeline = "polismath.run_math_pipeline:main" + +[tool.hatch.build.targets.wheel] +packages = ["polismath"] + +# Pyright: math/ is its own project root for files under math/polismath. +# The interpreter is SHARED with delphi (polismath is installed editable +# into delphi/.venv), hence the venvPath one level up and across. +[tool.pyright] +include = ["polismath"] +exclude = [ + "**/__pycache__", + "**/.pytest_cache", +] +venvPath = "../delphi" +venv = ".venv" +pythonVersion = "3.12" diff --git a/math/python_conversion/.gitignore b/math/python_conversion/.gitignore deleted file mode 100644 index fed8a5d32c..0000000000 --- a/math/python_conversion/.gitignore +++ /dev/null @@ -1,54 +0,0 @@ -# Python bytecode -__pycache__/ -*.py[cod] -*$py.class - -# Distribution / packaging -dist/ -build/ -*.egg-info/ -*.egg - -# Virtual environments -polis_env/ -new_polis_env/ -venv/ -ENV/ -env/ -.env -.venv - -# Jupyter Notebook -.ipynb_checkpoints -*/.ipynb_checkpoints/* - -# Data files -data/ -*.csv -*.json -*.npy -*.pkl -*.db -*.sqlite - -# Development files -.idea/ -.vscode/ -*.swp -*.swo -.DS_Store - -# Pytest cache -.pytest_cache/ -.coverage -htmlcov/ - -# Logs -*.log -logs/ - -# Environment variables -.env - -# Generated files -*.so \ No newline at end of file diff --git a/math/resources/config.edn b/math/resources/config.edn deleted file mode 100644 index dedffca616..0000000000 --- a/math/resources/config.edn +++ /dev/null @@ -1,81 +0,0 @@ -{ - ;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - - ;; TODO This is a work in progress... really think we need to move to aero though: https://github.com/juxt/aero - ;; Soon... - - :math-env #or [#keyword #env MATH_ENV, :dev] - :darwin {:server-port #or [#long #env SERVER_PORT, 3123]} - - :database {:pool-size #or [#long #env DATABASE_POOL_SIZE, 3] - :url #env DATABASE_URL} - - :math-schema-date "2014_08_22" - :export {:expiry-days 10} - - ;; TODO This poller config hasn't been hooked up yet... ghost - :poller {:votes {:polling-interval 2000} - :moderation {:polling-interval 5000}} - - :math {:matrix-implementation :vectorz - :cutoffs {:medium 100 - :large 10000}} - ;; These are copied over from the conv update opts; Should expose them through more generally - ;; A little messy because these get used in different places than above... - ;:n-comps 2 - ;:pca-iters 10 - ;:base-iters 10 - ;:base-k 50 - ;:max-k 5 - ;:group-iters 10 - ;:max-ptpts 80000 - ;:max-cmts 800 - ;:group-k-buffer 4}} - - :logging {:file "log/dev.log" - :level :warn} - - ;:math-env {:parse ->keyword} - ;; Have to use :port since that's what heroku expects... - ;:port {:path [:darwin :server-port] :parse ->long} - ;:database-url {:path [:database :url]} - :database-for-reads-name {:path [:database :reads-name]} - ;:database-pool-size {:path [:database :pool-size] :parse ->long} - :mongolab-uri {:path [:mongo :url]} - :mailgun-api-key {:path [:email :api-key]} - :mailgun-url {:path [:email :url]} - ;:math-matrix-implementation {:path [:math :matrix-implementation] :parse ->keyword} - :math-cutoff-medium {:path [:math :cutoffs :medium] :parse ->long - :doc "This is the maximum size of a conversation before running in :medium mode"} - :math-cutoff-large {:path [:math :cutoffs :large] :parse ->long - :doc "This is the maximum size of a conversation before running in :large mode"} - :math-cutoff-max-ptpts {:path [:math :cutoffs :max-ptpts] :parse ->long - :doc "This is the maximum number of participants before the conversation stops accepting new participants"} - :math-cutoff-max-cmnts {:path [:math :cutoffs :max-ptpts] :parse ->long - :doc "This is the maximum number of comments before the conversation stops accepting new comments"} - :math-schema-date {:doc "This helps us version our mongo buckets."} - ;; Should change these to be more abstract in key name; not hostedgraphite-apikey; just graphite-apikey etc XXX - :hostedgraphite-apikey {:path [:meta :graphite :api-key] - :doc "API key for graphite db (perf monitoring)"} - :hostedgraphite-hostname {:path [:meta :graphite :hostname] - :doc "The hostname for sending messages to graphite"} - :export-expiry-days {:path [:export :expiry-days] :parse ->long - :doc "The number of days before a mongo record representing a data exports gets removed"} - :vote-polling-interval {:parse ->long :path [:storm :spouts :votes :polling-interval] - :doc "The polling interval for votes, in milliseconds"} - :mod-polling-interval {:parse ->long :path [:storm :spouts :moderation :polling-interval] - :doc "The polling interval for moderation, in milliseconds"} - ;; Need to think more about the semantics of a recompute; once; always; only if not booted; etc? XXX - :recompute {:parse boolean - :doc "Whether or not to perform a recompute"} - ;; Need to think about how to handle options - :storm-execution {:path [:storm :execution] :options [:local :distributed] :parse ->keyword - :doc "Whether to run storm as a distributed cluster (StormSubmitter) or in local mode (LocalCluster)"} - :storm-cluster-name {:path [:storm :cluster-name] - :doc "Name of the cluster to run on in distributed mode"} - :storm-workers {:path [:storm :workers] :parse ->long - :doc "Number of storm cluster workers for distributed mode"} - :logging-level {:path [:logging :level] :parse ->keyword - :doc "Logging level for timbre; info, debug, error, etc"} - :logging-file {:path [:logging :file] - :doc "If set, a file to which the log will be appended"}} diff --git a/math/src/data_readers.clj b/math/src/data_readers.clj deleted file mode 100644 index 6e88088cd6..0000000000 --- a/math/src/data_readers.clj +++ /dev/null @@ -1,7 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -{mikera.vectorz.Vector clojure.core.matrix/matrix - mikera.arrayz.NDArray clojure.core.matrix/matrix - mikera.matrixx.Matrix clojure.core.matrix/matrix - ;clojure.core.matrix.impl.ndarray.NDArray clojure.core.matrix ;; Need to check this one... XXX - polismath.named-matrix.NamedMatrix polismath.named-matrix/named-matrix-reader} diff --git a/math/src/polismath/components/config.clj b/math/src/polismath/components/config.clj deleted file mode 100644 index b2a4eea571..0000000000 --- a/math/src/polismath/components/config.clj +++ /dev/null @@ -1,198 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.config - (:require [polismath.utils :as utils] - [taoensso.timbre :as log] - [com.stuartsierra.component :as component] - [environ.core :as environ] - [clojure.string :as string])) - - -;; I think we need to just move to this: https://github.com/juxt/aero -;; Good set of features without baking in too many assumptions or constraining design - - -(defn ->long [x] - (try - (Long/parseLong x) - (catch Exception e - (when-not (= x "") - ;; Otherwise, assume nil was intended... - (log/warn "Failed to parse ->long:" x)) - nil))) - -(defn ->keyword [x] - ;; Otherwise want to return nil, so merging works - (when (and x (not (= x ""))) - (keyword x))) - -(defn ->boolean [x] - (when x - ;; anything other than these values will be considered truthy - (not (#{"false" "0" "no"} x)))) - - -;; XXX This should be computed; that is be a function of the rules. -;; :default specification should be in the rules themselves -(def defaults - {:math-env :dev - :math-schema-date "2014_08_22" - :server {:port 8080} - ;:darwin {:server-port 3123} - :export {:expiry-days 6 - :temp-dir "/tmp/" - ;; Hmmm should be able to specify a dep on port; aero? - :private-url-base "http://localhost:8080"} - ;; Shit... should be able to specify dependency on preprod as well! - :database {:pool-size 3} - :poller {:votes {:polling-interval 1000} - :moderation {:polling-interval 1000} - :tasks {:polling-interval 1000} - :poll-from-days-ago 10} - :math {:matrix-implementation :vectorz} - :logging {:file "log/dev.log" - :level :warn}}) - -(defn ->long-list [x] - (when x - (->> (string/split x #",") - (map ->long) - (set)))) - - -(def rules - "Mapping of env keys to parsing options" - {:math-env {:parse ->keyword} - ;; Have to use :port since that's what heroku expects... - :port {:path [:server :port] :parse ->long} - :database-url {:path [:database :url]} - :database-for-reads-name {:path [:database :reads-name]} - :database-pool-size {:path [:database :pool-size] :parse ->long} - :database-ignore-ssl {:path [:database :ignore-ssl] :parse ->boolean} - :mailgun-api-key {:path [:email :api-key]} - :mailgun-url {:path [:email :url]} - :aws-secret-key {:path [:aws :secret-access-key]} - :aws-access-key {:path [:aws :access-key-id]} - :webserver-username {:path [:webserver-username]} - :webserver-pass {:path [:webserver-pass]} - :math-zid-blocklist {:path [:poller :zid-blocklist] :parse ->long-list} - :math-zid-allowlist {:path [:poller :zid-allowlist] :parse ->long-list} - :export-server-auth-username {:path [:darwin :server-auth-username]} - :export-server-auth-pass {:path [:darwin :server-auth-pass]} - :math-matrix-implementation {:path [:math :matrix-implementation] :parse ->keyword} - ;; TODO Put all these within a :conv-update opt so we can just pass that through to conv-update all at once - :math-cutoff-medium {:path [:math :cutoffs :medium] :parse ->long - :doc "This is the maximum size of a conversation before running in :medium mode"} - :math-cutoff-large {:path [:math :cutoffs :large] :parse ->long - :doc "This is the maximum size of a conversation before running in :large mode"} - :math-cutoff-max-ptpts {:path [:math :cutoffs :max-ptpts] :parse ->long - :doc "This is the maximum number of participants before the conversation stops accepting new participants"} - :math-cutoff-max-cmnts {:path [:math :cutoffs :max-ptpts] :parse ->long - :doc "This is the maximum number of comments before the conversation stops accepting new comments"} - :math-schema-date {:doc "This helps us version our mongo buckets."} - ;; Should change these to be more abstract in key name; not hostedgraphite-apikey; just graphite-apikey etc XXX - :hostedgraphite-apikey {:path [:meta :graphite :api-key] - :doc "API key for graphite db (perf monitoring)"} - :hostedgraphite-hostname {:path [:meta :graphite :hostname] - :doc "The hostname for sending messages to graphite"} - :export-expiry-days {:path [:export :expiry-days] :parse ->long - :doc "The number of days before a mongo record representing a data exports gets removed"} - :vote-polling-interval {:parse ->long :path [:poller :votes :polling-interval] - :doc "The polling interval for votes, in milliseconds"} - :mod-polling-interval {:parse ->long :path [:poller :moderation :polling-interval] - :doc "The polling interval for moderation, in milliseconds"} - ;:initial-polling-timestamp {:parse ->long :path [:poller :initial-polling-timestamp] - ; :doc "The initial vote and mod polling timestamp (only load convs with votes later than this)"} - :poll-from-days-ago {:parse ->long :path [:poller :poll-from-days-ago]} - ;; Need to think more about the semantics of a recompute; once; always; only if not booted; etc? XXX - :recompute {:parse ->boolean - :doc "Whether or not to perform a recompute"} - ;; Need to think about how to handle options - :logging-level {:path [:logging :level] :parse ->keyword - :doc "Logging level for timbre; info, debug, error, etc"} - :logging-file {:path [:logging :file] - :doc "If set, a file to which the log will be appended"}}) - ;; XXX TODO & Thoughts - ;; Mini batch sizes (see polismath.math.conversation) - - -(defn assoc-inferred-values - [{:as config-map :keys [math-env]}] - (let [math-env-string (name math-env) - webserver-url (str "https://" - (when-not (= math-env :prod) (str math-env-string ".")) - "pol.is/api/v3")] - (assoc config-map - :math-env-string math-env-string - :webserver-url webserver-url))) - - - -(defn get-environ-config [rules env] - ;; reduce over rules and assoc-in mappings into empty map - (reduce - (fn [config [name {:keys [parse path] :or {parse identity}}]] - (if-let [env-var-val (get env name)] - (assoc-in config (or path [name]) (parse env-var-val)) - config)) - {} - rules)) - -(defn deep-merge - "Like merge, but merges maps recursively." - [& maps] - (if (every? #(or (map? %) (nil? %)) maps) - (apply merge-with deep-merge maps) - (last maps))) - -(defn get-config - ([overrides] - ;; Then infer additional values - (assoc-inferred-values - (deep-merge defaults - ;(read-string (slurp "config.edn")) - (get-environ-config rules environ/env) - overrides))) - ([] (get-config {}))) - -(defrecord Config [overrides] - component/Lifecycle - (start [component] - (log/info ">> Starting config component") - (into component (get-config overrides))) - (stop [component] - (log/info "<< Stopping config component") - component)) - -(defn create-config - "Create a new instance of a Config component, with config-overrides." - ([config-overrides] - (Config. config-overrides)) - ([] (create-config {}))) - - -:ok - - -;; XXX This bit of refactoring is a WIP for switching to aero and mount (v component)... - -;(defmethod aero/reader `keyword -; [_ _ value] -; (keyword value)) -; -;(defn get-config -; ([profile overrides] -; (deep-merge (get-environ-conifg rules))) -; ([profile] (get-config profile {})) -; ([] (get-config :dev))) -; -;;; We use the overrides atom to allow the system start function to specify specific overrides -;(defonce profile (atom :dev)) -;(defonce overrides (atom nil)) -; -;(defstate config -; :start (do (log/info ">> Starting config component") -; (get-config profile overrides)) -; :stop (do (log/info "<< Stopping config component") -; (reset! overrides nil))) - diff --git a/math/src/polismath/components/core_matrix_boot.clj b/math/src/polismath/components/core_matrix_boot.clj deleted file mode 100644 index 6556f6bce5..0000000000 --- a/math/src/polismath/components/core_matrix_boot.clj +++ /dev/null @@ -1,69 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.core-matrix-boot - (:require [polismath.utils :as utils] - [clojure.core.matrix :as matrix] - [clojure.core.matrix.impl.ndarray :as ndarray] - [taoensso.timbre :as log] - [com.stuartsierra.component :as component] - [cheshire.generate :refer [add-encoder encode-seq remove-encoder]])) - - -;; Maybe this just shouldn't be a component, but an environment thing? Or does it work cause things won't be loaded on -;; static compile anyway? Ug... - -(defn matrix-encoder - [v jsonGenerator] - (encode-seq (into-array v) jsonGenerator)) - -;; Should maybe make this dynamically computed based on the implementation... Annoying that we have to do this at all; -;; Maybe there is something better we can hook on in our json encoding multimethod (the reason we -;; need any of this mess) -(def matrix-types ['mikera.vectorz.Vector - ;; XXX Need to figure out why this doesn't exist yet - ;'clojure.core.matrix.impl.ndarray.NDArray]) - ;; Could this be the new ns? - 'mikera.arrayz.NDArray - 'mikera.matrixx.Matrix - ;'clojure.core.matrix.impl.ndarray.NDArray - 'clojure.core.matrix.impl.ndarray_object.NDArray]) - - -(def dummy-matrix [[1 2 3] [4 5 6]]) - -(defrecord CoreMatrixBooter [config] - component/Lifecycle - ;; Load the matrix implementation and create a dummy matrix to ensure classes get loaded properly. - (start [component] - (let [implementation (get-in config [:math :matrix-implementation])] - (log/info ">> Starting CoreMatrixBooter with implementation:" implementation) - (matrix/matrix dummy-matrix) - (matrix/set-current-implementation implementation) - (matrix/matrix dummy-matrix) - ;(matrix/matrix :ndarray-object dummy-matrix) - (matrix/matrix :ndarray-double dummy-matrix) - (matrix/matrix :ndarray dummy-matrix) - ;; Adding encoders so things work properly with serialization - ;(require '[clojure.core.matrix.impl.ndarray]) - (doseq [t matrix-types] - (add-encoder (ns-resolve *ns* t) matrix-encoder))) - component) - - (stop [component] - (log/info "<< Stopping CoreMatrixBooter") - ;; Actually... let's just stop this. - ;(doseq [t matrix-types] - ; (remove-encoder (ns-resolve *ns* t))) - component)) - -(defn create-core-matrix-booter - ([] - (create-core-matrix-booter {})) - ([options] - (map->CoreMatrixBooter options))) - - -;(component/start (create-core-matrix-booter {:config {:math {:matrix-implementation :vectorz}}})) - -:ok - diff --git a/math/src/polismath/components/env.clj b/math/src/polismath/components/env.clj deleted file mode 100644 index d7c6fb2fbe..0000000000 --- a/math/src/polismath/components/env.clj +++ /dev/null @@ -1,14 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.env - (:require [environ.core :as environ])) - -;; Deprecating... Will remove by the end of refactor XXX - -(def ^:dynamic env environ/env) - -(defmacro with-env-overrides - [overrides & body] - `(binding [env (merge env ~overrides)] - ~@body)) - diff --git a/math/src/polismath/components/logger.clj b/math/src/polismath/components/logger.clj deleted file mode 100644 index f0517cfead..0000000000 --- a/math/src/polismath/components/logger.clj +++ /dev/null @@ -1,41 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.logger - (:require [com.stuartsierra.component :as component] - [taoensso.timbre :as log] - [taoensso.timbre.appenders.core :as appenders])) - - -;; XXX Hmmm.. this doesn't seem to be working yet. Maybe we need to use timbre for log calls instead of -;; clojure.tools.logging -(defrecord Logger [config] - component/Lifecycle - (start [component] - ;; Set minimum level first before any other logging happens - (when-let [level (get-in config [:logging :level])] - (log/set-level! level)) - ;; First merge the logging config from system config - (log/merge-config! - (:logging config)) - ;; Then merge appender config, inheriting the min-level from system config - (log/merge-config! - {:appenders {:println-appender - {:enabled? true - :async? false - :min-level (get-in config [:logging :level] :warn) ; Use system config level or default to :warn - :rate-limit [[1 250] [10 5000]] ; 1/250ms, 10/5s - :output-fn :inherit - :fn ; Appender's fn - (fn [data] - (let [{:keys [output-fn]} data - formatted-output-str (output-fn data)] - (println formatted-output-str)))} - :file-appender - {:spit (appenders/spit-appender {:fname (get-in config [:logging :file] "dev.log")})}}}) - component) - (stop [component] - (log/info "<< Stopping config component") - component)) - -(defn create-logger [] - (map->Logger {})) diff --git a/math/src/polismath/components/postgres.clj b/math/src/polismath/components/postgres.clj deleted file mode 100644 index a8b65f9668..0000000000 --- a/math/src/polismath/components/postgres.clj +++ /dev/null @@ -1,478 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.postgres - (:require [cheshire.core :as cheshire] - [taoensso.timbre :as log] - [com.stuartsierra.component :as component] - [clojure.java.jdbc :as jdbc] - [honeysql.core :as sql] - [honeysql.helpers :as honey]) - (:import (org.postgresql.util PGobject) - (com.zaxxer.hikari HikariConfig HikariDataSource))) - ;[alex-and-georges.debug-repl :as dbr] - - - -(defn create-hikari-datasource - "Create a HikariCP datasource for better connection pooling" - [db-uri pool-config] - (let [[_ user password host port db] (re-matches #"postgres://(?:(.+):(.*)@)?([^:]+)(?::(\d+))?/(.+)" db-uri) - pool-size (get pool-config :pool-size 10) - config (doto (HikariConfig.) - (.setJdbcUrl (str "jdbc:postgresql://" host ":" (or port 5432) "/" db)) - (.setUsername user) - (.setPassword password) - (.setDriverClassName "org.postgresql.Driver") - ;; Connection pool settings optimized for concurrent workloads - (.setMaximumPoolSize pool-size) - (.setMinimumIdle (max 1 (int (/ pool-size 4)))) ; 25% of max as minimum - (.setConnectionTimeout 30000) ; 30 seconds - (.setIdleTimeout 600000) ; 10 minutes - (.setMaxLifetime 1800000) ; 30 minutes - (.setLeakDetectionThreshold 60000) ; 1 minute - helps detect connection leaks - ;; Validation settings - (.setConnectionTestQuery "SELECT 1") - (.setValidationTimeout 5000) - ;; Performance optimizations - (.addDataSourceProperty "cachePrepStmts" "true") - (.addDataSourceProperty "prepStmtCacheSize" "250") - (.addDataSourceProperty "prepStmtCacheSqlLimit" "2048") - (.addDataSourceProperty "useServerPrepStmts" "true") - (.addDataSourceProperty "useLocalSessionState" "true") - (.addDataSourceProperty "rewriteBatchedStatements" "true") - (.addDataSourceProperty "cacheResultSetMetadata" "true") - (.addDataSourceProperty "cacheServerConfiguration" "true") - (.addDataSourceProperty "elideSetAutoCommits" "true") - (.addDataSourceProperty "maintainTimeStats" "false"))] - (HikariDataSource. config))) - -(defn heroku-db-spec - "Create a korma db-spec given a heroku db-uri with HikariCP connection pooling" - [db-uri _ignore-ssl pool-config] ; ignore-ssl parameter kept for compatibility but not used with HikariCP - (let [datasource (create-hikari-datasource db-uri pool-config)] - {:datasource datasource})) - - - -;; The executor function for honeysql queries (which we'll be rewriting everything in over time) - -(defn health-check - "Check if the database connection is healthy" - [component] - (try - (let [result (jdbc/query (:db-spec component) ["SELECT 1 as health"])] - (= 1 (:health (first result)))) - (catch Exception e - (log/error "Database health check failed:" (.getMessage e)) - false))) - -(defn query-with-retry - "Execute a query with retry logic for connection failures" - [component query-data & [retry-count]] - (let [max-retries (or retry-count 3)] - (loop [attempts 0] - (let [result (try - {:success (if (map? query-data) - (jdbc/query (:db-spec component) (sql/format query-data)) - (jdbc/query (:db-spec component) query-data))} - (catch java.sql.SQLException e - (if (and (< attempts max-retries) - (or (.contains (.getMessage e) "connection") - (.contains (.getMessage e) "timeout"))) - {:retry true :error e :attempts attempts} - {:error e})) - (catch Exception e - {:error e}))] - (cond - (:success result) (:success result) - (:retry result) (do - (log/warn "Database query failed, retrying... attempt" (inc (:attempts result)) ":" (.getMessage (:error result))) - (Thread/sleep (* 1000 (inc (:attempts result)))) ; Exponential backoff - (recur (inc (:attempts result)))) - :else (do - (log/error "Database query failed:" (.getMessage (:error result))) - (throw (:error result)))))))) - -(defn query - "Takes a postgres component and a query, and executes the query. The query can either be a postgres vector, or a map. - Maps will be compiled via honeysql/format." - [component query-data] - (query-with-retry component query-data)) - -(defrecord Postgres [config db-spec] - component/Lifecycle - (start [component] - (log/info ">> Starting Postgres component") - (let [database-url (-> config :database :url) - pool-config (-> config :database)] - (assert database-url "Missing database url. Make sure to set env variables.") - (log/info "Configuring PostgreSQL connection pool with size:" (get pool-config :pool-size 10)) - (assoc component :db-spec (heroku-db-spec database-url - (-> config :database :ignore-ssl) - pool-config)))) - (stop [component] - (log/info "<< Stopping Postgres component") - ;; Properly close the HikariCP connection pool - (when-let [db-spec (:db-spec component)] - (try - (when-let [datasource (:datasource db-spec)] - (log/info "Closing HikariCP connection pool") - (when (instance? HikariDataSource datasource) - (.close ^HikariDataSource datasource))) - (catch Exception e - (log/warn "Error closing connection pool:" (.getMessage e))))) - (assoc component :db-spec nil))) - -(defn create-postgres - "Creates a new Postgres component" - [] - (map->Postgres {})) - - -(defn poll - "Query for all data since last-vote-timestamp, given a db-spec" - [component last-vote-timestamp] - (log/info "poll" last-vote-timestamp) - (try - (query component - {:select [:*] - :from [:votes] - :order-by [:zid :tid :pid :created] - :where [:> :created last-vote-timestamp]}) - (catch Exception e - (log/error "polling failed " (.getMessage e)) - (.printStackTrace e) - []))) - - -(defn mod-poll - "Moderation query: basically look for when things were last modified, since this is the only time they will - have been moderated." - [component last-mod-timestamp] - (log/info "modpoll" last-mod-timestamp) - (try - (query component - {:select [:*] - :from [:comments] - :order-by [:zid :tid :modified] - :where [:> :modified last-mod-timestamp]}) - (catch Exception e - (log/error "moderation polling failed " (.getMessage e)) - []))) - - -(defn get-zid-from-zinvite - [component zinvite] - (log/debug "get-zid-from-zinvite for zinvite" zinvite) - (-> - (query component - {:select [:zid :zinvite] - :from [:zinvites] - :where [:= :zinvite zinvite]}) - first - :zid)) - -(defn get-meta-tids - [component zid] - (->> - (query component - {:select [:tid] - :from [:comments] - :where [:and [:= :zid zid] - :is_meta]}) - (map :tid) - (into #{}))) - -(defn get-zinvite-from-zid - [component zid] - (log/debug "get-zinvite-from-zid for zid" zid) - (-> - (query component - {:select [:zid :zinvite] - :from [:zinvites] - :where [:= :zid zid]}) - first - :zinvite)) - -(defn conv-poll - "Query for all vote data since last-vote-timestamp for a given zid, given an implicit db-spec" - [component zid last-vote-timestamp] - (log/info "conv-poll for zid" zid ", last-vote-timestap" last-vote-timestamp) - (try - (query component - {:select [:*] - :from [:votes] - :order-by [:zid :tid :pid :created] - :where [:and - [:> :created last-vote-timestamp] - [:= :zid zid]]}) - (catch Exception e - (log/error "polling failed for conv zid =" zid ":" (.getMessage e)) - (.printStackTrace e) - []))) - -(defn conv-mod-poll - "Query for all mod data since last-vote-timestamp for a given zid, given an implicit db-spec" - [component zid last-mod-timestamp] - (log/info "conv-mod-poll for zid" zid ", last-vote-timestap" last-mod-timestamp) - (query - component - {:select [:*] - :from [:comments] - :order-by [:tid :modified] - :where [:and - [:> :modified last-mod-timestamp] - [:= :zid zid]]})) - - -(defn format-as-json-for-db - "Formats data for pg json, first passing through a prep function which may strip out uneeded junk or - reshape things. Takes conv and lastVoteTimestamp, though the latter may be moved into the former in update" - [conv] - (-> conv - ; core.matrix & monger workaround: convert to str with cheshire then back - cheshire/generate-string - cheshire/parse-string)) - -; (defn collection-name -; "math_env name based on math-env and math-schema-date config variables. Makes sure that -; prod, preprod, dev (and subdevs like chrisdev or mikedev) have their own noninterfering collections." -; ([mongo rootname] -; (let [{:keys [math-schema-date math-env]} (:config mongo) -; math-env (or math-env :dev)] -; (str rootname "_" (name math-env) "_" math-schema-date))) -; ([mongo rootname basename] (str (collection-name mongo rootname) "_" basename))) - - - -(defn poll-tasks - [component last-timestamp] - (->> - (query - component - (sql/format - {:select [:*] - :from [:worker_tasks] - :where [:and - [:> :created last-timestamp] - [:= :math_env (-> component :config :math-env-string)] - [:= :finished_time nil]]})) - (map (fn [task-record] - (-> task-record - (update :task_type keyword) - (update :task_data (comp #(cheshire/parse-string % true) #(.toString %)))))))) - -(defn zid-from-rid - [rid] - {:select [:zid] - :from [:reports] - :where [:= :rid rid]}) - -(defn report-tids - [rid] - {:select [:tid] - :from [:report_comment_selections] - :where [:and - [:= :rid rid] - [:> :selection 0]]}) - -(defn ptpt-counts [postgres] - (query - postgres - {:select [:*] - :from [[{:select [:zid [:%count-distinct.pid :ptpt_cnt]] - :from [:votes] - :group-by [:zid]} - :counts]] - :where [:> :counts.ptpt_cnt 5]})) - -(defn query-zid-from-rid [component rid] - (query component (zid-from-rid rid))) - -(defn inc-math-tick - [postgres zid] - (log/info "inc-math-tick" zid) - (:math_tick (first (query postgres ["insert into math_ticks (zid, math_env) values (?, ?) on conflict (zid, math_env) do update set modified = now_as_millis(), math_tick = (math_ticks.math_tick + 1) returning math_tick;" zid (-> postgres :config :math-env-string)])))) - -(defn pg-json - [data] - (doto (PGobject.) - (.setType "json") - (.setValue (cheshire/encode data)))) - -(defn insert-correlationmatrix! - [postgres rid math-tick data] - (query postgres ["insert into math_report_correlationmatrix (rid, math_env, math_tick, data) values (?,?,?,?) on conflict (rid, math_env) do update set data = excluded.data, math_tick = excluded.math_tick returning rid;" rid (-> postgres :config :math-env-string) math-tick (pg-json data)])) - - -;; TODO Fix this; need task-type in here as well for this to work -;(defn mark-task-complete! -; [postgres task-type task-bucket] -; (log/info "mark-task-complete called for task-type, task-bucket:" task-type task-bucket) -; (jdbc/update! -; (:db-spec postgres) -; :worker_tasks -; {:finished_time (System/currentTimeMillis)} -; ["task_type = ? and task_bucket = ?" task-type task-bucket])) -;; Marks all tasks with the same task_bucket as done. -(defn mark-task-complete! - [postgres task_type task_bucket] - (log/info "mark-task-complete" task_bucket) - (query postgres ["update worker_tasks set finished_time = now_as_millis() where math_env = (?) and task_type = (?) and task_bucket = (?) returning finished_time;" (-> postgres :config :math-env-string) task_type task_bucket])) - -(defn upload-math-main - [postgres zid math-tick data] - (log/info "upload-math-main for zid" zid) - (let [math-env (-> postgres :config :math-env-string)] - (query postgres - ["insert into math_main (zid, math_env, last_vote_timestamp, math_tick, data, caching_tick) - values (?,?,?,?,?, COALESCE((select max(caching_tick) + 1 from math_main where math_env = (?)), 1)) - on conflict (zid, math_env) - do update set modified = now_as_millis(), - data = excluded.data, - last_vote_timestamp = excluded.last_vote_timestamp, - math_tick = excluded.math_tick, - caching_tick = excluded.caching_tick - returning zid;" - ;; I believe math env is twice here because it gets used in two separate ? - zid math-env (:lastVoteTimestamp data) math-tick (pg-json data) math-env]))) - -(defn upload-math-profile - [postgres zid data] - (log/info "upload-math-profile for zid" zid) - (query postgres - ["insert into math_profile (zid, math_env, data) - values (?,?,?) on conflict (zid, math_env) - do update set modified = now_as_millis(), data = excluded.data - returning zid;" - zid (-> postgres :config :math-env-string) (pg-json data)])) - -(defn upload-math-ptptstats - [postgres zid math-tick data] - (log/info "upload-math-ptptstats for zid" zid) - (query postgres - ["insert into math_ptptstats (zid, math_env, math_tick, data) - values (?,?,?,?) - on conflict (zid, math_env) - do update set modified = now_as_millis(), - data = excluded.data, - math_tick = excluded.math_tick - returning zid;" - zid (-> postgres :config :math-env-string) math-tick (pg-json data)])) - -;; XXX Not using this anywhere apparently so should remove -;(defn upload-math-cache -; [postgres zid data] -; (log/info "upload-math-cache for zid" zid) -; (query postgres ["insert into math_cache (zid, math_env, data) values (?,?,?) on conflict (zid, math_env) do update set modified = now_as_millis(), data = excluded.data returning zid;" zid (name (-> postgres :config :math-env)) (pg-json data)])) - -(defn upload-math-bidtopid - [postgres zid math-tick data] - (log/info "upload-math-bidtopid for zid" zid) - (query postgres - ["insert into math_bidtopid (zid, math_env, math_tick, data) - values (?,?,?,?) - on conflict (zid, math_env) - do update set modified = now_as_millis(), - data = excluded.data, - math_tick = excluded.math_tick - returning zid;" - zid (-> postgres :config :math-env-string) math-tick (pg-json data)])) - -(defn upload-math-exportstatus - [postgres zid filename data] - {:pre [postgres zid filename data]} - (log/info "upload-math-exportstatus for zid" zid) - (query - postgres - ["insert into math_exportstatus (zid, math_env, filename, data, modified) - values (?,?,?,?, now_as_millis()) - on conflict (zid, math_env) - do update set modified = now_as_millis(), - data = excluded.data, - filename = excluded.filename - returning zid;" - zid - (-> postgres :config :math-env-string) - filename - (pg-json data)])) - - -(defn decode-pg-json - [data] - (-> data .getValue cheshire/decode)) - -(defn get-math-exportstatus - [postgres zid filename] - (log/info "get-math-exportstatus for zid" zid) - (->> - (query postgres ["select * from math_exportstatus where zid = (?) and math_env = (?) and filename = (?);" zid (-> postgres :config :math-env-string) filename]) - first - :data - decode-pg-json)) - -(defn get-math-tick - [postgres zid] - (:math_tick (first (query postgres ["select math_tick from math_ticks where zid = (?) and math_env = (?);" zid (-> postgres :config :math-env-string)])))) - - -(defn load-conv - "Very bare bones reloading of the conversation; no cleanup for keyword/int hash-map key mismatches, - as found in the :repness" - [postgres zid] - (log/info "load-conv called for zid" zid) - (let [row (first (query postgres ["select * from math_main where zid = (?) and math_env = (?);" zid (-> postgres :config :math-env-string)]))] - (if row - ;; TODO Make sure this loads with keywords for map keys, except where they should be integers - (cheshire/parse-string - (.toString (:data row)) - (fn [x] - (try - (Long/parseLong x) - (catch Exception _ - (keyword x))))) - row))) - - ; (mc/find-one-as-map - ; (:db mongo) - ; (math-collection-name mongo "main") - ; {:zid zid})) - - -(comment - (require '[polismath.runner :as runner]) - (def postgres (:postgres runner/system)) - (def config (:config postgres)) - (query postgres ["select * from zinvites limit 10"]) - - (conv-poll postgres 18747 0) - (get-zinvite-from-zid postgres 18747) - (conv-mod-poll postgres 18747 0) - - - (get-math-exportstatus postgres 15077 "polis-export-9ma5xnjxpj-1491632824548.zip") - ;(query postgres ["insert into math_ticks (zid) values (?) on conflict (zid) do update set modified = now_as_millis(), math_tick = (math_ticks.math_tick + 1) returning *;" 12480]) - (poll-tasks postgres 0) - (query - postgres - (-> (honey/update :worker_tasks) - (honey/values [{}]))) - - (jdbc/execute! - (:db-spec postgres) - (-> (honey/update :worker_tasks) - (honey/value))) - - (try - (mark-task-complete! postgres :task-type 1) ; Fixed: added missing task-type parameter - (catch Exception e (log/error (.getNextException e)))) - - - (query - postgres - (report-tids 1)) - :endcomment) - -:ok - - diff --git a/math/src/polismath/components/random.clj b/math/src/polismath/components/random.clj deleted file mode 100644 index b781fbccf7..0000000000 --- a/math/src/polismath/components/random.clj +++ /dev/null @@ -1,7 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.random - "This will eventually house a random number generator component which assists in the tracking of seed state") - - - diff --git a/math/src/polismath/components/server.clj b/math/src/polismath/components/server.clj deleted file mode 100644 index 7ffa57793f..0000000000 --- a/math/src/polismath/components/server.clj +++ /dev/null @@ -1,72 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.components.server - (:require - ;; XXX Deprecate; use component config directly - [clojure.core.async :as async :refer [chan >!! ! > Starting server component with config:" (:server config)) - (let [wrapped-handler (wrap-handler (:handler app)) - jetty-server (jetty/run-jetty wrapped-handler - (merge {:join? false :port 8080} - (:server config) - opts))] - (.start jetty-server) - (assoc component :jetty-server jetty-server))) - (stop [component] - (log/info "<< Stopping server component") - (try - (.stop jetty-server) - (catch Exception e - (log/error e "Unable to stop server!"))) - component)) - - -(defn create-server - ;; Default handler just return the healthcheck response always - ([opts] - (let [opts (merge {:app {:handler healthcheck-handler}} - opts)] - (map->Server opts))) - ([] - (create-server {}))) diff --git a/math/src/polismath/conv_man.clj b/math/src/polismath/conv_man.clj deleted file mode 100644 index 8183a7ed3c..0000000000 --- a/math/src/polismath/conv_man.clj +++ /dev/null @@ -1,473 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.conv-man - "This is the namespace for the conversation manager" - (:require [polismath.math.named-matrix :as nm] - [polismath.math.conversation :as conv] - [polismath.math.clusters :as clust] - [polismath.meta.metrics :as met] - [polismath.meta.notify :as notify] - [polismath.components.env :as env] - [polismath.components.postgres :as db] - [polismath.math.corr :as corr] - [polismath.utils :as utils] - [clojure.core.matrix.impl.ndarray] - [clojure.core.async :as async :refer [go go-loop ! !! alts!! alts! chan dropping-buffer put! take!]] - [taoensso.timbre :as log] - [com.stuartsierra.component :as component] - [plumbing.core :as pc] - [schema.core :as s] - [polismath.components.postgres :as postgres])) - - - -;; Here, we organize things in terms of a conversation manager, which maintains a cache of conversations and routes -;; messages and updates to these conversations. -;; The conversations themselves are organized around conversation actors, which maintain the actual state of the -;; conversation, as well as the message handling machinery (chans and such). -;; Updates to the conversation actors are broken down via a react-to-messages multimethod, which allows for message -;; handling extensibility. - - -;; Utility update functions -;; ------------------------ - -(defn prep-bidToPid - "Prep function for passing to db/format-as-json-for-db given bidToPid data" - [results] - {:zid (:zid results) - :bidToPid (:bid-to-pid results) - :lastVoteTimestamp (:last-vote-timestamp results)}) - - -(defn prep-main - "Prep function for passing to db/format-as-json-for-db and on the main polismath collection." - [results] - (-> results - ; REFORMAT BASE CLUSTERS - (update-in [:base-clusters] clust/fold-clusters) - ; IMPORTANT! Whitelist of keys to be included in json output; used for removing intermediates - (assoc :lastVoteTimestamp (:last-vote-timestamp results)) - (assoc :lastModTimestamp (:last-mod-timestamp results)) - (utils/hash-map-subset #{:base-clusters - :group-clusters - :subgroup-clusters - :in-conv - :mod-out - :mod-in - :meta-tids - :lastVoteTimestamp - :lastModTimestamp - :n - :n-cmts - :pca - :repness - :group-aware-consensus - :consensus - :zid - :tids - :user-vote-counts - :votes-base - :group-votes - :subgroup-votes - :subgroup-repness - :comment-priorities}))) - ;:subgroup-ptpt-stats}))) - - -(defn columnize - ([stats keys] - (->> keys - (map - (fn [k] - (let [vals (map (fn [stat] (get stat k)) stats)] - [k vals]))) - (into {}))) - ([stats] - (let [keys (-> stats first keys)] - (columnize stats keys)))) - -(defn prep-ptpt-stats - [results] - {:zid (:zid results) - :ptptstats (columnize (:ptpt-stats results)) - :lastVoteTimestamp (:last-vote-timestamp results)}) - - -(defn handle-profile-data - "For now, just log profile data. Eventually want to send to influxDB and graphite." - [{:as conv-man :keys [postgres config]} conv & {:keys [recompute n-votes finish-time] :as extra-data}] - (if-let [prof-atom (:profile-data conv)] - (let [prof @prof-atom - tot (apply + (map second prof)) - prof (assoc prof :total tot)] - (try - (-> prof - (assoc :n-ptps (:n conv)) - (merge (utils/hash-map-subset conv #{:n-cmts :zid :last-vote-timestamp}) - extra-data) - (->> (db/upload-math-profile postgres (:zid conv)))) - (catch Exception e - (log/warn "Unable to submit profile data for zid:" (:zid conv)) - (.printStackTrace e))) - (log/debug "Profile data for zid" (:zid conv) ": " prof)))) - - - -;; Conversation update functions -;; ============================= - -;; XXX This is temporary; should be switching to new spec work in conversation ns -(def Conversation - "A schema for what valid conversations should look like (WIP)" - {:zid s/Int - ;; Note: we let all other key-value pairs pass through - s/Keyword s/Any}) - -;; Should rename this not to conflict with clojure.core/update... poor form -(defn conv-update - "This function is what actually gets sent to the conv-manager. In addition to the conversation and vote batches - up in the channel, we also take an error-callback. Eventually we'll want to pass opts through here as well." - [conv-man conv votes] - (let [start-time (System/currentTimeMillis) - config (:config conv-man) - pg (:postgres conv-man)] - (log/info "Starting conversation update for zid:" (:zid conv)) - ;; Need to expose opts for conv-update through config... XXX - (let [updated-conv (conv/conv-update conv votes) - zid (:zid updated-conv) - finish-time (System/currentTimeMillis) - ; If this is a recompute, we'll have either :full or :reboot, ow/ want to send false - recompute (or (:recompute conv) false)] - (log/info "Finished computng conv-update for zid" zid "in" (- finish-time start-time) "ms") - (handle-profile-data conv-man - updated-conv - :finish-time finish-time - :recompute recompute - :n-votes (count votes)) - ;; Make sure our data has the right shape - (when-let [validation-errors (s/check Conversation updated-conv)] - ;; XXX Should really be using throw+ (slingshot) here and throutout the code base - ;; Also, should put in code for doing smart collapsing of collections... - (log/error "Validation error: Conversation value does not match schema for conv:" updated-conv) - (throw (Exception. (str "Validation error: Conversation Value does not match schema: " - validation-errors)))) - ; Return the updated conv - updated-conv))) - -(defn write-conv-updates! - [{:as conv-man :keys [postgres]} {:as updated-conv :keys [zid]} math-tick] - ;; TODO Really need to extract these writes so that mod updates do whta they're supposed to! And also run in async/thread for better parallelism - ; Format and upload main results - (async/thread - (doseq [[prep-fn upload-fn] [[prep-main db/upload-math-main] ; main math results, for client - [prep-bidToPid db/upload-math-bidtopid] ; bidtopid mapping, for server - [prep-ptpt-stats db/upload-math-ptptstats]]] - (->> updated-conv - prep-fn - (upload-fn postgres zid math-tick))) - (log/info "Finished uploading math results for zid:" zid))) - -(defn restructure-json-conv - [conv] - (-> conv - (utils/hash-map-subset #{:math_tick :raw-rating-mat :rating-mat :lastVoteTimestamp :mod-out :mod-in :zid :pca :in-conv :n :n-cmts :group-clusters :base-clusters :repness :group-votes :subgroup-clusters :subgroup-votes :subgroup-repness :group-aware-consensus :comment-priorities :meta-tids}) - (assoc :last-vote-timestamp (get conv :lastVoteTimestamp) - :last-mod-timestamp (get conv :lastModTimestamp)) - ; Make sure there is an empty named matrix to operate on - (assoc :raw-rating-mat (nm/named-matrix)) - ; Update the base clusters to be unfolded - (update :base-clusters clust/unfold-clusters) - ; Make sure in-conv is a set - (update :in-conv set) - (update :mod-out set) - (update :mod-in set) - (update :meta-tids set))) - - -(defn load-or-init - "Given a zid, either load a minimal set of information from postgres, or if a new zid, create a new conv" - [conv-man zid & {:keys [recompute]}] - ;; TODO On recompute should try to preserve in conv and such - (log/info "Running load or init") - (if-let [conv (and (not recompute) (db/load-conv (:postgres conv-man) zid))] - (-> conv - ;(->> (tr/trace "load-or-init (about to restructure):")) - restructure-json-conv - ;(->> (tr/trace "load-or-init (post restructure):")) - ;; What the fuck is this all about? Should this really be getting set here? - (assoc :recompute :reboot) - (assoc :raw-rating-mat - (-> (nm/named-matrix) - (nm/update-nmat (->> (db/conv-poll (:postgres conv-man) zid 0) - (map (fn [vote-row] (mapv (partial get vote-row) [:pid :tid :vote]))))))) - (conv/mod-update - (db/conv-mod-poll (:postgres conv-man) zid 0))) - ; would be nice to have :recompute :initial - (assoc (conv/new-conv) :zid zid :recompute :full))) - -(defn generate-report-data! - [{:as conv-man :keys [postgres]} conv math-tick report-data] - (log/error "Report generation requested; No longer supported:" report-data)) - ;(log/info "Generating report data for report:" report-data)) - ;(let [rid (:rid report-data) - ;tids (map :tid (postgres/query (:postgres conv-man) (postgres/report-tids rid))) - ;corr-mat (corr/compute-corr conv tids)] - ;(async/thread - ;(postgres/insert-correlationmatrix! postgres rid math-tick corr-mat) - ;;; TODO update to submit usng task type and task bucket - ;(postgres/mark-task-complete! postgres "generate_report_data" rid)))) - - - -;; Message processing functions -;; ---------------------------- - - -(defmacro take-all! [c] - "Given a channel, takes all values currently in channel and places in a vector. Must be called - within a go block." - `(loop [acc# []] - (let [[v# ~c] (alts! [~c] :default nil)] - (if (not= ~c :default) - (recur (conj acc# v#)) - acc#)))) - - -(defn take-all!! [c] - "Given a channel, takes all values currently in channel and places in a vector. Must be called - within a go block." - (loop [acc []] - (let [[v c] (alts!! [c] :default nil)] - (if (not= c :default) - (recur (conj acc v)) - acc)))) - - -(defn split-batches - "This function splits message batches as sent to conv actor up by the first item in batch vector (:votes :moderation) - so messages can get processed properly" - [messages] - (->> messages - (group-by :message-type) - (pc/map-vals - (fn [labeled-batches] - (->> labeled-batches - (map :message-batch) - (flatten)))))) - - - -;; Message processing for conv-actor -;; ================================= - -;; Here's the multimethod at the core of the messages the conv actor may act upon. - -(defmulti react-to-messages - (fn [conv-man conv message-type messages] - message-type)) - -(defmethod react-to-messages :votes - [conv-man conv _ messages] - (conv-update conv-man conv messages)) - -(defmethod react-to-messages :moderation - [conv-man conv _ messages] - (conv/mod-update conv messages)) - -(defmethod react-to-messages :generate_report_data - [conv-man conv _ messages] - (let [math-tick (or (:math-tick conv) (:math_tick conv))] - (doseq [report-task messages] - (try - (generate-report-data! conv-man conv math-tick report-task) - (catch Exception e (log/error e (str "Unable to generate report " (pr-str report-task))))))) - ;; explicitly return nil so we don't trigger an update - nil) - - -;; Error handling - -(defn handle-errors - [conv-man conv-actor conv message-type messages update-error start-time] - (let [zid (:zid conv-actor) - zid-str (str "zid=" zid) - retry-chan (:retry-chan conv-actor) - notify-message (str "Failed conversation update on " (-> conv-man :config :math-env) " for message-type " message-type " and " zid-str)] - (try - (let [stack-trace (notify/error-message-body update-error)] - (notify/notify-team (:config conv-man) (str "Polismath conv-man error: " message-type) zid notify-message stack-trace)) - (catch Exception e - (log/error e "Unable to notify team"))) - (log/error update-error notify-message) - (.printStackTrace update-error) - ; Try requeing the votes that failed so that if we get more, they'll get replayed - (try - (log/info "Re-queueing messages for failed update for" zid-str) - (async/go (async/>! retry-chan {:message-type message-type :message-batch messages})) - (catch Exception qe - (log/error qe (str "MAJOR ERROR! Unable to re-queue votes after conversation update failed for " zid-str)) - (.printStackTrace qe))) - ; Try to send some failed conversation time metrics, but don't stress if it fails - (try - (let [end (System/currentTimeMillis) - duration (- end start-time)] - ;; Update to use MetricSender component XXX - (met/send-metric (:metrics conv-man) "math.pca.compute.fail" duration)) - (catch Exception e - (log/error "Unable to send metrics for failed compute for" zid-str))) - ; Try to save conversation state for debugging purposes - (try - (conv/conv-update-dump conv messages update-error) - (catch Exception e - (log/error "Unable to perform conv-update dump for" zid-str))))) - - -;; Wrapper around the multimethod above, which handles errors and such - -(defn react-to-messages! - [conv-man conv-actor message-type messages] - (let [start-time (System/currentTimeMillis) - {:keys [zid conv retry-chan]} conv-actor - update-fn - (fn [conv'] - (try - (if-let [updated-conv (react-to-messages conv-man conv' message-type messages)] - (do - (let [math-tick (or (:math-tick updated-conv) ;; pass through for report generation - (postgres/inc-math-tick (:postgres conv-man) zid))] - (write-conv-updates! conv-man updated-conv math-tick)) - updated-conv) - ;; if nil, don't update, for just side effects - conv') - (catch Exception e - (handle-errors conv-man conv-actor conv' message-type messages e start-time) - conv')))] - (swap! conv update-fn))) - - -;; Start the actor - -(defn go-act! - [conv-man conv-actor] - (let [{:keys [kill-chan conversations]} conv-man - {:keys [zid conv message-chan retry-chan]} conv-actor] - (go-loop [] - ;; If nil comes through as the first message, then the chan is closed, and we should be done, not continue looping forever - (let [[first-msg c] (async/alts! [kill-chan message-chan] :priority true)] - (when-not (= c kill-chan) - (log/debug "Message chan put in queue-message-batch! for zid:" (:zid first-msg)) - ;; If there are any retry messages from a failed recompute, we put them at the front of the message stack. - ;; But retry messages only get processed in this way if there are new messages that have come in triggering the - ;; first-msg take above, ensuring we don't just loop forever on a broken message/update. - (let [retry-msgs (async/poll! retry-chan) - msgs (vec (concat retry-msgs [first-msg] (take-all! message-chan))) - ;; Regardless, now we split the messages by message type and process them as below - split-msgs (split-batches msgs)] - ;; This acts as a whitelist for messages to run, and also an ordering of preference - (doseq [message-type [:votes :moderation :generate_report_data]] - (when-let [messages (get split-msgs message-type)] - (react-to-messages! conv-man conv-actor message-type messages))) - (recur))))))) - -;; Put the actor together and - -(defn conv-actor - [{:as conv-man :keys [conversations kill-chan config]} zid] - (log/info "Starting message batch queue and handler routine for conv zid:" zid) - (let [conv (load-or-init conv-man zid :recompute (:recompute config)) - _ (log/info "Conversation loaded for conv zid:" zid) - ;; Set up our main message chan - message-chan (chan 10000) - ;; Separate channel for messages that we've tried to process but that haven't worked for one reason or another (buffer size not important here) - retry-chan (chan 10) - actor {:zid zid :conv (atom conv) :message-chan message-chan :retry-chan retry-chan :conv-man conv-man}] - (go-act! conv-man actor) - ;; Trigger a conv update as the conv loads, so that state is always consistent - (react-to-messages! conv-man actor :votes []) - actor)) - - -(defn add-conv-actor-listener! - [conv-actor f] - (add-watch (:conv conv-actor) - ::conv-actor-state-watch - (fn [_ _ old-value new-value] - (when-not (= old-value new-value) - (f new-value))))) - -;; At the moment this is mostly used for testing, and I'm not sure that we'll want to keep it long term. But let's see... -(defn add-listener! - "Adds a watch to the conv-actor for the given zid such that the function f will be called when the given conversation - changes or initializes." - [conv-man zid f] - ;; If the conv-actor has already been created, then we add a watch for its :conv atom - (when-let [conv-actor (-> conv-man :conversations deref (get zid))] - (add-conv-actor-listener! conv-actor f)) - (add-watch (:conversations conv-man) - [::conversations-watch zid] - (fn [_ _ old-value new-value] - ;; If a conv-actor is added for this zid, set the watch now - (when-let [new-conv-actor (and (not (get old-value zid)) - (get new-value zid))] - (add-conv-actor-listener! new-conv-actor f))))) - - - -;; Conversation manager system component -;; ===================================== - -;; Puts together the conversation actors into a system component. - -(defrecord ConversationManager [config postgres metrics conversations kill-chan] - component/Lifecycle - (start [component] - (log/info ">> Starting ConversationManager") - (let [conversations (atom {}) - kill-chan (async/promise-chan)] - (assoc component :conversations conversations :kill-chan kill-chan))) - (stop [component] - (log/info "<< Stopping ConversationManager") - (try - ;; Close all our message channels for good measure - (log/debug "conversations:" conversations) - (go (>! kill-chan :kill)) - (doseq [[zid {:keys [message-chan]}] @conversations] - (async/close! message-chan)) - ;; Not sure, but we might want this for GC - (reset! conversations nil) - (catch Exception e - (log/error e "Unable to stop ConvMan component"))) - component)) - -(defn create-conversation-manager - [] - (map->ConversationManager {})) - - -;; How we queue messages up to the conversation manager - -;; Need to think about what to do if failed conversations lead to messages piling up in the message queue XXX -(defn queue-message-batch! - "Queue message batches for a given conversation by zid" - [{:as conv-man :keys [conversations config kill-chan]} message-type zid message-batch] - (when-not (async/poll! kill-chan) - (if-let [{:keys [conv message-chan]} (get @conversations zid)] - ;; Then we already have a go loop running for this - (>!! message-chan {:message-type message-type :message-batch message-batch}) - ;; Then we need to initialize the conversation and set up the conversation channel and go routine - (let [conv-actor (conv-actor conv-man zid)] - (swap! conversations assoc zid conv-actor) - ;; Just call again to make sure the message gets on the chan (using the if-let fork above) :-) - (queue-message-batch! conv-man message-type zid message-batch))))) - - -;; Need to find a good way of making sure these tests don't ever get committed uncommented -(comment - (require '[clojure.test :as test]) - (test/run-tests 'conv-man-tests) - :end-comment) - - -:ok - diff --git a/math/src/polismath/darwin/core.clj b/math/src/polismath/darwin/core.clj deleted file mode 100644 index 368f0c1a89..0000000000 --- a/math/src/polismath/darwin/core.clj +++ /dev/null @@ -1,242 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.darwin.core - "The darwin.core namespace wraps the more pure (+ writing files/zips) code in darwin.export. Deals with parsing the - params, updating the db status, uploading to aws, etc. This gets hooked up in the tasks namespace." - (:require - [polismath.darwin.export :as export] - ;; XXX Deprecate; use component config directly - ;[polismath.components.env :as env] - [polismath.components.postgres :as db] - [clojure.core.async :as async :refer [chan >!! ! darwin :config :aws)) - -(defn full-aws-path - [darwin filename] - (str (-> darwin :config :math-env-string) "/" filename)) - -(defn upload-to-aws - [{:as darwin :keys [s3-client]} filename] - (aws/invoke s3-client - {:op :PutObject - :request {:Bucket "polis-datadump" - :Body (io/input-stream (io/file filename)) - :Key (full-aws-path darwin filename)}})) - -(defn send-email-notification-via-polis-api! - [darwin {:as params :keys [zinvite email filename]}] - (try - (let [darwin-config (-> darwin :config) - response - ;; The next three lines should probably be extracted - (client/post (str (:webserver-url darwin-config) "/sendEmailExportReady") - {:form-params {:webserver_username (:webserver-username darwin-config) - :webserver_pass (:webserver-pass darwin-config) - :email email - :filename filename - :raise false - :conversation_id zinvite} - :content-type :json - :throw-entire-message? true})] - (log/info "send email notification response:\n" (with-out-str (clojure.pprint/pprint response)))) - (catch Exception e - (log/error e "failed to send email:\n")))) - - -(defn handle-completion! - [{:as darwin :keys [postgres]} {:as params :keys [filename task_bucket]}] - (log/info "Completed export computation for filename" filename "params:" (with-out-str (str params))) - (upload-to-aws darwin filename) - ;;(upload-to-polis-polismath.darwin.core darwin filename) - (notify-of-status darwin params "complete") - (postgres/mark-task-complete! postgres "generate_export_data" task_bucket) - (when (:email params) (send-email-notification-via-polis-api! darwin params))) - - - -;; The filename is actually pretty iportant. -;; It should be unique between different exports, as is used as the identifying key for the aws buckets, postgres -;; tracking and as part of the API requests. - -(defn generate-filename - "Generates a filename based on request-params" - [{:as request-params :keys [zinvite at-time format]}] - {:pre [zinvite format]} - (let [last-updated (or at-time (System/currentTimeMillis)) - ext (case format :excel "xlsx" :csv "zip") - filename (str "polis-export-" zinvite "-" last-updated "." ext)] - filename)) - - - -;; The following is really just a bunch of parameter parsing stuff. -;; Tihs could all possibly be interwoven with the config component as well. - -(defn- ->long - "Try to parse as an integer as long; return nil if not possible." - [x] - (if (number? x) - (long x) - (try (Long/parseLong x) - (catch Exception e nil)))) - -(defn- between? [a b x] - (and x (< a x) (> b x))) - -(defn- parse-and-validate-timeout - [x] - (let [x (try (Long/parseLong x) - (catch Exception e (throw (Exception. "Invalid timeout value"))))] - (assert (and x (< 0 x) (>= 29000 x)) "Invalid timout value") - x)) - -(def parsers {:at-time ->long - :format keyword - :timeout parse-and-validate-timeout}) - -;; And finally, our param parser. - -(defn params-with-zid - [darwin params] - (assoc params - :zid (or (:zid params) - (db/get-zid-from-zinvite (:postgres darwin) (:zinvite params))))) - -(defn params-with-filename - [params] - (assoc params - :filename - (or (:filename params) - (generate-filename params)))) - -(defn params-with-zinvite - [darwin params] - (assoc params - :zinvite (or (:zinvite params) - (db/get-zinvite-from-zid (:postgres darwin) (:zid params))))) - - -(defn parsed-params - "Parses the params for a request, occording to parsers." - [darwin params] - ;(log/info "Here are the params:" params) - (->> - params - (reduce - (fn [m [k v]] - ;; Don't really need this if we have params instead of query params, but whateves - (let [k (keyword k)] - (assoc m k ((or (parsers k) identity) v)))) - {}) - (params-with-zid darwin) - (params-with-zinvite darwin) - (params-with-filename))) - - - - - -(comment - ;; Here we're putting everything together - ;; This may not all be 100% correct, as it was copied over from the repl... but I ran through all but the spit and sanity checks pass - (require '[polismath.runner :as runner] - '[polismath.system :as system]) - ;(runner/run! system/base-system {:math-env :preprod}) - ;; Load the data for 15117 (zinvite 2ez5beswtc) - ;(def focus-id 15117) - (def focus-zinvite "36jajfnhhn") - (def focus-id 15228) - (def conv (conv-man/load-or-init (:conversation-manager runner/system) focus-id)) - ;conv - (matrix/shape (nm/get-matrix (:rating-mat conv))) - ;; Compute hclust on a cleaned, transposed rating matrix - (time - (def hclusters (hclust (cleaned-rating-mat-transpose conv)))) - (count (-> hclusters first :members)) - ;; Compute corr matrix - (time - (def corr-mat (correlation-matrix conv))) - (keys corr-mat) - (matrix/shape (:matrix corr-mat)) - (def corr-mat' (blockify-corr-matrix corr-mat hclusters)) - (keys corr-mat') - (count (:comments corr-mat')) - ;; Spit this out - (spit-matrix (str focus-zinvite ".corrmat.json") corr-mat) - (spit-matrix (str focus-zinvite ".corrmat-whclust.json") corr-mat') - (spit-hclust (str focus-zinvite ".hclust.json") hclusters) - ;; All done - :endcomment) - -;; Route everything together, build handlers, etc - - -;; NOOP right now for compatibility -(defrecord Darwin [config postgres conversation-manager s3-client] - component/Lifecycle - (start [component] - (if-let [aws-config (get config :aws)] - (let [creds (aws-creds/basic-credentials-provider aws-config)] - (assoc component - :s3-client (aws/client {:api :s3 - :credentials-provider creds}))) - (do (log/info "Skipping AWS connection") - component))) - (stop [component] - component)) - - -(defn create-darwin - [] - (map->Darwin {})) - - -:ok - diff --git a/math/src/polismath/darwin/export.clj b/math/src/polismath/darwin/export.clj deleted file mode 100644 index e01e5adff4..0000000000 --- a/math/src/polismath/darwin/export.clj +++ /dev/null @@ -1,712 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.darwin.export - (:require [clojure.java.io :as io] - [polismath.components.postgres :as db] - ;; Probably should move the function we need out, since this should be meta only XXX - [polismath.meta.microscope :as micro] - [polismath.math.clusters :as clust] - [polismath.math.named-matrix :as nm] - [polismath.math.conversation :as conv] - [polismath.utils :as utils] - [clojure.math.numeric-tower :as math] - [clojure.core.matrix :as mat] - [clj-time.core :as t] - [clj-time.coerce :as co] - ;; Think I'm going to use the second one here, since it's simpler (less mutable) - ;[dk.ative.docjure.spreadsheet :as spreadsheet] - ;[clj-excel.core :as excel] - [semantic-csv.core :as scsv] - [clojure-csv.core :as csv] - [clojure.pprint :refer [pprint]] - [clojure.core.matrix :as mat] - [clojure.tools.trace :as tr] - [taoensso.timbre :as log] - [clojure.tools.cli :refer [parse-opts]] - [polismath.components.postgres :as postgres] - [polismath.conv-man :as conv-man] - [plumbing.core :as plmb]) - (:import [java.util.zip ZipOutputStream ZipEntry])) - -;(mat/set-current-implementation :vectorz) - -;; Here's rougly what we want for data export. We have the following sheets per conversation. -;; -;; Summary: -;; * N Views -;; * N Voters -;; * N Voters "in conv" -;; * N Commenters -;; * N Groups -;; * url -;; -;; -;; Stats History -;; time, votes, comments, unique-hits, voters, commenters -;; -;; -;; Votes (full votes matrix): -;; participant-id, group-id, n-votes, n-comments, n-aggre, n-disagree, -;; -;; -;; Comments: -;; cid, author, aggrees, disagrees, mod, text - - -(defn datetime [timestamp] - (str (java.util.Date. timestamp))) - - -(defn full-path [darwin filename] - (str (or (-> darwin :config :export :temp-dir) "/tmp") - "/" - filename)) - - -(defn db-spec [darwin] - (-> darwin :postgres :db-spec)) - -;; Database calls for various things - -(defn get-zids-for-uid - [darwin uid] - (map :zid - (db/query (:postgres darwin) - {:select [:*] - :from [:conversations] - :where [:= :owner uid]}))) - - -(defn get-zinvite-from-zid - [darwin zid] - (-> - (db/query (:postgres darwin) - {:select [:zid :zinvite] - :from [:zinvites] - :where [:= :zid zid]}) - first - :zinvite)) - - -(defn get-conversation-votes* - ([darwin zid] - (db/query (:postgres darwin) - {:select [:*] - :from [:votes] - :where [:= :zid zid] - :order-by [:zid :tid :pid :created]})) - ([darwin zid final-vote-timestamp] - (db/query (:postgres darwin) - {:select [:*] - :from [:votes] - :where [:and [:= :zid zid] - [:<= :created final-vote-timestamp]] - :order-by [:zid :tid :pid :created]}))) - -(defn get-corrected-conversation-votes - [darwin & args] - (->> (apply get-conversation-votes* darwin args) - ;; Flip the signs on the votes XXX (remove when we switch) - (map #(update-in % [:vote] (partial * -1))) - (map #(assoc % :datetime (datetime (:created %)))))) - -(defn get-uncorrected-conversation-votes - [darwin & args] - (->> (apply get-conversation-votes* darwin args) - (map #(assoc % :datetime (datetime (:created %)))))) - - -(defn get-conversation-data - "Return a map with :topic and :description keys" - [darwin zid] - (-> - (db/query (:postgres darwin) - {:select [:zid :topic :description :created] - :from [:conversations] - :where [:= :zid zid]}) - first)) - -(defn get-participation-data* - ([darwin zid] - (db/query (:postgres darwin) - {:select [:zid :pid :vote_count :created] - :from [:participants] - :where [:= :zid zid]})) - ([darwin zid final-timestamp] - (db/query (:postgres darwin) - {:select [:zid :pid :vote_count :created] - :from [:participants] - :where [:and [:= :zid zid] - [:<= :created final-timestamp]]}))) - -(defn get-participation-data - [& args] - (->> (apply get-participation-data* args) - (map (fn [data] - (assoc data :datetime (datetime (:created data))))))) - - -(defn- get-comments-data-query - ([zid] - {:select [:zid :tid :pid :is_meta :is_seed :txt :mod :created] - :from [:comments] - :where [:= :zid zid]}) - ([zid final-timestamp] - (update (get-comments-data-query zid) - :where - (fn [zid-clause] - [:and zid-clause [:<= :created final-timestamp]])))) - -(defn get-comments-data - ([darwin zid] - (db/query (:postgres darwin) - (get-comments-data-query zid))) - ([darwin zid final-timestamp] - (db/query (:postgres darwin) - (get-comments-data-query zid final-timestamp)))) - - - -;; First the summary data -;; ====================== - -(defn- count-distinct-col - ([data] - (->> data distinct count)) - ([f data] - (->> data (map f) distinct count))) - -(defn summary-data - "Takes in rating matrix and set of all votes, and computes the summary stats for the - conversation" - [darwin - {:keys [n n-cmts group-clusters base-clusters zid rating-mat] :as conv} - votes - comments-data - participants] - ;; Fire off a bunch of database calls - (let [zinvite (future (get-zinvite-from-zid darwin zid)) - conv-data (future (get-conversation-data darwin zid)) - ;; Do anything needed with the data to prep - {:keys [topic description]} @conv-data - url (str "https://pol.is/" @zinvite)] - ;; Return the table of stuff to go into excel - {:topic topic - :url url - :n-views (count-distinct-col :pid participants) - :n-voters (count-distinct-col :pid votes) - :n-voters-in n - :n-commenters (count-distinct-col :pid comments-data) - :n-comments n-cmts - :n-groups (count group-clusters) - :description description})) - -(defn render-summary-with - "Intended for formatting the summary data's psuedo-headers for either excel or csv. Takes the - data as produced by summary data, and a key-mapping collection of keys as in the data to header - names to output, returning a collection of rows to be spit out." - [data key-mapping] - (for [[k v] key-mapping] - [v (get data k)])) - - - -;; Now the history data -;; ==================== - -(defn merge-histories - [votes participants comments] - (sort-by :created - (mapcat - (fn [collection tag] - (map #(assoc % :tag tag) collection)) - [votes participants comments] - [:vote :participant :comment]))) - -(defmulti update-history* - (fn [last-history datom] (:tag datom))) - -(defn update-history - "For stats-history; takes the last history value and updates it with the given record. Delegates - to a multimethod which dispatches on :tag, to determine what values need to be updated, and how." - [history datom] - (conj history - (-> datom - (->> (update-history* (or (last history) - {:n-votes 0 :n-comments 0 :n-visitors 0 :n-voters 0 :n-commenters 0 :ctxt {:voters #{} - :commenters #{}}}))) - (assoc :time (:created datom))))) - -(defmethod update-history* :vote - [last-history datom] - (-> last-history - (update-in [:n-votes] inc) - (update-in [:n-voters] - (fn [n-voters] (if-not (get-in datom [:ctxt :voters (:pid datom)]) - (inc n-voters) - n-voters))) - (update-in [:ctxt :voters] conj (:pid datom)))) - -(defmethod update-history* :participant - [last-history datom] - (-> last-history - (update-in [:n-visitors] inc))) - -(defmethod update-history* :comment - [last-history datom] - (-> last-history - (update-in [:n-comments] inc) - (update-in [:n-commenters] - (fn [n-commenters] (if-not (get-in datom [:ctxt :commenters (:pid datom)]) - (inc n-commenters) - n-commenters))) - (update-in [:ctxt :commenters] conj (:pid datom)))) - -(defn stats-history - "Returns rows of {time, n-votes, n-comments, n-visitors, n-voters, n-commenters}" - [votes participants comments] - (reduce update-history [] (merge-histories votes participants comments))) - - - -;; Full votes matrix (plus some other participant summaries) -;; ================= - -(defn reconstruct-vote-matrix - [votes] - (let [new-nmat (nm/named-matrix) - vote-tuples (map #(map % [:pid :tid :vote]) votes)] - (nm/update-nmat new-nmat vote-tuples))) - -(defn ffilter - "Given pred and coll, return the first x in coll for which (pred x) is truthy" - [pred coll] - (first (filter pred coll))) - -(defn flatten-clusters - "Takes group clusters and base clusters and flattens them out into a cluster mapping to ptpt ids directly" - [group-clusters base-clusters] - (map - (fn [gc] - (update-in gc - [:members] - (fn [members] - (mapcat - (fn [bid] - ;; get the base cluster, then get it's members, mapcat them (a level up) - (:members (ffilter #(= (:id %) bid) base-clusters))) - members)))) - group-clusters)) - - -(defn participant-xids - [darwin conv] - (->> - (db/query - (:postgres darwin) - {:select [:participants.pid :xids.uid :xid] - :from [:xids] - :join [:conversations [:= :conversations.owner :xids.owner] - :participants [:= :participants.uid :xids.uid]] - :where [:and - [:= :conversations.zid (:zid conv)] - [:= :participants.zid (:zid conv)]]}) - (map (fn [{:keys [pid xid]}] - [pid xid])) - (into {}))) - - -;; participant-id, group-id, n-votes, n-comments, n-aggre, n-disagree, -(defn participants-votes-table - [darwin conv votes comments {:as kw-args :keys [include-xid]}] - (let [mat (reconstruct-vote-matrix votes) - flattened-clusters (flatten-clusters (:group-clusters conv) (:base-clusters conv)) - xids (participant-xids darwin conv)] - (concat - ;; The header - [(concat ["participant"] - (when include-xid ["xid"]) - ["group-id" "n-comments" "n-votes" "n-agree" "n-disagree"] - (nm/colnames mat))] - ;; The rest of the data - (map - (fn [ptpt row] - (concat [ptpt] - (when include-xid [(get xids ptpt)]) - [(:id (ffilter #(some #{ptpt} (:members %)) flattened-clusters)) - (count (filter #(= (:pid %) ptpt) comments)) - (count (remove nil? row)) - ;; XXX God damn aggree vs disagree... - ;; Fixed this upstream, for now; so should be good to go once we've fixed it at the source. But - ;; keep an eye on it for now... XXX - (count (filter #{1} row)) - (count (filter #{-1} row))] - row)) - (nm/rownames mat) - (.matrix mat))))) - -(defn format-vote-matrix-header - "Apply format-header function to each element of header in a collection of row vectors" - [data format-header] - (concat [(mapv format-header (first data))] - (rest data))) - - -;; Comments -;; ======== - -(defn add-group-data-to-comments - [comments {:as conv :keys [group-aware-consensus]}] - (log/info (keys conv)) - (map (fn [{:as comment-data :keys [tid]}] - (assoc comment-data :group-informed-consensus (get group-aware-consensus tid))) - comments)) - -;(sort (keys theconv)) -;(:subgroup-repness theconv) - -;(:group-aware-consensus theconv) - -;(comment - ;(require '[polismath.runner :as runner]) - ;(runner/run! polismath.system/export-system) - ;(load-conv (:darwin runner/system) :zid 12798) - ;(try - ;(export-conversation (:darwin runner/system) - ;{;;:zid 310273 - ;:zinvite "2demo" - ;:format :csv - ;:filename "2demo.zip" - ;:math-env "prod"}) - ;(catch Exception e - ;(.printStackTrace e)))) - -(defn enriched-comments-data - "Just adds vote counts to the comments data" - [comments votes] - (map - (fn [{:keys [tid] :as comment-data}] - (let [comment-votes (filter #(= tid (:tid %)) votes) - ;; Fixed this upstream, for now; so should be good to go once we've fixed it at the source. But - ;; keep an eye on it for now... XXX - aggrees (filter #(= 1 (:vote %)) comment-votes) - disagrees (filter #(= -1 (:vote %)) comment-votes)] - (assoc comment-data - :aggrees (count aggrees) - :disagrees (count disagrees) - :datetime (datetime (:created comment-data))))) - comments)) - - -;; Now some reshaping stuff for excel; mostly just applying headers here -;; Actaul excel things - -(defn stringify-keys - ([m] - (stringify-keys m #(-> % str (clojure.string/replace ":" "")))) - ([m f] - (into {} (map (fn [[k v]] - [(if-not (string? k) (f k) k) v]) - m)))) - - -(defn update-from-map-or-leave - "Little utility for generating a function that either updates based on a map (closed over) or leaves value unchanged" - [m] - (fn [k] - (if-let [v (get m k)] - v - k))) - -(defn excel-format - [export-data] - (-> export-data - (update-in [:summary] - render-summary-with - [[:topic "Topic"] - [:url "URL"] - [:n-views "Views"] - [:n-voters "Voters"] - [:n-voters-in "Voters (in conv)"] - [:n-commenters "Commenters"] - [:n-comments "Comments"] - [:n-groups "Groups"] - [:description "Conversation Description"]]) - (update-in [:stats-history] - (partial scsv/vectorize {:header [:n-votes :n-comments :n-visitors :n-voters :n-commenters] - :format-header {:n-votes "Votes" - :n-visitors "Visitors" - :n-voters "Voters" - :n-comments "Comments" - :n-commenters "Commenters"}})) - (update-in [:comments] - (partial scsv/vectorize {:header [:created :tid :pid :aggrees :disagrees :mod :is_meta :is_seed :group-informed-consensus :txt] - :format-header {:created "Timestamp" - :tid "Comment ID" - :pid "Author" - :aggrees "Aggrees" - :disagrees "Disagrees" - :mod "Moderated" - :is_meta "Metadata" - :is_seed "Seed" - :group-informed-consensus "Group informed consensus" - :txt "Comment body"}})) - (update-in [:votes] - (partial scsv/vectorize {:header [:created :tid :pid :vote] - :format-header {:created "Timestamp" - :tid "Comment ID" - :pid "Voter" - :vote "Vote"}})) - (update-in [:participants-votes] - format-vote-matrix-header - ;; flesh out... - (update-from-map-or-leave {"participant" "Participant" - "group-id" "Group ID" - "n-comments" "Comments" - "n-votes" "Votes" - "n-agree" "Agrees" - "n-disagree" "Disagrees"})) - (stringify-keys (array-map :summary "Summary" - :stats-history "Stats History" - :comments "Comments" - :votes "Votes" - :participants-votes "Participants Votes")))) - - -(defn csv-format - [export-data] - (-> export-data - (update-in [:summary] - render-summary-with - [[:topic "topic"] - [:url "url"] - [:n-views "views"] - [:n-voters "voters"] - [:n-voters-in "voters-in-conv"] - [:n-commenters "commenters"] - [:n-comments "comments"] - [:n-groups "groups"] - [:description "conversation-description"]]) - (update-in [:stats-history] - (partial scsv/vectorize {:header [:n-votes :n-comments :n-visitors :n-voters :n-commenters]})) - (update-in [:votes] - (partial scsv/vectorize {:header [:created :datetime :tid :pid :vote] - :format-header {:created "timestamp" - :datetime "datetime" - :tid "comment-id" - :pid "voter-id" - :vote "vote"}})) - (update-in [:comments] - (partial scsv/vectorize {:header [:created :tid :pid :aggrees :disagrees :mod :is_meta :is_seed :group-informed-consensus :txt] - :format-header {:created "timestamp" - :datetime "datetime" - :tid "comment-id" - :pid "author-id" - :aggrees "agrees" - :disagrees "disagrees" - :mod "moderated" - :is_meta "is-meta" - :is_seed "is-seed" - :group-informed-consensus "group-informed-consensus" - :txt "comment-body"}})))) - - - -;; This zip nonsense is stolen from http://stackoverflow.com/questions/17965763/zip-a-file-in-clojure -(defmacro ^:private with-entry - [zip entry-name & body] - `(let [^ZipOutputStream zip# ~zip] - (.putNextEntry zip# (ZipEntry. ~entry-name)) - ~@body - (flush) - (.closeEntry zip#))) - -(defn move-to-zip-stream - [zip-stream input-filename entry-point] - (with-open [input (io/input-stream input-filename)] - (with-entry zip-stream entry-point - (io/copy input zip-stream)))) - -(defn print-csv - [table] - (->> table - (map (partial mapv str)) - csv/write-csv - print)) - -(defn zipfile-basename - [filename] - (-> filename - (clojure.string/split #"\/") - last - (clojure.string/replace #"\.zip$" ""))) - - -;; Must assert .zip in filenames or things will break on unzipping XXX -(defn save-to-csv-zip - ([filename data] - (with-open [file (io/output-stream filename) - zip (ZipOutputStream. file) - writer (io/writer zip)] - (save-to-csv-zip zip writer (zipfile-basename filename) data))) - ([zip-stream writer entry-point-base data] - (binding [*out* writer] - (doto zip-stream - (with-entry (str entry-point-base "/summary.csv") - (print-csv (:summary data))) - (with-entry (str entry-point-base "/stats-history.csv") - (print-csv (:stats-history data))) - (with-entry (str entry-point-base "/comments.csv") - (print-csv (:comments data))) - (with-entry (str entry-point-base "/votes.csv") - (print-csv (:votes data))) - (with-entry (str entry-point-base "/participants-votes.csv") - (print-csv (:participants-votes data))))))) - - -;(defn save-to-excel - ;([filename data] - ;(-> (excel/build-workbook data) - ;(excel/save filename))) - ;;; Should really change both this and the above to use the .zip filename, and take basename for the main dir - ;;; XXX - ;([zip-stream entry-point data] - ;;; Would be nice if we could write directly to the zip stream, but the excel library seems to be doing - ;;; weird things... - ;(let [tmp-file-path (str "tmp/rand-" (rand-int Integer/MAX_VALUE) ".xml")] - ;(save-to-excel tmp-file-path data) - ;(move-to-zip-stream zip-stream tmp-file-path entry-point)))) - - -;; Putting it all together -;; ======================= - -(defn kw->int - [kw] - (-> kw - (str) - (clojure.string/replace ":" "") - (Integer/parseInt))) - -;; Should be using the conv-man to load the conversation if it's already there -(defn load-conv - [darwin & {:keys [zid zinvite] :as args}] - (let [zid (or zid (postgres/get-zid-from-zinvite (:postgres darwin) zinvite))] - (log/debug "Loading conversation" zid) - (-> - (db/load-conv (:postgres darwin) zid) - ;; This should be ok here right? - (conv-man/restructure-json-conv) - (update-in - [:repness] - (partial plmb/map-keys kw->int))))) - - -(defn get-export-data - [darwin {:keys [zid zinvite update-math] :as kw-args}] - (let [zid (or zid (postgres/get-zid-from-zinvite (:postgres darwin) zinvite)) - ;; assert zid - votes (get-corrected-conversation-votes darwin zid) - participants (get-participation-data darwin zid) - ;; Should factor out into separate function - conv (cond-> (utils/apply-kwargs load-conv darwin kw-args) - update-math (conv/conv-update votes)) - comments (-> (get-comments-data darwin zid) - (enriched-comments-data votes) - (add-group-data-to-comments conv))] - (when update-math (log/info "UPDATED MATH!")) - {:votes votes - :summary (summary-data darwin conv votes comments participants) - :stats-history (stats-history votes participants comments) - :participants-votes (participants-votes-table darwin conv votes comments kw-args) - :comments comments})) - -(defn get-export-data-at-time - [darwin {:keys [zid zinvite env-overrides at-time update-postgres] :as kw-args}] - (let [zid (or zid (postgres/get-zid-from-zinvite (:postgres darwin) zinvite)) - votes (get-corrected-conversation-votes darwin zid at-time) - meta-tids (postgres/get-meta-tids (:postgres darwin) zid) - ;_ (log/info "META TIDS " meta-tids) - conv (-> (assoc (conv/new-conv) :zid zid :meta-tids meta-tids) - ;; The core math still uses the uncorrected votes - (conv/conv-update (get-uncorrected-conversation-votes darwin zid at-time))) - _ (log/info "Done with conv update") - comments (-> (get-comments-data darwin zid at-time) - (enriched-comments-data votes) - (add-group-data-to-comments conv)) - participants (get-participation-data darwin zid at-time)] - (log/info "Done with conv update") - (when update-postgres - (log/info "Updating postgres") - (let [math-tick (postgres/inc-math-tick (:postgres darwin) zid)] - (conv-man/write-conv-updates! darwin conv math-tick))) - {:votes votes - :summary (assoc (summary-data darwin conv votes comments participants) :at-time at-time) - :stats-history (stats-history votes participants comments) - :participants-votes (participants-votes-table darwin conv votes comments kw-args) - :comments comments})) - - -;(defn load-from-timestamp - ;[darwin {:keys [zid zivite at-time] :as kw-args}] - ;(let [zid (or zid (postgres/get-zid-from-zinvite (:postgrs darwin) zinvite)) - ;votes (get-conversation-votes darwin zid at-time) - ;math-tick (postgres/inc-math-tick (:postgres darwin) zid) - ;conv (-> (assoc (conv/new-conv) :zid zid) - ;(conv/conv-update votes))] - ;(conv-man/write-conv-updates! {:postgres (:postgres darwin)} - ;conv - ;math-tick) - ;(log/info "Done with conv update"))) - - - -(defn export-conversation - "This is the main API endpoint for the export functionality. Given either :zid or :zinvite, export data to - the specified :format and spit results out to :filename. Optionally, a :zip-stream and :entry point may be - specified, which can be used for building up items in a zip file. This is used in export/-main to export all - convs for a given uid, for example." - ;; Don't forget env-overrides {:math-env "prod"}; should clean up with system - [darwin {:keys [zid zinvite format filename zip-stream writer entry-point env-overrides at-time include-xid] :as kw-args}] - (log/info "Exporting data for zid =" zid ", zinvite =" zinvite) - (log/info "at-time:" at-time) - (let [export-data (if at-time - (get-export-data-at-time darwin kw-args) - (get-export-data darwin kw-args)) - ;; Had to remove excel support because lib outdated vs tech.ml.dataset java dep - ;[formatter saver] (case format :excel [excel-format save-to-excel] :csv [csv-format save-to-csv-zip]) - [formatter saver] [csv-format save-to-csv-zip] - formatted (formatter export-data)] - (if zip-stream - (if (-> export-data :summary :n-voters (> 0)) - (saver zip-stream writer entry-point formatted) - (log/debug "Skipping conv" zid zinvite ", since no votes")) - (saver (full-path darwin filename) formatted))) - (log/info "Finished exporting data for zid =" zid ", zinvite =" zinvite)) - - - - - -; A testbench for using the main API function -(comment - (require '[polismath.runner :as runner]) - (def darwin (:darwin runner/system)) - runner/system - (export-conversation runner/system - {;;:zid 310273 - :zinvite "7scufp" - :format :csv - :filename "cljwebdev.zip" - :math-env "prod"}) - (export-conversation darwin - {;;:zid 310273 - :zinvite "7scufp" - :format :excel - :filename "cljwebdev.xlsx" - :math-env "prod"}) - :end-comment) - - -:ok - diff --git a/math/src/polismath/math/clusters.clj b/math/src/polismath/math/clusters.clj deleted file mode 100644 index 4b8fffb624..0000000000 --- a/math/src/polismath/math/clusters.clj +++ /dev/null @@ -1,441 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.clusters - (:refer-clojure :exclude [* - + == /]) - (:require [taoensso.timbre :as log] - [plumbing.core :as pc - :refer (fnk map-vals <-)] - [plumbing.graph :as gr] - [clojure.tools.trace :as tr] - ; [alex-and-georges.debug-repl :as dbr] - [polismath.utils :as utils] - [polismath.math.stats :as stats] - [polismath.math.named-matrix :as nm] - [clojure.spec.alpha :as s] - [clojure.core.matrix :as matrix] - [clojure.core.matrix.stats :as matrix-stats] - [clojure.core.matrix.selection :as matrix.selection] - [clojure.core.matrix.operators :refer :all])) - - -;; We may want to make formal that it's an int? Cause we add later? -(s/def ::id (constantly true)) -(s/def ::members seq?) -(s/def ::center seq?) - -(s/def ::cluster - (s/keys :req-un [::id ::members ::center])) - -(s/def ::clustering - (s/* ::cluster)) - -(matrix/set-current-implementation :vectorz) - - -(defn clst-append - "Append an item to a cluster, where item is a (mem_id, vector) pair" - [clst item] - (assoc clst - ; Note that order is important here, and assumed to be the same for the weighted-mean call in `cluster-step`. - :members (conj (:members clst) (first item)) - :positions (conj (:positions clst) (last item)))) - - -(defn add-to-closest - "Find the closest cluster and append item (mem_id, vector) to it" - [clusts item] - (let [[clst-id clst] (apply min-key - (fn [[clst-id clst]] - (matrix/distance (last item) (:center clst))) - clusts)] - (assoc clusts clst-id - (clst-append clst item)))) - - -(defn init-clusters - "Effectively random initial clusters for initializing a new kmeans comp" - [data k] - (->> data - nm/get-matrix - matrix/rows - vec ;; Have to cast to vec before we can distinct cause rows gives us a matrixlist - distinct - (map-indexed - (fn [id position] {:id id :members [] :center (matrix/matrix position)})) - (take k))) - - -(defn same-clustering? - "Determines whether clusterings are within tolerance by measuring distances between - centers. Note that cluster centers here must be vectors and not NDArrays" - [clsts1 clsts2 & {:keys [threshold] :or {threshold 0.01}}] - (letfn [(cntrs [clsts] (sort (map :center clsts)))] - (every? - (fn [[x y]] - (< (matrix/distance x y) threshold)) - (utils/zip (cntrs clsts1) (cntrs clsts2))))) - - -(defn cleared-clusters - "Clears a cluster's members so that new ones can be assoced on a new clustering step" - [clusters] - (->> clusters - (map (fn [cluster] - [(:id cluster) - (assoc cluster :members [] :positions [])])) - (into {}))) - - -(defmulti weighted-mean - "Compute either the mean or weighted mean (if :weights is passed) of either a matrix or - named matrix. For a matrix, :weights should be vector, the ith element of which is the - weight corresponding to row i of mat. For a named matrix, it should be a hash." - (fn [& args] - [(matrix/matrix? (first args)) - (matrix/vec? (first args))])) - -; It's a matrix -(defmethod weighted-mean [true false] - [mat & {:keys [weights]}] - (if weights - (weighted-mean - (* (/ (count weights) (pc/sum weights)) - (reduce - (fn [m [row-i weight]] - (matrix/multiply-row m row-i weight)) - mat - (utils/with-indices weights)))) - (matrix-stats/mean (matrix/matrix mat)))) - -; It's a vector... -(defmethod weighted-mean [false true] - [v & {:keys [weights]}] - (if weights - (/ (matrix/dot weights v) (pc/sum weights)) - (matrix-stats/mean v))) - -; It's a named matrix... -(defmethod weighted-mean [false false] - [nmat & {:keys [weights]}] - (weighted-mean (nm/get-matrix nmat) - :weights - (when weights - (reduce - #(conj %1 (weights %2)) - [] - (nm/rownames nmat))))) - -;; Rename all above to weighted-mean* for profiling -;(defnp weighted-mean -; [& args] -; (apply weighted-mean* args)) - -(defn cluster-weights - "Get a weights seq given a cluster with :members and a hash-map of weights. Returns nil - if hm-weights is falsey." - [cluster hm-weights] - (when hm-weights - (->> (:members cluster) - (map hm-weights)))) - - -(defn cluster-step - "Performs one step of an iterative K-means: - data-iter: seq of pairs (id, position), eg (pid, person-rating-row) - clusters: array of clusters" - [data-iter k clusters & {:keys [weights]}] - (->> data-iter - ; Reduces a "blank" set of clusters w/ centers into clusters that have elements - (reduce add-to-closest (cleared-clusters clusters)) - vals - ; Filter out clusters that don't have any members (should maybe log on verbose?) - (filter #(> (count (:members %)) 0)) - ; Apply mean to get updated centers - (map (fn [clst] - (-> clst - (assoc :center (weighted-mean (:positions clst) - :weights (cluster-weights clst weights))) - (dissoc :positions)))))) - - -(defn recenter-clusters - "Replace cluster centers with a center computed from new positions" - [data clusters & {:keys [weights]}] - (map - (fn [clst] - (assoc clst :center (weighted-mean (nm/rowname-subset data (:members clst)) - :weights weights))) - clusters)) - - -(defn safe-recenter-clusters - "Replace cluster centers with a center computed from new positions" - [data clusters & {:keys [weights]}] - (as-> clusters clsts - ; map every cluster to the newly centered cluster or to nil if there are no members in data - (map - (fn [clst] - (let [rns (nm/safe-rowname-subset data (:members clst))] - (if (empty? (nm/rownames rns)) - nil - (assoc clst :center (weighted-mean rns :weights weights))))) - clsts) - ; Remove the nils, they break the math - (remove nil? clsts) - ; If nothing is left, make one great big cluster - so that things don't break in most-distal later - ; XXX - Should see if there is a cleaner place/way to handle this... - (if (empty? clsts) - [{:id (inc (apply max -1 (map :id clusters))) - :members (nm/rownames data) - :center (weighted-mean data :weights weights)}] - clsts))) - - -(defn merge-clusters [clst1 clst2] - (let [new-id (:id (max-key #(count (:members %)) clst1 clst2))] - {:id new-id - :members (into (:members clst1) (:members clst2)) - :center (weighted-mean (map :center [clst1 clst2]) - :weights (map (comp count :members) [clst1 clst2]))})) - - -(defn most-distal - "Finds the most distal point in all clusters" - [data clusters] - (let [[dist clst-id id] - ; find the maximum dist, clst-id, mem-id triple - (apply max-key #(get % 0) - (map - (fn [mem] - ; Find the minimum distance, cluster-id pair, and add the member name to the end - (conj (apply min-key #(get % 0) - (map - #(vector (matrix/distance (nm/get-row-by-name data mem) (:center %)) (:id %)) - clusters)) - mem)) - (nm/rownames data)))] - {:dist dist :clst-id clst-id :id id})) - - -(defn uniqify-clusters [clusters] - (reduce - (fn [clusters clst] - (let [identical-clst (first (filter #(= (:center clst) (:center %)) clusters))] - (if identical-clst - (assoc clusters (utils/typed-indexof clusters identical-clst) (merge-clusters identical-clst clst)) - (conj clusters clst)))) - [] clusters)) - - -(defn clean-start-clusters - "This function takes care of some possible messy situations which can crop up with using 'last-clusters' - in kmeans computation, and generally gets the last set of clusters ready as the basis for a new round of - clustering given the latest set of data. - - WARNING!!! This function is very slow for building up a new clustering from scratch, and thus for an empty - clustering we just call out to init-clusters. If you try building off of just a few seed clusters though, - this function will behave very poorly. This could potentially be optimized for down the road, but for now - be warned." - [data clusters k & {:keys [weights]}] - ; First recenter clusters (replace cluster center with a center computed from new positions) - (if (seq clusters) - ;; In most cases, we have existing clusters we want to optimize for as we break things apart, and want - ;; to break things apart in such a way as to preserve as much structure as possible - (let [clusters (safe-recenter-clusters data clusters :weights weights) - ; next make sure we're not dealing with any clusters that are identical to eachother - uniq-clusters (uniqify-clusters clusters) - ; count uniq data points to figure out how many clusters are possible - ;; I don't like the into here, but seems necessary to solve a weird core.matrix issue with lists not having nth - possible-clusters (min k (count (distinct (into [] (matrix/rows (nm/get-matrix data))))))] - (loop [clusters uniq-clusters] - ; Whatever the case here, we want to do one more recentering - (let [clusters (recenter-clusters data clusters :weights weights)] - (if (> possible-clusters (count clusters)) - ; first find the most distal point, and the cluster to which it's closest - (let [outlier (most-distal data clusters)] - (if (> (:dist outlier) 0) - ; There is work to be done, so do it - (recur - (-> - ; first remove the most distal point from the cluster it was in; - (mapv - (fn [clst] - (assoc clst :members - (remove (set [(:id outlier)]) (:members clst)))) - clusters) - ; next add a new cluster containing only said point. - (conj {:id (inc (apply max (map :id clusters))) - :members [(:id outlier)] - :center (nm/get-row-by-name data (:id outlier))}))) - ; Else just return recentered clusters - clusters)) - ; Else just return recentered clusters - clusters)))) - ;; Otherwise, just initialize a set of dummy clusters - (do - (log/warn "Had to initialize clusters from clean-start-clusters. This shouldn't have to happen, and maybe means bidToPid mapping didn't load into new conv.") - (init-clusters data k)))) - - -(defn setify-members - [clsts & {:keys [trans] :or {trans identity}}] - (->> clsts - (map (pc/fn->> :members (map trans) set)) - (set))) - - -(defn simplify-clsts - "Given a clustering, creates a set of member sets. This makes it easy to compare clusters for equality. - Optional `:trans` keyword args lets you perform a transformation to the member names included in member - sets." - [clsts & {:keys [trans] :or {trans identity}}] - {:members (map - (pc/fn->> :members (map trans) set) - clsts) - :center (map - (pc/fn->> :center (mapv #(utils/round-to % 4))) - clsts)}) - - -; Each cluster should have the shape {:id :members :center} -(defn kmeans - "Performs a k-means clustering." - [data k & {:keys [last-clusters max-iters weights] :or {max-iters 20}}] - (let [data-iter (utils/zip (nm/rownames data) (matrix/matrix (nm/get-matrix data))) - clusters (if last-clusters - (clean-start-clusters data last-clusters k :weights weights) - (init-clusters data k))] - (loop [clusters clusters iter max-iters] - (let [new-clusters (cluster-step data-iter k clusters :weights weights)] - (if (or (= iter 0) (same-clustering? clusters new-clusters)) - new-clusters - (recur new-clusters (dec iter))))))) - - -(defn dist-matrix - "Dist matrix" - ([m] (dist-matrix m m)) - ([m1 m2] - (matrix/matrix - (map - (fn [r1] - (map - (fn [r2] - (matrix/distance r1 r2)) - m2)) - m1)))) - - -(defn named-dist-matrix - "Distance matrix with rownames and colnames corresponding to rownames of nm1 and nm2 respectively." - ([nm] (named-dist-matrix nm nm)) - ([nm1 nm2] - (nm/named-matrix - (nm/rownames nm1) - (nm/rownames nm2) - (dist-matrix (nm/get-matrix nm1) (nm/get-matrix nm2))))) - - -(defn silhouette - "Compute the silhoette coefficient for either a cluster member, or for an entire clustering. Currently, - the latter just averages over the former for all members - it's likely there is a more efficient way - to block things up. If distmat has distances in it not in clusters, those distances are ignored." - ([distmat clusters member] - (let [dist-row (nm/rowname-subset distmat [member]) - [a b] - (reduce - (fn [[a b] clst] - (let [memb-clst? (some #{member} (:members clst)) - membs (remove #{member} (:members clst))] - ; This is a little bit silly, but will basically trigger returning 0 if member is in a - ; singleton cluster - (if (and memb-clst? (empty? membs)) - (reduced [1 1]) - ; Otherwise, continue... - (as-> membs data - ; Subset to just the columns for this clusters - (nm/colname-subset dist-row data) - (nm/get-matrix data) - ; This is a 2D row vector; we want 1D, so take first - (first data) - ; Take the mean of the entries - (matrix-stats/mean data) - (if memb-clst? - [data b] - [a (min data (or b data))]))))) - [nil nil] - clusters)] - ; The actual silhouette computation - (/ (- b a) (max b a)))) - ([distmat clusters] - (let [cluster-distmat (nm/rowname-subset distmat (mapcat :members clusters))] - (weighted-mean - (map - (partial silhouette distmat clusters) - (nm/rownames cluster-distmat)))))) - - -(defn group-members - "Given group-clusters group and base clusters, get the members for the group" - [group base-clusters] - (let [group-bids (set (:members group))] - (->> base-clusters - (filter #(group-bids (:id %))) - (map :members) - (apply concat)))) - - -;; XXX This is a little too opinionated imo; Should just map to center and leave unfolding x y for elsewhere for better generality -(defn fold-clusters - "Takes clusters -- a seq of maps `{:members :id :center}` -- and transforms into a single map - `{:id :members :x :y :count}`, where each key points to a seq of the values associated with each - cluster. In particular, this is what's used to format base clusters for mongo uploading (for the - sake of compression)." - [clusters] - {:id (map :id clusters) - :members (map :members clusters) - :x (map (comp first :center) clusters) - :y (map (comp second :center) clusters) - :count (map (comp count :members) clusters)}) - - -(defn unfold-clusters - "The inverse of `fold-clusters`; takes folded clusters and puts them into standard form. - i.e. `(= identity (comp unfold-clusters fold-clusters))`" - [{:keys [members id x y] :as folded-clusters}] - (map - (fn [ms id x y] - {:id id - :members ms - :center [x y]}) - members - id - x - y)) - - -(defn xy-clusters-to-nmat [clusters] - (let [nmat (nm/named-matrix)] - (nm/update-nmat - nmat - (apply concat ; flatten the list of lists below - (mapv - (fn [cluster] - (let [center (:center cluster) - id (:id cluster)] - ; Return some values that we can feed to update-nmat - [[id :x (first center)] - [id :y (second center)]])) - - clusters))))) - - -(defn xy-clusters-to-nmat2 [clusters] - (nm/named-matrix - (map :id clusters) ; row names - [:x :y] ; column names - (matrix/matrix (map :center clusters)))) - - -:ok - diff --git a/math/src/polismath/math/conversation.clj b/math/src/polismath/math/conversation.clj deleted file mode 100644 index eb2fab2845..0000000000 --- a/math/src/polismath/math/conversation.clj +++ /dev/null @@ -1,945 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.conversation - (:refer-clojure :exclude [* - + == /]) - (:require - [polismath.utils :as utils] - [polismath.math.pca :as pca] - [polismath.math.clusters :as clusters] - [polismath.math.repness :as repness] - [polismath.math.named-matrix :as nm] - [clojure.core.matrix :as matrix] - [clojure.spec.alpha :as s] - [clojure.tools.reader.edn :as edn] - [clojure.tools.trace :as tr] - [clojure.math.numeric-tower :as math] - [clojure.core.matrix :as matrix] - [clojure.core.matrix.operators :refer :all] - [plumbing.core :as plmb] - [plumbing.graph :as graph] - [bigml.sampling.simple :as sampling] - ;[alex-and-georges.debug-repl :as dbr] - [taoensso.timbre :as log] - [clojure.spec.gen.alpha :as gen] - [clojure.test.check.generators :as generators] - [tech.v3.dataset :as ds] - [tech.v3.datatype :as dt])) - ;[semantic-csv.core :as s.csv])) - - -;; Starting to spec out our domain model here and build generators for the pieces -;; This will let us do generative testing and all other ilk of awesome things - -(defn pos-int [gen-bound] - (s/with-gen - (s/and int? pos?) - (s/gen (s/int-in 0 gen-bound)))) - -(s/def ::zid (pos-int 20)) -(s/def ::tid (pos-int 100)) -(s/def ::pid (pos-int 100)) -(s/def ::gid (pos-int 8)) -;; QUESTION How do we have a generator that gives us monotonically increasing values? -(s/def ::created (s/and int? pos?)) -(s/def ::vote #{-1 0 1 -1.0 1.0 0.0}) - -(s/def ::vote-map - (s/keys :req-un [::zid ::tid ::pid ::vote ::created])) - -(s/def ::maybe-vote (s/or :missing nil? :voted ::vote)) - -(s/def ::zid int?) -(s/def ::last-vote-timestamp int?) -(s/def ::group-votes - (constantly true)) -(s/def ::sub-group-votes - (s/map-of ::gid ::group-votes)) - -(s/def ::group-clusters ::clusters/clustering) -(s/def ::subgroup-clusters - (s/map-of ::gid ::clusters/clustering)) - -(s/def ::repness ::repness/clustering-repness) - -(s/def ::subgroup-repness - (s/map-of ::gid ::repness/clustering-repness)) - -(s/def ::rating-mat - ;(s/with-gen - (s/and ::nm/NamedMatrix)) - ;every element is a ::maybe-vote - ;#(every?)))) - ;; Let's just use a generator that generates votes and places them in a rating mat? - ;()) - -(s/def ::raw-rating-mat ::rating-mat) - -;; May want to speck out "bare" vs "fleshed out" conversations, as well as loaded vs raw, etc -(s/def ::new-conversation - (s/keys :req-un [::raw-rating-mat ::last-vote-timestamp])) -(s/def ::full-conversation - (s/keys :req-un [::zid ::last-vote-timestamp ::group-votes ::raw-rating-mat ::rating-mat ::group-clusters ::subgroup-clusters ::repness ::subgroup-repness])) - -(s/def ::conversation - (s/or :new ::new-conversation - :full ::full-conversation)) - - -(defn new-conv [] - "Minimal structure upon which to perform conversation updates" - {:raw-rating-mat (nm/named-matrix) - :last-vote-timestamp 0 - :lastVoteTimestamp 0 - :rating-mat (nm/named-matrix)}) - - -;; I think this is old and can be removed -;(defn choose-group-k [base-clusters] -; (let [len (count base-clusters)] -; (cond -; (< len 99) 3 -; :else 4))) - - -(defn agg-bucket-votes-for-tid [bid-to-pid rating-mat filter-cond tid] - (if-let [idx (nm/index (nm/get-col-index rating-mat) tid)] - ; If we have data for the given comment... - (let [pid-to-row (zipmap (nm/rownames rating-mat) (range (count (nm/rownames rating-mat)))) - person-rows (nm/get-matrix rating-mat)] - (mapv ; for each bucket - (fn [pids] - (->> pids - ; get votes for the tid from each ptpt in group - (map (fn [pid] - (if-let [row-idx (pid-to-row pid)] - (get (get person-rows row-idx) idx) - nil))) - ; filter votes you don't want to count - (filter filter-cond) - ; count - (count))) - bid-to-pid)) - ; Otherwise return an empty vector - [])) - - - -; conv - should have -; * last-updated -; * pca -; * center -; * pcs -; * base-clusters -; * group-clusters -; * repness -; [hidden] -; * rating matrix -; * base-cluster-full [ptpt to base mpa] - -(def base-conv-update-graph - "Base of all conversation updates; handles default update opts and does named matrix updating" - {:opts' (plmb/fnk [opts] - "Merge in opts with the following defaults" - ;; TODO Answer and resolve this question: - ;; QUESTION Does it make senes to have the defaults here or in the config.edn or both duplicated? - (merge {:n-comps 2 ; does our code even generalize to others? - :pca-iters 100 - :base-iters 100 - :base-k 100 - :max-k 5 - :group-iters 100 - ;; These three in particular we should be able to tune quickly - :max-ptpts 100000 - :max-cmts 10000 - :group-k-buffer 4} - opts)) - - :zid (plmb/fnk [conv votes] - (or (:zid conv) - (:zid (first votes)))) - - :last-vote-timestamp - (plmb/fnk [conv votes] - (apply max - (or (:last-vote-timestamp conv) 0) - (map :created votes))) - - :customs (plmb/fnk [conv votes opts'] - ; Decides whether there is room for new ptpts/cmts, and which votes should be allowed in - ; based on which ptpts/cmts have already been seen. This is a simple prevention against - ; conversations that get too large. Returns {:pids :tids :votes}, where the first two kv - ; pairs are persisted and built upon and persisted; :votes is used downstream and tossed - (reduce - (fn [{:keys [pids tids] :as result} - {:keys [pid tid] :as vote}] - (let [pid-room (< (count pids) (:max-ptpts opts')) - tid-room (< (count tids) (:max-cmts opts')) - pid-in (pids pid) - tid-in (tids tid)] - (if (and (or pid-room pid-in) - (or tid-room tid-in)) - (assoc result - :pids (conj (:pids result) pid) - :tids (conj (:tids result) tid) - :votes (conj (:votes result) vote)) - result))) - ; Customs collection off which to base reduction; note that votes get cleared out - (assoc (or (:customs conv) {:pids #{} :tids #{}}) - :votes []) - votes)) - - :keep-votes - (plmb/fnk [customs] - (:votes customs)) - - :raw-rating-mat - (plmb/fnk [conv keep-votes] - (nm/update-nmat - (:raw-rating-mat conv) - (map (fn [v] (vector (:pid v) (:tid v) (:vote v))) keep-votes))) - - :rating-mat - (plmb/fnk [conv raw-rating-mat] - ;; This if-let here is just a simple performance optimization - (let [mat - (if-let [mod-out (:mod-out conv)] - (nm/zero-out-columns raw-rating-mat mod-out) - raw-rating-mat)] - mat)) - - :tids - (plmb/fnk [conv rating-mat] - (nm/colnames rating-mat)) - - :n (plmb/fnk [rating-mat] - (count (nm/rownames rating-mat))) - - :n-cmts (plmb/fnk [rating-mat] - (count (nm/colnames rating-mat))) - - :user-vote-counts - (plmb/fnk [raw-rating-mat] - ; For deciding in-conv below; filter ptpts based on how much they've voted - (->> (mapv - (fn [rowname row] - [rowname (count (remove nil? row))]) - (nm/rownames raw-rating-mat) - (nm/get-matrix raw-rating-mat)) - (into {}))) - - ;; Ugg... right, have to clarify that we don't want to drop this; are we leaving anything else out like this? - :mod-out - (plmb/fnk [conv] - (:mod-out conv)) - :mod-in - (plmb/fnk [conv] - (:mod-in conv)) - :meta-tids - (plmb/fnk [conv] - (:meta-tids conv)) - - ;; There should really be a nice way for us to specify that we want a full recompute on everything except in-conv, - ;; since in meta-tids we don't want to loose people in that process. - :in-conv (plmb/fnk [conv user-vote-counts n-cmts] - ; This keeps track of which ptpts are in the conversation (to be considered - ; for base-clustering) based on home many votes they have. Once a ptpt is in, - ; they will remain in. - (as-> (or (:in-conv conv) #{}) in-conv - ; Start with whatever you have, and join it with anything that meets the criteria - (into in-conv - (map first - (filter - (fn [[rowname cnt]] - ; We only start looking at a ptpt if they have rated either all the comments or at - ; least 7 if there are more than 7 - (>= cnt (min 7 n-cmts))) - user-vote-counts))) - ; If you are left with fewer than 15 participants, take the top most contributing - ; participants - (let [greedy-n 15 - n-in-conv (count in-conv)] - (if (< n-in-conv greedy-n) - (->> user-vote-counts - (remove - (fn [[k v]] (in-conv k))) - (sort-by (comp - second)) - (map first) - (take (- greedy-n n-in-conv)) - (into in-conv)) - in-conv))))}) - ; End of base conv update - - - -(defn max-k-fn - [data max-max-k] - (min - max-max-k - (+ 2 - (int (/ (count (nm/rownames data)) 12))))) - -(defn group-votes - "Returns a map of group-clusters ids to {:votes { {:A _ :D _ :S _}}} :n maps." - [group-clusters base-clusters votes-base] - (let [bid-to-index (zipmap (map :id base-clusters) - (range))] - (into {} - (map - (fn [{:keys [id members] :as group-cluster}] - (letfn [(count-fn [tid vote] - (->> - members - (mapv bid-to-index) - (mapv #(((votes-base tid) vote) %)) - (apply +)))] - [id - {:n-members (let [bids (set members)] - ; Add up the count of members in each base-cluster in this group-cluster - (->> base-clusters - (filter #(bids (:id %))) - (map #(count (:members %))) - (reduce + 0))) - :votes (plmb/map-from-keys - (fn [tid] - {:A (count-fn tid :A) - :D (count-fn tid :D) - :S (count-fn tid :S)}) - (keys votes-base))}])) - group-clusters)))) - - -(defn importance-metric - [A P S E] - (let [p (/ (+ P 1) (+ S 2)) - a (/ (+ A 1) (+ S 2))] - (* (- 1 p) (+ E 1) a))) - -;; This could be -;; TODO TUNE -(def meta-priority 7) - -(defn priority-metric - [is-meta A P S E] - ;; We square to deepen our bias - (matrix/pow - (if is-meta - meta-priority - (* (importance-metric A P S E) - ;; scale by a factor which lets new comments bubble up - (+ 1 (* 8 (matrix/pow 2 (/ S -5)))))) - 2)) - - -(comment - ;; testing values - (float (importance-metric 1 0 1 0)) - (priority-metric false 1 0 1 0) - (priority-metric false 20 3 20 0) - (priority-metric false 18 3 20 1) - :end-comment) - -(defn with-proj-and-extremtiy - "Compute projection and extremity and merge in to pca results" - [pca] - (let [cmnt-proj (pca/pca-project-cmnts pca) - cmnt-extremity - (mapv - (fn [row] - (matrix/length row)) - (matrix/rows cmnt-proj))] - (assoc pca - :comment-projection (matrix/transpose cmnt-proj) - :comment-extremity cmnt-extremity))) - -(def small-conv-update-graph - "For computing small conversation updates (those without need for base clustering)" - (merge - base-conv-update-graph - {:mat (plmb/fnk [rating-mat] - ; swap nils for per column average - most things need the 0s, but repness needs the nils - (let [mat (nm/get-matrix rating-mat) - ;; TODO column-averages should be a separate build task - ;; Can toggle here for nil? or (nil or = 0) - ;replace? #(or (nil? %) (= 0 %)) - replace? nil? - column-averages - (mapv - (fn [col] - (let [col (remove replace? col)] - (double - (/ (reduce + col) - (count col))))) - (matrix/columns mat))] - (mapv - (fn [row] - (mapv - (fn [i x] - (if (replace? x) (get column-averages i) x)) - (range) - row)) - mat))) - :pca (plmb/fnk [conv mat opts'] - (let [pca - (pca/wrapped-pca mat - (:n-comps opts') - :start-vectors (get-in conv [:pca :comps]) - :iters (:pca-iters opts'))] - (with-proj-and-extremtiy pca))) - - - :proj - (plmb/fnk [rating-mat pca] - (pca/sparsity-aware-project-ptpts (nm/get-matrix rating-mat) pca)) - - ;:cmnt-proj - ;(plmb/fnk [pca] - ; (pca/pca-project-cmnts pca)) - - ;; QUESTION Just have proj return an nmat? - :proj-nmat - (plmb/fnk [rating-mat proj] - (nm/named-matrix (nm/rownames rating-mat) ["x" "y"] proj)) - - :base-clusters - (plmb/fnk [conv proj-nmat in-conv opts'] - (let [in-conv-mat (nm/rowname-subset proj-nmat in-conv)] - (sort-by :id - (clusters/kmeans in-conv-mat - (:base-k opts') - :last-clusters (:base-clusters conv) - :max-iters (:base-iters opts'))))) - - :base-clusters-proj - (plmb/fnk [base-clusters] - (clusters/xy-clusters-to-nmat2 base-clusters)) - - :bucket-dists - (plmb/fnk [base-clusters-proj] - (clusters/named-dist-matrix base-clusters-proj)) - - :base-clusters-weights - (plmb/fnk [base-clusters] - (into {} - (map - (fn [clst] - [(:id clst) (count (:members clst))]) - base-clusters))) - - - ;; Here we compute the top level clusters; These are the traditional clusters we've been using for some time now. - ;; Below we'll use these as the basis for a second level of subgroup clusters - - ; Compute group-clusters for multiple k values - :group-clusterings - (plmb/fnk [conv base-clusters-weights base-clusters-proj opts'] - (plmb/map-from-keys - (fn [k] - (sort-by :id - (clusters/kmeans base-clusters-proj k - :last-clusters - ; A little pedantic here in case no clustering yet for this k - (when-let [last-clusterings (:group-clusterings conv)] - (last-clusterings k)) - :cluster-iters (:group-iters opts') - :weights base-clusters-weights))) - (range 2 (inc (max-k-fn base-clusters-proj (:max-k opts')))))) - - ; Compute silhouette values for the various clusterings - :group-clusterings-silhouettes - (plmb/fnk [group-clusterings bucket-dists] - (plmb/map-vals (partial clusters/silhouette bucket-dists) group-clusterings)) - - ; This smooths changes in cluster counts (K-vals) by remembering what the last K was, and only changing - ; after (:group-k-buffer opts') many times on a new K value - :group-k-smoother - (plmb/fnk - [conv group-clusterings group-clusterings-silhouettes opts'] - (let [{:keys [last-k last-k-count smoothed-k] :or {last-k-count 0}} - (:group-k-smoother conv) - count-buffer (:group-k-buffer opts') - ; Find best K value for current data, given silhouette - this-k (apply max-key group-clusterings-silhouettes (keys group-clusterings)) - ; If this and last K values are the same, increment counter - same (if last-k (= this-k last-k) false) - this-k-count (if same (+ last-k-count 1) 1) - ; if seen > buffer many times, switch, OW, take last smoothed - smoothed-k (if (>= this-k-count count-buffer) - this-k - (if smoothed-k smoothed-k this-k)) - ; Clamp: if smoothed-k no longer exists in this iteration's - ; clusterings (e.g. base-cluster count shrank), fall back to - ; the best available k by silhouette. - clamped-smoothed-k - (if (contains? group-clusterings smoothed-k) - smoothed-k - this-k)] - {:last-k this-k - :last-k-count this-k-count - :smoothed-k clamped-smoothed-k})) - - ; Pick the cluster corresponding to smoothed K value from group-k-smoother - :group-clusters - (plmb/fnk [group-clusterings group-k-smoother] - (get group-clusterings - (:smoothed-k group-k-smoother))) - - - ;; Now we're going to do the same thing for subclusters, or 2-level, 2-down hierarchical clustering - ;; This will more or less look the same, except that we'll have to do it for each group - - ;; Compute subgroup-clusters for each group id, for multiple k values, returning a nested map of: - ;; {:group-id {:k [clusters] ...} ...} - :subgroup-clusterings - (plmb/fnk [conv base-clusters-weights base-clusters-proj group-clusters opts'] - (into {} - ;; For each group cluster id - (map - (fn [group-cluster] - (let [gid (:id group-cluster) - group-members (:members group-cluster) - group-cluster-proj (nm/rowname-subset base-clusters-proj group-members)] - ;; For each k value... - ;; TODO Here we should really: - ;; * not return anyting if group cluster less than 10% (or some such) of population - ;; * not return anything if group cluster generally too small to subcluster (10?) - ;; * how do we handle these possibilties downstream? - [gid - (plmb/map-from-keys - (fn [k] - ;; We sort by id just to have a canonical repr - (sort-by - :id - (clusters/kmeans group-cluster-proj k - ;; This is where we grab the last-clusters from the last update's conv - :last-clusters (get-in conv [:subgroup-clusterings gid k]) ;; ok if nil - ;; TODO: Really need to properly account for what happens when group k changes... - ;; QUESTION: Add subgroup iters? For now assume same parameter value as group; level - :cluster-iters (:group-iters opts') - :weights base-clusters-weights))) - (range 2 (inc (max-k-fn group-cluster-proj (:max-k opts')))))])) - group-clusters))) - - ; Compute silhouette values for the various clusterings - :subgroup-clusterings-silhouettes - (plmb/fnk [subgroup-clusterings bucket-dists] - (plmb/map-vals - (partial - plmb/map-vals - (partial clusters/silhouette bucket-dists)) - subgroup-clusterings)) - - ;; Should call this the k-selector: - ; This smooths changes in cluster counts (K-vals) by remembering what the last K was, and only changing - ; after (:group-k-buffer opts') many times on a new K value - :subgroup-k-smoother - (plmb/fnk - [conv subgroup-clusterings subgroup-clusterings-silhouettes opts'] - (into {} - (map - (fn [[gid group-subgroup-clusterings]] - (let [group-subgroup-silhouettes (get subgroup-clusterings-silhouettes gid) - {:keys [last-k last-k-count smoothed-k] :or {last-k-count 0}} - (get-in conv [:subgroup-k-smoother gid]) - count-buffer (:group-k-buffer opts') - ; Find best K value for current data, given silhouette - this-k (apply max-key group-subgroup-silhouettes (keys group-subgroup-clusterings)) - ; If this and last K values are the same, increment counter - same (if last-k (= this-k last-k) false) - this-k-count (if same (+ last-k-count 1) 1) - ; if seen > buffer many times, switch, OW, take last smoothed - smoothed-k (if (>= this-k-count count-buffer) - this-k - (if smoothed-k smoothed-k this-k)) - ; Clamp: if smoothed-k no longer exists in THIS group's current - ; subgroup clusterings (e.g. the group's base-cluster count shrank, - ; lowering M), fall back to the best available k by silhouette. - ; Mirrors the group-k-smoother clamp added in #2536; see issue #2575. - clamped-smoothed-k - (if (contains? group-subgroup-clusterings smoothed-k) - smoothed-k - this-k)] - ;; We return a map of key-value pairs that look like this: - ;; This is maybe where we could put information about whether the last count matches for the sake of subgroups... - [gid - {:last-k this-k - :last-k-count this-k-count - :smoothed-k clamped-smoothed-k}])) - subgroup-clusterings))) - - ;; This is a little different from the group version above; - ;; For each group, we take the subgroup clustering the best silhouette, and keep it. - ;; To each of the clusters in this clustering, we assoc the :parent-id of that cluster. - ;; We end up with a map that looks like { clusters}, where clusters looks as it does for `:group-clusters`, - ;; excepting the each cluster has a `:parent-id ` attr/value pair. - :subgroup-clusters - (plmb/fnk [subgroup-clusterings subgroup-k-smoother] - (into - {} - (map - (fn [[gid group-subgroup-clusterings]] - (if-let [smoothed-k (get-in subgroup-k-smoother [gid :smoothed-k])] - (do - (log/debug "Found smoothed-k:" smoothed-k) - [gid - (map - (plmb/fn-> (assoc :parent-id gid)) - (get group-subgroup-clusterings smoothed-k))]) - (log/warn "Didn't find smoothed-k for gid:" gid))) - subgroup-clusterings))) - - - - ;; a vector of member vectors, sorted by base cluster id - :bid-to-pid (plmb/fnk [base-clusters] - (mapv :members (sort-by :id base-clusters))) - - ;; returns {tid { - ;; :agree [0 4 2 0 6 0 0 1] - ;; :disagree [3 0 0 1 0 23 0 ]} - ;; where the indices in the arrays correspond NOT directly to the bid, but to the index of the - ;; corresponding bid in a hypothetically sorted list of the base cluster ids - :votes-base (plmb/fnk [bid-to-pid raw-rating-mat] - (->> raw-rating-mat - nm/colnames - (plmb/map-from-keys - (fn [tid] - {:A (agg-bucket-votes-for-tid bid-to-pid raw-rating-mat utils/agree? tid) - :D (agg-bucket-votes-for-tid bid-to-pid raw-rating-mat utils/disagree? tid) - :S (agg-bucket-votes-for-tid bid-to-pid raw-rating-mat number? tid)})))) - - - ;; In most or all of the below, we need to do things for both groups and subgroups. However, the logic for these - ;; isn't well refactored. There should ideally be some representation of the computations operating on group - ;; clusters that naturally generalizes or can be applied to the nesting of the subgroup-clusters. However, it also - ;; raises the question of how we could be structuring the subgroup clusters data differently to facilitate simpler - ;; provessing. There's some tradeoff between ease of initial processing (dealing with ids and such) and ease of - ;; downstream processing and data consumption. Will hvae to strike a balance here... For now, again, things are a - ;; little verbose, but you can hopefully see some of the patters emerging for where these things may generalize. - - ; {gid {:votes { {A _ D _ S}}}} - :group-votes - (plmb/fnk [group-clusters base-clusters votes-base] - (group-votes group-clusters base-clusters votes-base)) - ;; ? - :subgroup-votes - (plmb/fnk [subgroup-clusters base-clusters votes-base] - (->> subgroup-clusters - (plmb/map-vals - (fn [subgroup-clusters'] - (group-votes subgroup-clusters' base-clusters votes-base))))) - - - ; {tid consensus} - :group-aware-consensus - (plmb/fnk [group-votes] - (let [tid-gid-probs - (reduce - (fn [result [gid gid-stats]] - (reduce - (fn [result [tid {:keys [A S] :or {A 0 S 0}}]] - (let [prob (/ (+ A 1.0) (+ S 2.0))] - (assoc-in result [tid gid] prob))) - result - (:votes gid-stats))) - ;; +1 acts as a dumb prior - {} - group-votes) - tid-consensus - (plmb/map-vals - (fn [tid-stats] - (->> tid-stats - (map second) - (reduce *))) - tid-gid-probs)] - tid-consensus)) - - :comment-priorities - (plmb/fnk [conv group-votes pca tids meta-tids] - (let [group-votes (:group-votes conv) - extremities (into {} (map vector tids (:comment-extremity pca)))] - (plmb/map-from-keys - (fn [tid] - (let [{:as total-votes :keys [A D S P]} - ;; reduce over votes per group, already aggregated - (reduce - (fn [votes [gid data]] - ;; not sure why we have to do the or here? how would this ever come up nil? small - ;; convs? - (let [{:as data :keys [A S D] :or {A 0 S 0 D 0}} (get-in data [:votes tid]) - data (assoc data :P (+ (- S (+ A D))))] - ;; Add in each of the data's kv count pairs - (reduce - (fn [votes' [k v]] - (update votes' k + v)) - votes - data))) - {:A 0 :D 0 :S 0 :P 0} - group-votes) - extremity (or (get extremities tid) - (do - (log/warn "No extremity for tid" tid "zid" (:zid conv)) - 0))] - ;; Pass a real boolean. (get meta-tids tid 0) defaults non-meta - ;; tids to 0, which is TRUTHY in Clojure, so priority-metric took - ;; the meta branch for every comment (regression #1961). See #2571 - ;; and delphi/docs/MATH_ALGORITHM_HISTORY.md. - (priority-metric (contains? meta-tids tid) A P S extremity))) - tids))) - - - ;; ATTENTION! The following uses of :mod-out should be ideally taking into account strict/vs non-strict - :repness - (plmb/fnk [conv rating-mat group-clusters base-clusters] - (-> (repness/conv-repness rating-mat group-clusters base-clusters) - (repness/select-rep-comments (:mod-out conv)))) - :subgroup-repness - (plmb/fnk [conv rating-mat subgroup-clusters base-clusters] - (->> subgroup-clusters - (plmb/map-vals - (fn [subgroup-clusters'] - (-> (repness/conv-repness rating-mat subgroup-clusters' base-clusters) - (repness/select-rep-comments (:mod-out conv))))))) - - :ptpt-stats - (plmb/fnk [group-clusters base-clusters proj-nmat user-vote-counts] - (repness/participant-stats group-clusters base-clusters proj-nmat user-vote-counts)) - :subgroup-ptpt-stats - (plmb/fnk [subgroup-clusters base-clusters proj-nmat user-vote-counts] - (->> subgroup-clusters - (plmb/map-vals - (fn [subgroup-clusters'] - (repness/participant-stats subgroup-clusters' base-clusters proj-nmat user-vote-counts))))) - - - :consensus - (plmb/fnk [conv rating-mat] - (-> (repness/consensus-stats rating-mat) - (repness/select-consensus-comments (:mod-out conv)))) - - ; End of large-update - #_:end})) - - - -(defn partial-pca - "This function takes in the rating matrix, the current pca and a set of row indices and - computes the partial pca off of those, returning a lambda that will take the latest PCA - and make the update on that in case there have been other mini batch updates since started" - [mat pca indices & {:keys [n-comps iters learning-rate] - :or {n-comps 2 iters 10 learning-rate 0.01}}] - (let [rating-subset (utils/filter-by-index mat indices) - part-pca (pca/powerit-pca rating-subset n-comps - :start-vectors (:comps pca) - :iters iters) - forget-rate (- 1 learning-rate) - learn (fn [old-val new-val] - (let [old-val (matrix/join old-val (repeat (- (matrix/dimension-count new-val 0) - (matrix/dimension-count old-val 0)) 0))] - (+ (* forget-rate old-val) (* learning-rate new-val))))] - (fn [pca'] - ; Actual updater lambda - {:center (learn (:center pca') (:center part-pca)) - :comps (mapv #(learn %1 %2) (:comps pca') (:comps part-pca))}))) - - -(defn sample-size-fn - "Return a function which decides how many ptpts to sample for mini-batch updates; the input - parameters correspond to a line of sample sizes to interpolate. Beyon the bounds of these - points, the sample sizes flatten out so all sample sizes lie in [start-y stop-y]" - [start-y stop-y start-x stop-x] - (let [slope (/ (- stop-y start-y) (- stop-x start-x)) - start (- (* slope start-x) start-y)] - (fn [size] - (max - (long (min (+ start (* slope size)) stop-y)) - start-y)))) -; For now... Will want this constructed with opts eventually XXX -(def sample-size (sample-size-fn 100 1500 1500 150000)) - - -(def large-conv-update-graph - "Same as small-conv-update-graph, but uses mini-batch PCA" - (merge small-conv-update-graph - {:pca (plmb/fnk [conv mat opts'] - (let [n-ptpts (matrix/dimension-count mat 0) - sample-size (sample-size n-ptpts)] - (loop [pca (:pca conv) iter (:pca-iters opts')] - (let [rand-indices (take sample-size (sampling/sample (range n-ptpts) :generator :twister)) - pca ((partial-pca mat pca rand-indices) pca)] - (if (= iter 0) - ;; Then done, but don't forget to merge in the comment extremtiy, etc - (with-proj-and-extremtiy pca) - ;; Recur - (recur pca (dec iter)))))))})) - - -(def eager-profiled-compiler - ;(comp graph/eager-compile (partial graph/profiled :profile-data)) - (comp graph/par-compile (partial graph/profiled :profile-data))) - -(def small-conv-update (eager-profiled-compiler small-conv-update-graph)) -(def large-conv-update (eager-profiled-compiler large-conv-update-graph)) - - -(defn conv-update - "This function dispatches to either small- or large-conv-update, depending on the number - of participants or comments. Uses large-conv-update if either: - - Number of participants exceeds ptpt-cutoff (default 10000) - - Number of comments exceeds cmt-cutoff (default 5000)" - ([conv votes] - (conv-update conv votes {})) - ;; TODO Need to pass through these options from all the various places where we call this function... - ;; XXX Also need to set the max globally and by conversation for plan throttling - ([conv votes {:keys [ptpt-cutoff cmt-cutoff] - :or {ptpt-cutoff 10000 - cmt-cutoff 5000} - :as opts}] - (let [zid (or (:zid conv) (:zid (first votes))) - ptpts (nm/rownames (:raw-rating-mat conv)) - cmnts (nm/colnames (:raw-rating-mat conv)) - n-ptpts (count (distinct (into ptpts (map :pid votes)))) - n-cmts (count (distinct (into cmnts (map :tid votes))))] - ;; This is a safety measure so we can call conv-update on an empty conversation after adding mod-out - ;; Note though that as long as we have a non-empty conv, updating with empty/nil votes should still trigger recompute - (if (and (= 0 n-ptpts n-cmts) - (empty? votes)) - conv - (do - (log/info (str "Starting conv-update for zid " zid ": N=" n-ptpts ", C=" n-cmts ", V=" (count votes))) - (-> - ; dispatch to the appropriate function based on either participant or comment count - ((cond - (or (> n-ptpts ptpt-cutoff) - (> n-cmts cmt-cutoff)) large-conv-update - :else small-conv-update) - {:conv conv :votes votes :opts opts}) - ;; This seems hackish... XXX - ; Remove the :votes key from customs; not needed for persistence - (assoc-in [:customs :votes] []) - (dissoc :keep-votes))))))) - - - -(defn conv-shape - [conv] - (matrix/shape (:raw-rating-mat conv))) - -(defn not-smaller? - [conv1 conv2] - (let [[p1 c1] (conv-shape conv1) - [p2 c2] (conv-shape conv2)] - (and (>= p1 p2) (>= c1 c2)))) - -(s/fdef conv-update - :args (s/cat :conv ::conversation :votes (s/* ::vote)) - :ret ::full-conversation - :fn (s/and #(not-smaller? (:ret %) (-> % :args :conv)))) - -(comment - (require '[clojure.spec.gen :as gen]) - (gen/generate (s/gen ::conversation)) - :end) - - - -;; TODO The language here, in the poller, the config, the conv structure, etc should all reflect that this isn't just moderation but also is_meta -(defn mod-update - "Take a conversation record and a seq of moderation data and updates the conversation's mod-out attr" - [conv mods] - (try - ;; We use reduce here instead of a (->> % filter into) because we need to make sure that we are - ;; processing things in order for mod in then out vs out then in - (let [mod-out - (reduce - (fn [mod-out {:keys [tid is_meta mod]}] - (if (or is_meta (= mod -1)) - (conj mod-out tid) - (disj mod-out tid))) - (set (:mod-out conv)) - mods) - mod-in - (reduce - (fn [mod-in {:keys [tid is_meta mod]}] - (if (or is_meta (= mod 1)) - (conj mod-in tid) - (disj mod-in tid))) - (set (:mod-in conv)) - mods) - meta-tids - (reduce - (fn [meta-tids {:keys [tid is_meta]}] - (if is_meta - (conj meta-tids tid) - (disj meta-tids tid))) - (set (:meta-tids conv)) - mods)] - (-> conv - (assoc :mod-out mod-out - :mod-in mod-in - :meta-tids meta-tids) - (update :last-mod-timestamp #(apply max (or % 0) (map :modified mods))))) - (catch Exception e - (log/error "Problem running mod-update with mod-out:" (:mod-out conv) "and mods:" mods ":" e) - (.printStackTrace e) - conv))) - - -;; Creating some overrides for how core.matrix instances are printed, so that we can read them back via our -;; edn reader - -(def ^:private ipv-print-method (get (methods print-method) clojure.lang.IPersistentVector)) - -(defmethod print-method mikera.matrixx.Matrix - [o ^java.io.Writer w] - (.write w "#mikera.matrixx.Matrix ") - (ipv-print-method - (mapv #(into [] %) o) - w)) - -(defmethod print-method mikera.vectorz.Vector - [o ^java.io.Writer w] - (.write w "#mikera.vectorz.Vector ") - (ipv-print-method o w)) - -(defmethod print-method mikera.arrayz.NDArray - [o ^java.io.Writer w] - (.write w "#mikera.arrayz.NDArray ") - (ipv-print-method o w)) - - -(defn misc-reader [type] - (fn [& args] - {:type type - :args args})) - -; a reader that uses these custom printing formats -(defn read-vectorz-edn [text] - (edn/read-string - {:readers {'mikera.vectorz.Vector matrix/matrix - 'mikera.arrayz.NDArray matrix/matrix - 'mikera.matrixx.Matrix matrix/matrix - 'polismath.named-matrix.NamedMatrix nm/named-matrix-reader - 'object (misc-reader :object) - 'error (misc-reader :error)}} - text)) - - - -(defn conv-update-dump - "Write out conversation state, votes, computational opts and error for debugging purposes." - [conv votes & [opts error]] - (spit (str "errorconv." (. System (nanoTime)) ".edn") - (prn-str - {:conv (into {} - (assoc-in conv [:pca :center] (matrix/matrix (into [] (:center (:pca conv)))))) - :votes votes - :opts opts - :error (str error)}))) - - -(defn load-conv-update [filename] - (read-vectorz-edn (slurp filename))) - - -:ok - diff --git a/math/src/polismath/math/corr.clj b/math/src/polismath/math/corr.clj deleted file mode 100644 index cf18053082..0000000000 --- a/math/src/polismath/math/corr.clj +++ /dev/null @@ -1,247 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.corr - (:refer-clojure :exclude [* - + == /]) - (:require - [taoensso.timbre :as log] - ;[plumbing.core :as pc - ; :refer (fnk map-vals <-)] - ;[plumbing.graph :as gr] - ;[clojure.tools.trace :as tr] - ;[polismath.utils :as utils] - ;[polismath.math.stats :as stats] - [polismath.math.named-matrix :as nm] - [clojure.spec.alpha :as s] - [clojure.core.matrix :as matrix] - ;[clojure.core.matrix.stats :as matrix-stats] - [clojure.core.matrix.selection :as matrix.selection] - [clojure.core.matrix.operators :refer :all] - ;; - ;[incanter.charts :as charts] - ;[polismath.conv-man :as conv-man] - [clojure.set :as set] - [polismath.components.postgres :as postgres])) - ;; Can't use ic-stats anymore; It's turned to shit and can't be imported in clj/deps.edn - ;[incanter.stats :as ic-stats])) - - - -;; The following is a naive hclust implementation for clustering comments using the rating matrix. -;; This is so that we can rearrange a correlation matrix according to the hclust ordering for one of them slick heatmap things. -;; This is not a particularly efficient algorithm, but it gets the job done for now. - -(defn -hclust - "Implements the inner recursive agglomeration step for the hclust function." - ([clusters distances] - (if (= (count clusters) 1) - clusters - (let [[c1 c2] (->> (for [c1 clusters - c2 clusters - ;; Make sure we don't traverse [c c], or both [x y] and [y x] - :when (< (:id c1) (:id c2))] - [c1 c2]) - (apply min-key - (fn [[c1 c2]] - (let [key [(:id c1) (:id c2)]] - (if-let [dist (get @distances key)] - dist - (let [dist (matrix/distance (:center c1) (:center c2))] - (swap! distances assoc key dist) - dist)))))) - size (+ (:size c1) (:size c2)) - clusters (-> clusters - (->> (remove #{c1 c2})) - (conj {:id (inc (apply max (map :id clusters))) - :children [c1 c2] - :members (concat (:members c1) (:members c2)) - :distance (get @distances [(:id c1) (:id c2)]) - :center (/ (+ (* (:size c1) (:center c1)) - (* (:size c2) (:center c2))) - size) - :size size}))] - (-hclust clusters)))) - ([clusters] - (-hclust clusters (atom {})))) - - -(defn hclust - "Performs hclust on a named matrix" - [nmatrix] - (log/debug "hclust on matrix of shape" (matrix/shape (nm/get-matrix nmatrix))) - (-hclust - (mapv (fn [id row] - {:center row :id id :size 1 :members [id]}) - (nm/rownames nmatrix) - (matrix/rows (nm/get-matrix nmatrix))))) - - -(defn flatten-hclust - "Extracts out the tip/leaf node ordering from the hclust results. - (This is rendered somewhat irrelevant by the fact that the hclust algorithm now tracks this data using :members attr)" - [clusters] - (mapcat - (fn [cluster] - (if-let [children (:children cluster)] - (flatten-hclust children) - [(:id cluster)])) - clusters)) - - -(defn blockify-corr-matrix - "Rearrange the given correlation matrix ({:comments :matrix}) such that" - [corr-matrix clusters] - (let [comments (:comments corr-matrix) - matrix (:matrix corr-matrix) - members (:members (first clusters)) - member-indices (mapv #(.indexOf comments %) members)] - {:matrix (matrix.selection/sel matrix member-indices member-indices) - :comments members})) - - -;; Need to remove nulls to compute cov/corr - -;; Ideally we'd have something like a null senstive implementation of corr/cov? Restrict to overlapping dimensions? -;; What to do about 0 variance comments (all pass/null)? -;; These return NaN; treat as 0? - - -;; These things really should be in the named matrix namespace? -(defn cleaned-nmat [nmat] - (nm/named-matrix - (nm/rownames nmat) - (nm/colnames nmat) - (matrix/matrix - (mapv (fn [row] (map #(or % 0) row)) - (matrix/rows (nm/get-matrix nmat)))))) - -(defn transpose-nmat [nmat] - (nm/named-matrix - (nm/colnames nmat) - (nm/rownames nmat) - (matrix/transpose (nm/get-matrix nmat)))) - -;(defn cleaned-matrix [conv] -; (mapv (fn [row] (map #(or % 0) row)) -; (matrix/rows (nm/get-matrix (:rating-mat conv))))) - - -;; The actual correlation matrix stuff - - -;; See above -;; Can't use ic-stats anymore; It's turned to shit and can't be imported in clj/deps.edn -;(defn correlation-matrix - ;[nmat] - ;{:matrix (ic-stats/correlation (matrix/matrix (nm/get-matrix nmat))) - ;:comments (nm/colnames nmat)}) - - -;; Here's some stuff for spitting out the actual results. - -(require '[cheshire.core :as cheshire]) - -(defn prepare-hclust-for-export - [clusters] - (mapv - (fn [cluster] - (-> cluster - (update :center (partial into [])))) - clusters)) - - -(defn default-tids - ;; Gonna need to get all of this under group clusters vs subgroups - [{:as conv :keys [repness consensus group-clusters rating-mat pca]}] - (let [{:keys [extremity]} pca - {:keys [agree disagree]} consensus] - (set - (concat - (map :tid (concat agree disagree)) - (mapcat - (fn [[gid gid-repness]] - (map :tid gid-repness)) - repness))))) - - -;; See above: -;; Can't use ic-stats anymore; It's turned to shit and can't be imported in clj/deps.edn -;(defn compute-corr - ;([conv tids] - ;(let [matrix (:rating-mat conv) - ;subset-matrix (if tids (nm/colname-subset matrix tids) matrix) - ;cleaned-matrix (cleaned-nmat subset-matrix) - ;transposed-matrix (transpose-nmat cleaned-matrix) - ;; note: taoensso.timbre.profiling was removed from timbre; use com.taoensso/tufte now - ;corr-mat (prof/profile :info :corr-mat (correlation-matrix cleaned-matrix)) - ;hclusters (prof/profile :info ::hclust (hclust transposed-matrix)) - ;corr-mat' (blockify-corr-matrix corr-mat hclusters) - ;corr-mat' - ;;; Prep for export... - ;(update corr-mat' :matrix (comp (partial mapv (fn [row] (into [] row))) - ;matrix/rows))] - ;corr-mat')) - ;([conv] - ;(compute-corr conv (default-tids conv)))) - - -(defn spit-json - [filename data] - (spit - filename - (cheshire/encode data))) - -(defn spit-hclust - [filename clusters] - (spit-json - filename - (prepare-hclust-for-export clusters))) - -;; We should add a little clarity on where things are regular matrices vs our {:comments :matrix} matrices for final export... -;; And probably just use named matrices everywhere -(defn spit-matrix - [filename matrix] - (spit-json - filename - (update matrix :matrix (comp (partial mapv (fn [row] (into [] row))) - matrix/rows)))) - - -;; We have metadata/tags on the argument symbols of function vars -;; So what if we built a macro that copied over entire namespaces of functions to a new ns, but binds them to some sort of default system? -;; System bind! -;; Then you can access a mirror api that has sans-1 arity versions of all the functions, so you don't have to think about system! -;; This could solve the battle between component and mount - - - - -(comment - ;; Here we're putting everything together - ;; This may not all be 100% correct, as it was copied over from the repl... but I ran through all but the spit and sanity checks pass - (require '[incanter.charts :as charts] - '[polismath.runner :as runner] - '[polismath.conv-man :as conv-man] - '[polismath.system :as system]) - ;(runner/run! system/base-system {:math-env :preprod}) - ;; Load the data for 15117 (zinvite 2ez5beswtc) - ;(def focus-id 15117) - (def focus-zinvite "36jajfnhhn") - (def focus-id 15228) - (def conv (conv-man/load-or-init (:conversation-manager runner/system) focus-id)) - - (compute-corr conv) - :ok - - ;conv - (matrix/shape (nm/get-matrix (:rating-mat conv))) - ;; Compute hclust on a cleaned, transposed rating matrix - (def tids (nm/rownames (:rating-mat conv))) - tids - (def corr-mat' (compute-corr (:darwin runner/system))) - ;; Spit this out - (spit-matrix (str focus-zinvite ".corrmat.json") corr-mat) - (spit-matrix (str focus-zinvite ".corrmat-whclust.json") corr-mat') - (spit-hclust (str focus-zinvite ".hclust.json") hclusters) - ;; All done - :endcomment) - diff --git a/math/src/polismath/math/named_matrix.clj b/math/src/polismath/math/named_matrix.clj deleted file mode 100644 index d19ace2cf5..0000000000 --- a/math/src/polismath/math/named_matrix.clj +++ /dev/null @@ -1,278 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.named-matrix - (:require [clojure.core.matrix :as matrix] - [clojure.spec.alpha :as s] - [clojure.spec.gen.alpha :as gen] - [clojure.test.check.generators :as generators]) - ;; Again, move to - (:use polismath.utils)) - - -(defprotocol AutoIncIndex - "Things which implement this are intended to be used as data structures for use as rownames or colnames. - They are basically overglorified hash maps" - (get-names [this] "In order of index, the names associated with the data") - (next-index [this] "Get the next available index") - (index [this keyname] "Get the index corresponding to keyname") - (append [this keyname] "Append keyname") - (append-many [this keynames] "Append many") - (subset [this keynames] "Subset the index, and update index/hash")) - - -(deftype IndexHash - [^java.util.Vector names ^clojure.lang.PersistentArrayMap index-hash] - AutoIncIndex - (get-names [this] (.names this)) - (next-index [this] (count (.names this))) - (index [this keyname] (index-hash keyname)) - (append [this keyname] - (if ((.index-hash this) keyname) - this - (IndexHash. - (conj (get-names this) keyname) - (assoc (.index-hash this) keyname (next-index this))))) - (append-many [this keynames] - ; potentially faster - ;(let [uniq-kns (distinct keynames) - ;new-kns (remove (.index-hash this) uniq-kns)] - ;(IndexHash. - ;(into (.names this) new-kns) - ;(into (.index-hash) - ;(map - ;#(vector %1 (+ (count (.names this)) %2)) - ;new-kns - ;(range))))) - ; much simpler - (reduce append this keynames)) - (subset [this keynames] - (let [kn-set (set keynames) - new-kns (filter kn-set (.names this))] - (IndexHash. new-kns (into {} (map vector new-kns (range))))))) - -(defn index-hash - "Construct a new IndexHash with the given keynames" - [keynames] - (let [uniq-kns (into [] (distinct keynames))] - (IndexHash. uniq-kns - (into {} (map vector uniq-kns (range)))))) - - -(defprotocol PNamedMatrix - ; should really just change to update, but for now... - (update-nmat [this values] - "Updates a nameable matrix given a seq of (rowname, colname, value) tuples") - (rownames [this] "Vector of row names, in order") - (colnames [this] "Vector of column names, in order") - (get-matrix [this] "Extract the matrix object") - ; XXX - Should probably set these up map [this key] -> (index (.row-index this) key) since that's whats needed - (get-row-index [this] "Extract the row-index object") - (get-col-index [this] "Extract the col-index object") - (rowname-subset [this names] "Get a new PNamedMatrix subsetting to just the given rownames") - (colname-subset [this names] "Get a new PNamedMatrix subsetting to just the given columns")) - - -(defn add-padding - "Adds specified value padding to 2d matrices" - [mat ^Integer dim ^Integer n & [value]] - (let [other-dim (mod (inc dim) 2) - other-dimcount (matrix/dimension-count mat other-dim)] - (case dim - 0 (let [padding (into [] (repeat other-dimcount value))] - (into mat (repeat n padding))) - 1 (let [padding (repeat n value)] - (mapv #(into % padding) mat))))) - - -(deftype NamedMatrix - [row-index col-index matrix] - PNamedMatrix - (update-nmat [this values] - ; First find the row and column names that aren't yet in the data - (let [[missing-rows missing-cols] - (reduce - (fn [[missing-rows missing-cols] [row col value]] - [(if (nil? (index (.row-index this) row)) - (conj missing-rows row) - missing-rows) - (if (nil? (index (.col-index this) col)) - (conj missing-cols col) - missing-cols)]) - [[] []] - values) - ; Construct new rowname and colname hash-indices - new-row-index (append-many (.row-index this) missing-rows) - new-col-index (append-many (.col-index this) missing-cols) - new-row-count (count (set missing-rows)) - new-col-count (count (set missing-cols))] - ; Construct a new NamedMatrix - (NamedMatrix. - new-row-index - new-col-index - ; Construct new matrix - (as-> (.matrix this) mat - (if (= 0 (matrix/dimension-count mat 1)) - ; If the matrix is empty, just create the shape needed - (matrix/coerce [[]] - (matrix/broadcast nil [new-row-count new-col-count])) - ; OW, add padding of nils for new rows/cols - (-> mat - (add-padding 1 (count (set missing-cols))) - (add-padding 0 (count (set missing-rows))))) - ; Next assoc-in all of the new votes - (reduce - (fn [mat' [row col value]] - (let [row-i (index new-row-index row) - col-i (index new-col-index col)] - (assoc-in mat' [row-i col-i] value))) - mat - values))))) - (rownames [this] (get-names (.row-index this))) - (colnames [this] (get-names (.col-index this))) - (get-matrix [this] (.matrix this)) - (get-row-index [this] (.row-index this)) - (get-col-index [this] (.col-index this)) - (rowname-subset [this names] - (let [row-indices (map (partial index (.row-index this)) names) - row-index (subset (.row-index this) names)] - (NamedMatrix. - row-index - (.col-index this) - (filter-by-index (.matrix this) row-indices)))) - (colname-subset [this names] - (let [col-indices (map (partial index (.col-index this)) names) - col-index (subset (.col-index this) names)] - (NamedMatrix. - (.row-index this) - col-index - (matrix/select (.matrix this) :all col-indices))))) - - -(defn named-matrix - "Generator function for a new named matrix" - [& [rows cols matrix]] - (NamedMatrix. - (index-hash (or rows [])) - (index-hash (or cols [])) - (or matrix [[]]))) - - - -(defn named-matrix-gen-of - [gen] - (gen/bind - (gen/tuple - (gen/such-that not-empty (gen/vector-distinct (gen/such-that pos? (gen/int)))) - (gen/such-that not-empty (gen/vector-distinct (gen/such-that pos? (gen/int))))) - (fn [[rownames colnames]] - (gen/fmap - (fn [matrix] (named-matrix rownames colnames matrix)) - (gen/vector - (gen/vector gen (count colnames)) - (count rownames)))))) - -;(gen/sample (named-matrix-gen-of (s/gen #{-1 0 1 nil}))) - - -(s/def ::NamedMatrix - (s/with-gen - (s/and - (partial satisfies? PNamedMatrix) - ;(comp get-matrix matrix/matrix?) - (comp matrix/vec? rownames) - (comp matrix/vec? colnames) - (comp distinct rownames) - (comp distinct colnames)) - ;; Silly... need way smarter generators here; And maybe actually the generator shouldn't be on this entity, since a named - ;(clojure.test.check.generators/elements) - #(named-matrix-gen-of (gen/double)))) - -;(gen/sample (s/gen ::NamedMatrix)) - - -;(gen/frequency) -;(gen/double*) -;(gen/vector-of) -;(gen/sample (gen/double* {:min 0 :max 1 :NaN? false :infinite? false}) 100) - - -(defmethod print-method NamedMatrix - [nm ^java.io.Writer w] - (.write w - (str "#polismath.named-matrix.NamedMatrix " - "{:rownames " (into [] (rownames nm)) - " :colnames " (into [] (colnames nm)) - " :matrix " (get-matrix nm) - "}"))) - -;(gen/sample (s/gen (s/cat :nm ::NamedMatrix :columns coll?))) - -(s/fdef zero-out-columns - :args (s/cat :nm ::NamedMatrix :columns coll?) - :ret ::NamedMatrix) - -(defn zero-out-columns - "Creates a new rating matrix which has had the specied columns zeroed out" - [nm columns] - (let [m (get-matrix nm) - cols (colnames nm) - rows (rownames nm) - col-index (get-col-index nm) - m' - (reduce - (fn [m' col] - ;; It's possible to is_meta out a comment that hasn't yet been repsonded to - (if-let [i (index col-index col)] - (matrix/set-column m' i 0) - m')) - m - columns)] - (named-matrix rows cols m'))) - -; Just trying to figure out how the hell function checking really works... -(require '[clojure.spec.test.alpha :as stest]) -(stest/check `zero-out-columns) - - -(s/fdef whatever - :args (s/cat :x (s/and number? #(not= ##NaN %) #(not= 0 %))) - :ret number? - :fn #(= (-> % :ret (* 2) (-> % :args :x)))) -(defn whatever - [x] - (/ 2 x)) - -;(gen/sample (s/gen number?)) -(stest/check `whatever) - - - - -(defn named-matrix-reader - [{:keys [rownames colnames matrix]}] - (named-matrix rownames colnames matrix)) - - -; Put in interface? ... - -(defn safe-rowname-subset - "This version of rowname-subset filters out negative indices, so that if not all names in row-names - are in nmat, it just subsets to the rownames that are. Should scrap other one?" - [nmat names] - (let [safe-names (filter - (partial index (get-row-index nmat)) - names)] - (rowname-subset nmat names))) - - -(defn get-row-by-name [nmat row-name] - (matrix/get-row (get-matrix nmat) (index (get-row-index nmat) row-name))) - - -(defn inv-rowname-subset [nmat row-names] - "Returns named matrix which has been subset to all the rows not in row-names" - (rowname-subset nmat - (remove (set row-names) (rownames nmat)))) - -:ok - diff --git a/math/src/polismath/math/pca.clj b/math/src/polismath/math/pca.clj deleted file mode 100644 index bc98d518fc..0000000000 --- a/math/src/polismath/math/pca.clj +++ /dev/null @@ -1,182 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.pca - (:refer-clojure :exclude [* - + == / min max]) - (:require [taoensso.timbre :as log] - [polismath.utils :as utils] - [clojure.core.match :refer [match]] - [clojure.core.matrix :as matrix] - [clojure.core.matrix.stats :as matrix-stats] - [clojure.core.matrix.operators :refer :all])) - -(matrix/set-current-implementation :vectorz) - -(set! *unchecked-math* true) - - -(defn repeatv - "Utility function for making a vector of length n composed entirely of x" - [n x] - (matrix/matrix (into [] (repeat n x)))) - - -; Should maybe hide this inside power-iteration and have that function wrap it's current -; recursive functionality -(defn xtxr - "Will need to rename this and some of the inner variables to be easier to read... - Computes an inner step of the power-iteration process" - [data start-vec] - (let [n-cols (matrix/dimension-count data 1) - curr-vec (matrix/transpose (repeatv n-cols 0))] - (loop [data (matrix/rows data) curr-vec curr-vec] - (if-let [row (first data)] - (recur (rest data) - (+ curr-vec (* (matrix/inner-product start-vec row) row))) - curr-vec)))) - - -(defn power-iteration - "This function produces the first eigenvector of data using the power iteration method with - iters iterations and starting vector start-vector (defaulting to 100 and 111111 resp)." - [data & [iters start-vector]] - ; need to clean up some of these variables names to be more descriptive - (let [iters (or iters 100) - n-cols (matrix/dimension-count data 1) - start-vector (or start-vector (repeatv n-cols 1)) - ; XXX - this add extra cols to the start vector if we have new comments... should test - start-vector (matrix/matrix - (concat start-vector - (repeatv (- n-cols (matrix/dimension-count start-vector 0)) 1)))] - (loop [iters iters start-vector start-vector last-eigval 0] - (let [product-vector (xtxr data start-vector) - eigval (matrix/length product-vector) - normed (matrix/normalise product-vector)] - (if (or (= iters 0) (= eigval last-eigval)) - normed - (recur (dec iters) normed eigval)))))) - - -(defn proj-vec - "This computes the projection of ys orthogonally onto the vector spanned by xs" - [xs ys] - (let [coeff (/ (matrix/dot xs ys) (matrix/dot xs xs))] - (* coeff xs))) - - -(defn factor-matrix - "As in the Gram-Shmidt process; we can 'factor out' the vector xs from all the vectors in data, - such that there is no remaining variance in the xs direction within the data." - [data xs] - ; If we have a zero eigenvector, it's safe to just assume that 0 should remain the matrix - (if (#{0 0.0} (matrix/dot xs xs)) - data - ; Fucking weird... sometimes getting this "can't convert to persistent vector array: inconcsistent shape" - ; error when we don't do the into [] here. Should need to though. Will have to figure out if there is some - ; better solution - (matrix/matrix (mapv #(into [] (- % (proj-vec xs %))) data)))) - - -(defn rand-starting-vec [data] - ;; Should really throw a parallelizable random number generator in the equation here... - ;; With seeds fed in and persisted... XXX - (matrix/matrix (for [x (range (matrix/dimension-count data 1))] (rand)))) - - -; Will eventually also want to add last-pcs -(defn powerit-pca - "Find the first n-comps principal components of the data matrix; iters defaults to iters of - power-iteration" - [data n-comps & {:keys [iters start-vectors]}] - (let [center (matrix-stats/mean data) - cntrd-data (- data center) - start-vectors (or start-vectors []) - data-dim (min (matrix/row-count cntrd-data) (matrix/column-count cntrd-data))] - {:center center - :comps - (loop [data' cntrd-data n-comps' (min n-comps data-dim) pcs [] start-vectors start-vectors] - ; may eventually want to return eigenvals... - (let [start-vector (or (first start-vectors) (rand-starting-vec data)) - pc (power-iteration data' iters start-vector) - pcs (conj pcs pc)] - (if (= n-comps' 1) - pcs ; return if done - (let [data' (factor-matrix data' pc) - n-comps' (dec n-comps')] - (recur data' n-comps' pcs (rest start-vectors))))))})) - - -(defn wrapped-pca - "This function gracefully handles weird edge cases inherent in the messiness of real world data" - [data n-comps & {:keys [iters start-vectors] :as kwargs}] - (match (map (partial matrix/dimension-count data) [0 1]) - [1 n-cols] - {:center (matrix/matrix (repeatv n-comps 0)) - :comps (into [(matrix/normalise (matrix/get-row data 0))] - (repeat (dec n-comps) (repeatv n-cols 0)))} - [n-rows 1] - {:center (matrix/matrix [0]) - :comps (matrix/matrix [1])} - :else - (utils/apply-kwargs powerit-pca data n-comps - (assoc kwargs :start-vectors - (if start-vectors - (map #(if (every? #{0 0.0} %) nil %) start-vectors) - nil))))) - - -(defn pca-project - "Apply the principal component projection specified by pcs to the data" - [data {:keys [comps center]}] - ; Here we map each row of data to its projection - (matrix/mmul (- data center) (matrix/transpose comps))) - - -(defn sparsity-aware-project-ptpt - [votes {:keys [comps center]}] - ; reduce into the projected point [p1, p2] - (let [n-cmnts (count votes) - [pc1 pc2] comps - [n-votes p1 p2] ; (p1, p2) is the projection we build - (reduce - ; _-n is the nth entry in _ - (fn [[n-votes p1 p2] [x-n cntr-n pc1-n pc2-n]] - ; if we have voted, do the thing - (if x-n - ; first subtract center - (let [x-n' (- x-n cntr-n)] - ; then do a step in the dot product, and inc n-votes seen - [(inc n-votes) - (+ p1 (* x-n' pc1-n)) - (+ p2 (* x-n' pc2-n))]) - ; ... ow (if haven't voted) return what was there - [n-votes p1 p2])) - [0 0.0 0.0] - (utils/zip votes center pc1 pc2))] - ; Now scale the projection by the following value, which pushes us out from the center - (* (Math/sqrt (/ n-cmnts (max n-votes 1))) - [p1 p2]))) - - -(defn sparsity-aware-project-ptpts - "Like pca-project, but only performs the projection on non-null values, and scales the projection - according to how many responses we have gotten" - [data pca] - (mapv #(sparsity-aware-project-ptpt % pca) data)) - - -(defn pca-project-cmnts - [{:as pca :keys [comps center]}] - (let [n-cols (matrix/column-count comps)] - (sparsity-aware-project-ptpts - (map - (fn [i] - (assoc - (vec (repeat n-cols nil)) - i - -1)) - (range n-cols)) - pca))) - - -:ok - diff --git a/math/src/polismath/math/repness.clj b/math/src/polismath/math/repness.clj deleted file mode 100644 index 79b59071fa..0000000000 --- a/math/src/polismath/math/repness.clj +++ /dev/null @@ -1,417 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.repness - (:require [polismath.utils :as utils] - [polismath.math.stats :as stats] - [polismath.math.named-matrix :as nm] - [polismath.math.clusters :as clusters] - [clojure.spec.alpha :as s] - [clojure.core.matrix :as mat] - [clojure.core.matrix.operators :refer :all] - [clojure.tools.trace :as tr] - [plumbing.core :as pc :refer [fnk map-vals <-]] - [plumbing.graph :as gr] - [clojure.core.matrix :as matrix] - [taoensso.timbre :as log]) - ;[alex-and-georges.debug-repl :as dbr] - - (:refer-clojure :exclude [* - + == /])) - -(mat/set-current-implementation :vectorz) - - -(s/def ::repful-for - #{:agree :disagree}) - - -(s/def ::probability - (s/and number? - (partial >= 1) - (partial <= 0))) - -(s/def ::test-val - (s/and number?)) - - -(s/def ::tid int?) -(s/def ::gid int?) -(s/def ::n-success int?) -(s/def ::n-trials int?) -(s/def ::p-success - (partial s/valid? ::probability)) -(s/def ::p-test number?) - -(s/def ::repress - (s/and number? - (partial < 0))) - -(s/def ::comment-repness - (s/keys :req-un [::tid ::test-val ::repful-for ::n-success ::n-trials ::p-success ::p-test ::repness])) - -(s/def ::clustering-repness - (s/map-of ::gid - (s/* ::repness))) - - -(defn- count-votes - [votes & [vote]] - "Utility function for counting the number of votes matching `vote`. Not specifying `vote` returns - length of vote vector, sans any missed votes." - (let [filt-fn (if vote #(= vote %) identity)] - (count (filter filt-fn votes)))) - - -(def comment-stats-graphimpl - ; The graph implementation is in a closure so that we don't reify classes every time function is called. - ; This can cause PermGen space crashes. This is used for both consensus and repness comments. - (gr/eager-compile - {:na (fnk [votes] (count-votes votes -1)) - :nd (fnk [votes] (count-votes votes 1)) - :ns (fnk [votes] (count-votes votes)) - ; XXX - Change when we flip votes!!! - :pa (fnk [na ns] (/ (+ 1 na) (+ 2 ns))) - :pd (fnk [nd ns] (/ (+ 1 nd) (+ 2 ns))) - :pat (fnk [na ns] (stats/prop-test na ns)) - :pdt (fnk [nd ns] (stats/prop-test nd ns))})) - - -(defn- comment-stats - "Vote count stats for a given vote column. This vote column should represent the votes for a specific - comment and group. Comparisons _between_ groups happen later. See `(doc conv-repness)` for key details." - [vote-col] - (comment-stats-graphimpl {:votes vote-col})) - - -(defn- add-comparitive-stats - "Builds on results of comment-stats by doing comparisons _between_ groups. Args are - in group stats (as returned by comment-stats), and list of stats for rest of groups. - See `(doc conv-repness)` for key details." - [in-stats rest-stats] - (assoc in-stats - :ra (/ (:pa in-stats) - (/ (+ 1 (pc/sum :na rest-stats)) - (+ 2 (pc/sum :ns rest-stats)))) - :rd (/ (:pd in-stats) - (/ (+ 1 (pc/sum :nd rest-stats)) - (+ 2 (pc/sum :ns rest-stats)))) - :rat (stats/two-prop-test (:na in-stats) (pc/sum :na rest-stats) - (:ns in-stats) (pc/sum :ns rest-stats)) - :rdt (stats/two-prop-test (:nd in-stats) (pc/sum :nd rest-stats) - (:ns in-stats) (pc/sum :ns rest-stats)))) - - -(defn conv-repness - "Computes representativeness (repness) for a given data set (named-matrix), group clustering and base clustering, - returned as a map `{:ids __ :stats __}`. The `:stats` key maps to a sequences of results, one for each - group cluster, each of which is a sequence where the positional index corresponds to a particular - comment, and each value is a hash-map of statistics for the given group/comment. The hash keys obey the - following legend: - n=#, p=prob, r=rep, t=test, a=agree, d=disagree, s=seen - The :ids key maps to a vector of group ids in the same order they appear in the :stats sequence." - [data group-clusters base-clusters] - (when (empty? group-clusters) - (throw (IllegalArgumentException. - "conv-repness: group-clusters is nil or empty — investigate upstream why group-clusters is nil."))) - {:ids (map :id group-clusters) - :tids (nm/colnames data) - :stats - (->> group-clusters - ; XXX - Base clusters may not be necessary if we use the already computed bucket vote stats - ; A vector, where each entry is a column iterator for the matrix subset of the given group - (mapv (comp mat/columns - nm/get-matrix - (partial nm/rowname-subset data) - #(clusters/group-members % base-clusters))) - (apply - map - (fn [& vote-cols-for-groups] - (->> - (mapv comment-stats vote-cols-for-groups) - (utils/mapv-rest add-comparitive-stats)))))}) - - -(defn- beats-best-by-test? - "Returns true if a given comment/group stat hash-map has a more representative z score than - current-best-z. Used for making sure we have at least _one_ representative comment for every group, - even if none are left over after the more thorough filters." - [{:keys [rat rdt] :as comment-conv-stats} current-best-z] - (or (nil? current-best-z) - (> (max rat rdt) current-best-z))) - - -(defn- beats-best-agr? - "Like beats-best-by-test?, but only considers agrees. Additionally, doesn't focus solely on repness, - but also on raw probability of agreement, so as to ensure that there is some representation of what - people in the group agree on. Also, note that this takes the current-best, instead of just current-best-z." - [{:keys [na nd ra rat pa pat ns] :as comment-conv-stats} current-best] - (cond - ; Explicitly don't let something that hasn't been voted on at all come into repness - (= 0 na nd) - false - ; If we have a current-best by repness estimate, use the more robust measurement - (and current-best (> (:ra current-best) 1.0)) - ; XXX - a litte messy, since we're basicaly reimplimenting the repness sort function - (> (* ra rat pa pat) (apply * (map current-best [:ra :rat :pa :pat]))) - ; If we have current-best, but only by prob estimate, just shoot for something that is generally agreed upon - current-best - (> (* pa pat) (apply * (map current-best [:pa :pat]))) - ; Otherwise, accept if either repness or probability look generally good - :else - (or (stats/z-sig-90? pat) - (and (> ra 1.0) - (> pa 0.5))))) - - -(defn- passes-by-test? - "Decide whether we should count a given comment-conv-stat hash-map as being representative." - ;; Should set the significance as a input variable or env variable XXX - [{:keys [pat rat pdt rdt] :as comment-conv-stats}] - (or (and (stats/z-sig-90? rat) (stats/z-sig-90? pat)) - (and (stats/z-sig-90? rdt) (stats/z-sig-90? pdt)))) - - -(defn finalize-cmt-stats - "Formats comment/repness stats to be consumable by clients. In particular, chooses between - agree/disagree as to which is more representative, and populates a more regular structure accordingly." - [tid {:keys [na nd ns pa pd pat pdt ra rd rat rdt] :as comment-conv-stats}] - (let [[n-success n-trials p-success p-test repness repness-test repful-for] - (if (> rat rdt) - [na ns pa pat ra rat :agree] - [nd ns pd pdt rd rdt :disagree])] - {:tid tid - :n-success n-success - :n-trials n-trials - :p-success p-success - :p-test p-test - :repness repness - :repness-test (float repness-test) - :repful-for repful-for})) - - -(defn repness-metric - [{:keys [repness repness-test p-success p-test]}] - (* repness repness-test p-success p-test)) - - -(defn repness-sort - [repdata] - (sort-by - (comp - repness-metric) - repdata)) - - -(defn agrees-before-disagrees - "Always put agrees before disagrees" - [repdata] - (concat - ; Need vector so into appends, not prepends - (filter #(= :agree (:repful-for %)) repdata) - (filter #(= :disagree (:repful-for %)) repdata))) - - -(defn select-rep-comments - "Selects representative comments based on beats-best-by-test? and passes-by-test?. Always ensures - there is at least one representative comment for a given cluster. Takes the results of conv-repness - and returns a map of group cluster ids to representative comments and stats." - [{:keys [ids tids stats] :as repness-stats} & [mod-out]] - ; Reduce statistics into a results hash mapping group ids to rep comments - (->> - ; reduce with indices, so we have tids - (utils/zip tids stats) - ; Apply moderation - (remove (comp (set mod-out) first)) - (reduce - (fn [result [tid stats]] - ; Inner reduce folds data into result for each group in comment stats - ; XXX - could this be an assoc-in? - (reduce - (fn [group-result [gid comment-conv-stats]] - ; Heplper functions for building our result; abbrv. gr = group-result - (letfn [(gr-get [gr & ks] - (get-in gr (into [gid] ks))) - (gr-assoc [gr & ks-and-val] - (assoc-in gr (into [gid] (butlast ks-and-val)) (last ks-and-val)))] - (as-> group-result gr - ; First check to see if the comment data passes, and add if it does - (if (passes-by-test? comment-conv-stats) - (->> comment-conv-stats - (finalize-cmt-stats tid) - (conj (gr-get gr :sufficient)) - (gr-assoc gr :sufficient)) - gr) - ; Keep track of what the best comment so far is, even if it doesn't pass, so we always have at - ; least one comment - (if (and (empty? (gr-get gr :sufficient)) - (beats-best-by-test? comment-conv-stats (gr-get gr :best :repness-test))) - (gr-assoc gr :best (finalize-cmt-stats tid comment-conv-stats)) - gr) - ; Also keep track of best agree comment is, so we can throw that the front preferentially - (if (beats-best-agr? comment-conv-stats (gr-get gr :best-agree)) - (gr-assoc gr :best-agree (assoc comment-conv-stats :tid tid)) - gr)))) - result - (utils/zip ids stats))) - ; initialize result hash - (into {} (map #(vector % {:best nil :best-agree nil :sufficient []}) ids))) - ; If no sufficient, use best; otherwise sort sufficient and take 5 - (map-vals - (fn [{:keys [best best-agree sufficient]}] - (let [best-agree (when best-agree - ; finalize, and assoc in a :best-agree attribute, so the client can render - ; accordingly - (assoc (finalize-cmt-stats (:tid best-agree) best-agree) - :n-agree (:na best-agree) - :best-agree true)) - ; Use best agree if that's what we have; otherwise best (possibly disagree); otherwise nothing - best-head (cond - best-agree [best-agree] - best [best] - :else [])] - ; If there weren't any matches of the criteria, just take the best match, and take the best agree if - ; possible, and if not just the best general - (if (empty? sufficient) - best-head - (->> sufficient - ; Remove best agree if in list, since we'll be putting it in manually; don't need to do this - ; with best, since in any case we'd let it sort itself out - (remove #(= (:tid best-agree) (:tid %))) - (repness-sort) ; sort by our complicated repness metric - (concat (if best-agree [best-agree] [])) ; put best agree at front, if possible - (take 5) ; take the top 5 - (agrees-before-disagrees)))))))) - - -(defn consensus-stats - [data] - (->> data - nm/get-matrix - mat/columns - (map comment-stats) - (map #(assoc %2 :tid %1) (nm/colnames data)))) - - -(defn select-consensus-comments - [cons-stats & [mod-out]] - (let [stats (->> cons-stats - (remove (comp (set mod-out) :tid)) - (map - (fn [stat] - (assoc stat - :dm (* (:pd stat) (:pdt stat)) - :am (* (:pa stat) (:pat stat)))))) - format-stat - (fn [cons-for stat] - (let [[n-success p-success p-test] (if (= cons-for :agree) - [:na :pa :pat] - [:nd :pd :pdt])] - {:tid (:tid stat) - :n-success (n-success stat) - :n-trials (:ns stat) - :p-success (p-success stat) - :p-test (p-test stat)})) - top-5 (fn [cons-for] - (let [[metric prob test] (if (= cons-for :agree) - [:am :pa :pat] - [:dm :pd :pdt])] - (->> stats - (filter #(and (> (prob %) 0.5) - (stats/z-sig-90? (test %)))) - (sort-by (comp - metric)) - (take 5) - (map (partial format-stat cons-for)))))] - {:agree (top-5 :agree) - :disagree (top-5 :disagree)})) - - - -;; Participant analyses -;; ==================== - -;; We'd like to characterize various interesting participant archetypes. - -;; * bridge: individuals who have the potential of "bridging" the divide between groups -;; * high a-repness in multiple groups, or -;; * high repness in one group and high majority aggreement -;; * look for people non-trivially in the center !!! -;; * core/extremist: (are these the same thing); Those who are most solidly within one group -;; * high repness in a group -;; * low consensus rep ??? -;; * lowest possible repness out of group -;; * same as looking at how far away from center you are !!! - -;; Does it make sense to even talk about representativeness of a participant to a group? -;; I think it does. -;; And I think it naturally corresponds to the partial projection k - -;; repness to each group: This could be the partial projection for repful comments; do we restrict by positive? -;; consensus-followingness; don't know if we really need this... - -;; Summarizing: -;; -;; I’m thinking we can implement that with two new computed things (edited) -;; the first would be a representativeness score for each participant -;; and the second would be a score for how strongly each ptpt agrees with majority opinions - - -;; XXX This should probably be in the clusters ns -(defn ids->members - [clusters] - (->> clusters - (map (fn [{:keys [id members]}] - [id members])) - (into {}))) - - -(defn mat-center - [matrix] - (if (satisfies? nm/PNamedMatrix matrix) - (mat-center (nm/get-matrix matrix)) - (/ (reduce + matrix) - (matrix/row-count matrix)))) - -(defn closeness - [ptpt-position center] - (- 1 - (mat/length - (- ptpt-position center)))) - -(defn extremeness - [ptpt-position group-center group-extreme-direction] - (mat/dot (- ptpt-position group-center) - group-extreme-direction)) - -(defn participant-stats - [group-clusters base-clusters proj-nmat ptpt-vote-counts] - (let [bid->pid (ids->members base-clusters) - ;; Map group ids to participant ids (pushing through base-clusters) - global-center (mat-center proj-nmat) - group-data - (->> group-clusters - (map (fn [{:keys [id members]}] - (let [pids (apply concat (map bid->pid members)) - size (count pids) ; group size - ;; XXX Oh... actually we already have what we need to do the group centers from base; Should do that for perf - center (mat-center (nm/rowname-subset proj-nmat pids)) - extreme-direction (mat/normalise (- center global-center))] - [id {:gid id :pids pids :center center :size size :extreme-direction extreme-direction}]))) - (into {})) - ;; Compute the global center - ;; For each group, the normalized vector pointing from center to group - the-stats - (->> (vals group-data) - (mapcat (fn [{:keys [gid pids center extreme-direction]}] - (map (fn [pid] - (let [ptpt-position (nm/get-row-by-name proj-nmat pid)] - {:pid pid - :gid gid - :n-votes (get ptpt-vote-counts pid) - ;; Here we compute the stats - :centricness (closeness ptpt-position global-center) - :coreness (closeness ptpt-position center) - :extremeness (extremeness ptpt-position center extreme-direction)})) - pids))))] - the-stats)) - - -:ok - diff --git a/math/src/polismath/math/stats.clj b/math/src/polismath/math/stats.clj deleted file mode 100644 index 17399aa722..0000000000 --- a/math/src/polismath/math/stats.clj +++ /dev/null @@ -1,49 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.math.stats - ;; XXX Moe to require - (:require [plumbing.core :as pc] - [clojure.core.matrix :as matrix] - [taoensso.timbre :as log])) - - -(defn prop-test - [succ n] - (let [[succ n] (map inc [succ n])] - (* 2 - (matrix/sqrt n) - (+ (/ succ n) -0.5)))) - - -(defn two-prop-test - [succ-in succ-out pop-in pop-out] - (let [[succ-in succ-out pop-in pop-out] - (map inc [succ-in succ-out pop-in pop-out]) - pi1 (/ succ-in pop-in) - pi2 (/ succ-out pop-out) - pi-hat (/ (+ succ-in succ-out) (+ pop-in pop-out))] - (if (= pi-hat 1) - ; XXX - this isn't quite right... could actually solve this using limits. I think there is some theorem - ; that lets you take the ratio of the derivatives or something... - 0 - (/ (- pi1 pi2) - (matrix/sqrt - (* pi-hat - (- 1 pi-hat) - (+ (/ 1 pop-in) (/ 1 pop-out)))))))) - - -(defn z-sig-90? - "Test whether z-statistic is significant at %90 confidence with alternative hypothesis >" - [z-val] - (> z-val 1.2816)) - - -(defn z-sig-95? - "Test whether z-statistic is significant at %95 confidence with alternative hypothesis >" - [z-val] - (> z-val 1.6449)) - - -:ok - diff --git a/math/src/polismath/meta/metrics.clj b/math/src/polismath/meta/metrics.clj deleted file mode 100644 index b61be65b49..0000000000 --- a/math/src/polismath/meta/metrics.clj +++ /dev/null @@ -1,89 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.meta.metrics - (:require [clojure.tools.logging :as log] - [com.stuartsierra.component :as component])) - - -;; Should maybe make this component depend upon mongo so that it handles both metrics that get to mongo and -;; those sent to graphitedb. Either that or we should rename the component and namespace graphitedb... - -(defn- make-socket - "Make a datagram socket; optional port parameter is the local port for the socket. If ommitted (or if nil is passed), - the Java implementation will pick some available port and bind it." - ([] (new java.net.DatagramSocket)) - ([port] (if port (new java.net.DatagramSocket port) (make-socket)))) - -(defrecord MetricSender [config send-socket] - component/Lifecycle - (start [component] - (log/info "Starting metric sender") - (let [send-socket (make-socket (-> config :meta :graphite :local-port))] - (assoc component :send-socket send-socket))) - (stop [component] - (log/info "Closing metric sender") - (.close send-socket) - (assoc component :send-socket nil))) - -(defn- get-config - [metric-sender] - (-> metric-sender :config :meta :graphite)) - -(defn- send-data - [metric-sender data] - (let [send-socket (:send-socket metric-sender) - {:keys [hostname remote-port]} (get-config metric-sender) - ipaddress (java.net.InetAddress/getByName hostname) - send-packet (new java.net.DatagramPacket (.getBytes data) (.length data) ipaddress remote-port)] - (.send send-socket send-packet))) - -(defn- make-send-string - "All keys are name keys are prepended with 'math..'..." - [math-env api-key values] - (str api-key ".math." math-env "." (partial clojure.string/join " " values) \n)) - -(defn- send-metric-values - [metric-sender values] - (let [{:keys [api-key hostname remote-port]} (get-config metric-sender)] - (log/info "sending metric data " values " to " hostname ":" remote-port) - (send-data metric-sender - (make-send-string (-> metric-sender :config :math-env) - api-key - values)))) - -;; ## Public API - -(defn send-metric - "Sends metric using metric sender, with optional timestamp attribute" - ([metric-sender name value timestamp] - (send-metric-values metric-sender [name value timestamp])) - ([metric-sender name value] - (send-metric-values metric-sender [name value]))) - -(defmacro meter - "Macro wrapping send-metric which runs a computation and sends the metrics to graphitedb" - [metric-sender metric-name & expr] - `(let [start# (System/currentTimeMillis) - ret# ~@expr - end# (System/currentTimeMillis) - duration# (- end# start#)] - (send-metric ~metric-sender ~metric-name duration# end#) - (log/debug (str end# " " ~metric-name " " duration# " millis")) - ret#)) - -;; It looks like we can optionally get responses from graphitedb? We're not using that capability for now -;; though. - -;(defn receive-data [receive-socket] - ;(let [receive-data (byte-array 1024), - ;receive-packet (new java.net.DatagramPacket receive-data 1024)] - ;(.receive receive-socket receive-packet) - ;(new java.lang.String (.getData receive-packet) 0 (.getLength receive-packet)))) - - -;(defn make-receive [receive-port] - ;(let [receive-socket (make-socket receive-port)] - ;(fn [] (receive-data receive-socket)))) - -:ok - diff --git a/math/src/polismath/meta/microscope.clj b/math/src/polismath/meta/microscope.clj deleted file mode 100644 index 7face2d0c9..0000000000 --- a/math/src/polismath/meta/microscope.clj +++ /dev/null @@ -1,88 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.meta.microscope - (:require [polismath.conv-man :as cm] - [polismath.components.env :as env] - [polismath.components.postgres :as db] - [polismath.math.conversation :as conv] - [polismath.math.named-matrix :as nm] - [polismath.utils :as utils] - [clojure.tools.trace :as tr] - [clojure.tools.logging :as log] - [clojure.tools.cli :refer [parse-opts]] - [com.stuartsierra.component :as component] - [plumbing.core :as pc] - [korma.core :as ko] - [korma.db :as kdb] - [polismath.components.postgres :as postgres])) -; -; -; -;(defrecord Microscope [postgres mongo conversation-manager config] -; component/Lifecycle -; ;; Don't know if there will really be any important state here; -; ;; Seems like an anti-pattern and like we should just be operating on some base-system... -; (start [this] this) -; (stop [this] this)) - -;; Should really just move this to the conv-man namespace; this namespace can go for now - -(defn recompute - [{:as system :keys [conversation-manager postgres]} & {:keys [zid zinvite] :as args}] - (assert (utils/xor zid zinvite)) - (let [zid (or zid (postgres/get-zid-from-zinvite (:postgres system) zinvite)) - new-votes (db/conv-poll postgres zid 0)] - (log/info "Running a recompute on zid:" zid "(zinvite:" zinvite ")") - (cm/queue-message-batch! conversation-manager :votes zid new-votes))) - -(comment - (require '[polismath.runner :as runner :refer [system]]) - (postgres/get-zid-from-zinvite (:postgres system) "7scufp") - (recompute system :zinvite "7scufp") - :end-example-comment) - - -(defn kw->int - [kw] - (-> kw - (str) - (clojure.string/replace ":" "") - (Integer/parseInt))) - - - -;(defn replay-conv-update -; "Can be run as a shell command on a error file to replay what happened." -; [filename] -; (let [data (conv/load-conv-update filename) -; {:keys [conv votes opts]} data -; {:keys [rating-mat base-clusters pca]} conv] -; (println "Loaded conv:" filename) -; (println "Dimensions:" (count (nm/rownames rating-mat)) "x" (count (nm/colnames rating-mat))) -; (conv/conv-update conv votes))) -; -; -;(def cli-options -; [["-z" "--zid ZID" "ZID on which to do a rerun" :parse-fn #(Integer/parseInt %)] -; ["-Z" "--zinvite ZINVITE" "ZINVITE code on which to perform a rerun"] -; ["-r" "--recompute" "If set, will run a full recompute"]]) -; -; -;(defn -main -; [& args] -; (let [{:keys [options arguments errors summary]} (parse-opts args cli-options) -; conv-actor (utils/apply-kwargs recompute options) -; done? (atom false)] -; (add-watch -; (:conv conv-actor) -; :shutdown-watch -; (fn [k r o n] -; (println n) -; (swap! done? (fn [_] true)) -; (shutdown-agents))) -; (loop [] -; (Thread/sleep 1000) -; (when-not @done? -; (recur))))) -; - diff --git a/math/src/polismath/meta/notify.clj b/math/src/polismath/meta/notify.clj deleted file mode 100644 index f2e8e7cb2c..0000000000 --- a/math/src/polismath/meta/notify.clj +++ /dev/null @@ -1,59 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.meta.notify - (:require [clj-http.client :as client] - [taoensso.timbre :as log])) - - -;; Keep track of the type of notifications sent, and when they were sent, so you can avoid spamming. -;; Maps from message type to a list of timestamps. If there are more than N timestamps within T seconds, it prevents sending. Timestamps older than T seconds will be cleared. -(def team-notifications-per-message-type-throttle (atom {})) -;; Maps from zid,message-type to time last sent in millis. -(def team-notifications-per-convo (atom {})) - - -(def throttle-count 5) ; number of messages of message-type that can be sent during a throttle-time amount of time -(def throttle-time (* 60 60 1000)) -(def min-wait (* 60 60 1000)) - -(defn send-team-notification [config subject body] - (try - (let [response (client/post (str (:webserver-url config) "/notifyTeam") - {:form-params {:webserver_username (:webserver-username config) - :webserver_pass (:webserver-pass config) - :subject subject - :body body} - :content-type :json - :throw-entire-message? true})] - (log/info "notifyTeam response:\n" (with-out-str (clojure.pprint/pprint response)))) - (catch Exception e - (log/error e "failed to notifyTeam:\n")))) - - -(defn error-message-body - [error] - (let [{:as error-map :keys [cause via trace]} (Throwable->map error)] - (str "Cause: " (:cause error-map) "\n" - "Via:" (:via error-map) "\n\n" - "Trace:\n" - (apply str - (for [row trace] - (str " " row "\n")))))) - -(defn notify-team [config message-type zid subject body] - (let [now (quot (System/currentTimeMillis) 1000) - per-convo-key [zid message-type] - oldest (- now throttle-time) - trimmed-throttle-list (filter (fn [t] (> t oldest)) (get @team-notifications-per-message-type-throttle message-type)) - message-type-throttle-count (count trimmed-throttle-list)] - (if (and - (< message-type-throttle-count throttle-count) ; don't send more than 5 per hour for a given message type - (or ; don't send more than one per zid,message-type per hour - (not (contains? @team-notifications-per-convo per-convo-key)) - (>= now (+ min-wait (get @team-notifications-per-convo per-convo-key))))) - (do - (send-team-notification config subject body) - (swap! team-notifications-per-convo (fn [old] (assoc old per-convo-key now))) - (swap! team-notifications-per-message-type-throttle (fn [old] (assoc old message-type (conj trimmed-throttle-list now)))))))) - -:ok diff --git a/math/src/polismath/meta/simulation.clj b/math/src/polismath/meta/simulation.clj deleted file mode 100644 index 94fa7cdef5..0000000000 --- a/math/src/polismath/meta/simulation.clj +++ /dev/null @@ -1,270 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.meta.simulation) -; (:require [polismath.math.conversation :as conversation] -; [polismath.math.named-matrix :as nm] -; [polismath.utils :as utils] -; [clojure.tools.cli :refer [parse-opts]] -; [clojure.string :as string] -; [plumbing.core :as pc] -; [bigml.sampling [reservoir :as reservoir] -; [simple :as simple]] -; [clj-time.coerce :as time-coerce] -; [clj-time.local :as time-local])) -; -; -;(defprotocol Voteable -; (cast-vote! [this] [this cmnt] [this member cmnt])) -; -; -;(defrecord CommentVoteDist -; [p-agree p-disagree] -; Voteable -; (cast-vote! [_] -; (let [r (rand)] -; (cond -; (> r p-agree) -1 -; (> r (+ p-agree p-disagree)) 1 -; :else 0)))) -; -; -;(defn comment-vote-dist! -; "A somewhat bimodal random vote distribution generator. Could be more principaled about this, but I think -; it's fine for starters." -; ([] -; (let [p1 (rand) -; p2 (rand (- 1 p1)) -; p1-for-aggree (= (rand-int 2) 0) -; p-agree (if p1-for-aggree p1 p2) -; p-disagree (if p1-for-aggree p2 p1)] -; (CommentVoteDist. p-agree p-disagree))) -; ([p-agree] -; (let [p-disagree (rand (- 1 p-agree))] -; (CommentVoteDist. p-agree p-disagree))) -; ([p-agree p-disagree] -; (assert (<= (+ p-agree p-disagree) 1)) -; (CommentVoteDist. p-agree p-disagree))) -; -; -;(defrecord Group -; [comment-dists] -; Voteable -; (cast-vote! [_ cmnt] -; (cast-vote! (get comment-dists cmnt)))) -; -; -;(defn new-group [] -; (Group. [])) -; -; -;(defn grp-add-cmnt -; [grp & {:keys [dist]}] -; (update-in grp [:comment-dists] conj (or dist (comment-vote-dist!)))) -; -; -;(defn random-grp-i -; [grp-dists] -; (first -; (simple/sample (range (count grp-dists)) -; :weight grp-dists))) -; -; -;(defn add-member -; [conv & [grp-i]] -; (let [member (count (:members conv))] -; (-> conv -; (update-in [:members] conj (or grp-i (random-grp-i (:grp-dists conv)))) -; (update-in [:unvoted] into (for [c (range (:comments conv))] [member c]))))) -; -; -;(defn conv-add-cmnt -; [conv] -; (let [cmnt (:comments conv)] -; (-> conv -; (update-in [:groups] (partial mapv grp-add-cmnt)) -; (update-in [:unvoted] into (for [m (range (count (:members conv)))] [m cmnt])) -; (update-in [:comments] inc)))) -; -; -;(defn fn-exp -; [f n] -; (apply comp (repeat n f))) -; -; -;(defn add-members -; [conv n] -; ((fn-exp add-member n) conv)) -; -; -;(defn add-cmnts -; [conv n] -; ((fn-exp conv-add-cmnt n) conv)) -; -; -;(defn sim-conv -; [& {:keys [zid n-grps grp-dists n-ptpts n-cmnts] :or {n-grps 3 n-ptpts 0 n-comnts 0}}] -; (-> {:zid (or zid (rand-int 1000)) -; :groups (repeat n-grps (new-group)) -; :unvoted [] -; :members [] ; actually a map of member indices to group indices -; :comments 0 -; :grp-dists (or grp-dists (repeat n-grps (/ 1 n-grps)))} -; (add-members n-ptpts) -; (add-cmnts n-cmnts))) -; -; -;(defn conv-votes -; [conv & [n]] -; (let [n (or n 1)] -; (let [zid (:zid conv) -; picks (reservoir/sample (:unvoted conv) n) -; new-conv (update-in conv [:unvoted] (partial remove (set picks))) -; ; Handle the situation where we want more votes than there are :unvoted (ptpt, cmnt) pairs by having -; ; revotes -; picks (if (= (count picks) n) -; picks -; (concat picks (for [_ (range (- n (count picks)))] -; [(rand-int (count (:members conv))) -; (rand-int (:comments conv))]))) -; votes (map -; (fn [[m c]] -; (let [grp-i (get-in conv [:members m]) -; grp ((:groups conv) grp-i)] -; {:zid zid :pid m :tid c :vote (cast-vote! grp c)})) -; picks)] -; ; Return the new conversation state so it can be looped on -; [new-conv votes]))) -; -; -;(defprotocol Pollable -; (poll! [this state last-vote-timestamp])) -; -; -;(defn unzip -; [xys] -; (reduce -; (fn [[xs ys] xy] -; [(conj xs (first xy)) (conj ys (second xy))]) -; [[] []] -; xys)) -; -; -;(defn rand-ms -; [max] -; (+ (* (rand-int (int (/ max 1000))) -; 1000) -; (rand-int 1000))) -; -; -; -;(defn add-timestamps -; [[conv votes] last-vote-timestamp] -; (let [old-last-ts (or last-vote-timestamp 0) -; now (System/currentTimeMillis) -; ts-diff (- now old-last-ts)] -; [conv -; (map #(assoc % :created (+ (rand-ms ts-diff) old-last-ts)) votes)])) -; -; -;(defn new-poller -; "Takes a collection of conversation simulators, and a set of growth functions, each of which takes an argument of -; the conversation sims, and uses infomation there to decide how to modify the conversation and how to poll" -; [& {:keys [ptpt-growth-fn -; cmnt-growth-fn -; poll-count-fn]}] -; (reify -; Pollable -; (poll! [this conv last-vote-timestamp] -; (let [[new-ptpts new-cmnts n-votes] (map #(% conv) [ptpt-growth-fn cmnt-growth-fn poll-count-fn])] -; (-> conv -; ((fn-exp add-member new-ptpts)) -; ((fn-exp conv-add-cmnt new-cmnts)) -; (conv-votes n-votes) -; ; Adds :created timestamps -; (add-timestamps last-vote-timestamp)))))) -; -; -;(def simple-poller -; (new-poller -; :ptpt-growth-fn (fn [conv] 10) -; :cmnt-growth-fn (fn [conv] 1) -; :poll-count-fn (fn [conv] -; (* (inc (count (:members conv))) 2)))) -; -; -;(defn comp-poller -; [& pollers] -; (reify -; Pollable -; (poll! [this convs last-vote-timestamp] -; (let [[new-convs vote-batches] -; (unzip -; (map #(poll! %1 %2 last-vote-timestamp) pollers convs))] -; [new-convs -; (->> vote-batches -; (apply concat) -; (sort-by :created))])))) -; -; -;(defn int-opt [& args] -; (conj (into [] args) -; :parse-fn #(Integer/parseInt %))) -; -; -;(def cli-options -; [(int-opt "-i" "--poll-interval INTERVAL" "Milliseconds between randomly generated polls" :default 1000) -; (int-opt "-z" "--n-convs NUMBER" "Number of conversations to simulate" :default 3) -; (int-opt "-p" "--person-count-start COUNT" :default 4) -; (int-opt "-P" "--person-count-growth COUNT" :default 3) -; (int-opt "-c" "--comment-count-start COUNT" :default 3) -; (int-opt "-C" "--comment-count-growth COUNT" :default 1) -; ["-h" "--help"]]) -; -; -;(defn usage [options-summary] -; (->> ["Polismath simulations" -; "Usage: lein run -m polismath.simulation [options]" -; "" -; "Options:" -; options-summary] -; (string/join \newline))) -; -; -;;; Just commenting out the redis/carmine specific stuff -; -;;; See `wcar` docstring for opts -;;(def server1-conn {:pool {} -; ;:spec {}}) -;;(defmacro wcar* [& body] `(car/wcar server1-conn ~@body)) -;;(def wcar-worker* (partial car-mq/worker server1-conn)) -; -;;(defn simulate! -; ;[{:keys [n-convs poll-interval]}] -; ;(let [pollers (repeat n-convs simple-poller) -; ;poller (apply comp-poller pollers)] -; ;(loop [convs (for [i (range n-convs)] -; ;(sim-conv :n-ptpts 4 :n-cmnts 5 :zid i)) -; ;last-vote-timestamp 0 -; ;polls 0] -; ;(Thread/sleep poll-interval) -; ;(let [[new-convs votes] (poll! poller convs last-vote-timestamp) -; ;new-last-vote-timestamp (apply max (map :created votes))] -; ;(println "Simulating" (count votes)) -; ;(wcar* (car-mq/enqueue "simvotes" (vec votes))) -; ;(recur new-convs new-last-vote-timestamp (inc polls)))))) -; -;;(comment -; ;(simulate! {:n-convs 3 :poll-interval 3}) -; ;) -; -; -;;;; XXX Move to a system -;;(defn -main [& args] -; ;(println "Starting simulations") -; ;(let [{:keys [options arguments errors summary]} (parse-opts args cli-options)] -; ;(cond -; ;(:help options) (exit 0 (usage summary)) -; ;(:errors options) (exit 1 (str "Found the following errors:" \newline (:errors options))) -; ;:else (simulate! options)))) -; - diff --git a/math/src/polismath/poller.clj b/math/src/polismath/poller.clj deleted file mode 100644 index 2830c1da4b..0000000000 --- a/math/src/polismath/poller.clj +++ /dev/null @@ -1,60 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.poller - (:require [clojure.core.async :as async :refer [go go-loop >! !! config :poller) - start-polling-from (- (System/currentTimeMillis) (* (:poll-from-days-ago poller-config) 1000 60 60 24)) - polling-interval (or (-> poller-config (get message-type) :polling-interval) 1000) - {:keys [zid-blocklist zid-allowlist]} poller-config - [poll-function timestamp-key] - (get {:votes [postgres/poll :created] - :moderation [postgres/mod-poll :modified]} - message-type)] - (go-loop [last-timestamp start-polling-from] - (when-not (async/poll! kill-chan) - (log/info "Polling" message-type ">" last-timestamp) - (let [results (poll-function postgres last-timestamp) - grouped-batches (group-by :zid results) - last-timestamp (apply max 0 last-timestamp (map timestamp-key results))] - ; For each chunk of votes, for each conversation, send to the appropriate spout - (doseq [[zid batch] grouped-batches] - (when (cond zid-allowlist (get zid-allowlist zid) - zid-blocklist (not (get zid-blocklist zid)) - :else true) - ;; TODO; need to sort out how we're using blocking vs non-blocking ops for threadability - (conv-man/queue-message-batch! conversation-manager message-type zid batch))) - ; Update timestamp - (> Starting" message-type "poller component") - (let [kill-chan (async/promise-chan) - component (assoc component :kill-chan kill-chan)] - (poll component message-type) - component)) - (stop [component] - (log/info "<< Stopping" message-type "poller component") - (go (>! kill-chan :kill)) - component)) - -(defn create-poller - ([message-type] - (create-poller message-type {})) - ([message-type options] - (map->Poller (merge options {:message-type message-type})))) - - diff --git a/math/src/polismath/runner.clj b/math/src/polismath/runner.clj deleted file mode 100644 index 18ed78e812..0000000000 --- a/math/src/polismath/runner.clj +++ /dev/null @@ -1,265 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.runner - "This namespace is responsible for running systems" - (:refer-clojure :exclude [run!]) - (:require [polismath.system :as system] - ;[polismath.stormspec :as stormspec :refer [storm-system]] - [polismath.utils :as utils] - ;; TODO Replace this with the canonical clojure.tools.cli once storm is removed - [clojure.tools.cli :as cli] - [clojure.tools.namespace.repl :as namespace.repl] - [taoensso.timbre :as log] - [clojure.string :as string] - [clj-time.core :as t] - [clojure.edn :as edn] - [clj-time.coerce :as co] - [clojure.java.io :as io] - [com.stuartsierra.component :as component] - [polismath.darwin.export :as export] - [polismath.conv-man :as conv-man] - [polismath.math.conversation :as conv] - [polismath.components.postgres :as postgres] - [clojure.pprint :as pprint]) - (:import [java.util.zip ZipOutputStream ZipEntry])) - - -(defonce system nil) -;(def system nil) - -;; Should build this to be an atom, and build something that intiates this state from a config file. -;; So when you reset, it reboots all systems. - -(defn init! - ([system-map-generator config-overrides] - (alter-var-root #'system - (constantly (utils/apply-kwargs component/system-map (system-map-generator config-overrides))))) - ([system-map-generator] - (init! system-map-generator {}))) - -(defn start! [] - ;; Make sure there's _something_ to start... - (when-not system - (init! system/poller-system)) - (alter-var-root #'system component/start)) - -(defn stop! [] - (alter-var-root #'system - (fn [s] (when s (component/stop s))))) - -(defn run! - ([system-map-generator config-overrides] - ;; Make sure if we already have a system that it has been stopped - (when system (stop!)) - (init! system-map-generator config-overrides) - (start!)) - ([system-map-generator] - (run! system-map-generator {})) - ([] - (run! system/poller-system))) - -(defn system-reset! - ([] - (stop!) - (namespace.repl/refresh :after 'polismath.runner/run!))) - - -(def subcommands - {;"storm" stormspec/storm-system ;; remove... - ;"onyx" system/onyx-system ;; soon... - "update-all" system/base-system - "update" system/base-system - "poller" system/poller-system - "tasks" system/task-system - "full" system/full-system - "simulator" system/simulator-system - "export" system/export-system}) - -;; TODO Build nice cli settings forking on subcommand, and tie in sanely with options comp - -;; QUESTION How do we fork things nicely on command that get run and exit, vs long lived? - -(def cli-options - "Has the same options as simulation if simulations are run" - [["-r" "--recompute" "Recompute conversations from scratch instead of starting from most recent values"] - ["-h" "--help" "Print help and exit"] - ["-z" "--zid ZID" "ZID on which to do an export" :parse-fn #(Integer/parseInt %)] - ["-Z" "--zinvite ZINVITE" "ZINVITE code on which to perform an export"]]) - -(defn usage [options-summary] - (->> ["Usage: lein run [subcommand] [options]" - "" - (str "Subcommand options: " (string/join " " (keys subcommands))) - - "" - "Other options:" - options-summary] - (string/join \newline))) - - - -(defn update-conv - [conv-man zid] - (try - (let [conv (conv-man/load-or-init conv-man zid) - updated-conv (conv/conv-update conv []) - math-tick (postgres/inc-math-tick (:postgres conv-man) zid)] - (conv-man/write-conv-updates! conv-man updated-conv math-tick)) - (catch Exception e (log/error e (str "Unable to complete conversation update for zid " zid))))) - - -(defn update-all-convs - [{:as system :keys [conversation-manager postgres]}] - (->> - (postgres/ptpt-counts postgres) - (map :ptpt_cnt) - (pmap (partial update-conv conversation-manager)) - (doall))) - - - -;; This needs to be cleaned up and integrated somehow; Different mode though, not sure exactly how to.. -; -(defn parse-time - [s] - (cond - ;; Then assume a ms timestamp - (re-matches #"\d*" s) - ;(re-matches #"\d*" "1649721997107") - (Long/parseLong s) - ;; Then assume a string of format YYYY-MM-DD-HH-MM-SS - (re-matches #"\d*(-\d*)+" s) - (->> (clojure.string/split s #"-") - (map parse-time) - (apply t/date-time) - co/to-long) - :else - (log/error "Unable to parse time: " s "\nMust use ms since epoch timestamp, or YYYY-MM-DD-HH-MM-SS"))) - -(defn parse-times - [s] - (->> (clojure.string/split s #":") - (map parse-time))) - -;(map Long/parseLong) - - -(def export-cli-options - [["-z" "--zid ZID" "ZID on which to do an export" :parse-fn #(Integer/parseInt %)] - ["-Z" "--zinvite ZINVITE" "ZINVITE code on which to perform an export"] - ["-X" "--include-xid" "Include user xids in output"] - ["-u" "--user-id USER_ID" "Export all conversations associated with USER_ID, and place in zip file" :parse-fn #(Integer/parseInt %)] - ["-f" "--filename FILENAME" "filename" "Name of output file (should be zip for csv out)"] - ["-t" "--at-time AT_TIME" "A string of YYYY-MM-DD-HH-MM-SS (in UTC) or ms-timestamp since epoch" :parse-fn parse-time] - ["-T" "--at-times AT_TIMES" "A vector of strings of --at-time format" :parse-fn parse-times] - ["-F" "--format FORMAT" "Either csv, excel or (soon) json" :parse-fn keyword :validate [#{:csv :excel} "Must be either csv or excel"] :default :csv] - ["-M" "--update-math" "Update math"] - ["-P" "--update-postgres" "Update postgres"] - ["-h" "--help" "Print help and exit"]]) - -(defn error-msg [errors] - (str "The following errors occurred while parsing your command:\n\n" - (clojure.string/join \newline errors))) - -(defn help-msg [options] - (str "Export a conversation or set of conversations according to the options below:\n\n" - \tab - "filename" \tab "Filename (or file basename, in case of zip output, implicit or explicit" \newline - (clojure.string/join \newline - (for [opt cli-options] - (apply str (interleave (repeat \tab) (take 3 opt))))))) - -(defn run-export - [system {:as options - :keys [filename user-id at-times] - :or {filename (str "export." (System/currentTimeMillis) ".zip")}}] - (let [darwin (:darwin system)] - (cond - ;; In this case, build all exports for this particular user - user-id - ;; maybe here check if filename ends in zip and add if not; safest, and easiest... XXX - (with-open [file (io/output-stream filename) - zip (ZipOutputStream. file) - writer (io/writer zip)] - (doseq [zid (export/get-zids-for-uid darwin user-id)] - (let [zinvite (export/get-zinvite-from-zid darwin zid)] - (log/info "Now working on conv:" zid zinvite) - (export/export-conversation darwin - (assoc options - :zid zid - :zip-stream zip - :writer writer - :entry-point (str (export/zipfile-basename filename) "/" zinvite)))))) - ;; Want to print the status of the conversation at each time t in at-times - at-times - (with-open [file (io/output-stream filename) - zip (ZipOutputStream. file) - writer (io/writer zip)] - (doseq [at-time at-times] - (log/info "Now working on conv:" (:zid options) (:zinvite options)) - (export/export-conversation darwin - (assoc options - :at-time at-time - :zip-stream zip - :writer writer - :entry-point (str (export/zipfile-basename filename) "/" at-time))))) - :else - (export/export-conversation darwin options)) - (utils/exit 0 "Export complete"))) - - -(defn -main [& args] - (log/info "Runner main function executed") - ;; default to poller subcommand - (let [subcommand (or (first args) "poller") - parser-spec (if (= subcommand "export") export-cli-options cli-options) - {:as parse-results :keys [arguments options errors summary]} (cli/parse-opts args parser-spec)] - (log/info "CLI arguments and options:\n" (with-out-str (pprint/pprint (select-keys parse-results [:arguments :options])))) - (cond - ;; Help message - (:help options) - (utils/exit 0 (usage summary)) - ;; Error in parsing (this should really catch the below condition as well - errors - (utils/exit 1 (apply str "Found the following errors:\n" - (clojure.string/join "\n" errors) "\n" - usage summary)) - ;; otherwise, run the thing - :else - (let [system-map-generator (subcommands subcommand) - sys-options (if (#{"export"} subcommand) - ;; pretty much always want prod here, regardless of env variable; overrideable? - {:math-env :prod :export {:temp-dir "."}} - options) - system (system/create-and-run-system! system-map-generator sys-options)] - (case subcommand - "update-all" - (update-all-convs system) - "export" - (run-export system options) - "update" - (let [zid (:zid options)] - (println "zid is: " zid) - (update-conv system zid)) - ;; Otherwise, default to keeping the main thread spinning while the system runs - (loop [] - (Thread/sleep 1000) - (recur))))))) - - -(comment - ;(run! system/poller-system) - (run! system/base-system) - - (conv-man/load-or-init) - - ;(require '[polismath.conv-man :as conv-man]) - ;(let [conv-man (:conversation-manager system)] - ;(conv-man/queue-message-batch! conv-man )) - - (stop!) - :endcomment) - - - - diff --git a/math/src/polismath/stormspec.clj b/math/src/polismath/stormspec.clj deleted file mode 100644 index 4cb3afe3e6..0000000000 --- a/math/src/polismath/stormspec.clj +++ /dev/null @@ -1,142 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.stormspec) - ;(:import [backtype.storm StormSubmitter LocalCluster]) - ;(:require [polismath.system :as system] - ; [polismath.conv-man :as cm] - ; [polismath.components.postgres :as postgres] - ; [polismath.components.env :as env] - ; [polismath.math.named-matrix :as nm] - ; [polismath.utils :as utils] - ; [polismath.math.conversation :as conv] - ; ;[polismath.simulation :as sim] - ; [clojure.core.matrix :as matrix] - ; [clojure.string :as string] - ; [clojure.tools.logging :as log] - ; [clojure.core.async :as async :refer [go ! !! alts!! alts! chan dropping-buffer put! take!]] - ; [com.stuartsierra.component :as component] - ; [backtype.storm [clojure :as storm] [config :as storm-config]] - ; [plumbing.core :as pc]) - ;;; I don't think we need this anymore on newer Clojures, so should remove XXX - ;(:gen-class)) - - -;; XXX Maybe loading the current implementation can be a system boot step? -; XXX - storm hack. Solves issue where one process or thread has started loading vectorz, but the other -; doesn't know to wait (at least this is what seems to be the case) -;(matrix/set-current-implementation :vectorz) -;(matrix/matrix [[1 2 3] [4 5 6]]) - - -(comment - (storm/defspout poll-spout ["message-type" "zid" "batch"] - ;; Should fork timestamp key on message-type as well XXX - {:params [system-config message-type timestamp-key] :prepare true} - [conf context collector] - ;(let [last-timestamp (atom 0) - ;;; Need config for storm to be set under keys of :vote and :mod, so the spouts are configurable - ;;; more simply from this; But I guess some of these things should maybe not be constructable this - ;;; way... - ;;system (-> system-config system/base-system component/system-map component/start) - ;system (system/create-and-run-base-system! system-config) - ;(storm/spout - (let [system (system/create-and-run-base-system! system-config) - polling-interval (-> system-config :storm :spouts message-type :polling-interval) - postgres (-> system :postgres) - last-timestamp (atom (try (java.lang.Long/parseLong (:initial-polling-timestamp env/env)) (catch Exception e (log/warn "INITIAL_POLLING_TIMESTAMP not set; setting to 0") 0)))] - ;(if (= poll-fn :sim-poll) - ;(log/info "Starting sim poller!") - ;(start-sim-poller!)) - (storm/spout - (nextTuple [] - (try - (log/info "Polling" message-type ">" @last-timestamp) - (let [poll-function (get {:votes postgres/poll :moderation postgres/mod-poll} message-type) - results (poll-function postgres @last-timestamp) - grouped-batches (group-by :zid results)] - ; For each chunk of votes, for each conversation, send to the appropriate spout - (doseq [[zid batch] grouped-batches] - (storm/emit-spout! collector [message-type zid batch])) - ; Update timestamp if needed - (swap! last-timestamp - (fn [last-ts] (apply max 0 last-ts (map timestamp-key results))))) - (catch Exception e - (log/error "Exception bubbled up in poll-spout!") - (.printStackTrace e))) - (Thread/sleep polling-interval)) - (ack [id])))) - - ;; XXX Should build a recompute trigger spout, which just sends a recompute message message-type with nil batch payload - - (storm/defbolt conv-update-bolt [] - {:params [system-config] :prepare true} - [conf context collector] - (let [system (system/create-and-run-base-system! system-config) - conv-man (:conversation-manager system)] - (storm/bolt - (execute [tuple] - (try - (let [[message-type zid batch] (.getValues tuple)] - ;; XXX Need to think more about recompute... - (cm/queue-message-batch! conv-man message-type zid batch)) - (storm/ack! collector tuple) - (catch Exception e - (log/error "Exception bubbled up in conv-update-bold!") - (.printStackTrace e))))))) - - (defn make-topology - ;; Note here that we're pushing through the full system configuration as config overrides, because we need - ;; something serializable for the nodes to boot their systems from in the prepare steps - [{:keys [config] :as storm-cluster}] - (let [config (into {} config) ;; O/w have a non-serializable object - spouts {"vote-spout" (storm/spout-spec (poll-spout config :votes :created)) - "mod-spout" (storm/spout-spec (poll-spout config :moderation :modified))} - bolt-inputs (into {} (for [s (keys spouts)] [s ["zid"]]))] - (assoc storm-cluster - :topology - (storm/topology - spouts - {"conv-update" (storm/bolt-spec bolt-inputs (conv-update-bolt config))})))) - - (defn submit-topology! - [storm-cluster] - (let [{:keys [execution cluster-name workers]} (-> storm-cluster :config :storm) - cluster-submitter (case execution - :local (LocalCluster.) - :distributed (StormSubmitter.))] - (.submitTopology cluster-submitter - cluster-name - {storm-config/TOPOLOGY-DEBUG false - storm-config/TOPOLOGY-WORKERS workers} - (:topology storm-cluster)) - (assoc storm-cluster :cluster-submitter cluster-submitter))) - - (defn stop-cluster! - [{:as storm-cluster :keys [cluster-submitter]}] - (let [cluster-name (-> storm-cluster :config :storm :cluster-name)] - ;(.killTopology topology cluster-name) - (.shutdown cluster-submitter))) - - - (defrecord StormCluster [config conversation-manager topology cluster-submitter] - component/Lifecycle - (start [component] - (-> component - make-topology - submit-topology!)) - (stop [component] - (stop-cluster! component) - (assoc component :topology nil))) - - - (defn create-storm-cluster [] - (map->StormCluster {})) - - - (defn storm-system - "Creates a base-system and assocs in polismath storm worker related components." - [config-overrides] - (merge (system/base-system config-overrides) - {:storm-cluster (component/using (create-storm-cluster) [:config :conversation-manager])}))) - - diff --git a/math/src/polismath/system.clj b/math/src/polismath/system.clj deleted file mode 100644 index afa511e231..0000000000 --- a/math/src/polismath/system.clj +++ /dev/null @@ -1,78 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.system - (:require [polismath.utils :as utils] - [polismath.components [config :as config] - [postgres :as postgres] - [core-matrix-boot :as core-matrix-boot] - [logger :as logger]] - ;[server :as server]] - [polismath.conv-man :as conv-man] - [polismath.poller :as poller] - [polismath.tasks :as tasks] - [polismath.darwin.core :as darwin] - [taoensso.timbre :as log] - [clojure.tools.namespace.repl :as namespace.repl] - [clojure.string :as string] - [clojure.tools.cli :as cli] - [com.stuartsierra.component :as component])) - -(defn base-system - "This constructs an instance of the base system components, including config, db, etc." - [config-overrides] - {:config (config/create-config config-overrides) - :logger (component/using (logger/create-logger) [:config]) - :core-matrix-boot (component/using (core-matrix-boot/create-core-matrix-booter) [:config :logger]) - :postgres (component/using (postgres/create-postgres) [:config :logger]) - :conversation-manager (component/using (conv-man/create-conversation-manager) [:config :core-matrix-boot :postgres :logger])}) - -(defn poller-system - "Creates a base-system and assocs in darwin server related components." - [config-overrides] - (merge - (base-system config-overrides) - {:vote-poller (component/using (poller/create-poller :votes) [:config :postgres :conversation-manager :logger]) - :mod-poller (component/using (poller/create-poller :moderation) [:config :postgres :conversation-manager :logger])})) - -(defn task-system - [config-overrides] - (merge - (base-system config-overrides) - {:darwin (component/using (darwin/create-darwin) [:config :postgres :conversation-manager]) - :task-poller (component/using (tasks/create-task-poller) [:config :darwin :postgres :conversation-manager])})) - -(defn export-system - [config-overrides] - (log/info "export system overrides:" config-overrides) - (merge - (base-system config-overrides) - {:darwin (component/using (darwin/create-darwin) [:config :postgres :conversation-manager])})) - -(defn full-system - [config-overrides] - (merge - (poller-system config-overrides))) - ;; This is a little silly to do this here, since we can just change the commands that get called to only - ;; run the poller system, but this is more expedient for the moment, and trying to get things smoothed out - ;; for developer meetup tomorrow :grimacing: - ;(task-system config-overrides))) - -(defn onyx-system - "Creates a base-system and assocs in polismath onyx worker related components." - [config-overrides] - :TODO) - -(defn simulator-system - "Creates a base-system and assocs in a simulation engine." - [config-overrides] - :TODO) - -(defn create-and-run-system! - [system config] - (->> config system (utils/apply-kwargs component/system-map) component/start)) - -(defn create-and-run-base-system! - [config] - (create-and-run-system! base-system config)) - - diff --git a/math/src/polismath/tasks.clj b/math/src/polismath/tasks.clj deleted file mode 100644 index 97f9fee01a..0000000000 --- a/math/src/polismath/tasks.clj +++ /dev/null @@ -1,103 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.tasks - (:require - [clojure.core.async :as async :refer [go go-loop >! !! > task-record :task_data :rid (postgres/query-zid-from-rid (:postgres poller)) first :zid)] - (conv-man/queue-message-batch! (:conversation-manager poller) - :generate_report_data - zid - [(:task_data task-record)])))) - - -(defmethod dispatch-task! :generate_export_data - [{:as poller :keys [darwin]} task-record] - (log/debug "Dispatching generate_export_data for" task-record) - (let [params (assoc (:task_data task-record) :task_bucket (:task_bucket task-record)) - ;; Need to think about security implications more thoroughly before we can really include this from - ;; requests triggered by server (needs thorough audit to make sure user can't trigger this via web - ;; api request) - params (dissoc task-record :include-xid)] - (try - (let [params (darwin/parsed-params darwin params)] - (darwin/notify-of-status darwin params "processing") - (export/export-conversation darwin params) - (darwin/handle-completion! darwin params)) - (catch Exception e - (log/error e (str "Error generating export data for " task-record ":")) - (darwin/notify-of-status darwin (:task_data params) "failed"))))) - - -(defmethod dispatch-task! :update_math - [{:as poller :keys [darwin conversation-manager]} task-record] - (log/debug "Dispatching update_math task for:" task-record) - (async/thread - (conv-man/queue-message-batch! conversation-manager :votes (-> task-record :task_data :zid) []))) - - -(defn poll - [{:as poller :keys [config postgres kill-chan]}] - (log/debug ">> Initializing task poller loop") - (let [poller-config (-> config :poller) - start-polling-from (- (System/currentTimeMillis) (* (:poll-from-days-ago poller-config) 1000 60 60 24)) - polling-interval (or (-> poller-config :tasks :polling-interval) 1000)] - (async/thread - (loop [last-timestamp start-polling-from] - (log/debug "polling tasks from" last-timestamp) - (when-not (async/poll! kill-chan) ;; TODO When we upgrade clojure & clojure.core.async, should do this - (let [results (postgres/poll-tasks postgres last-timestamp) - last-timestamp (apply max 0 last-timestamp (map :created results))] - (log/debug "new last-timestamp" last-timestamp) - ;; For each chunk of votes, for each conversation, send to the appropriate spout - (doseq [task-record results] - ;; TODO; Erm... need to sort out how we're using blocking vs non-blocking ops for threadability - (log/debug "dispatch task" task-record) - (dispatch-task! poller task-record)) - (> Starting task poller component") - (let [kill-chan (async/promise-chan) - component (assoc component :kill-chan kill-chan)] - (poll component) - component)) - (stop [component] - (log/info "<< Stopping task poller component") - (go (>! kill-chan :kill)) - component)) - -(defn create-task-poller - ([] - (create-task-poller {})) - ([options] - (map->TaskPoller options))) - diff --git a/math/src/polismath/util/pretty_printers.clj b/math/src/polismath/util/pretty_printers.clj deleted file mode 100644 index 562a477f85..0000000000 --- a/math/src/polismath/util/pretty_printers.clj +++ /dev/null @@ -1,29 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.util.pretty-printers - ;; Shoud just switch to fipp - (:require [clojure.pprint :refer :all])) - - -(defn str-repeat [s n] - (apply str (repeat n s))) - - -(defn prindent [base-indent main-indent & other-strings] - (print (str-repeat " " (* 2 (+ base-indent main-indent)))) - (apply println other-strings)) - - -(defn print-repness [repness & {:keys [indent] :or {indent 0}}] - (doseq [[gid comments] repness] - (prindent indent 0 gid) - (doseq [c comments] - (prindent indent 2 c)))) - - -(defn wide-pp [x & {:keys [width] :or {width 400}}] - (binding [*print-miser-width* width - *print-right-margin* width] - (pprint x))) - - diff --git a/math/src/polismath/utils.clj b/math/src/polismath/utils.clj deleted file mode 100644 index 28c0792595..0000000000 --- a/math/src/polismath/utils.clj +++ /dev/null @@ -1,173 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns polismath.utils - (:use clojure.core.matrix) - (:require [clojure.core.matrix :as matrix] - [clojure.math.numeric-tower :as math] - [clojure.core.matrix :as mat] - [clojure.tools.trace :as tr])) - -(matrix/set-current-implementation :vectorz) - - -(defn xor - [a b] - (and - (or a b) - (not (and a b)))) - - -(defn round-to - "Round x to n demical places" - [x n] - (let [tens (math/expt 10.0 n)] - (-> (* x tens) - (math/round) - (/ tens)))) - - -(defn gets - "Like get, but gives a coll mapped from all the keys" - [m ks & [not-found]] - (mapv #(get m % not-found) ks)) - - -(defn exit [status msg] - (println msg) - (System/exit status)) - - -(defn agree? [n] - (and - (not (nil? n)) - (< n 0))) - - -(defn disagree? [n] - (and - (not (nil? n)) - (> n 0))) - - -(defmacro time2 - [tag & expr] - `(let [start# (. System (nanoTime)) - ret# ~@expr] - (println (str (System/currentTimeMillis) " " ~tag " " (/ (double (- (. System (nanoTime)) start#)) 1000000.0) " msecs")) - ret#)) - - -(defmacro f?>> - "Modified 'penguin' operator from plumbing.core, where do-it? is a function of the threaded value - instead of a static value. E.g.: (->> nums (f?>> #(even? (count %)) (map inc)))" - [do-it? & args] - `(if (~do-it? ~(last args)) - (->> ~(last args) ~@(butlast args)) - ~(last args))) - - -(defmacro f?> - "Modified 'penguin' operator from plumbing.core, where do-it? is a function of the threaded value - instead of a static value. E.g.: (-> n inc (f?> even? (* 2)))" - [arg do-it? & rest] - `(if (~do-it? ~arg) - (-> ~arg ~@rest) - ~arg)) - - -(defn zip [& xss] - "Like haskell or python zip" - ;;should we redo this like the with-indices below, using a map? - (if (> (count xss) 1) - (partition (count xss) (apply interleave xss)) - xss)) - - -(defn map-rest - "Like map, but the mapper function takes both the item and the rest of the items in the collection, - letting you operate on each item in comparison with all the others easily" - [f col] - (for [i (range (count col))] - (f (get col i) - (concat - (subvec col 0 i) - (subvec col (inc i)))))) - - -(defn mapv-rest - "Like map-rest, but returns a vector instead of a lazy seq" - [f col] - (vec (map-rest f col))) - - -;; XX This should be an env variable -(let [greedy? true] - (defn greedy [iter] - (if greedy? - (into [] iter) - iter))) - - -(defn greedy-false [iter] iter) - - -(defn ^long typed-indexof [^java.util.List coll item] - (.indexOf coll item)) - - -(defmacro endlessly [interval & forms] - `(doseq [~'x (range)] - ~@forms - (Thread/sleep ~interval))) - - -(defn with-indices [coll] - (map #(vector %1 %2) (range) coll)) - - -(defn filter-by-index [coll idxs] - (greedy-false - (let [idx-set (set idxs)] - (->> (with-indices coll) - (filter #(idx-set (first %))) - (map second))))) - - - -(defn apply-kwargs - "Takes a function f, any number of regular args, and a final kw-args argument which will be - splatted in as a final argument" - [f & args] - (apply (apply partial f (butlast args)) (apply concat (last args)))) - - -(defn hash-map-subset - "Create a new map which is given by subsetting to the given keys (ks)" - [m ks] - (let [ks (set ks)] - (into {} - (filter - (fn [[k v]] (ks k)) - m)))) - - -(defn clst-trace - ([clsts] (clst-trace "" clsts)) - ([k clsts] - (println "TRACE" k ":") - (doseq [c clsts] - (println " " c)) - clsts)) - - -(comment - (require '[cemerick.pomegranate :refer [add-dependencies]]) - (defn load-dep - [dep] - (add-dependencies :coordinates [dep] :repositories (merge cemerick.pomegranate.aether/maven-central {"clojars" "http://clojars.org/repo"}))) - (load-dep '[clj-time "0.10.0"]) - (load-dep '[clj-excel "0.0.1"])) - - -:ok - diff --git a/math/system.properties b/math/system.properties deleted file mode 100644 index 0dc726cece..0000000000 --- a/math/system.properties +++ /dev/null @@ -1 +0,0 @@ -java.runtime.version=17 \ No newline at end of file diff --git a/math/test/cluster_tests.clj b/math/test/cluster_tests.clj deleted file mode 100644 index bfdba61fb3..0000000000 --- a/math/test/cluster_tests.clj +++ /dev/null @@ -1,296 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns cluster-tests - (:use polismath.utils - test-helpers) - (:require [clojure.test :refer :all] - [plumbing.core :as pc] - [polismath.math.named-matrix :refer :all] - [polismath.math.clusters :refer :all])) - -(defn size-correct [clusters n] - ; XXX - really need to rewrite as macro - (is (= n (count clusters)))) - - -(def real-nmat (named-matrix - ["p1" "p2" "p3" "p4" "p5" "p6"] - ["c1" "c2" "c3" "c4"] - [[ 0 1 0 1] - [ 0 1 0 1] - [-1 0 -1 1] - [ 1 0 1 0] - [-1 -1 0 1] - [-1 1 0 1]])) - - -; Initialization test -(deftest init-test - (let [n-clusts 2 - ic (init-clusters real-nmat n-clusts)] - (testing "gives right number of clusters" - (size-correct ic 2)) - (testing "doesn't give duplicate clusters" - (is (not (= (:center (first ic)) (:center (second ic)))))))) - -(deftest basic-test - (let [n-clusts 2 - clusts (kmeans real-nmat n-clusts)] - (testing "gives right number of clusters" - (size-correct clusts 2)) - (testing "doesn't give duplicate clusters" - (is (not (= (:center (first clusts)) (:center (second clusts)))))) - (testing "coordinate transforms should be handled gracefully" - (let [real-nmat (named-matrix (rownames real-nmat) (colnames real-nmat) - [[0.5 0.3] - [4.3 0.5] - [2.3 8.1] - [1.2 4.2] - [1.2 2.3] - [4.3 1.2]])] - ; basically just testing here that things don't break - (is (= 2 (count (kmeans real-nmat 2 :last-clusters clusts)))))))) - -(deftest growing-clusters - (let [data (named-matrix - ["p1" "p2" "p3" "p4" "p5"] - ["c1" "c2" "c3"] - [[ 1 1 1] - [ 1 1 0] - [-1 -1 -1] - [-1 -1 0] - [-1 1 0]]) - last-clusts [{:id 1 :members ["p1" "p2"]} {:id 2 :members ["p3" "p4"]}] - clusts (kmeans data 3 :last-clusters last-clusts)] - (testing "should give the right number of clusters" - (size-correct clusts 3)))) - - -(deftest less-than-k-test - (testing "k-means on n < k items gives n clusters" - (let [data (named-matrix - ["p1" "p2"] - ["c1" "c2" "c3"] - [[ 0 1 0] - [-1 1 0]])] - (size-correct (kmeans data 3) 2)))) - - -(let [data (named-matrix - ["p1" "p2" "p3"] - ["c1" "c2" "c3"] - [[ 0 1 0] - [ 0 1 0] - [-1 1 0]])] - (deftest identical-mem-pos-test - (testing "k-means gives n-1 clusters when precisely 2 items have identical positions" - (size-correct (kmeans data 3) 2))) - - (deftest shrinking-test - (testing "k-means gives n-1 clusters even when n last-clusters have been specified - if two members have identical positions" - (let [last-clusts [{:id 1 :members ["p1"]} - {:id 2 :members ["p2"]} - {:id 3 :members ["p3"]}]] - (size-correct (kmeans data 3 :last-clusters last-clusts) 2))))) - -(deftest edge-cases (less-than-k-test) (identical-mem-pos-test) (shrinking-test)) - - -(deftest most-distal-test - (let [data (named-matrix - ["p1" "p2" "p3" "p4" "p5"] - ["c1" "c2" "c3"] - [[ 1 1 1] - [ 1 1 0] - [-1 -1 -1] - [-1 -1 0] - [-1 1 0]]) - clusts [{:id 1 :members ["p1" "p2"]} {:id 2 :members ["p3" "p4" "p5"]}] - clusts (recenter-clusters data clusts)] - (testing "correct clst-id" - (is (= (:clst-id (most-distal data clusts)) 2))) - (testing "correct member id" - (is (= (:id (most-distal data clusts)) "p5"))) - (testing "correct distance" - (is (< - (- (:dist (most-distal data clusts)) 1.37436854) - 0.0001))))) - - -(deftest uniqify-clusters-test - (let [last-clusts [{:id 1 :members ["p1"] :center [1 1 1]} - {:id 2 :members ["p2"] :center [1 1 1]} - {:id 3 :members ["p3"] :center [1 0 -1]}]] - (testing "correct size" - (size-correct (uniqify-clusters last-clusts) 2)) - (testing "correct members" - (is (some #{["p1" "p2"]} (map :members (uniqify-clusters last-clusts))))))) - - -(deftest merge-clusters-test - (let [clst1 {:id 1 :center [1 1 1] :members ["a" "b"]} - clst2 {:id 2 :center [1 0 0] :members ["c" "d"]}] - (is (= #{"a" "b" "c" "d"} - (set (:members (merge-clusters clst1 clst2))))))) - - -(let [last-clusters [{:members [1 2] :id 1} - {:members [3 4] :id 2}] - nmat (fn [rows] (named-matrix rows [:x :y] - [[1.2 0.4] - [1.0 0.3] - [-0.2 -0.4] - [-0.7 -0.7]])) - kmeanser (fn [new-data] (kmeans new-data 2 :last-clusters last-clusters))] - - (deftest missing-some-members - (let [new-data (nmat [1 5 3 4])] - (is (kmeanser new-data)) - (size-correct (kmeanser new-data) 2))) - - (deftest missing-all-members - (let [new-data (nmat [6 5 3 4])] - (is (kmeanser new-data)) - (size-correct (kmeanser new-data) 2))) - - (deftest missing-all-members-global - (let [new-data (nmat [6 5 8 7])] - (is (kmeanser new-data)) - (size-correct (kmeanser new-data) 2)))) - -(deftest missing-members - (missing-some-members) - (missing-all-members) - (missing-all-members-global)) - - -(deftest dropped-cluster - ; Goal is to split clst 2 so that one mem goes to 1 and other goes to 3; empty cluster should handle - (let [last-clusters [{:members [1 2] :id 1} - {:members [3 4] :id 2} - {:members [5 6] :id 3}] - nmat (named-matrix [1 2 3 4 5 6] [:x :y] - [[ 1.1 1.1] - [ 1.0 1.0] - [ 0.9 0.9] - [-0.9 -0.9] - [-1.0 -1.0] - [-1.1 -1.1]])] - (is (kmeans nmat 3 :last-clusters last-clusters)))) - - -; Reproducible random generator for weighted kmeans test; if this gets used anywhere else, need to move -; to a closure -(def rand-gen - (atom - (java.util.Random. 1234))) - - -(defn random-vec* - "Reproducible random vector" - [n] - (for [_ (range n)] - (.nextFloat @rand-gen))) - - -(defn rand-int* - "Reproducible random integer" - [n] - (.nextInt @rand-gen n)) - -(defn dup-matrix-from-weights - "Take a matrix and weights, and produce a new matrix with duplicate rows according to `weights` integers." - [mat weights] - (reduce - (fn [duped-mat [weight row]] - (concat duped-mat (replicate weight row))) - [] - (map vector weights mat))) - -(defn dup-rownames-from-weights - "Create a new rownames vector with names duplicated according to weights, as with dup-matrix-from-weights." - [rownames weights] - (reduce - (fn [duped-rownames [weight rowname]] - (concat duped-rownames - (for [i (range weight)] - [rowname i]))) - [] - (map vector weights rownames))) - -(defn print-mat - "Helper for looking at matrices" - [names mat] - (doseq [[n r] (map vector names mat)] - (println n ":" r))) - -(defn init-clsts - [f coll] - (->> coll - (group-by f) - (map (fn [[i mems]] {:id i :members mems})))) - -(deftest weighted-kmeans - (testing (str "Should give the same result as regular kmeans where rows have been duplicated according to a set " - "of integer weights") - (let [j 5] - (swap! rand-gen (fn [_] (java.util.Random. j))) - (let [n-uniq 20 - n-dups-max 5 - n-cmnts 3 - ; First create the base matrix, and a random set of integer weights - uniq-positions (into [] (for [_ (range n-uniq)] (random-vec* n-cmnts))) - ptpt-names (into [] (for [i (range n-uniq)] i)) - weights (into [] (for [_ (range n-uniq)] (inc (rand-int* n-dups-max)))) - weights-hash (into {} (map vector ptpt-names weights)) - deduped-data (named-matrix ptpt-names [:x :y :z] uniq-positions) - ; Use the weights to create a matrix where the rows have been duplicated according to weights - duped-positions (dup-matrix-from-weights uniq-positions weights) - duped-names (dup-rownames-from-weights ptpt-names weights) - duped-data (named-matrix duped-names [:x :y :z] duped-positions) - ; Some initial data to play with - uniq-init (recenter-clusters - deduped-data - (init-clsts #(mod % 3) ptpt-names)) - weighted-init (recenter-clusters - deduped-data - (init-clsts #(mod % 3) ptpt-names) - :weights weights-hash) - duped-init (recenter-clusters - duped-data - (init-clsts #(mod (first %) 3) duped-names)) - ; Run kmeans on the various data - weighted-clsts (kmeans deduped-data 3 - :last-clusters uniq-init - :weights weights-hash) - duped-clsts (kmeans duped-data 3 - :last-clusters duped-init)] - (is (= (setify-members weighted-clsts) - (setify-members duped-clsts :trans first))) - (doseq [[weighted-clst duped-clst] - (map vector weighted-init duped-init)] - (is (almost=? (:center weighted-clst) - (:center weighted-clst)))))))) - - -(deftest folding-test - (let [clusters [{:id 0 :members [0 1 2] :center [0.4 8.5]} - {:id 1 :members [3 4] :center [-0.4 -5.6]} - {:id 2 :members [5 6 7] :center [0.3 0.4]}]] - (testing "folding should work" - (is (= (fold-clusters clusters) - {:id [0 1 2] - :members [[0 1 2] [3 4] [5 6 7]] - :x [0.4 -0.4 0.3] - :y [8.5 -5.6 0.4] - :count [3 2 3]}))) - (testing "unfolding should get you back" - (is (= ((comp unfold-clusters fold-clusters) clusters) - clusters))))) - - -(defn -main [] - (run-tests 'cluster-tests)) - - diff --git a/math/test/conv_edge_cases_test.clj b/math/test/conv_edge_cases_test.clj deleted file mode 100644 index f51427b6d2..0000000000 --- a/math/test/conv_edge_cases_test.clj +++ /dev/null @@ -1,218 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns conv-edge-cases-test - "Tests for edge cases in the conversation update pipeline that can cause - crashes on large or evolving conversations." - (:require [clojure.test :refer [deftest testing is]] - [polismath.math.repness :as repness] - [polismath.math.named-matrix :as nm] - [polismath.math.conversation :as conversation])) - - -;; ============================================================================ -;; Bug 1: conv-repness crashes with empty group-clusters -;; -;; When group-clusters is nil or empty, (apply map f []) returns a transducer -;; (function object) instead of a sequence. Downstream code that tries to -;; iterate over :stats gets: -;; IllegalArgumentException: Don't know how to create ISeq from: clojure.core$map$fn__XXXX -;; -;; This happens in production when the k-smoother holds a stale smoothed-k -;; that no longer exists in group-clusterings (see Bug 2 below). -;; ============================================================================ - -(def test-rating-mat - (nm/named-matrix - [:p1 :p2 :p3 :p4] - [:c1 :c2 :c3] - [[-1 1 0] - [ 1 -1 1] - [-1 -1 0] - [ 1 1 -1]])) - -(def test-base-clusters - [{:id :b1 :members [:p1 :p2]} - {:id :b2 :members [:p3 :p4]}]) - - -(deftest conv-repness-with-empty-group-clusters - (testing "conv-repness throws an informative error when group-clusters is empty" - ;; Without the guard, (apply map f []) silently returns a transducer that - ;; crashes downstream with a cryptic "Don't know how to create ISeq" error. - ;; The guard throws immediately with a message pointing to the root cause. - (is (thrown-with-msg? IllegalArgumentException - #"group-clusters is nil or empty" - (repness/conv-repness test-rating-mat [] test-base-clusters)))) - - (testing "conv-repness throws an informative error when group-clusters is nil" - (is (thrown-with-msg? IllegalArgumentException - #"group-clusters is nil or empty" - (repness/conv-repness test-rating-mat nil test-base-clusters))))) - - -;; ============================================================================ -;; Bug 2: stale smoothed-k in group-k-smoother -;; -;; The k-smoother preserves the old smoothed-k for :group-k-buffer iterations -;; to dampen oscillation. But it doesn't check whether the old k still exists -;; in the current group-clusterings. When the number of non-empty base-clusters -;; drops (e.g. due to PCA rotation after a batch of new votes), the k-range -;; shrinks and (get group-clusterings old-smoothed-k) returns nil. -;; -;; We test this by feeding the smoother a conv with a previous smoothed-k=5 -;; and group-clusterings that only contain k=2,3. -;; ============================================================================ - -(deftest stale-smoothed-k-is-clamped-to-available-group-clusters - (testing "smoothed-k is clamped to an available range so group-clusters stays non-nil" - ;; Simulate: previous iteration had smoothed-k=5, but current base-cluster - ;; count only supports k=2,3 - (let [;; Minimal group clusterings for k=2 and k=3 - dummy-clustering-k2 [{:id 0 :members [:b1]} {:id 1 :members [:b2]}] - dummy-clustering-k3 [{:id 0 :members [:b1]} {:id 1 :members [:b2]} {:id 2 :members []}] - group-clusterings {2 dummy-clustering-k2 - 3 dummy-clustering-k3} - ;; Old smoother state with smoothed-k=5 (stale!) - old-smoother {:last-k 5 :last-k-count 1 :smoothed-k 5} - ;; Current smoother picks the best available k, but preserves old smoothed-k - ;; because buffer hasn't been exceeded - group-clusterings-silhouettes {2 0.6, 3 0.8} - smoother-fnk (:group-k-smoother conversation/small-conv-update-graph) - new-smoother (smoother-fnk - {:conv {:group-k-smoother old-smoother} - :group-clusterings group-clusterings - :group-clusterings-silhouettes group-clusterings-silhouettes - :opts' {:group-k-buffer 4}}) - ;; Now look up group-clusters using the smoother's smoothed-k - group-clusters (get group-clusterings (:smoothed-k new-smoother))] - - ;; With the current (unfixed) code, smoothed-k stays at 5 and the lookup returns nil. - ;; After the fix, smoothed-k should be clamped to an available k. - (testing "smoothed-k should be a key that exists in group-clusterings" - (is (contains? group-clusterings (:smoothed-k new-smoother)) - (str "smoothed-k=" (:smoothed-k new-smoother) - " not in " (keys group-clusterings)))) - - (testing "group-clusters should not be nil" - (is (some? group-clusters) - "group-clusters lookup must not return nil"))))) - - -;; ============================================================================ -;; Bug 2b: stale smoothed-k in :subgroup-k-smoother (issue #2575) -;; -;; The identical bug as Bug 2, one level down. #2536 clamped smoothed-k in -;; group-k-smoother but NOT in :subgroup-k-smoother. A group's subgroup -;; clustering runs k-means for k in (range 2 (inc M)), where M is count-based -;; on the group's base-cluster count. When group membership drops below a /12 -;; boundary, M falls; the carried smoothed-k can exceed M; and downstream -;; (get group-subgroup-clusterings smoothed-k) returns nil → an empty subgroup -;; clustering → conv-repness crash. The fix mirrors #2536: clamp the carried -;; smoothed-k to a key that exists in THIS group's current subgroup clusterings. -;; ============================================================================ - -(deftest stale-subgroup-smoothed-k-is-clamped-to-available-subgroup-clusters - (testing "subgroup smoothed-k is clamped per group so subgroup-clusters stays non-nil" - ;; Simulate: a group :g0 whose previous subgroup smoothed-k was 5, but whose - ;; current base-cluster count only supports subgroup k=2,3 (M stepped down). - (let [gid :g0 - ;; Minimal subgroup clusterings for k=2 and k=3 for this group - dummy-clustering-k2 [{:id 0 :members [:b1]} {:id 1 :members [:b2]}] - dummy-clustering-k3 [{:id 0 :members [:b1]} {:id 1 :members [:b2]} {:id 2 :members []}] - group-subgroup-clusterings {2 dummy-clustering-k2 - 3 dummy-clustering-k3} - subgroup-clusterings {gid group-subgroup-clusterings} - ;; Best available k by silhouette is 3; but buffer hasn't been exceeded, - ;; so the smoother preserves the (stale) old smoothed-k of 5. - subgroup-clusterings-silhouettes {gid {2 0.6, 3 0.8}} - old-smoother {:last-k 5 :last-k-count 1 :smoothed-k 5} - smoother-fnk (:subgroup-k-smoother conversation/small-conv-update-graph) - new-smoother (smoother-fnk - {:conv {:subgroup-k-smoother {gid old-smoother}} - :subgroup-clusterings subgroup-clusterings - :subgroup-clusterings-silhouettes subgroup-clusterings-silhouettes - :opts' {:group-k-buffer 4}}) - smoothed-k (get-in new-smoother [gid :smoothed-k]) - ;; Downstream :subgroup-clusters does exactly this lookup per group. - subgroup-clusters (get group-subgroup-clusterings smoothed-k)] - - ;; With the current (unfixed) code, smoothed-k stays at 5 and the lookup returns nil. - ;; After the fix, smoothed-k should be clamped to an available k. - (testing "smoothed-k should be a key that exists in this group's subgroup clusterings" - (is (contains? group-subgroup-clusterings smoothed-k) - (str "smoothed-k=" smoothed-k - " not in " (keys group-subgroup-clusterings)))) - - (testing "subgroup-clusters lookup should not be nil" - (is (some? subgroup-clusters) - "subgroup-clusters lookup must not return nil"))))) - - -;; ============================================================================ -;; Bug 3 (colleague's fix): agg-bucket-votes-for-tid with unknown pids -;; -;; When base-cluster members include pids not present in the rating matrix -;; (e.g. after incremental updates where clusters lag behind the matrix), -;; the pid-to-row lookup returns nil, and (get person-rows nil) can fail -;; depending on the matrix implementation. -;; ============================================================================ - -(deftest agg-bucket-votes-unknown-pid - (testing "agg-bucket-votes-for-tid handles pids not in the rating matrix" - (let [;; rating-mat only has :p1 and :p2 - rating-mat (nm/named-matrix - [:p1 :p2] - [:c1 :c2] - [[-1 1] - [ 1 0]]) - ;; but bid-to-pid references :p3 which doesn't exist in rating-mat - bid-to-pid [[:p1 :p3] [:p2]] - result (conversation/agg-bucket-votes-for-tid - bid-to-pid rating-mat number? :c1)] - ;; Should not throw; :p3's vote is nil, filtered out by number? - (is (vector? result)) - ;; bucket 0 has :p1 (voted) and :p3 (unknown → nil, filtered out) → count 1 - (is (= 1 (first result))) - ;; bucket 1 has :p2 (voted) → count 1 - (is (= 1 (second result)))))) - - -;; ============================================================================ -;; Bug 4: comment-priorities collapsed every comment to META_PRIORITY^2 (=49) -;; -;; #1961 (2025-03-15) changed the meta-tid lookup in the :comment-priorities fnk -;; from (meta-tids tid) to (get meta-tids tid 0). For a non-meta tid, -;; (get meta-tids tid 0) returns 0 — and 0 is TRUTHY in Clojure — so -;; priority-metric took the meta branch for EVERY comment, collapsing all -;; priorities to meta-priority^2 = 49 and degrading routing to uniform-random. -;; Fixed by passing a real boolean: (contains? meta-tids tid). See #2571. -;; ============================================================================ - -(deftest comment-priorities-only-meta-tids-get-meta-priority - (testing "only genuine meta tids get meta-priority^2; non-meta tids get varied importance-based priorities" - (let [priorities-fnk (:comment-priorities conversation/small-conv-update-graph) - tids [1 2 3] - meta-tids #{2} ; only tid 2 is a meta comment - group-votes {0 {:votes {1 {:A 5 :D 1 :S 8} - 2 {:A 3 :D 0 :S 6} - 3 {:A 1 :D 2 :S 7}}} - 1 {:votes {1 {:A 2 :D 1 :S 5} - 2 {:A 1 :D 1 :S 4} - 3 {:A 4 :D 0 :S 9}}}} - conv {:zid 1 :group-votes group-votes} - pca {:comment-extremity [0.5 1.2 0.8]} ; one per tid, in tids order - meta-priority-sq (double (* conversation/meta-priority conversation/meta-priority)) - priorities (priorities-fnk {:conv conv - :group-votes group-votes - :pca pca - :tids tids - :meta-tids meta-tids})] - (testing "the meta tid gets exactly meta-priority^2" - (is (== meta-priority-sq (double (get priorities 2))))) - ;; Regression guard for #1961: with the truthy-0 bug, non-meta tids also - ;; hit the meta branch and returned meta-priority^2. - (testing "non-meta tids do NOT get meta-priority^2" - (is (not (== meta-priority-sq (double (get priorities 1))))) - (is (not (== meta-priority-sq (double (get priorities 3)))))) - (testing "non-meta priorities are varied, not a single constant" - (is (not (== (double (get priorities 1)) (double (get priorities 3))))))))) diff --git a/math/test/conv_man_tests.clj b/math/test/conv_man_tests.clj deleted file mode 100644 index 772638a303..0000000000 --- a/math/test/conv_man_tests.clj +++ /dev/null @@ -1,62 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns conv-man-tests - (:use polismath.utils - test-helpers) - (:require [clojure.test :refer :all] - ;[clojure.test.check :as tc] - ;[clojure.test.check.generators :as gen] - ;[clojure.test.check.properties :as prop] - [clojure.core.async :as async] - [clojure.tools.logging :as log] - [polismath.math.conversation :as conv] - [polismath.math.named-matrix :as nm] - [polismath.conv-man :as conv-man] - [polismath.system :as system] - [polismath.components.config :as config])) - - - -;(gen/sample gen/->Generator - -;(let [random-matrix (gen/vector (partial gen/vector (ge - -;; Should make this generative -(deftest integration-test - (testing "of some votes" - (let [messages [{:vote 0 :pid 0 :tid 0 :zid 0 :created 9000} - {:vote 1 :pid 1 :tid 1 :zid 0 :created 9003} - {:vote -1 :pid 2 :tid 0 :zid 0 :created 9005} - {:vote -1 :pid 1 :tid 2 :zid 0 :created 9009}] - config {:math-env :test - :poller {:initial-polling-timestamp (System/currentTimeMillis)}}] - (testing "in a new conv" - (let [system (system/create-and-run-base-system! config) - conv-man (:conversation-manager system) - result-chan (async/chan) - ;; TODO Need to hook this up - error-chan (async/chan)] - ;; Since the conversation manager is asynchronous, we'll create a channel where we'll put the - ;; results of the conversation update. - ;; Should add an error listener as well... so we can catch information there as well - (conv-man/add-listener! conv-man 0 (fn [conv] (async/>!! result-chan conv))) - (conv-man/queue-message-batch! conv-man :votes 0 messages) - ;; For right now, let's just use a timeout to check for whether the conversation updated; Really - ;; need to fix this... XXX - (is (if-let [conv (first (async/alts!! [result-chan (async/timeout 20000)]))] - (let [matrix (-> conv :rating-mat nm/get-matrix)] - (is (m=? [[0 nil nil] - [nil 1 -1] - [-1 nil nil]] - matrix) - "The conversation does not have the correct rating matrix post conversation update") - (not (nil? conv))) - ;; There's a problem; the timeout was hit - (do (log/error "There was an update timeout in integration-test; check log...") - (/ 1 0))))))))) - - -(defn -main [] - (run-tests 'conv-man-tests)) - - diff --git a/math/test/conversation_test.clj b/math/test/conversation_test.clj deleted file mode 100644 index ddc4142e3f..0000000000 --- a/math/test/conversation_test.clj +++ /dev/null @@ -1,174 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns conversation-test - (:require [clojure.test :refer :all] - [polismath.math.conversation :as conversation] - [polismath.math.named-matrix :as named-matrix] - [clojure.tools.trace :as tr])) - - -(def rat-mat - (named-matrix/named-matrix [:a :b] [:x :y] [[1 0] [-1 1]])) - - -(let [empty-conv {:raw-rating-mat (named-matrix/named-matrix)} - small-conv {:raw-rating-mat rat-mat - :rating-mat rat-mat} - - single-vote [{:created 500 :pid :a, :tid :x, :vote 3}] - wonky-votes [{:created 600 :pid :b, :tid :x, :vote 0} - {:created 700 :pid :a, :tid :y, :vote 1}] - several-votes (concat single-vote wonky-votes) - full-votes (for [pid [:a :b :c] tid [:x :y :z]] - {:created 100 :pid pid :tid tid :vote (rand)}) - - big-conv-matrix - (named-matrix/named-matrix - [:p1 :p2 :p3 :p4 :p5] - [:c1 :c2 :c3 :c4] - [[ 0 1 0 -1 1] - [-1 0 -1 -1 0] - [ 1 0 1 0 1] - [ 1 -1 0 -1 0] - [-1 -1 0 1 0]]) - big-conv {:raw-rating-mat big-conv-matrix - :rating-mat big-conv-matrix - :pca {:center [0.0 -0.2 -0.0 0.4] ;actually calculated this... - :comps [[0.4 0.2 -0.3 0.7] ;this is faked... - [0.1 -0.5 0.2 0.2]]}}] - - (deftest init-conv-update-test - (testing "empty matrix and one vote" - (is (conversation/conv-update empty-conv single-vote))) - (testing "empty matrix and no vote" - (is (conversation/conv-update empty-conv []))) - (testing "empty matrix and several votes" - (is (conversation/conv-update empty-conv several-votes))) - (testing "empty matrix and full votes" - (is (conversation/conv-update empty-conv full-votes))) - (testing "empty matrix and wonky votes" - (is (conversation/conv-update empty-conv wonky-votes)))) - - ; Note that this assumes that a small conversation will actually use the small-conv-update implementation - (deftest small-conv-update-test - (testing "small matrix and one vote" - (is (conversation/conv-update small-conv single-vote))) - (testing "small matrix and no vote" - (is (conversation/conv-update small-conv []))) - (testing "small matrix and several votes" - (is (conversation/conv-update small-conv several-votes))) - (testing "small matrix and full votes" - (is (conversation/conv-update small-conv full-votes))) - (testing "small matrix and wonky votes" - (is (conversation/conv-update small-conv wonky-votes)))) - - - (let [votes-base-fnk (:votes-base conversation/small-conv-update-graph) - group-votes-fn (fn [conv] - ((:group-votes conversation/small-conv-update-graph) - (assoc conv :votes-base (votes-base-fnk conv)))) - - group-clusters [{:id :g1 :members [:b1 :b2]} {:id :g2 :members [:b3 :b4 :b5]}] - base-clusters (mapv (fn [[id m]] {:id id :members [m]}) - [[:b1 :p1] [:b2 :p2] [:b3 :p3] [:b4 :p4] [:b5 :p5]]) - - conv (assoc big-conv - :group-clusters group-clusters - :base-clusters base-clusters) - conv (assoc conv - :bid-to-pid ((:bid-to-pid conversation/small-conv-update-graph) conv))] - - (deftest vote-base-test - (letfn [(get-count [tid vote bid] - (-> conv votes-base-fnk tid vote (nth bid)))] - - (testing "agree counts" - (doseq [pid [1 4]] - (is (= (get-count :c1 :A pid) - 1))) - (doseq [pid [0 2 3]] - (is (= (get-count :c1 :A pid) - 0)))) - - (testing "disagree counts" - (is (= (get-count :c1 :D 2) - 1)) - (doseq [pid [0 1]] - (is (= (get-count :c1 :D pid) - 0)))) - - (testing "disagree counts" - (doseq [pid [0 1]] - (is (= (get-count :c1 :S pid) - 1)))))) - - (deftest group-votes-test - (letfn [(get-count [gid tid vote] - (-> conv group-votes-fn gid :votes tid vote))] - (testing "members counts" - (is (= (-> conv group-votes-fn :g1 :n-members) 2)) - (is (= (-> conv group-votes-fn :g2 :n-members) 3))) - (testing "aggrees" - (is (= (get-count :g1 :c1 :A) 1)) - (is (= (get-count :g2 :c1 :A) 1))) - (testing "aggrees" - (is (= (get-count :g1 :c1 :D) 0)) - (is (= (get-count :g2 :c1 :D) 2)))))) - - - ; Test that iterating on previous pca/clustering results makes sense - ;; TODO Put this conv-update inside a deftest, so that errors are handled appropriately - (deftest iterative-results - (let [fleshed-conv (conversation/conv-update small-conv single-vote)] - (println "yeah...") - (testing "small-conv-iterative-test" - (testing "fleshed conv and full matrix" - (is (conversation/conv-update fleshed-conv full-votes))) - (testing "fleshed conv and wonky vote" - (is (conversation/conv-update fleshed-conv [{:created 999 :pid :j :tid :k :vote -1}])))) - - (testing "moderation-test" - (testing "from scratch with fleshed out conv" - (is (= (:mod-out (conversation/mod-update fleshed-conv - [{:tid :x :mod -1 :modified 1234}])) - #{:x}))) - (testing "based on previous moderations" - (is (= (:mod-out (conversation/mod-update (assoc big-conv :mod-out #{:x}) - [{:tid :y :mod -1 :modified 1856}])) - #{:x :y}))) - (testing "modding back in" - (is (= (:mod-out (conversation/mod-update (assoc big-conv :mod-out #{:x :y}) - [{:tid :y :mod 1 :modified 1856}])) - #{:x}))) - (testing "'moderating' out is_meta" - (is (= (:mod-out (conversation/mod-update (assoc big-conv :mod-out #{:x}) - [{:tid :y :is_meta true :modified 1856}])) - #{:x :y}))) - (testing "undoing is_meta" - (is (= (:mod-out (conversation/mod-update (assoc big-conv :mod-out #{:x :y}) - [{:tid :y :is_meta false :modified 1856}])) - #{:x})))) - (testing "with only approve mods" - (is (= (:mod-out (conversation/mod-update big-conv [{:tid :x :mod 1 :modified 8576} - {:tid :y :mod 1 :modified 9856}])) - #{}))) - (testing "with only approve mods" - (is (= (:mod-out (conversation/mod-update big-conv [{:tid :x :mod 1 :modified 7654} - {:tid :y :mod -1 :modified 8567}])) - #{:y}))))) - - (deftest large-conv-update-test - (testing "should work with votes for only existing ptpts/cmts" - (is (conversation/large-conv-update {:conv big-conv :opts {} - :votes [{:created 100 :pid :p1 :tid :c1 :vote 1} - {:created 200 :pid :p3 :tid :c3 :vote -1}]}))) - (testing "should work with votes for new cmts" - (is (conversation/large-conv-update {:conv big-conv :opts {} - :votes [{:created 100 :pid :p3 :tid :c5 :vote 1} - {:created 200 :pid :p5 :tid :c5 :vote -1}]}))))) - -(defn -main [] - (run-tests 'conversation-test)) - - - diff --git a/math/test/index_hash_test.clj b/math/test/index_hash_test.clj deleted file mode 100644 index 3f1d38bf39..0000000000 --- a/math/test/index_hash_test.clj +++ /dev/null @@ -1,54 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns index-hash-test - (:require [clojure.test :refer :all] - [polismath.math.named-matrix :refer :all])) - - -(defn ih-equals [ih1 ih2] - (letfn [(hash-rep [h] (sort (into [] h))) - (hash-equal [h1 h2] (= (hash-rep h1) (hash-rep h2)))] - (and (= (get-names ih1) (get-names ih2)) - (= (hash-equal (.index-hash ih1) (.index-hash ih2)))))) - - -; XXX - should really go through here and clean up not to use .names -(deftest test-index-hash-creation - (testing "Creating a new index-hash using `index-hash`" - (testing "with no elements" - (let [empty-ih (index-hash [])] - (is (= (.names empty-ih) [])) - (is (= (.index-hash empty-ih) {})))) - (testing "with some thing" - (let [ih (index-hash ["this" "that"])] - (is (= (.names ih) ["this" "that"])) - (is (= (.index-hash ih) {"this" 0 "that" 1})))) - (testing "with duplicate things" - (let [ih (index-hash ["this" "that" "this"])] - (is (= (.names ih) ["this" "that"])) - (is (= (.index-hash ih) {"this" 0 "that" 1})))) - (testing "with another ordering" - (let [ih (index-hash ["that" "this"])] - (is (= (.names ih) ["that" "this"])) - (is (= (.index-hash ih) {"that" 0 "this" 1})))))) - -(deftest test-append - (testing "Appending a new item to the index" - (let [base (index-hash ["this" "that"])] - (is (ih-equals (index-hash ["this" "that" "more"]) (append base "more"))) - (testing "when it's already in the index" - (is (ih-equals base (append base "this"))))))) - -(deftest test-append-many - (testing "Appending several items to an index" - (let [base (index-hash ["this"]) - expected (index-hash ["this" "that" "more"])] - (is (ih-equals expected (append-many base ["that" "more"]))) - (testing "when some are duplicates" - (is (ih-equals expected (append-many base ["that" "this" "more" "that"]))))))) - -(deftest test-subset - (testing "Creatign a subset" - (let [base (index-hash ["this" "that" "stuff" "great"])] - (testing "should give the right stuff" - (is (ih-equals (index-hash ["this" "stuff"]) (subset base ["this" "stuff"]))))))) diff --git a/math/test/named_matrix_test.clj b/math/test/named_matrix_test.clj deleted file mode 100644 index 876e31bfec..0000000000 --- a/math/test/named_matrix_test.clj +++ /dev/null @@ -1,146 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns named-matrix-test - (:use test-helpers) - (:require [clojure.test :refer :all] - [polismath.math.named-matrix :refer :all] - [polismath.math.named-matrix :as nm])) - -(def reactions [ - '("p1" "c1" 0) - '("p1" "c2" 1) - '("p2" "c1" -1) - '("p2" "c2" 1) - '("p3" "c1" -1) - '("p2" "c3" 1)]) - - -(def real-nmat (named-matrix - ["p1" "p2" "p3"] - ["c1" "c2" "c3"] - [[ 0 1 nil] - [-1 1 1] - [-1 nil nil]])) - - -(defn unpack-nmat [nmat] - (map #(% nmat) [rownames colnames get-matrix])) - -(def m [[1 2 3] [4 5 6]]) - -(deftest simple-add-padding-test - (testing "Adding nil" - (testing "verticle" - (is (m=? [[1 2 3] [4 5 6] [nil nil nil]] (add-padding m 0 1 nil)))) - (testing "horizontal" - (is (m=? [[1 2 3 nil] [4 5 6 nil]] (add-padding m 1 1 nil)))) - (testing "adding 0" - (testing "verticle" - (is (m=? [[1 2 3] [4 5 6] [0 0 0]] (add-padding m 0 1 0)))) - (testing "horizontal" - (is (m=? [[1 2 3 0] [4 5 6 0]] (add-padding m 1 1 0))))))) - - -(deftest test-zero-out-columns - (testing "sanity check" - (is (m=? (nm/get-matrix (nm/zero-out-columns real-nmat ["c2"])) - [[0 0 nil] [-1 0 1] [-1 0 nil]])) - (is (m=? (nm/get-matrix (nm/zero-out-columns real-nmat [])) - (nm/get-matrix real-nmat)))) - (testing "mod out a comment without votes yet" - (is (m=? (nm/get-matrix (nm/zero-out-columns real-nmat ["c999"])) - (nm/get-matrix real-nmat))))) - -; Simple creation test -(deftest create-matrix - (testing "Creation of matrix from scratch" - (let [nmat (named-matrix) - [rows cols matrix] (unpack-nmat (update-nmat nmat reactions))] - (is (= rows (rownames real-nmat))) - (is (= cols (colnames real-nmat))) - (is (= matrix (get-matrix real-nmat)))))) - -(defn nmat-equal [nm1 nm2] - (is - (and - (= (rownames nm1) (rownames nm2)) - (= (colnames nm1) (colnames nm2)) - (m=? (get-matrix nm1) (get-matrix nm2))))) - -(deftest test-update-nmat - (testing "Updating an existing nmatrix" - (testing "with existing cols" - (let [nmat (update-nmat real-nmat - ['("p1" "c3" 1) '("p3" "c3" -1)])] - (is (m=? (get-matrix nmat) - [[ 0 1 1] - [-1 1 1] - [-1 nil -1]])) - (is (= (rownames nmat) - (rownames real-nmat)) - "Nothing should change rows") - (is (= (colnames nmat) - (colnames real-nmat)) - "Nothing should change with cols"))) - - (testing "with new rows" - (let [nmat (update-nmat real-nmat - ['("p4" "c3" 1) '("p3" "c3" -1)])] - (is (m=? (get-matrix nmat) - [[ 0 1 nil] - [-1 1 1] - [-1 nil -1] - [ nil nil 1]])) - (is (= (rownames nmat) - ["p1" "p2" "p3" "p4"]) - "the new participant should be in rows") - (is (= (colnames nmat) - (colnames real-nmat)) - "Nothing should change with cols"))) - - (testing "with new cols" - (let [nmat (update-nmat real-nmat - ['("p3" "c4" 1) '("p2" "c3" -1)])] - (is (m=? (get-matrix nmat) - [[ 0 1 nil nil] - [-1 1 -1 nil] - [-1 nil nil 1]])) - (is (= (rownames nmat) - (rownames real-nmat)) - "Nothing should change with rows") - (is (= (colnames nmat) - ["c1" "c2" "c3" "c4"]) - "the new cols should be in cols"))) - - (testing "with new cols and rows" - (let [nmat (update-nmat real-nmat - ['("p4" "c3" 1) '("p3" "c4" -1)])] - (is (m=? (get-matrix nmat) - [[ 0 1 nil nil] - [-1 1 1 nil] - [-1 nil nil -1] - [ nil nil 1 nil]])) - (is (= (rownames nmat) - ["p1" "p2" "p3" "p4"]) - "the new participant should be in rows") - (is (= (colnames nmat) - ["c1" "c2" "c3" "c4"]) - "there new participants should be in rows"))))) - - -(deftest matrix-subsetting-test - (testing "by row" - (let [submat (named-matrix - ["p1" "p2"] ["c1" "c2" "c3"] - [[ 0 1 nil] - [-1 1 1]])] - (testing "using rowname-subset" - (is (nmat-equal (rowname-subset real-nmat ["p1" "p2"]) submat))) - (testing "using inv-rowname-subset" - (is (nmat-equal (inv-rowname-subset real-nmat ["p3"]) submat)))))) - - -(deftest get-row-by-name-test - (is (= (get-row-by-name real-nmat "p2") [-1 1 1]))) - - diff --git a/math/test/pca_test.clj b/math/test/pca_test.clj deleted file mode 100644 index e873c04208..0000000000 --- a/math/test/pca_test.clj +++ /dev/null @@ -1,61 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns pca-test - (:use test-helpers) - (:require [clojure.test :refer :all] - [polismath.math.named-matrix :refer :all] - [clojure.math.numeric-tower :refer :all] - [clojure.core.matrix :as m] - [polismath.math.pca :refer :all])) - - -(deftest powerit - (testing "Should generally work" - (let [data (m/matrix [[ 1 0 0] - [-1 1 0.1] - [ 0 1 0.1] - [ 0 1 -0.1]]) - expected [-0.34217 0.93906 0.032633] - pc-from-start (fn [start] (power-iteration data 2 start))] - - (testing "from scratch" - (is (= true (almost=? (pc-from-start nil) expected)))) - (testing "from scratch" - (is (almost=? (pc-from-start [1 1 1]) expected))) - (testing "from scratch" - (is (almost=? (pc-from-start [1 1]) expected)))))) - - -(deftest wrapped-pca-test - (testing "Should not fail and have the right shape for for" - (letfn [(right-shape [data] - (let [res (wrapped-pca (m/matrix data) 2)] - (and (:center res) (:comps res))))] - (testing "1x1" - (is (right-shape [[1]]))) - (testing "1x2" - (is (right-shape [[1 0]]))) - (testing "2x1" - (is (right-shape [[1] [0]]))) - (testing "2x2" - (is (right-shape [[1 0] [-1 1]]))))) - - (testing "zero matrices" - (let [data (m/matrix [[1 -1 0] - [1 -1 0] - [1 -1 0]])] - ; need to test not only that we get something sensible here, but also that if 0 vectors are returned, - ; that our wrapped-pca still be able to start up again - (is (almost=? [0 0 0] (first (:comps (wrapped-pca data 2))))) - (is (almost=? [0 0 0] (second (:comps (wrapped-pca data 2))))))) - - (testing "zero start-vectors" - (let [data (m/matrix [[1 0 0] - [0 -1 1] - [1 1 0]]) - zero-vec (m/matrix [0 0 0]) - zero-vecs [zero-vec zero-vec] - comps (:comps (wrapped-pca data 2 :start-vectors zero-vecs))] - (is (not (almost=? zero-vec (first comps)))) - (is (not (almost=? zero-vec (second comps))))))) - diff --git a/math/test/ptpt_stats_test.clj b/math/test/ptpt_stats_test.clj deleted file mode 100644 index 8c1278f2a9..0000000000 --- a/math/test/ptpt_stats_test.clj +++ /dev/null @@ -1,31 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns ptpt-stats-test - ;(:use test-helpers) - (:require [clojure.test :refer [deftest testing is]] - [polismath.math.named-matrix :as nm] - [polismath.math.repness :as repness])) - - -(deftest test-participant-stats - (testing "should generally be sane" - (let [group-clusters [{:id 1 :members [1 2]} - {:id 2 :members [3 4]}] - base-clusters [{:id 1 :members [1 2]} - {:id 2 :members [3 4]} - {:id 3 :members [5 6]} - {:id 4 :members [7 8]}] - proj-nmat (nm/named-matrix - [1 2 3 4 5 6 7 8] - ["x" "y"] - [[0.1 0.2] - [0.3 0.4] - [0.54 0.63] - [0.7 0.8] - [-0.1 -0.2] - [-0.31 -0.4] - [-0.5 -0.6] - [-0.7 -0.8]]) - ptpt-vote-counts {1 3, 2 5, 3 2, 4 5, 5 9, 6 3, 7 5, 8 10}] - (is (not-empty (vec (repness/participant-stats group-clusters base-clusters proj-nmat ptpt-vote-counts))))))) - diff --git a/math/test/silhouette_test.clj b/math/test/silhouette_test.clj deleted file mode 100644 index 5b610c8bae..0000000000 --- a/math/test/silhouette_test.clj +++ /dev/null @@ -1,60 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns silhouette-test - (:use test-helpers) - (:require [clojure.test :refer :all] - [polismath.math.named-matrix :refer :all] - [polismath.math.clusters :refer :all] - [clojure.core.matrix :as m])) - -(deftest named-dist-matrix-test - (let [m (named-matrix [:a :b :c] - [:x :y :z :w] - [[1 2 3 4] - [5 6 7 8] - [9 9 9 9]])] - (testing "With one arg" - (testing "should have the correct rownames" - (is (= [:a :b :c] (rownames (named-dist-matrix m))))) - (testing "should have the correct colnames" - (is (= [:a :b :c] (colnames (named-dist-matrix m))))) - (testing "should have the correct distance matrix" - (is (almost=? - (get-matrix (named-dist-matrix m)) - ; From R - [[ 0.0 8.0 13.1909] - [ 8.0 0.0 5.4772] - [13.1909 5.4772 0.0]] - :tol 0.001)))) - (testing "With two args" - (let [m2 (named-matrix [:q :r :t] - [:x :y :z :w] - [[0 2 4 8] - [10 12 14 16] - [0 0 0 0]])] - (testing "should have correct rownames" - (is (= [:a :b :c] (rownames (named-dist-matrix m m2))))) - (testing "should have the correct colnames" - (is (= [:q :r :t] (colnames (named-dist-matrix m m2))))))))) - - -; Some pretty simple, basic examples -(let [distmat - (named-matrix [:a :b :c :d] - [:a :b :c :d] - [[0 1 3 4] - [1 0 3 4] - [3 3 0 2] - [4 4 2 0]]) - clusters - [{:members [:a :b]} {:members [:c :d]}]] - (deftest member-silhouette-test - ; Compute by hand - (is (almost=? (silhouette distmat clusters :a) 0.71428))) - (deftest clutering-silhouette-test - ; Again, by hand - (is (almost=? (silhouette distmat clusters) 0.5654))) - (deftest singleton-cluster-test - (is (= 0 (silhouette distmat [{:members [:a]} {:members [:c :d]}] :a))))) - - diff --git a/math/test/stats_test.clj b/math/test/stats_test.clj deleted file mode 100644 index d6006f7032..0000000000 --- a/math/test/stats_test.clj +++ /dev/null @@ -1,27 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns stats-test - (:use test-helpers) - (:require [clojure.test :refer :all] - [polismath.math.clusters :refer :all] - [polismath.math.stats :refer :all])) - - -(deftest test-two-prop-test - (testing "should have correct z-score" - (is (almost=? - ; from example here - http://www.cliffsnotes.com/math/statistics/univariate-inferential-tests/test-for-comparing-two-proportions - ; note this has dec applied to all inputs since we use psuedocounts to prevent / 0 - (two-prop-test 15 56 24 71) - -1.518 - :tol 0.01)))) - - -(deftest test-prop-test - (testing "should have correct z-score" - (is (almost=? - ; Silly little hack - as we become more certain out -> 0.5, these should converge - (two-prop-test 30 1000000 50 2000000) - (prop-test 30 50))))) - - diff --git a/math/test/test_helpers.clj b/math/test/test_helpers.clj deleted file mode 100644 index eab4d3292a..0000000000 --- a/math/test/test_helpers.clj +++ /dev/null @@ -1,42 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns test-helpers - (:require [clojure.core.matrix :as m])) - - -(defn dimension-counts [xs] - (map #(m/dimension-count xs %) (range (m/dimensionality xs)))) - - -(defn dims-equal? - "Checks that the dimension-counts are the same for all dimensions" - [xs ys] - (every? identity - (map = (dimension-counts xs) (dimension-counts ys)))) - - -(defmulti almost=? - "Check if values are almost equal, regardless of dimension (supports tolerance)" - (fn [xs ys & args] - (let [dims (mapv m/dimensionality [xs ys])] - (assert (= (first dims) (last dims)) - "xs and ys should have the same dimesnionality") - dims))) -(defmethod almost=? [0 0] - [x y & {:keys [tol] :or {tol 0.001}}] - (or - (and (nil? x) (nil? y)) - (>= tol - (m/abs (- (float x) (float y)))))) -(defmethod almost=? :default - [xs ys & {:keys [tol] :or {tol 0.001}}] - (and - (dims-equal? xs ys) - (every? identity - (map #(almost=? %1 %2 :tol tol) xs ys)))) - - -(defn m=? [xs ys] - (almost=? xs ys :tol 0)) - - diff --git a/math/test/test_runner.clj b/math/test/test_runner.clj deleted file mode 100644 index 1302879a77..0000000000 --- a/math/test/test_runner.clj +++ /dev/null @@ -1,42 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns test-runner - (:require [cluster-tests] - [conv-man-tests] - [conversation-test] - [conv-edge-cases-test] - [index-hash-test] - [named-matrix-test] - [pca-test] - [silhouette-test] - [stats-test] - [utils-test] - [ptpt-stats-test] - [clojure.test :as test])) - - -;; Print each deftest name as it starts, so we can see progress and detect hangs. -(defmethod test/report :begin-test-var [m] - (let [v (:var m)] - (println " " (-> v meta :name)) - (flush))) - -(defn -main - "Run all the pure tests for polisapp. The one integration test is in conv-man-tests, and should be run separately (and - needs to be cleaned up to run on a separate poller system)" - [] - (apply - test/run-tests - '[cluster-tests - conversation-test - conv-edge-cases-test - index-hash-test - named-matrix-test - pca-test - silhouette-test - stats-test - utils-test - ptpt-stats-test])) - -;(-main) -;(test/run-tests 'conversation-test) diff --git a/math/test/utils_test.clj b/math/test/utils_test.clj deleted file mode 100644 index 6e4c5f3e54..0000000000 --- a/math/test/utils_test.clj +++ /dev/null @@ -1,20 +0,0 @@ -;; Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . - -(ns utils-test - (:require [clojure.test :refer :all] - [polismath.utils :refer :all])) - - -(deftest apply-kwargs-test - (letfn [(fun [a b & {:keys [c d] :as kw-args}] - {:a a :b b :c c :d d :kw-args kw-args})] - (let [res (apply-kwargs fun "this" "that" {:c "crap" :d "stuff" :m "more"})] - (testing "should pass through regular args" - (is (= (res :a) "this")) - (is (= (res :b) "that"))) - (testing "should pass through kw-args" - (is (= (res :c) "crap")) - (is (= (res :d) "stuff")) - (is (= ((res :kw-args) :m) "more")))))) - -