Contrib data handling - #2117

Draft

bfoley12 wants to merge 243 commits into

devfrom

contrib-data-handling

Commits on Jun 17, 2026

Initial ruff formatting and uv project init
bfoley12
committed
94d7227
Added .venv and .env to gitignore
bfoley12
committed
3c79379
Moved dev optional-dependencies to a dependency group
bfoley12
committed
4d5c64f
Synced lock file
bfoley12
committed
8ebb3f6
Example env vars
bfoley12
committed
98d867f
Moved to a src/proj style repo. Began creating fastapi server
bfoley12
committed
6558bc7
Removed project-level uv ennvironment
bfoley12
committed
04da662
Updated python version to 3.14
bfoley12
committed
b84f81f
Added projects router and reintroduced prefix
bfoley12
committed
a3624de
Added Beanie init
bfoley12
committed
f31519b
Broke redis and kong settings into their own classes for modularity
bfoley12
committed
4de7efd
Added shared types
bfoley12
committed
48f19a6
Removed unnecessary files
bfoley12
committed
4e44d03
Added packages to uv
bfoley12
committed
c44dc0e
Built out project domain for getting data
bfoley12
committed
da8d2e5
Added shared dependencies for injection
bfoley12
committed
b130b33
Added AuthenticationError and GatewayError for more control over thrown errors
bfoley12
committed
c3004a0
Ruff formatting and checking
bfoley12
committed
a5b5849
PrefixedEmail now throws ValidationError instead of ValueError
bfoley12
committed
74ac09f
Changed ProjectResource for ProjectOut for consistency with ProjectIn. Added create_project
bfoley12
committed
6baffad
Changed create_project to insert_project
bfoley12
committed
d761d3a
Added Project patch for partial udpates
bfoley12
committed
bd2dbbb
Added Project delete
bfoley12
committed
9271c5b
Added Projet Upsert (put)
bfoley12
committed
0ac8204
Added docstrings and removed dead code
bfoley12
committed
b8bbb4d
Removed outdated runtime_checkable and awaited async function to get returned value
bfoley12
committed
617afad
Created a parent class to abstract the handling of fields requested from query. Users can now request partial documents with _fields
bfoley12
committed
22ad07d
modified docstrings to match function definition
bfoley12
committed
a4d2330
Fixed issue where find_one syntax was used with get
bfoley12
committed
5808483
Removed dead ProjectSummary usages
bfoley12
committed
2998c02
Automatic projection processing now hanldes dot-paths and validates existence against the Document, unless field is a dict
bfoley12
committed
41dd24b
Initial Contributions route setup
bfoley12
committed
95d3398
Added stub methods
bfoley12
committed
8053215
Added base contributions class and ContributionPatch. Contribution now sets last_modified on any modification result
bfoley12
committed
1d30533
Added settings to Project to name proper collection and prevent nulls from populating
bfoley12
committed
97f01bd
Added contributions router
bfoley12
committed
a2d31fb
Sketched out endpoints and repo methods
bfoley12
committed
191c4c0
Mostly formatting and type checking changes. Added stub files for component attachments. Added project-level claude.md. Added just file for development ease.
bfoley12
committed
96832e8
Abstracted repository and models
bfoley12
committed
25bdc30
Implemented classes and methods with new abstractions
bfoley12
committed
6392fb2
Implemented delete_contributions
bfoley12
committed
960ab66
Added linked-document filtering for contributions
bfoley12
committed
de54f47
Anyonymous users can no longer be admin. Fixed imports
bfoley12
committed
78f32b2
Added unit tests
bfoley12
committed
ee75871
added check fix for normal fixes
bfoley12
committed
dbd664d
Various small fixes in function signatures and usages
bfoley12
committed
71aaf9d
Added integration tests
bfoley12
committed
573aeb5
Added test commands to justfile
bfoley12
committed
0f84dd4
Added tests to live dev db
bfoley12
committed
8c0060b
Fixed importing from src
bfoley12
committed
d7b5b8b
OpenAPI UI improvement. Added app info and tags to routers
bfoley12
committed
d3496ab
Moved more openapi info dicts to _openapi
bfoley12
committed
4333511
Changed imports so they resolve
bfoley12
committed
2a2d03e
Removed old repo
bfoley12
committed
7a8181c
Fixed basedpyright not knowing proper import path. Removed old ignores to 'old' folder. Added bfoley as author :)
bfoley12
committed
0c45e9d
Removed dependencies. Some temporarily
bfoley12
committed
38b2b14
Reordered deps
bfoley12
committed
f21e781
fields argument in routers now expects a list instead of comma delimited string
bfoley12
committed
eb2ddf9
Fleshed out contribution repo methods
bfoley12
committed
fb176b4
Pulled repeated repo logic into parent class
bfoley12
committed
9d83970
Changed logging middleware to allow contextvars being passed upwards
bfoley12
committed
792f83a
basedpyright ignores tests
bfoley12
committed
870456a
Infra changes for rewrite
bfoley12
committed
ba08993
Added healthcheck endpoint
bfoley12
committed
c7d7bcf
Forward username and is_admin values for logging
bfoley12
committed
5991392
Convert ids coming into Contributions to PydanticObjectIds
bfoley12
committed
eac368f
Fleshed out Contribution component models
bfoley12
committed
d77cf4b
Init component models
bfoley12
committed
7a5dbf8
Bulk upload of contributions with components now bulk inserts components first
bfoley12
committed
9a96716
Made component repositories to pull insert logic out of contributions repo
bfoley12
committed
4d860f9
Moved cross-repo insert logic into service layer
bfoley12
committed
7435e7d
Documents now handle decoding cursor to accomodate differing id types (OId vs str)
bfoley12
committed
ca962a4
Added docstrings
bfoley12
committed
4bb92e2
Inserting many contributions now uses multi-document transactions rather than unsafe raw async
bfoley12
committed
8276016
upsert_contributions uses semaphore
bfoley12
committed
3725643
Added project_identifier index on ContributionsBase
bfoley12
committed
610ca6d
clamp max_concurrent_transactions
bfoley12
committed
9ab2a6a
Atomically upsert a contribution by identifiers rather than _id
bfoley12
committed
d63edcb
Contributions delete children on bulk delete
bfoley12
committed
2bce5ed
Deleting contribution by id deletes all children
bfoley12
committed
d3c8bf0
Further parameterized the AsyncMongoClient
bfoley12
committed
6d6b4ed
Fixed spacing issues
bfoley12
committed
24fa834
Added table routes
bfoley12
committed
8f965d0
Moved types to shared domains
bfoley12
committed
c979f3d
Removed unused import
bfoley12
committed
ddb8b7f
Added session to delete_by_id and made a DeleteResponse model
bfoley12
committed
fa456ad
Fleshed out basic CRUD operations for Tables
bfoley12
committed
93d60e0
Added missing update to patch endpoint
bfoley12
committed
b6278e0
Added missing colon in patch function def
bfoley12
committed
be8b7fd
Fleshed out streaming download to user with compression
bfoley12
committed
b1977f8
Removed unused response paramter from download_table
bfoley12
committed
f199f9f
Added shared repository for components
bfoley12
committed
dc8e53b
Added Structures endpoint and refatored to use shared components repo
bfoley12
committed
2572d1d
Added Attachments endpoint
bfoley12
committed
7277a99
Removed get and patch capabilities from attachment endpoint
bfoley12
committed
dd91fbc
Removed logic for inserting attachments during contribution insert
bfoley12
committed
b6c9753
Moved _children to a property
bfoley12
committed
8cb3422
Added attachments, structures, and tables routers to main router
bfoley12
committed
3fb7e01
Made PolarsFrame type to handle coercion and serialization
bfoley12
committed
c844e0a
Removed 'components' tag from components routers
bfoley12
committed
4f7b57d
Added additional logging context to each request
bfoley12
committed
b776d2d
Removed unused gateway secret check
bfoley12
committed
73e5acf
Moved router tagging to the api router
bfoley12
committed
724698a
Updated tests to account for new code
bfoley12
committed
c126522
Changed exception handling and modified variable names to be descriptive rather than one letter
bfoley12
committed
75d4bdb
Removed old settings reference and scoped contribution access to project user.groups
bfoley12
committed
a934bdd
DeleteResponse now consumes a DeleteResult to allow for easier expansion later. MongoDbRepo now allows deleting multiple ids at once
bfoley12
committed
79ef63f
Removed vestigial GatewayError references
bfoley12
committed
93e6bf4
Contribution.data validation for depth <= 7
bfoley12
committed
946e2e7
Disallow lists in Contribution.data
bfoley12
committed
210b0e1
Added more validation to Contribution.data dict
bfoley12
committed
cb9b506
Removed useless thin wrapper method
bfoley12
committed
39c032f
Inserting many components now validates by md5 hash and returns documents with their id
bfoley12
committed
3e21a4d
Component.insert_component delegates to insert_components
bfoley12
committed
f415323
Attachments document model now verifies extension of file. Modified imports.
bfoley12
committed
e92977f
Greatly expanded unit testing
bfoley12
committed
7e359ea
Fixed domain routers not prefixing '/'
bfoley12
committed
0413425
Added more thorough integration test coverage
bfoley12
committed
d89e40c
Moved download logic to base shared repository
bfoley12
committed
86ba711
Added csv serialization for downloads
bfoley12
committed
8e83e65
Added tests to cover downloads
bfoley12
committed
26d0398
Fixed gzip missing final packet; improved CSV serialization; improved download filenames
bfoley12
committed
12340a6
Moved DataFormat decision to a match statement
bfoley12
committed
381353a
Removed module-level docstrings
bfoley12
committed
ddcd41f
Setup app to use aioboto3 and load defaults from env
bfoley12
committed
b41a420
Added dependencies for boto and S3
bfoley12
committed
0264d35
Changed filename to authz to clarify that it is authorization
bfoley12
committed
a69e64d
Added S3Dep and s3 reliance to download methods; modified auth import to authz
bfoley12
committed
bfb32c9
Updated auth import to authz
bfoley12
committed
d4129b4
Uncommented kong in settings
bfoley12
committed
937be7d
Modified S3Dep to return the client rather than a generator; moved key_name and bucket_name out of router function defs since they are in flux
bfoley12
committed
453845c
Removed kong settings, since they are unused
bfoley12
committed
2249c68
Made a shared ComponentService to ensure deletion only occurs when no contributions reference the component in question
bfoley12
committed
6906556
Added .claude/ to gitignore
bfoley12
committed
6604e55
Moved component repo and service logic into a shared service. Component-specific repos still exist to declare their document and out_model types, but have no subclass-specific logic
bfoley12
committed
79fd99c
Added download tests, removed config test raising on creating Settings empty - we have defaults now
bfoley12
committed
d2ca35c
Reverted non-API changes to master versions
bfoley12
committed
5f75cf3
upgrade dependencies for deployment

github-actions
authored and
bfoley12
committed
d79ac6f
Infra changes for rewrite
bfoley12
committed
fa4886c
Stop tracking .claude/ directory
bfoley12
committed
a5d543b
Revert changes to deployment of other submodules
bfoley12
committed
24d7b5d

Commits on Jun 18, 2026

Commits on Jun 22, 2026

Added ty for typechecking
bfoley12
committed
408b6a6
Removed ty - not mature enough
bfoley12
committed
90eaca5
Changed docstring, imports, and added polars dep
bfoley12
committed
0d122bc

Commits on Jun 23, 2026

Commented out unused dependencies
bfoley12
committed
10ce393

Commits on Jun 24, 2026

Commits on Jun 25, 2026

Commits on Jun 26, 2026

Added jinja2 back to dependencies and sorted deps
bfoley12
committed
4359ee8
Build uv venv outside of app so dependencies load properly
bfoley12
committed
016546b
Fixed env var names to match new format
bfoley12
committed
05811ea

Commits on Jun 29, 2026

Commits on Jun 30, 2026

Contributions are now versioned. The server derives integer versions from the number of documents sharing (project, identifier) in projects where unique_identifiers=False, otherwise version === 1
bfoley12
committed
2c5cedb

Commits on Jul 1, 2026

Commits on Jul 8, 2026

Added settings specific to MPContribs business logic
bfoley12
committed
c625b3c
Added Pint and Uncertainties
bfoley12
committed
c1bb17e

Commits on Jul 9, 2026

Commits on Jul 10, 2026

Commits on Jul 13, 2026

Added comments
bfoley12
committed
167f177
Made AnnotatedData the source of data and unit handling. Implemented additional processing, and wired into pivot
bfoley12
committed
e5e68e4
Synced identifier to be SearchStr instead of bare str in Filter
bfoley12
committed
ed6380b
Allow manual specification of username and groups when in dev context
bfoley12
committed
e4a8022
Added tests
bfoley12
committed
b58500a
Added hypothesis package for testing annotated contribution data
bfoley12
committed
2a7672c
Contributions do not accept id for POST anymore
bfoley12
committed
ab6e41f

Commits on Jul 23, 2026

Commits on Jul 27, 2026

Set Project.owner to the caller that created the Project
bfoley12
committed
27d03b6

Commits on Jul 28, 2026

Commits on Jul 30, 2026

Merged dev into this branch to fix merge conflicts
bfoley12
committed
f28bd77

Uh oh!

Commits

Commits on Jun 17, 2026

Commits on Jun 18, 2026