basic-torch-optim

training loop optimization

this repo contain a simple baseline iteratively improved by the simple optimization techniques we discussed today, composed iteratively. each version introduce one change on top of the previous. in each file please find #& for the exact lines touched compared to the prev version

• v0.py: baseline MLP + SGD
• v1.py: v0 + AdamW, residual pre-norm blocks
• v2.py: v1 + AMP (autocast + GradScaler)
• v3.py: v2 + activation checkpointing
• v4.py: v3 + gradient accumulation
• v5.py: v4 + DataLoader tuning (workers, pin_memory, persistent_workers, prefetch_factor)

commit messages too carry the explanation per change