Skip to content
MondegoPublic

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

34 Commits

Folders and files

Repository files navigation

STRAT (STratified Registers And Types). The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning

Rethinking the Transformer: separate subspaces for data and control.

Python 3.10+ PyTorch License: GPL-3.0

Overview · Repository layout · Quick start

Overview

STRAT partitions the residual stream of a Transformer into two orthogonal subspaces, Data and Type. Attention reads and updates only Type, and a gate conditioned on Type controls every update of Data, so numerical values stay apart from routing decisions. This repository holds the experiments with small STRAT models on arithmetic and logical tasks.

The STRAT block

The STRAT block on the arithmetic example 12 + 3 − 5. Type (blue) drives attention and the gate; the gated transform is added back to Data (orange).

Repository layout

Directory What it does
hand-coded-alu/ A hand-coded arithmetic unit that computes signed sums without training
dense-failures/ An ablation of dense models on the signed-sum task, with probes that measure how each model fails
lite/ STRAT-Lite on the signed-sum task, with a predefined Type and with a learned Type
efficiency/ A sample-efficiency sweep in which every model trains on 10 sequences
multitask/ STRAT and a Transformer baseline on 11 datasets, with the data of every dataset

Quick start

The code needs Python 3.10 or later, PyTorch and NumPy; dense-failures/ also needs pandas, and the SAT datasets in multitask/ need pycosat. Run from the repository root.

# Hand-coded arithmetic unit on two examples.
python hand-coded-alu/GeomALU_sle.py

# A small run of the ablation, written to its own directory.
python dense-failures/run_ablation.py --configs A Bfl --seeds 5 --out ablation-small

# STRAT-Lite with a predefined Type and with a learned Type.
python lite/strat_lite.py
python lite/strat_lite_learned_types.py

# The sample-efficiency sweep.
python efficiency/driver-efficiency-experiment.py

# STRAT and the Transformer on one dataset, then their ID and OOD accuracy.
bash multitask/run.sh parity5
python -m multitask.evaluate_checkpoints --type-mode onehot --tasks parity5

Each directory holds further scripts. multitask/README.md describes the datasets and options.

License

This repository is released under the GNU General Public License v3.0; see LICENSE.

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages