Handwriting recognition from a sensor pen. imu2text trains on the Fraunhofer IIS OnHW datasets (13 IMU channels at 100 Hz) and reports accuracy on writers the model has never seen.
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
python -m imu2text.download onhw_chars --out ./data # 896 MB, once
python -m imu2text.models --models cnn_bilstm_attn \
--onhw-chars data/onhw-chars_2021-06-30 --case both --dependency indep --fold 0 \
--augment 2 --aug-policy extended --label-smoothing 0.1 --lr-schedule --epochs 30No download needed to check the sequence pipeline: python -m imu2text.seq2seq --demo.
Official splits, writer-independent (fold 0 where the archive ships folds); one seed unless stated.
| Task | Metric | imu2text | Published |
|---|---|---|---|
| OnHW-chars, 52 classes | accuracy | 72.5% | 68.06% (Ott et al., ACM MM 2022, CNN+BiLSTM) |
| OnHW-chars, 52 classes, 5 seeds averaged | accuracy | 74.46% | n/a |
| OnHW-chars, left-handed writers, 52 classes, 5 seeds averaged | accuracy, our own split of the 9 writers | 59.47% | n/a |
| OnHW-equations, 15 symbols, split per symbol | accuracy | 86.16% | 83.88% (same paper, Table 2) |
| OnHW-equations, digits 0-9 | accuracy | 87.35% | n/a |
| OnHW-equations, whole numbers | every digit right | 76.10% | n/a |
| OnHW-Words500, 59 characters | greedy CER, refit on all training writers | 53.95% | not compared |
| OnHW-Words500, 59 characters | words read exactly, word list of training words | 40.78% | not compared |
On the characters, 43% of the remaining errors are a letter read as its other case; scored case-insensitively the same model reaches 84.3%. On the equations, a number of four or more digits is read right 56.6% of the time, about what 87% per digit predicts. The equations are already cut into symbols by the dataset authors, so these scores do not include finding the boundaries.
Details, every other split and the reproduction commands: docs/benchmarks.md.
- Vahini playground: one letter through the pipeline, in the browser (source)
- Getting started: the problem, the code, where to begin
- Benchmarks: all results and how they were measured
- Uncertainty: ensembles, calibration, and when to trust a prediction
- Datasets: the OnHW archives and how to load them
- Roadmap: what is done and what comes next
- Glossary: the abbreviations
Read AGENTS.md first. CI runs pytest, black, pylint and a
Markdown style check.
The code is Apache License 2.0; see LICENSE and NOTICE. Maintained by @vahinitech.
The OnHW datasets are by Fraunhofer IIS, for non-commercial use only. They are
not covered by the Apache license and this repository does not contain them:
python -m imu2text.download fetches them from Fraunhofer. If you use them,
cite Ott et al., "The OnHW Dataset: Online Handwriting Recognition from
IMU-Enhanced Ballpoint Pens with Machine Learning", Proc. ACM IMWUT 4(3), 2020.