onnx-light

Documentation

See also the ONNX roadmap for the upstream project's direction and priorities.

Note: onnx-light started from the upstream ONNX pull request onnx/onnx#7208, which is the initial code base from which this project diverged.

onnx without protobuf

• ONNX Files larger than 2 GB (protobuf is limited to 2Gb)
• Parallel loading and saving: significantly faster compared to the single-threaded path
• Zero-copy parsing – creates the ModelProto without any tensor copy
• Aligned external tensor offsets – external tensor data can be written with explicit offset alignment
• No serialize/parse round-trip for C++ tools – the Python ModelProto is the C++ ModelProto
• Supports protobuf (onnx) and flatbuffers (onnxruntime) format.

Modular C++ libraries

The C++ code is split into several small libraries so a downstream project can link only what it needs:

• onnx_light::lib_onnx_proto – protobuf-compatible message types, parser / serializer, external data, optional encrypted save / load (AES-256-CBC or ChaCha20-Poly1305).
• onnx_light::lib_onnx_core – implements all the generic functionalities (runtime value types and execution engine, the LightOpSchema data structures, the symbolic expression engine and the kernel / shape-inference dispatch tables) but ships no concrete operators. The dispatch tables start empty and are filled by the extension libraries below.
• onnx_light::lib_onnx_op – lightweight LightOpSchema registrations for ONNX operator domains, with no shape inference.
• onnx_light::lib_onnx_lib – full ONNX-compatible schemas (with history), checker, inliner, shape inference and version converter.
• onnx_light::lib_onnx_shape – shape-inference dispatch table, expression engine and graph optimization helpers.
• onnx_light::lib_onnx_kernels – C++ kernels used to generate the backend test outputs and to evaluate models with the reference runtime.
• onnx_light::lib_onnx_backend_test – C++ backend test infrastructure and reference operator kernels.

onnx_core only implements the mechanisms: the actual operator schemas, kernels, shape-inference and peak-memory functions are registered into the shared dispatch tables owned by onnx_core by the extension libraries (onnx_op, onnx_shapes, onnx_kernels, ...) through their Register*Functions() entry points. This keeps the extensions independent from each other while sharing the same core engine.

Kernels and Backend Tests

• Each operator has a corresponding runtime implementation in C++, it is used to generate the C++ output of the backend tests.
• Fully written in C++, it can be used in any language.
• Outputs are always generated with a C++ kernel.
• The kernels can be used without the backend tests.

Runtime, Optimization and Gradients

• Reference runtime – the kernels double as a self-contained C++ runtime for the ONNX operator set, so a model can be evaluated in C++ or from Python without a third-party runtime. A RuntimeSession parses a model once, builds an execution plan and can then be run repeatedly on runtime Tensor inputs.
• Graph optimization – a model is optimized by repeatedly matching small PatternOptimization subgraphs and replacing them with a simplified equivalent. Patterns are implemented in C++ and registered into a shared dispatch table so a downstream project can add its own; every applied rewrite is recorded and can be replayed.
• Gradients – gradients of an ONNX graph can be computed and used to train a model.
• Encrypted save / load – models can be encrypted with AES-256-CBC (ONNXCRY1) or ChaCha20-Poly1305 (ONNXCRY2), both using PBKDF2-HMAC-SHA256 key derivation, and saved to a single self-contained .onnxc file or serialized to an in-memory bytes object.

Software Bill of Materials (SBOM)

A CycloneDX 1.7 Software Bill of Materials is shipped at the root of the repository as sbom.cdx.json and is also included in the source distribution. It lists the third-party components bundled into the built artifacts (currently only nanobind, used to expose the C++ extension to Python). The file is validated against the CycloneDX 1.7 schema by the SBOM GitHub Actions workflow.

Getting started

Install the package in editable mode:

pip install -e .[dev] -v

or

python setup.py build_ext --inplace

The setup.py build_ext --inplace command refuses to run when an editable onnx-light installation points to another source tree, or when an import hook left behind by a removed installation is still present, because such a hook hides the extensions built in the current source tree. An editable installation of the current source tree is fine. Uninstall the conflicting one with python -m pip uninstall onnx-light; the error reports every location that must be removed.

setup.py build_ext configures CMake with -DCMAKE_BUILD_TYPE=Release by default (unless CMAKE_ARGS already sets CMAKE_BUILD_TYPE). --cpp-tests can be used to build the C++ unit tests and run them with ctest.

To speed up compilation with multiple threads, pass --parallel (or -j) with the number of jobs:

python setup.py build_ext --inplace --parallel 8

By default, python setup.py build_ext now auto-enables parallel builds (--parallel <cpu_count>) unless CMAKE_BUILD_PARALLEL_LEVEL is already set.

Alternatively, when installing with pip, you can control parallel builds using the CMAKE_BUILD_PARALLEL_LEVEL environment variable:

CMAKE_BUILD_PARALLEL_LEVEL=8 pip install -e .[dev] -v

Run a quick check:

python -c "import onnx_light; print(onnx_light.__version__)"

Build and run the C++ unit tests from the editable build:

With pip install:

pip install -C build-dir=build -C cmake.build-type=Debug -C cmake.define.ONNX_LIGHT_BUILD_TESTS=ON -e .[dev] -v
ctest --test-dir build --output-on-failure

With setup.py, --cpp-tests builds the C++ unit tests and runs them with ctest in one step:

python setup.py build_ext --inplace --build-temp build --cpp-tests

The Python package is built and installed inplace before the C++ unit tests are built and run, so the editable install is always available even if a C++ test fails to build or run.

On multi-config generators such as Visual Studio, add the matching configuration to ctest: use -C Debug when the build was configured with cmake.build-type=Debug, and -C Release after python setup.py build_ext --cpp-tests.

Load a model with parallel tensor parsing:

import onnx_light.onnx

model = onnx_light.onnx.load("model.onnx", num_threads=4)
print(model.ir_version)

Using onnx_light as a C++ library

Installing the C++ library

Build and install the static library and headers to a local prefix (Python extension not required):

cmake -S . -B build-install
  -DCMAKE_BUILD_TYPE=Release \
  -DONNX_LIGHT_BUILD_PYTHON=OFF \
  -DCMAKE_INSTALL_PREFIX=/usr/local
cmake --build  build-install
cmake --install build-install

This installs:

• liblib_onnx_proto.a, liblib_onnx_op.a, and liblib_onnx_lib.a (the static libraries) into <prefix>/lib
• All public C++ headers into <prefix>/include/onnx_light
• CMake package config files into <prefix>/lib/cmake/onnx_light

Using find_package(onnx_light) in your project

Once installed, any CMake project can locate and link the library with:

find_package(onnx_light REQUIRED)
target_link_libraries(my_target PRIVATE onnx_light::lib_onnx_lib)

If the code only needs protobuf-compatible message parsing/serialization and does not need operator schemas, checker, or shape inference, it can link against the lighter onnx_light::lib_onnx_proto target instead:

find_package(onnx_light REQUIRED)
target_link_libraries(my_target PRIVATE onnx_light::lib_onnx_proto)

If the code needs lightweight math operator schemas without shape inference, it can link against onnx_light::lib_onnx_op and query onnx_op::math::GetAllOnnxOpMathSchemasWithHistory().

Pass -DCMAKE_PREFIX_PATH=<prefix> when configuring your project if the library was installed to a non-standard prefix.