- Created: 2026-09-22
- Last Updated: 2026-10-24
LionSQL is an enterprise-grade, high-concurrency embedded OLTP database engine written in pure Go and Cgo (WiredTiger). Packaged as a single standalone binary with zero external operational dependencies, it combines multi-engine ACID storage, full Microsoft T-SQL procedural execution, multi-wire protocol interoperability (PostgreSQL and TDS/MSSQL), an embedded connection pooler, and an embedded web management studio.
+---------------------------------------------------------------------------------------------------------+
| Client Ecosystem & Interfaces |
| +--------------------------+ +--------------------------+ +-----------------------------------+ |
| | PostgreSQL Tools / ORMs | | MSSQL / Azure Data Studio| | HTTP REST / Embedded Web Studio | |
| | (psql, DBeaver, GORM) | | (sqlcmd, SSMS, pymssql) | | (Browser Dashboard, Realtime SSE) | |
| | Port: 5433 | | Port: 1433 | | Port: 8090 | |
+---+--------------------------+---+--------------------------+---+-----------------------------------+---+
|
v
+---------------------------------------------------------------------------------------------------------+
| Session & Connection Pooler Layer (pkg/api) |
| - Goroutine Connection Multiplexer (Replaces PgBouncer; 10,000+ client connections) |
| - Bounded Worker Pool (MaxWorkers = NumCPU * 4, MaxIdleSessions = 64) |
| - Realtime Broker (Server-Sent Events pub/sub) & Superuser JWT / API Key Authentication |
+---------------------------------------------------------------------------------------------------------+
|
v
+---------------------------------------------------------------------------------------------------------+
| Compiler, Optimizer & Plan Cache Layer (pkg/proc, pkg/planner) |
| +-------------------------------------------------------------------------------------------------+ |
| | Dual-Parser Engine: | |
| | - Fast-Path Native Pratt Parser (sub-microsecond, zero-allocation) | |
| | - Full-Spec ANTLR4 Fallback Parser (Official Microsoft T-SQL Grammar) | |
| +-------------------------------------------------------------------------------------------------+ |
| | Cost-Based Optimizer (CBO): | |
| | - Table & Column Statistics Catalog (ANALYZE TABLE, Equi-depth Histograms, NDV, Selectivity) | |
| | - Physical Plan Cost Evaluation (Index Scan vs Seq Scan, Hash Join vs Nested Loop Join) | |
| | - EXPLAIN & EXPLAIN ANALYZE Plan Tree Visualizer | |
| +-------------------------------------------------------------------------------------------------+ |
| | Plan Cache: LRU Thread-Safe AST & Bytecode Program Cache | |
+---+-------------------------------------------------------------------------------------------------+---+
|
v
+---------------------------------------------------------------------------------------------------------+
| Unified Execution Layer: Native VDBE Bytecode Virtual Machine (pkg/proc/runtime) |
| - Single-Engine Parity: Interactive DQL & DML (SELECT, INSERT, UPDATE, DELETE) & Procedures/Triggers |
| - Zero-Parse Prepared Statement Bytecode Caching across PostgreSQL Wire and Microsoft TDS |
| - sync.Pool Register Allocator, Ambient ACID Transaction Binding, and Opcode Dispatcher |
| |
| ┌───────────────────────────────────────────────────┬──────────────────────────────────┐ |
| ▼ ▼ ▼ |
| [Volcano Query Pipeline (OpRelQuery)] [Secondary Index Scan (OpIndexScan)] [Direct DML Ops] |
| - Composable Iterators (HashJoin, etc.) - SkipList & B+Tree Range Iteration - OpInsert/Update |
| - work_mem Memory Governor & External Sort - Automatic Predicate Index Matching - OpDelete |
+---------------------------------------------------------------------------------------------------------+
|
v
+---------------------------------------------------------------------------------------------------------+
| Unified Storage Layer (pkg/storage) |
| storage.Engine Contract: TableSchema, IndexSchema, Value, Record, Session, Transaction |
| +----------------------------------------------+ +----------------------------------------------+ |
| | WiredTiger Engine (pkg/storage/wiredtiger) | | Memory Engine (pkg/storage/memengine) | |
| | - Default production engine (Cgo) | | - Pure Go fallback / hermetic test engine | |
| | - Lock-free B+Tree with hazard pointers | | - Optimistic Concurrency Control (OCC) | |
| | - MVCC read-only snapshot isolation | | - MessagePack Base Snapshot (tables.mpk) | |
| | - Low-level disk checkpointing & write log | | - Append-Only Write-Ahead Log (wal.aof) | |
| +----------------------------------------------+ +----------------------------------------------+ |
+---------------------------------------------------------------------------------------------------------+
|
v
+---------------------------------------------------------------------------------------------------------+
| High Availability, Replication & Backup Layer |
| - 1-Writer Multi-Reader Topology: Linear WAL Streaming & Replica Broadcaster (pkg/replication) |
| - Zero-Downtime Failover & Replica Promotion (/api/replication/promote, /api/replication/demote) |
| - Non-Blocking Online Snapshot Hot Backup (.tar.gz with manifest, schema, catalog, mpk data) |
| - Point-in-Time Restore Engine (CLI: -backup / -restore, HTTP: /api/backup, /api/restore) |
| - Daemon Lifecycle Management (start, stop, restart, status with PID and lockfiles) |
+---------------------------------------------------------------------------------------------------------+
To balance sub-microsecond parsing latency with complete syntax compatibility for complex enterprise Microsoft T-SQL scripts, LionSQL uses a dual-parser architecture:
flowchart TD
A["T-SQL Client Script / Query"] --> B["Primary: Native Pratt Parser\n(pkg/proc/lexer.go, parser.go)"]
B -- "Success\n(Standard DDL/DML, Flow, Select)" --> E["Unified AST\n(*proc.CreateProcStmt)"]
B -- "Syntax Failure / Complex Syntax\n(WITH RECOMPILE, [dbo].[Proc], Vendor Options)" --> C["Secondary: ANTLR4 Fallback\n(pkg/proc/parser/antlr)"]
C -- "Official TSqlParser.g4 Parse Tree" --> D["ASTBuilder Transformation"]
D --> E
E --> F["Native VDBE Bytecode Compiler"]
F --> G["Bytecode VM & Storage Engine"]
- Primary Fast-Path (Native Pratt Parser):
- Hand-crafted zero-allocation lexer and recursive descent / Pratt expression parser.
- Tailored for high-frequency queries (
SELECT,INSERT,UPDATE,DELETE,IF...ELSE,WHILE,TRY...CATCH,BEGIN...COMMIT/ROLLBACK). - Achieves sub-microsecond parsing time per statement.
- Secondary Fallback (Official Microsoft T-SQL ANTLR4 Parser):
- Compiles the official ANTLR4
TSqlParser.g4grammar into Go. - Handles bracket escapes (
[dbo].[accounts]), complex vendor procedure options (WITH ENCRYPTION,WITH RECOMPILE), cursor declarations, and exotic DDL syntax.
- Compiles the official ANTLR4
- Pluggable Decoupling (Circular Dependency Immunity):
pkg/procdefines theProcedureFallbackParserfunction hook.pkg/proc/parser/antlr/ast_builder.goregisters its fallback implementation ininit().pkg/procnever importspkg/proc/parser/antlr, preserving clean dependency graphs and eliminating circular imports.
The query optimizer uses both logical equivalence transformations and statistical cost estimation:
[SQL Query]
│
▼
[AST Logical Plan]
│
▼
[Logical Planner] ───> Logical Operators (LogicalScan, LogicalFilter, LogicalJoin, LogicalAggregate)
│
▼
[Cost-Based Physical Optimizer (pkg/planner/cost.go)]
│ ├── Catalog Statistics (ANALYZE TABLE)
│ │ ├── Row Count, Page Count, Average Tuple Width
│ │ └── Column Stats (NDV, Null Count, Equi-Depth Histograms)
│ ├── Operator Cost Model:
│ │ ├── SeqScan vs IndexScan (Page I/O + Tuple CPU * Selectivity)
│ │ └── HashJoin vs IndexNestedLoopJoin (Build Table Selection)
│ └── Histogram Range Selectivity Estimation
▼
[Physical Plan Tree] ───> [Volcano Iterator Pipeline]
ANALYZE TABLE <name>: Scans the target table within an isolated read-only transaction. Computes:- Total row count and physical data size.
- Column distinct value counts (
NDV- Number of Distinct Values). - Null value fractions.
- 64-bucket Equi-Depth Histograms (
HistogramBucket{LowerBound, UpperBound, Count, DistinctCount}) for non-uniform data distribution tracking.
EXPLAIN&EXPLAIN ANALYZE: Displays estimated cost ranges (cost=0.00..45.20 rows=100) alongside actual wall-clock execution timings and tuple counts across all pipeline stages.
All relational queries execute via the Volcano Iterator Model (pkg/execution/):
type Iterator interface {
Init(ctx context.Context) error
Next() (Tuple, error)
Close() error
}SeqScanIterator&IndexScanIterator: Stream rows sequentially or through secondary index ranges.FilterIterator&ProjectIterator: On-the-fly expression evaluation with zero heap allocations for filtered-out rows.HashJoinIterator: Builds an in-memory hash table on the smaller relation (determined by CBO) and probes with the streaming outer relation.IndexNestedLoopJoinIterator: Probes the inner index directly for each outer tuple.GroupByIterator: Hash-based and pre-sorted streaming group aggregations (COUNT,SUM,AVG,MIN,MAX).SortIterator(with External Merge Sort):- Tracks accumulated tuple memory footprints against
work_mem. - When memory exceeds
work_mem, sorts the in-memory chunk and spills it to a disk run file (lionsql-sort-run-*.tmp). - Merges spilled run files and in-memory tuples via an N-way Min-Heap (
tupleMergeHeap). - Deterministically closes and unlinks all temporary run files in
Close().
- Tracks accumulated tuple memory footprints against
- Server-wide default:
DefaultWorkMem = 64MB(set via CLI-work-mem=64MB). - Dynamic session overrides:
SET work_mem = '128MB',SHOW work_mem,SELECT @@work_mem.
LionSQL unifies all procedural logic and interactive queries (SELECT, INSERT, UPDATE, DELETE) on the exact same register-based VDBE Virtual Machine:
[Interactive SQL / T-SQL Script / Prepared Stmt]
│
▼
[Bytecode Compiler (pkg/proc/compiler.go)]
- CompileQuery: SELECT queries ➔ OpRelQuery
- CompileStatement: INSERT/UPDATE/DELETE ➔ OpInsert/Update/Delete
- CompileProcedure: Stored procedures, triggers, UDFs
│
▼
[Bytecode Program (Opcode Stream)]
00: OpInit reg: 0
01: OpOpenRead table: "accounts"
02: OpIndexScan table: "accounts", idx: "idx_holder", val: "Alice"
03: OpColumn col: "balance", reg: 1
04: OpLt reg: 1, val: 500, jump: 08
05: OpUpdate table: "accounts", reg: 1
06: OpCommit
07: OpHalt code: 0
08: OpRaiseError msg: "Insufficient funds"
09: OpRollback
10: OpHalt code: -1
│
▼
[Virtual Machine (pkg/proc/runtime/vm.go)]
- Evaluates opcodes sequentially with program counter (IP)
- Zero-allocation sync.Pool register array recycling
- Executes within existing session transactions via s.executor.ExecuteInTx
- Connects Volcano iterator engine via OpRelQuery for joins & aggregations
- Zero-parse execution for prepared statements cached in PGWire & TDS
- Interactive DQL & DML Unification:
SELECTqueries compile into single-instruction programs (OpRelQuery) executing the Volcano physical query pipeline while binding ambient transaction and session parameters.INSERT,UPDATE, andDELETEcompile intoOpInsert,OpUpdate, andOpDeletewith schema validation, identity column tracking, and row affected reporting.
- Prepared Statement Bytecode Caching:
- Both PostgreSQL Wire Protocol (
Parse) and Microsoft TDS (sp_prepare) precompile parameterized queries into*proc.Programinstances. Executeandsp_executeinvokes.executor.Run(program, ...)directly, eliminating repeated lexical parsing and AST construction.
- Both PostgreSQL Wire Protocol (
- Secondary Index Opcode
OpIndexScan:- Scans records through B+Tree or SkipList secondary indexes, avoiding full table scans when filter predicates match indexed columns.
- Virtual Delta Tables for Triggers:
- Materializes ephemeral virtual delta tables (
INSERTEDandDELETED) during trigger execution.
- Materializes ephemeral virtual delta tables (
The database engine abstracts the storage tier behind the storage.Engine interface contract, providing two concrete implementations:
[ storage.Engine Interface ]
│
┌─────────────────────────┴─────────────────────────┐
│ │
▼ ▼
[ WiredTiger Storage Engine ] [ In-Memory OCC Storage Engine ]
(pkg/storage/wiredtiger) (pkg/storage/memengine)
- Cgo binding to libwiredtiger - Pure Go (Zero Cgo, cross-platform)
- Lock-free B+Tree indexing - Row-level versioning & OCC validation
- Hazard pointer memory protection - In-memory sorted secondary indexes
- MVCC transaction snapshot isolation - Hybrid persistence:
- Automated background checkpointing * Base MessagePack snapshot (tables.mpk)
- Low-level disk WAL write logging * Monotonic LSN Append-Only Log (wal.aof)
* Automatic AOF rewrite compaction
| Feature | WiredTiger (wiredtiger) |
Memory Engine (memengine) |
|---|---|---|
| Language & Build | Cgo (requires libwiredtiger) |
Pure Go (CGO_ENABLED=0 compatible) |
| Concurrency Control | Multi-Version Concurrency Control (MVCC) | Optimistic Concurrency Control (OCC) |
| Disk Data Structure | Lock-Free B+Tree with hazard pointers | Hash Map + Sorted Index Entry Slices |
| Transaction Durability | WiredTiger disk checkpoints & log | Hybrid MessagePack snapshot + AOF log |
| Fsync Policy | Handled by WiredTiger subsystem | FsyncEverySec, FsyncAlways, FsyncNone |
| Primary Use Case | Production enterprise multi-core OLTP | Embedded systems, dev/test, CI pipelines |
LionSQL exposes concurrent native wire protocol listeners on distinct ports, eliminating external protocol adapters:
sequenceDiagram
autonumber
participant Client as Client Application (psql / ORM / SSMS)
participant Multiplexer as Goroutine Session Multiplexer (pkg/api/pool.go)
participant Worker as Execution Worker Pool
participant Engine as Storage Engine (WiredTiger / Memory)
Client->>Multiplexer: Connect (TCP Handshake on :5433 or :1433)
Note over Multiplexer: Spawns lightweight ~4KB Go goroutine.<br/>Zero OS process overhead.
Multiplexer->>Multiplexer: Acquire idle session from SessionPool
Client->>Multiplexer: Query Packet ('Q' or SQLBatch)
Multiplexer->>Worker: Enqueue query job to worker channel
Worker->>Engine: Execute query under ACID transaction
Engine-->>Worker: Result rows / affected count
Worker-->>Multiplexer: Formatted response packet
Multiplexer-->>Client: Stream RowDescription, DataRows, CommandComplete
Multiplexer->>Multiplexer: Return session to pool; Worker waits for next job
- PostgreSQL Wire Protocol v3 (
pkg/wire/):- Port
:5433. - Supports Simple Query (
'Q') and Extended Query protocols ('P'Parse,'B'Bind,'D'Describe,'E'Execute,'S'Sync,'C'Close). - Full compatibility with
psql, DBeaver, GORM, Prisma, and PostgreSQL drivers.
- Port
- PostgreSQL Bulk Copy Streaming (
pkg/wire/copy.go):COPY <table> FROM STDIN [WITH (FORMAT CSV, HEADER)]:- Server issues
CopyInResponse ('G'). - Client streams
CopyData ('d')chunks. - Server parses CSV/TSV and batch-inserts rows inside an ACID transaction.
- Sends
CommandComplete ('C')with row count.
- Server issues
COPY <table> TO STDOUT [WITH (FORMAT CSV, HEADER)]:- Server issues
CopyOutResponse ('H'). - Streams table rows in ~32KB
CopyData ('d')chunks under a non-blocking snapshot transaction. - Concludes with
CopyDone ('c')andCommandComplete ('C').
- Server issues
- Fully compatible with
psql \copyandpg_dump.
- Microsoft SQL Server TDS 7.x/8.0 Protocol (
pkg/wire/tds/):- Port
:1433. - Supports PRELOGIN, LOGIN7, SQLBatch, and RPC token parsing for SSMS, Azure Data Studio, and
sqlcmd.
- Port
LionSQL includes a live database backup engine that operates without taking locks or pausing incoming read and write transactions:
Live Production Engine (Active Reads & Writes)
│
▼ (Open Read-Only Snapshot Transaction)
[Point-in-Time View T0 (MVCC/OCC)] ─── Zero lock contention; writers never blocked
│
├── Scan Schemas ──────────> schemas.json
├── Scan Catalog ──────────> catalog/ (procedures, functions, triggers)
├── Stream Table Records ──> data/<table_name>.jsonl (Buffered chunks)
└── Compute Summary ───────> manifest.json (Timestamp, row counts, checksums)
│
▼ (Gzip & Tar Stream)
[Atomic File Writer (Golden Rule 1)] ─── Write to lionsql-backup-*.tmp ➔ Atomic rename
│
▼
[Self-Contained Backup Archive: lionsql-backup-20240922-133000.tar.gz]
- HTTP Streaming API:
GET /api/backup&POST /api/backup: Direct streaming download asapplication/gzipor server-side disk snapshot via{"target": "path"}.POST /api/restore: Multipart form upload or file-path restore ({"source": "path"}).
- CLI Commands:
lionsql -dir data -backup backup.tar.gzlionsql -dir data -restore backup.tar.gz
- Embedded Web Studio:
- One-click backup download button and restore upload modal.
LionSQL embeds a PocketBase-style administration dashboard (pkg/ui/dashboard.html served from /_/ via embed.FS):
- Real-Time Data Sync: Server-Sent Events (SSE) stream (
/api/realtime) automatically pushes table mutation events (ActionCreate,ActionUpdate,ActionDelete,ActionImport) to active browser tabs. - Interactive SQL Query Studio: Real-time syntax editor with query execution history, saved query favorites, and
EXPLAIN ANALYZEexecution plan visualization. - Spreadsheet Table Editor: In-place row editing, column sorting, pagination, and full-text record search.
- Security & API Keys: Superuser JWT authentication with granular API key generation (read-only and table-scoped access tokens).
- System Metrics: Real-time worker pool and storage session metrics (
/api/pool/stats).
lionsql/
├── bin/ # Compiled binaries (server, bench)
├── cmd/
│ ├── server/ # Server entry point, CLI flags, daemon control
│ └── bench/ # High-concurrency OLTP & storage benchmarks
├── doc/
│ ├── ADR.md # Architecture Decision Records (ADRs 001 - 018)
│ └── ARCHITECTURE.md # Complete System Technical Specification (This document)
├── pkg/
│ ├── api/ # HTTP REST API, Session Pooler, Realtime Broker, Auth
│ ├── auth/ # Password hashing (bcrypt), JWT generation & revocation
│ ├── backup/ # Non-blocking snapshot backup & restore engine
│ ├── catalog/ # System catalog, table schemas, procedures, triggers, stats
│ ├── config/ # JSON and CLI configuration loader and hierarchical resolution
│ ├── execution/ # Volcano Iterator engine, memory governor, external sort
│ ├── planner/ # Cost-Based Optimizer (CBO), cost formulas, histograms
│ ├── proc/ # T-SQL compiler, native Pratt parser, plan cache
│ │ ├── parser/antlr/ # Official Microsoft T-SQL ANTLR4 fallback parser
│ │ └── runtime/ # Native VDBE Bytecode VM, ambient transactions
│ ├── replication/ # 1-Writer Multi-Reader WAL streaming, broadcaster, replica client
│ ├── storage/ # Unified storage.Engine interface, Value/Record abstractions
│ │ ├── memengine/ # Pure-Go OCC in-memory engine with WAL/AOF persistence
│ │ └── wiredtiger/ # WiredTiger Cgo storage engine implementation
│ ├── ui/ # Embedded HTML/CSS/JS dashboard (embed.FS)
│ └── wire/ # Multi-wire protocols: PostgreSQL v3, COPY, TDS 7.x/8.0
├── GEMINI.md # Project engineering rules and architectural invariants
├── GEMINI.ko.md # Project engineering rules and architectural invariants (Korean)
├── TODO.ko.md # Actionable daily engineering backlog and task tracker (Korean)
└── go.mod # Go module definitions
# Standard startup with WiredTiger engine (default)
./bin/lionsql -http=127.0.0.1:8090 -pg=127.0.0.1:5433 -tds=127.0.0.1:1433 -dir=data
# Pure-Go in-memory engine (zero Cgo dependency)
./bin/lionsql -engine=memory -http=127.0.0.1:8090 -dir=data
# Configure query memory governor budget
./bin/lionsql -work-mem=128MB -max-workers=32 -max-idle-sessions=128
# Daemon mode lifecycle controls
./bin/lionsql start -dev # Start daemon in background
./bin/lionsql status # Check running daemon status and PID
./bin/lionsql restart -dev # Restart running daemon cleanly
./bin/lionsql stop # Stop daemon gracefully# Offline or CLI-driven snapshot hot backup
./bin/lionsql backup -dir=data -target=/backups/lionsql-snapshot.tar.gz
# Restore database from archive
./bin/lionsql restore -dir=data -source=/backups/lionsql-snapshot.tar.gz
# Online live backup via curl
curl -OJ http://127.0.0.1:8090/api/backup
# Bulk copy import via psql
psql -h 127.0.0.1 -p 5433 -d postgres -c "\copy accounts from 'accounts.csv' with (format csv, header)"
# Bulk copy export via psql
psql -h 127.0.0.1 -p 5433 -d postgres -c "\copy accounts to 'accounts.csv' with (format csv, header)"All feature development, bug fixes, and refactorings must strictly satisfy the project's 3-Stage Verification Pipeline:
- Stage 1: Compilation Integrity (Build)
go build -tags wiredtiger -buildvcs=false -o bin/server ./cmd/server && \ go build -tags wiredtiger -buildvcs=false -o bin/lionsql ./cmd/lionsql && \ go build -tags wiredtiger -buildvcs=false -o bin/bench ./cmd/bench && \ go build -tags wiredtiger -buildvcs=false -o bin/tpcc-runtime ./cmd/tpcc_runtime
- Stage 2: Race-Free Test Verification (Test)
Must achieve 100% PASS across all packages with 0 data races.
go test -tags wiredtiger -count=1 -v -race ./... - Stage 3: Conventional Commit (Commit)
- Follow Conventional Commits format (
feat(...),fix(...),refactor(...), etc.). - Strictly avoid decorative emojis in commit messages, comments, and documentation (Golden Rule 19).
- Follow Conventional Commits format (