Skip to content
AditS-H
LORA
Public

About

A complete local coding AI pipeline that scrapes, processes, fine-tunes, benchmarks, exports, and serves a QLoRA-based coding model with hybrid RAG support, optimized for RTX 3060 12GB systems.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

Local Coder AI — v2.0

A 7-stage pipeline that scrapes → processes → fine-tunes → serves a local coding AI. Built for RTX 3060 12GB · 16GB RAM · Windows/Linux. Base model: Qwen2.5-Coder-14B-Instruct via QLoRA.

What This Builds

After running all phases you get:

Output Description
outputs/model_q4km.gguf Fine-tuned GGUF model (~8 GB, Q4_K_M) ready for Ollama
outputs/training_manifest.json Reproducibility record — links GGUF to exact dataset + config + scores
outputs/benchmark_results.json HumanEval + MBPP pass@1 scores vs base model
data/chromadb/ ChromaDB vector store (BGE-M3 embeddings, ~136k docs)
data/bm25_index.pkl BM25 exact-match index over the same corpus
http://localhost:8000 FastAPI chat server with hybrid RAG + web UI

System Requirements

  • GPU: NVIDIA RTX 3060 12 GB VRAM (minimum). RTX 3070+ works fine.
  • RAM: 16 GB minimum (32 GB recommended for processing 136k pairs)
  • Disk: ~80 GB free (base model ~28 GB + GGUF ~8 GB + dataset + ChromaDB)
  • OS: Windows 10/11 or Linux (Ubuntu 20.04+)
  • Python: 3.10 or 3.11
  • CUDA: 12.1+ (check with nvcc --version)

External Tools (install once)

1. Ollama — model serving

Linux:

curl -fsSL https://ollama.ai/install.sh | sh

Windows: Download from https://ollama.ai and run the installer.

Verify:

ollama --version

2. llama.cpp — GGUF conversion

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release -j8
pip install -r requirements.txt
cd ..

The llama.cpp folder must sit next to the project root (or update export.llama_cpp_path in config.yaml).

Installation

Step 0 — Clone and configure

git clone <your-repo-url> local-coder-ai
cd local-coder-ai

pip install -r requirements.txt

Copy and edit the config:

cp config.yaml config.yaml.bak   # keep a backup

Open config.yaml and set:

collectors:
  stackoverflow:
    api_key: "YOUR_SO_API_KEY"     # optional but raises quota 300 → 10000/day

  github:
    token: "ghp_YOUR_TOKEN_HERE"   # REQUIRED — https://github.com/settings/tokens

server:
  ingest_api_key: "some-random-secret-string"   # CHANGE THIS

GitHub token needs repo (read-only) scope. Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic).

Full Pipeline — Step by Step

Step 1 — Collect data

Collect from all sources (Stack Overflow, GitHub, docs, HuggingFace):

python orchestrator.py collect
python orchestrator.py collect --source all       # same as above

Collect from individual sources:

python orchestrator.py collect --source stackoverflow
python orchestrator.py collect --source github
python orchestrator.py collect --source docs        # cppreference + MDN + Python docs
python orchestrator.py collect --source hf          # HuggingFace datasets (no scraping)

Collection is resumable — if interrupted, it picks up from the last checkpoint:

python orchestrator.py collect --source github --resume    # default: resume=true
python orchestrator.py collect --source github --no-resume # start over

Expected time: 4–12 hours depending on network speed and API quota. Stack Overflow and GitHub are the longest. Run overnight. Output: data/raw/raw.db

Step 2 — Process data into training pairs

python orchestrator.py process

This runs the full pipeline:

  1. Load raw items from SQLite
  2. tree-sitter syntax validation
  3. MinHash + cross-dedup deduplication
  4. Heuristic quality filter (0–100 score, threshold 40)
  5. AI quality scoring pass (Qwen2.5-Coder-1.5B, threshold 3.0/5.0)
  6. Build single-turn instruction pairs
  7. Generate multi-turn conversation chains (15% of dataset)
  8. Balance language distribution (35% C++, 25% Python, …)
  9. Shuffle + stratified train/eval split (95/5)
  10. Write data/processed/train.jsonl, eval.jsonl, stats.json

Optional flags:

python orchestrator.py process --skip-dedup          # skip MinHash (faster, less clean)
python orchestrator.py process --skip-ai-score       # skip the 5-hour AI scoring pass
python orchestrator.py process --min-quality 50      # stricter heuristic filter (default: 40)
python orchestrator.py process --skip-ai-score --min-quality 45   # combine flags

Expected time: 5–8 hours (AI scoring dominates). Use --skip-ai-score for a quick test run. Output: data/processed/train.jsonl (~129k pairs), eval.jsonl (~7k pairs), stats.json

Step 3 — Install training dependencies

# Install PyTorch first (match your CUDA version)
pip install torch --index-url https://download.pytorch.org/whl/cu121

# Install Unsloth (REQUIRED for 12GB VRAM)
pip install "unsloth[cu121-torch210] @ git+https://github.com/unslothai/unsloth.git"

# Install the rest
pip install -r requirements-train.txt

Close LM Studio and Ollama before installing/training — they hold VRAM.

Step 4 — Train

python orchestrator.py train
  • Downloads Qwen2.5-Coder-14B-Instruct (~28 GB, first run only)
  • Trains 3 epochs with QLoRA (LR=1e-4, batch=1, grad_acc=8)
  • Saves checkpoints every 500 steps to outputs/lora_adapter/
  • Runs HumanEval + MBPP benchmarks after training completes

Optional flags:

python orchestrator.py train --epochs 1              # quick single-epoch run
python orchestrator.py train --model 7b              # use 7B model (faster, less capable)
python orchestrator.py train --skip-benchmark        # skip HumanEval + MBPP after training
python orchestrator.py train --epochs 2 --model 7b  # combine flags

Expected time: 6–10 hours for 3 epochs on RTX 3060. If you run out of VRAM: reduce max_seq_length to 2048 in config.yaml, or switch to 7B. Output: outputs/lora_adapter/

Step 5 — Export to GGUF

python orchestrator.py export

This:

  1. Merges LoRA adapter into base model (FP16, ~28 GB temp)
  2. Converts to GGUF Q4_K_M via llama.cpp (~8 GB final)
  3. Verifies the GGUF with a test prompt
  4. Writes outputs/training_manifest.json

Quantization options:

python orchestrator.py export --quant q4_k_m    # default — 8 GB, very good quality
python orchestrator.py export --quant q6_k      # 11 GB, excellent quality
python orchestrator.py export --quant q8_0      # 15 GB, near-perfect quality
python orchestrator.py export --quant q3_k_m    # 6 GB, good quality, fastest
python orchestrator.py export --quant q2_k      # 4.5 GB, acceptable quality

Disk warning: The merge step needs ~28 GB free for outputs/merged_hf/ (temp). It is safe to delete outputs/merged_hf/ after the GGUF is verified. Output: outputs/model_q4km.gguf

Step 6 — Load into Ollama

A Modelfile is generated by the export step. Register the model:

ollama create local-coder -f outputs/Modelfile

Test it:

ollama run local-coder "Write a TCP server in C++"
ollama run local-coder "Explain std::move with a complete example"
ollama run local-coder "Fix this segfault: [paste code]"

Step 7 — Build RAG indexes

python orchestrator.py ingest-rag
  • Embeds all processed pairs into ChromaDB (BGE-M3 embeddings)
  • Builds BM25 exact-match index
  • Only needs to run once (or when you add new data)

Force rebuild of BM25 index:

python orchestrator.py ingest-rag --rebuild-bm25

Expected time: 1–3 hours for ~136k pairs. Do NOT run while training — both need VRAM. Output: data/chromadb/, data/bm25_index.pkl

Step 8 — Start the chat server

python orchestrator.py serve

Then open http://localhost:8000/ui in your browser.

Override port:

python orchestrator.py serve --port 9000

The server starts at http://127.0.0.1:8000 by default (localhost only). To expose on your LAN, change server.host to 0.0.0.0 in config.yaml.

All CLI Commands Reference

collect

python orchestrator.py collect [OPTIONS]

Options:
  --source [all|stackoverflow|github|docs|hf]
                    Data source to collect from.  [default: all]
  --resume / --no-resume
                    Resume from last checkpoint.  [default: resume]
  --config PATH     Path to config.yaml  [default: config.yaml]

process

python orchestrator.py process [OPTIONS]

Options:
  --skip-dedup      Skip MinHash deduplication (faster, less clean).
  --skip-ai-score   Skip AI quality scoring pass (saves ~5 hours).
  --min-quality FLOAT
                    Minimum heuristic quality score (0-100).  [default: 40.0]
  --config PATH     Path to config.yaml  [default: config.yaml]

train

python orchestrator.py train [OPTIONS]

Options:
  --epochs INTEGER  Override number of training epochs.
  --model [7b|14b]  Base model size.  [default: 14b]
  --skip-benchmark  Skip HumanEval + MBPP benchmarks after training.
  --config PATH     Path to config.yaml  [default: config.yaml]

export

python orchestrator.py export [OPTIONS]

Options:
  --quant [q4_k_m|q6_k|q8_0|q3_k_m|q2_k]
                    GGUF quantization level.  [default: q4_k_m]
  --config PATH     Path to config.yaml  [default: config.yaml]

ingest-rag

python orchestrator.py ingest-rag [OPTIONS]

Options:
  --rebuild-bm25    Force rebuild of BM25 index even if it already exists.
  --config PATH     Path to config.yaml  [default: config.yaml]

serve

python orchestrator.py serve [OPTIONS]

Options:
  --port INTEGER    Override server port (0 = use config.yaml).  [default: 0]
  --config PATH     Path to config.yaml  [default: config.yaml]

benchmark

Run HumanEval + MBPP standalone without re-training:

python orchestrator.py benchmark [OPTIONS]

Options:
  --adapter PATH    Path to LoRA adapter.  [default: outputs/lora_adapter]
  --config PATH     Path to config.yaml  [default: config.yaml]

status

Print current pipeline progress, file sizes, benchmark scores, and GPU info:

python orchestrator.py status
python orchestrator.py status --config config.yaml

clean

Delete intermediate files for a pipeline phase:

python orchestrator.py clean [OPTIONS]

Options:
  --phase [raw|processed|outputs]
                    Which phase's data to delete.  [default: processed]

Notes:
  raw       → deletes data/raw/raw.db + data/checkpoints/
  processed → deletes data/processed/
  outputs   → deletes outputs/merged_hf/ ONLY (never auto-deletes GGUF or adapter)

rag-eval

Debug retrieval — runs the full hybrid search pipeline WITHOUT calling the model. Use to distinguish "is this a retrieval problem or a generation problem?":

python orchestrator.py rag-eval [OPTIONS]

Options:
  --query, -q TEXT      Query to test against the retrieval pipeline.  [required]
  --language, -l TEXT   Optional language filter (e.g. cpp, python).

Examples:
  python orchestrator.py rag-eval --query "std::move semantics C++"
  python orchestrator.py rag-eval -q "binary search tree" -l cpp
  python orchestrator.py rag-eval -q "async await javascript"

Note: The server must be running (python orchestrator.py serve) for this to work.

API Endpoints

Once serve is running at http://localhost:8000:

Method Endpoint Description
GET /ui Browser chat interface
GET /docs Auto-generated API docs (Swagger UI)
POST /chat RAG-augmented chat (JSON)
WS /chat/stream WebSocket streaming chat
GET /search Raw hybrid search
GET /eval Debug retrieval without calling the model
GET /status Model status, VRAM, uptime
POST /ingest Add document to knowledge base (auth required)
POST /feedback Thumbs up/down on retrieved sources

POST /chat

// Request
{
  "message": "How do I fix a segfault when iterating a vector?",
  "history": [],
  "use_rag": true,
  "language": "cpp",
  "stream": false
}

// Response
{
  "response": "The most common cause is...",
  "sources": [
    {
      "doc_id": "github_diff_00004521",
      "text": "...",
      "source": "github_diffs",
      "language": "cpp",
      "url": "https://github.com/...",
      "final_score": 0.7832
    }
  ],
  "rag_used": true,
  "retrieval_ms": 84,
  "generation_ms": 3201,
  "message_id": "a3f7c9d2-..."
}

POST /ingest (requires X-API-Key header)

curl -X POST http://localhost:8000/ingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key-from-config" \
  -d '{
    "text": "// Custom code snippet...",
    "source": "documentation",
    "language": "cpp",
    "url": "https://your-source.com"
  }'

POST /feedback

{
  "message_id": "a3f7c9d2-...",
  "source_ids": ["github_diff_00004521"],
  "rating": "up"
}

GET /eval

curl "http://localhost:8000/eval?q=std::move+semantics&language=cpp"

GET /search

curl "http://localhost:8000/search?q=binary+search+tree&top_k=5&language=cpp"

Directory Structure

local-coder-ai/
├── config.yaml                  ← all runtime parameters
├── orchestrator.py              ← CLI entry point (this file)
├── requirements.txt             ← base deps (collection, RAG, serving)
├── requirements-train.txt       ← training deps (torch, unsloth, trl)
│
├── collectors/                  ← Phase 1: data collection
│   ├── base.py
│   ├── stackoverflow.py
│   ├── github.py
│   ├── docs.py                  ← cppreference + MDN + Python docs
│   ├── hf_datasets.py           ← HuggingFace dataset loader
│   └── rate_limiter.py
│
├── storage/
│   ├── raw_store.py             ← SQLite persistence for raw items
│   └── schema.sql
│
├── processing/                  ← Phase 2: data pipeline
│   ├── extractor.py             ← tree-sitter AST extraction
│   ├── dedup.py                 ← MinHash LSH + cross-dedup
│   ├── quality.py               ← heuristic scoring
│   ├── ai_scorer.py             ← Qwen2.5-Coder-1.5B quality pass
│   ├── pairs.py                 ← instruction-tuning pair builder
│   ├── multiturn.py             ← multi-turn conversation chains
│   ├── balancer.py              ← language distribution enforcer
│   └── writer.py                ← JSONL dataset serializer
│
├── training/                    ← Phase 4: QLoRA fine-tuning
│   ├── config.py                ← TrainingConfig dataclass
│   ├── loader.py                ← Unsloth model + tokenizer init
│   ├── trainer.py               ← SFTTrainer wrapper
│   ├── benchmark.py             ← HumanEval + MBPP runner
│   └── chat_template.py
│
├── export/                      ← Phase 5: GGUF export
│   └── exporter.py              ← merge + convert + manifest
│
├── rag/                         ← Phase 6: retrieval
│   ├── embedder.py              ← BGE-M3 async embedder
│   ├── bm25.py                  ← BM25 exact-match index
│   ├── vectorstore.py           ← ChromaDB interface
│   ├── retriever.py             ← HybridRetriever (BM25 + semantic + reranker)
│   └── inference.py             ← Ollama/LM Studio client
│
├── server/                      ← Phase 7: API serving
│   ├── app.py                   ← FastAPI application
│   ├── feedback.py              ← FeedbackStore (SQLite)
│   ├── models.py                ← Pydantic request/response schemas
│   └── ui/
│       └── index.html           ← single-file chat UI
│
├── data/
│   ├── raw/                     ← raw.db (SQLite)
│   ├── processed/               ← train.jsonl, eval.jsonl, stats.json
│   ├── chromadb/                ← ChromaDB persistent storage
│   ├── bm25_index.pkl           ← BM25 index
│   ├── feedback.db              ← feedback votes
│   └── checkpoints/             ← collection checkpoints
│
└── outputs/
    ├── lora_adapter/            ← trained LoRA weights
    ├── merged_hf/               ← temp — delete after GGUF verified
    ├── model_q4km.gguf          ← final inference model
    ├── training_manifest.json   ← GGUF birth certificate
    └── benchmark_results.json

VRAM Budget (RTX 3060 12 GB)

Component VRAM
14B base model (4-bit) ~7.5 GB
LoRA adapter gradients ~1.5 GB
Optimizer state (8-bit Adam) ~1.5 GB
Activations (batch=1, seq=4096, grad ckpt) ~0.8 GB
Total ~11.3 GB ✅

If you OOM during training:

  1. Reduce max_seq_length from 4096 → 2048 in config.yaml
  2. Or switch to 7B: python orchestrator.py train --model 7b

Common Issues

"GitHub token not set" Set collectors.github.token in config.yaml. The pipeline skips GitHub without it.

OOM during training Set max_seq_length: 2048 in config.yaml or use --model 7b.

"Training data not found" You must run process before train. The file data/processed/train.jsonl must exist.

"GGUF not found" / Ollama model missing Run export first, then ollama create local-coder -f outputs/Modelfile.

Server starts but /chat returns 503 Ollama is not running or the model is not loaded. Run ollama run local-coder first.

Benchmark scores lower than base model Common causes and fixes:

  • Learning rate too high → try learning_rate: 5e-5 in config.yaml
  • AI scorer threshold too permissive → lower ai_scorer_threshold to 2.5
  • Too many multi-turn pairs → lower multiturn_ratio to 0.10

cross_dedup_fingerprints.pkl not found Download from the project releases page (~80 MB). Or disable cross-dedup: set cross_dedup_enabled: false in config.yaml.

rag-eval fails with "connection refused" The server must be running. Start it first: python orchestrator.py serve

Quick Reference — Common Workflows

First full run (everything):

pip install -r requirements.txt
# edit config.yaml: add GitHub token, set ingest_api_key
python orchestrator.py collect
python orchestrator.py process
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install "unsloth[cu121-torch210] @ git+https://github.com/unslothai/unsloth.git"
pip install -r requirements-train.txt
python orchestrator.py train
python orchestrator.py export
ollama create local-coder -f outputs/Modelfile
python orchestrator.py ingest-rag
python orchestrator.py serve
# Open http://localhost:8000/ui

Fast test run (no AI scoring, 1 epoch, 7B):

python orchestrator.py collect --source hf            # 15 min, no API keys needed
python orchestrator.py process --skip-ai-score --min-quality 50
python orchestrator.py train --model 7b --epochs 1 --skip-benchmark
python orchestrator.py export --quant q4_k_m

Rebuild RAG after adding new data:

python orchestrator.py collect --source stackoverflow
python orchestrator.py process --skip-ai-score
python orchestrator.py ingest-rag --rebuild-bm25

Debug why the model gives a bad answer:

# Is it retrieval failing, or generation failing?
python orchestrator.py rag-eval -q "your question here" -l cpp
# Check what context was injected — if bad context, fix retrieval
# If good context but bad answer, the model needs more training data on this topic

License

Training data sources:

  • Stack Overflow: CC BY-SA 4.0 — attribute, do not redistribute raw
  • GitHub repos: MIT / Apache 2.0 / BSD only (checked via API)
  • cppreference: CC BY-SA 3.0
  • MDN: CC BY-SA 2.5
  • HuggingFace datasets (The Stack v2, CodeSearchNet): Apache 2.0

The model weights, code, and configs in this repo: MIT License.

About

A complete local coding AI pipeline that scrapes, processes, fine-tunes, benchmarks, exports, and serves a QLoRA-based coding model with hybrid RAG support, optimized for RTX 3060 12GB systems.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages