DSCI 575 Project -- Amazon Product Query Assistant

Team: Prabuddha Tamhane (pat0216), Ojasv Issar (ojasv31)

Repository: https://github.com/UBC-MDS/Amazon_recommender_search_assistant

About

A retrieval-augmented generation (RAG) system for the Appliances category of the Amazon Reviews 2023 dataset. Given a natural language query (e.g. "energy efficient dishwasher under $500"), the system retrieves relevant products using BM25 keyword search and semantic embedding search, generates grounded answers via an open-source LLM, and presents results through a Streamlit web app.

LLM Choice

We use Ollama with llama3.2:3b (Llama 3.2, Meta, 3 billion parameters) as our default LLM for RAG generation.

• Why Llama 3.2 3B: After comparing qwen2.5:3b and llama3.2:3b on 5 identical queries with the same retrieved context, Llama 3.2 produced better-grounded answers with more citations and direct review quotes. See results/final_discussion.md for the full comparison.
• Why 3B: Good balance between answer quality and inference speed on laptop hardware (M2 Air, 16 GB RAM). Both 3B models run at ~15-25 tokens/sec locally.
• Why Ollama: Runs fully offline with no API key needed, making it easy for TAs and collaborators to reproduce.
• Alternative: The pipeline also supports HuggingFace Inference API (Qwen/Qwen3.5-2B) if Ollama is unavailable -- select it in the app sidebar.

Dataset

We use the Appliances category from the Amazon Reviews 2023 dataset (McAuley Lab, UCSD):

File	Description
Appliances.jsonl.gz	User reviews -- ratings, review text, timestamps, helpful votes
meta_Appliances.jsonl.gz	Product metadata -- titles, descriptions, features, price, categories

Key fields used for retrieval:

• Metadata: title, description, features, price, average_rating
• Reviews: text, rating, verified_purchase, helpful_vote
• Join key: parent_asin

For EDA we work with a smaller subset; for the retrieval pipeline we download the full Appliances category using DuckDB and store it as parquet in data/processed/. Raw files go in data/raw/ and are not committed.

Data Processing

1. Stream review and metadata files from the McAuley Lab servers using DuckDB
2. Inspect the first 200 entries in the EDA notebook to understand schema, sample text, and missing values
3. Convert full dataset to parquet format locally for fast repeated queries
4. Join reviews with metadata on parent_asin
5. Drop rows with missing review text or product title
6. Aggregate to product-level documents (top 5 reviews per product by helpfulness)

Preprocessing details and justifications are documented in notebooks/milestone1_exploration.ipynb.

Retrieval Methods

BM25 (keyword search)

Lexical retrieval using rank-bm25 (BM25Okapi) through our retriever layer. Documents and queries are tokenized with lowercasing and basic punctuation removal. Returns top-k results ranked by BM25 score. Main implementation is in src/ranking.py with API helpers in src/bm25.py.

Persistence:

• Tokenized corpus and BM25 index are saved to data/processed/bm25_index.pkl.

Semantic Search (embedding-based)

Uses sentence-transformers (all-MiniLM-L6-v2) when available, with a TF-IDF cosine-similarity fallback in this workspace. Main implementation is in src/ranking.py with API helpers in src/semantic.py.

Persistence:

• When sentence-transformers is available, a FAISS index and metadata are saved under data/processed/semantic_faiss/.

Step 2: Semantic RAG Pipeline

The Step 2 RAG implementation is in src/rag_pipeline.py.

It includes:

• Semantic retrieval with top-k support (retrieve, retrieve_indices)
• Structured context builder (build_context)
• Prompt template variants (strict, concise, analyst)
• End-to-end generation flow (answer) with open-source LLM backends

Workflow Diagram

flowchart LR
    A[User Query] --> B[Semantic Retriever]
    B --> C[Top-k Documents k=5]
    C --> D[Context Builder]
    D --> E[Prompt Template]
    E --> F[Open-Source LLM]
    F --> G[RAG Answer with Citations]

Loading

Quick usage

from src.rag_pipeline import build_default_rag_pipeline

pipeline = build_default_rag_pipeline(
    provider="ollama",
    model="llama3.2:3b",
    default_k=5,
)

result = pipeline.answer(
    query="quiet dishwasher for a small apartment",
    k=5,
    prompt_variant="strict",
)

print(result.retrieved_indices)
print(result.answer)

Step 3: Hybrid RAG (BM25 + Semantic)

The Step 3 hybrid implementation is in src/hybrid_rag_pipeline.py.

Hybrid retrieval options

• simple-merge: concatenate BM25 and semantic top-k results
• merge-dedup: merge and remove duplicates
• rrf (default): Reciprocal Rank Fusion re-ranking

Default fusion weights are BM25 0.4 and semantic 0.6.

Hybrid workflow

flowchart LR
    A[User Query] --> B1[BM25 Retriever]
    A --> B2[Semantic Retriever]
    B1 --> C[Hybrid Fusion RRF]
    B2 --> C
    C --> D[Top-k Hybrid Context]
    D --> E[Prompt Template]
    E --> F[Open-Source LLM]
    F --> G[Hybrid RAG Answer]

Loading

Quick usage

from src.hybrid_rag_pipeline import FusionConfig, build_default_hybrid_rag_pipeline

pipeline = build_default_hybrid_rag_pipeline(
    provider="ollama",
    model="llama3.2:3b",
    default_k=5,
    fusion=FusionConfig(mode="rrf", bm25_weight=0.4, semantic_weight=0.6),
)

result = pipeline.answer(
    query="dishwasher that runs quietly at night",
    k=5,
    prompt_variant="strict",
)

print(result.retrieved_indices)
print(result.answer)

Step 4: Qualitative Evaluation

The qualitative evaluation workflow lives in src/qualitative_eval.py and writes the step-4 report to results/milestone1_discussion.md.

It covers:

• A 10-query set spanning easy, medium, and complex query types
• Top-5 retrieval outputs for both BM25 and semantic search
• Side-by-side comparison for five selected queries
• A short discussion of strengths, weaknesses, and cases that may need reranking or RAG

Run the report generator after your processed corpus is available (defaults to the first matching parquet under data/processed/):

python -m src.qualitative_eval

To pin a specific file:

python -m src.qualitative_eval --data-path data/processed/appliances_merged.parquet

Web App

The Streamlit app is implemented in app/app.py with two tabs:

• Search Only (Milestone 1): keyword / semantic / hybrid retrieval with top-k results, ratings, scores, and feedback buttons
• RAG Mode (Milestone 2): hybrid RAG pipeline that generates an LLM answer above the retrieved source documents

LLM provider and model can be changed in the sidebar (defaults to Ollama llama3.2:3b).

Repository Structure

DSCI_575_project_ojasv31_pat0216/
|-- README.md
|-- requirements.txt
|-- .env                       # API keys (not committed)
|
|-- data/
|   |-- raw/                   # downloaded .jsonl.gz (gitignored)
|   |-- processed/             # cleaned parquet files (gitignored)
|
|-- notebooks/
|   |-- milestone1_exploration.ipynb
|   |-- milestone2_rag.ipynb
|
|-- src/
|   |-- bm25.py
|   |-- semantic.py
|   |-- hybrid.py
|   |-- rag_pipeline.py
|   |-- hybrid_rag_pipeline.py
|   |-- llm_pipeline.py
|   |-- retrieval_metrics.py
|   |-- data_io.py
|   |-- ranking.py
|   |-- feedback.py
|   |-- qualitative_eval.py
|   |-- download_full.py
|   |-- utils.py
|
|-- scripts/
|   |-- compare_llms.py         # LLM comparison script (final submission)
|
|-- results/
|   |-- milestone1_discussion.md
|   |-- milestone2_discussion.md
|   |-- final_discussion.md
|   |-- llm_comparison.json     # raw LLM comparison outputs
|
|-- app/
    |-- app.py

Setup and Reproduction

1. Clone the repository

git clone https://github.com/UBC-MDS/DSCI_575_project_ojasv31_pat0216.git
cd DSCI_575_project_ojasv31_pat0216

2. Create environment and install dependencies

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Alternative (conda + environment file):

conda env create -f environment.yml
conda activate dsci575-project

3. Install Ollama and pull the model

# Install Ollama: https://ollama.com/download
ollama pull llama3.2:3b

Alternatively, set a HuggingFace token in .env for the HF API fallback:

HF_TOKEN=your_huggingface_token_here

4. Run the EDA notebook

Open and run notebooks/milestone1_exploration.ipynb to explore a subset of the data and understand the schema.

4.1 Download the full dataset (required for retrieval)

python -m src.download_full

This streams the full Appliances category (~2.1M reviews, ~94K products) via DuckDB and produces data/processed/appliances_products.parquet with product-level documents.

5. Run the app

streamlit run app/app.py

5.1 Build persistent indexes (recommended before running app)

python -m src.build_indexes

This creates:

• data/processed/bm25_index.pkl
• data/processed/semantic_faiss/index.faiss
• data/processed/semantic_faiss/metadata.json

The app automatically loads these persisted indexes when present.

6. Generate the Step 4 report

python -m src.qualitative_eval