langchain-mrscraper

LangChain integration package for the MrScraper SDK.

This package exposes MrScraper capabilities as LangChain tools so agents can:

• Fetch rendered HTML from protected websites
• Scrape Google SERP (search results) synchronously
• Create AI scrapers from natural-language prompts
• Rerun AI/manual scrapers (single and bulk)
• List and fetch scraping results

Installation

pip install -U langchain-mrscraper

or:

uv add langchain-mrscraper

mrscraper-sdk is installed automatically as a dependency, so users do not need to install it separately.

Quick start

import os
from langchain_mrscraper import MrScraperToolkit

os.environ["MRSCRAPER_API_TOKEN"] = "your-token"

tools = MrScraperToolkit().get_tools()

Use with an agent

from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(token="your-token").get_tools()
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)

Available tools

• mrscraper_google_serp
• mrscraper_fetch_html
• mrscraper_create_scraper
• mrscraper_rerun_ai_scraper
• mrscraper_bulk_rerun_ai_scraper
• mrscraper_rerun_manual_scraper
• mrscraper_bulk_rerun_manual_scraper
• mrscraper_get_all_results
• mrscraper_get_result_by_id

API styles

You can initialize via:

• MrScraperToolkit(...).get_tools() (recommended)
• load_mrscraper_tools(...) convenience function
• per-tool constructors with token="..."
• environment variables MRSCRAPER_API_TOKEN

Tools vs. loaders

This integration is intentionally tools-first. MrScraper endpoints are action-oriented (fetch, create, rerun, list, retrieve) and best represented as BaseTool methods that agents can call explicitly.

A document loader abstraction is usually better when the primary job is deterministic "URL -> documents" ingestion into vector stores. MrScraper can support that in a separate package later, but this package should remain focused on agent tools.

Compliance & Legal Risk

WARNING Scraping login-protected pages carries serious legal and compliance risks. Many websites explicitly prohibit automated access in their Terms of Service, and bypassing authentication to scrape content may expose you to legal action including lawsuits, account termination, and financial penalties. By proceeding on scraping login-protected pages, you confirm that you have read and understood the target website's Terms of Service, and you fully accept all legal, financial, and ethical responsibility for your actions.