Skip to content

Readme.md

Latest commit

 

History

History
185 lines (144 loc) · 7.32 KB

Readme.md

File metadata and controls

185 lines (144 loc) · 7.32 KB
Raw

IMAP Grab

Quick Start

  1. Start the Dev Container
  2. Inside the container, run the following command to download the emails - provide the email address, password, server, and port.
# Call download_emails.sh with the parameters
bash ./download_emails.sh -e "$EMAIL" -p "$PASSWORD" -s "$SERVER" -P "$PORT"
  1. Check the downloaded emails in the downloaded_emails directory

Migration helpers (Yahoo → local mbox)

These wrappers sit on top of imapgrab3.py + getmail. They are aimed at large Yahoo mailboxes that stall, time out, or hit Yahoo’s ~10k FETCH ceiling. All of them are safe to re-run: progress is tracked in oldmail-* files under downloaded_emails/<email_sanitized>/.

Use a Yahoo app password. Omit -p to be prompted.

Recommended order for a new account

# 1) Inbox + Sent in monthly chunks (avoids Yahoo ~10k bulk FETCH drop)
bash ./chunked_download.sh -e "user@example.com" -m "Inbox,Sent"

# 2) Optional: compare remote vs local counts
python3 check_download_completeness.py -e "user@example.com" \
  -f downloaded_emails/user_example_com

# 3) Everything else (skips Inbox/Sent by default)
bash ./download_all_remaining.sh -e "user@example.com"

Then upload with the sibling imap-upload repo (upload_remaining.sh).

chunked_download.sh

Downloads one or more folders month-by-month via IMAP SEARCH (SINCE / BEFORE).

bash ./chunked_download.sh -e "user@example.com" -m "Inbox,Sent"
bash ./chunked_download.sh -e "user@example.com" -m "Inbox,Sent" --from 2020-01
Flag Meaning
-e / -p Email / app password
-m Comma-separated folders (default: Inbox,Sent)
--from / --to Month range YYYY-MM
-w Seconds to wait after a failed attempt (default: 30)
--stall-limit Give up on a month after N no-progress tries (default: 3)

download_all_remaining.sh

Lists all IMAP folders and downloads each one except those in --skip (default: Inbox,Sent). Retries on timeout until progress stalls.

bash ./download_all_remaining.sh -e "user@example.com"
bash ./download_all_remaining.sh -e "user@example.com" --skip "Inbox,Sent,Trash"

retry_download.sh

Retries full-folder downloads (no monthly windows) until OK or no progress. Useful when a non-chunked download died mid-folder.

bash ./retry_download.sh -e "user@example.com" -m "Inbox,Sent"

check_download_completeness.py

Prints Remote vs Local vs Missing for selected folders (default Inbox/Sent). May hang on Yahoo LIST after the table — Ctrl+C is fine if the counts already printed.

python3 check_download_completeness.py -e "user@example.com" \
  -f downloaded_emails/user_example_com

download_emails.sh

Original helper: list folders and download (good for small accounts or a first pass).

bash ./download_emails.sh -e "user@example.com"

Notes

  • Output: downloaded_emails/<email_with_@_and_._replaced>/
  • Sitting at ~10,000 local messages often means Yahoo’s FETCH cap — re-run chunked_download.sh to pull months that were never fetched.
  • Do not commit downloaded_emails/ (mail content).

About

This is a fork of the original IMAP Grab written by Daniel Roesler (Diafygi) for Python 2. The fork does not include the GUI but works with Python 3.

ImapGrab is a cli and gui program (written in Python) that allows you to log into an IMAP server, list mailboxes, and download selected mailboxes to mbox or maildir files. It requires getmail 4.8.2 or higher.

The basic functions of listing and downloading have been tested. No further development or maintenance is planned. The repository includes the cli code of the original 0.1.4 release for Python 2 as well as a version for Python 3 modified by me. I've set the version number to 0.2.0.

The code has been released under the GPLv2 license.

The following is copied and adapted from the -now abandoned- original project on sourceforge

Description

imapgrab.py is a command line interface (CLI) that allows you to log into an IMAP server and download selected mailboxes (i.e. folders/labels) to mbox files or maildir folders. You can also list the mailboxes on that server. It is basically a wrapper for getmail (a mail downloading program). When done downloading selected mailboxes, you end up with one .mbox file per mailbox or one set of maildir folders per mailbox.

Features

  • GUI for easy listing/downloading
  • Download in Mbox or Maildir formats
  • SSL connections possible
  • multiple select mailboxes may be downloaded in one command
  • all available folders may be downloaded
  • folders may be excepted when above is enabled
  • when mbox file is detected, only new mail is downloaded, but this can be changed (with -a)
  • special option to except Google Mail (Gmail) folders from download
  • GUI preset for Gmail's server settings

Dependancies

getmail 4.8.2 or higher (tested with 5.14) python 3.5 or higher (tested with 3.7.7)

CLI Usage

imapgrab [-ldaSv] [-s] SERVER [-P] PORT [-u] USERNAME [-p] PASSWORD [-m] "BOX1,BOX2,..." [-f] DIRECTORY

Possible arguments

  • --list -l
    • List the mailboxes available for download
  • --download -d
    • Download mailboxes to separate mbox files
  • --mbox -B
  • --maildir -M
  • --all -a
    • Force download all mail in a mailbox (optional)
  • --ssl -S
    • Use SSL connection (optional)
  • --server -s
    • IP or domain of server (required)
  • --port -P
    • Port of server (optional)
  • --username -u
    • Username for account (required)
  • --password -p
    • Password for account (required)
  • --mailboxes -m
    • Comma separated list of mailboxes to download (i.e. "Box1, Box2, Box3") ({,} for non-separating commas) ("ALL" for all mailboxes) ("ALL, -Box1" to except Box1 from ALL) ("ALL, -Gmail" to except Gmail* and Google Mail* folders) (required for -d)
  • --folder -f
    • Path to folder (optional, creates imapgrab folder in current directory as default)
  • --localuser -L
    • User that writes to the mailboxes (use only when involking imapgrab as root)
  • --quiet -q
    • Don't display any output
  • --verbose -v
    • Verbose output
  • --debug
    • Print debug output
  • --version
    • Print version
  • --about
    • Display detailed info
  • --help -h
    • Print help with command options

CLI Examples

  • List available mailboxes
    • imapgrab -l -s imap.example.com -u username -p password
  • Download "box1" and "box2" from server imap.example.com (save "box1.mbox" and "box2.mbox")
    • imapgrab -d -s imap.example.com -u username -p password -m "box1, box2"
  • Download all mailboxes except "box3"
    • imapgrab -d -s imap.example.com -u username -p password -m "_ALL_, -box3"
  • Download all Gmail custom labels and INBOX (none of the Gmail* or Google Mail* mailboxes)
    • imapgrab -d -S -s imap.gmail.com -u username -p password -m "_ALL_, -_Gmail_"
  • Download Gmail label "receipts"
    • imapgrab -d -S -s imap.gmail.com -u username -p password -m "receipts"