Skip to content

assign_and_map_hash_ids() reuses the same id for two elements of a page that is not contiguous #4620

Open

Description

@Eapp1e

Describe the bug

unstructured/documents/elements.py derives the "sequence number on page" with itertools.groupby(page_numbers), so the number is the index inside each contiguous run of equal page numbers and it restarts when a page number appears again later in the list. The comment above the code ("generate sequence number for each element on a page") and Element.id_to_hash() ("sequence number on page") both describe a per-page counter, and the sibling _assign_hash_ids() in unstructured/partition/common/metadata.py implements exactly that.

The visible effect: two distinct elements that share their text on a page that is not contiguous in the list get the same id, so old_to_new_mapping silently points both parent_ids at one element.

To reproduce

import copy
from unstructured.documents.elements import ElementMetadata, Text, assign_and_map_hash_ids

elements = [
    Text(text="Section", metadata=ElementMetadata(page_number=1)),
    Text(text="Interlude", metadata=ElementMetadata(page_number=2)),
    Text(text="Section", metadata=ElementMetadata(page_number=1)),
]
ids = [e.id for e in assign_and_map_hash_ids(copy.deepcopy(elements))]
print(ids, len(set(ids)))  # 2 unique ids for 3 elements

Expected behavior

Three unique ids: the third element is the second element of page 1, not the first one again.

Version info

Observed on main at 53a56a0a (unstructured 0.27.35); no extra packages are needed for the snippet above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions