Skip to content

Add RuntimeDataFrame<T> and RuntimeDataFrameFactory<T> interface - #1754

Open
CaiJimmy wants to merge 6 commits into
masterfrom
jimmy/generic-dataframe
Open

Add RuntimeDataFrame<T> and RuntimeDataFrameFactory<T> interface#1754
CaiJimmy wants to merge 6 commits into
masterfrom
jimmy/generic-dataframe

Conversation

@CaiJimmy

@CaiJimmy CaiJimmy commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

This is to prepare for the iterator refactor, which will have a centralized logic of conversion between local, RDD and dataframe representation, and avoid duplicated codes.

public interface RuntimeDataFrame<T> extends Serializable {
    /**
     * Returns the underlying physical Spark DataFrame.
     */
    Dataset<Row> getDataFrame();

    /**
     * Converts this DataFrame to its logical runtime representation.
     *
     * @param metadata query metadata used if a row cannot be decoded
     * @return an RDD of logical runtime values
     */
    JavaRDD<T> toRDD(ExceptionMetadata metadata);
}

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

QT3 Test Results - Parser: jsoniq

Regression baseline: run 30373527161

🎉 No regressions found for this build.

Summary of passed tests for jsoniq
Test Suite Passing Failing Errors Skipped Total
app 1983 39 120 15 2157
array 207 0 1 1 209
fn 10014 385 211 525 11135
map 220 0 0 0 220
math 149 0 0 0 149
misc 698 73 32 68 871
op 4147 72 75 72 4366
prod 9828 547 409 1423 12207
ser 321 21 1 0 343
xs 147 6 0 11 164
Total 27714 1143 849 2115 31821

Full analysis report: see analysis-jsoniq in artifacts.

Download detailed test results

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

QT3 Test Results - Parser: xquery

Regression baseline: run 30373527161

🎉 No regressions found for this build.

Summary of passed tests for xquery
Test Suite Passing Failing Errors Skipped Total
app 2023 38 89 7 2157
array 207 0 1 1 209
fn 10068 356 213 498 11135
map 220 0 0 0 220
math 149 0 0 0 149
misc 720 64 27 60 871
op 4270 23 32 41 4366
prod 10583 432 346 846 12207
ser 343 0 0 0 343
xs 153 0 0 11 164
Total 28736 913 708 1464 31821

Full analysis report: see analysis-xquery in artifacts.

Download detailed test results

@CaiJimmy
CaiJimmy marked this pull request as ready for review July 28, 2026 16:19
@CaiJimmy
CaiJimmy requested a review from ghislainfourny as a code owner July 28, 2026 16:19
… DataFrame creation from RDDs and local items
@CaiJimmy CaiJimmy changed the title Add RuntimeDataFrame<T> interface Add RuntimeDataFrame<T> and RuntimeDataFrameFactory<T> interface Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant