[DG47]

Description

Closes #639.

spark.SparkJDBCDataset can currently only load whole tables. Spark's JDBC source also supports a query option, which lets the database run the filtering, joins or aggregation and return only the result. This PR adds a query argument to the dataset, as discussed in the issue:

weather_summary:
  type: spark.SparkJDBCDataset
  query: SELECT city, AVG(temperature) AS avg_temperature FROM weather_table GROUP BY city
  credentials: db_credentials
  load_args:
    properties:
      driver: org.postgresql.Driver

Development notes

• table is now optional and query is new; exactly one of them must be provided (DatasetError if both or neither are given).
• With query, load() uses spark.read.format("jdbc") with the properties from load_args (credentials included, as before) plus the url and query options. Unlike the sketch in the issue thread, the connection properties and credentials are passed through, so authenticated databases work.
• A query-based dataset is read-only: save() raises a DatasetError asking for table instead.
• With query, load_args may only contain properties. Spark does not allow partitionColumn with query, and predicates relies on dbtable, so column / lowerBound / upperBound / numPartitions / predicates are rejected at construction with a clear message rather than failing later inside Spark.
• _describe() now includes query.
• Updated the docstring (new YAML example and argument docs), the spark.SparkJDBCDataset entry in static/jsonschema/kedro-catalog-1.0.0.json (query property, table no longer required, oneOf table/query like other datasets in the schema), RELEASE.md, and the line number in .secrets.baseline that moved with the docstring.

Testing:

• pytest tests/spark/test_spark_jdbc_dataset.py --cov-config pyproject.toml --cov=kedro_datasets: 22 passed, spark_jdbc_dataset.py at 100% coverage. The 7 new tests fail on main.
• pre-commit run --hook-stage manual --files <changed files> (ruff, black, detect-secrets, bandit, blacken-docs and the file checks) passes, and mypy kedro_datasets/spark/spark_jdbc_dataset.py --ignore-missing-imports reports no issues.
• I also ran it by hand against an in-memory Apache Derby database (the driver ships with PySpark) on PySpark 3.4.4: saved a table with table=..., loaded a filtered query with query=... using credentials.url and extra properties, and confirmed that save() on the query dataset raises the new error.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Updated jsonschema/kedro-catalog-X.XX.json if necessary
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes
• Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

[ankatiyar]

Thanks @DG47 for this PR! The issue associated with this PR has not been marked for external contribution yet. As a policy, we only accept contributions for issues marked as "Ready" on this board: OS contribution tasks
I'll mark this PR as draft and bring the ticket through internal discussion. We'll let you know when it's ready for contribution. In the meantime if you would like to pick up a ticket from the board mentioned above, you're more than welcome!

[jitu5]

Thanks @DG47 for this PR! The issue associated with this PR has not been marked for external contribution yet. As a policy, we only accept contributions for issues marked as "Ready" on this board: OS contribution tasks
I'll mark this PR as draft and bring the ticket through internal discussion.