Add AI docs support for publishing and resolving documentation artifacts - #41
Add AI docs support for publishing and resolving documentation artifacts#41oguzkocer wants to merge 7 commits into
Conversation
Introduces an `aiDocs` concept that allows libraries to publish structured
documentation alongside Maven artifacts, and consumers to resolve them
on-demand.
Publisher side (via existing `publish-to-s3` plugin):
- `aiDocs { from(file("docs/ai-reference")) }` zips and attaches as
`ai-docs` classifier artifact to all Maven publications
Consumer side (new `com.automattic.android.ai-docs` plugin):
- `aiDocs { resolve("group:artifact") }` declares docs to fetch
- `./gradlew resolveAiDocs` resolves, unpacks to `.ai-docs/`, and
cleans stale versions — resolution only happens at task execution
time, never during configuration
Write resolved/unpacked AI docs to `build/ai-docs` instead of the project-root `.ai-docs`. They are a build artifact, so this makes them cleaned by `clean` and implicitly gitignored, with no per-consumer `.gitignore` entry required.
There was a problem hiding this comment.
Pull request overview
Adds an aiDocs capability to the Gradle plugin ecosystem in this repo, enabling producers to publish a structured docs ZIP alongside Maven artifacts and consumers to resolve/unpack those docs on demand.
Changes:
- Introduces
ZipAiDocsTaskand wires it intocom.automattic.android.publish-to-s3to attach anai-docsclassifier ZIP to Maven publications. - Adds a new consumer plugin
com.automattic.android.ai-docswithResolveAiDocsTaskto fetch and unpack AI docs artifacts. - Adds functional tests and registers the new plugin in
plugin/build.gradle.kts.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| plugin/src/main/kotlin/com/automattic/android/publish/ZipAiDocsTask.kt | New task to zip a configured AI docs directory for publishing. |
| plugin/src/main/kotlin/com/automattic/android/publish/ResolveAiDocsTask.kt | New task to resolve and unpack ai-docs ZIP artifacts to a local directory. |
| plugin/src/main/kotlin/com/automattic/android/publish/PublishToS3Plugin.kt | Creates aiDocs extension and wires publishing configuration into the existing publish plugin. |
| plugin/src/main/kotlin/com/automattic/android/publish/AiDocsPlugin.kt | New consumer plugin + shared publishing/resolving configuration helpers. |
| plugin/src/main/kotlin/com/automattic/android/publish/AiDocsExtension.kt | New extension API (from(...), resolve(...)) for producer/consumer configuration. |
| plugin/src/functionalTest/kotlin/com/automattic/android/publish/AiDocsFunctionalTest.kt | Functional tests for zipping and consumer plugin application. |
| plugin/build.gradle.kts | Registers the new com.automattic.android.ai-docs Gradle plugin. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| val entryPath = file.relativeTo(sourceDir).path | ||
| zos.putNextEntry(ZipEntry(entryPath)) | ||
| file.inputStream().use { it.copyTo(zos) } |
| zip.entries().asSequence().forEach { entry -> | ||
| val targetFile = File(targetDir, entry.name) | ||
| if (entry.isDirectory) { | ||
| targetFile.mkdirs() | ||
| } else { | ||
| unpackEntry(zip, entry, targetFile) | ||
| } | ||
| } |
| val parts = notation.split(":") | ||
| val group = parts[0] | ||
| val artifact = parts[1] | ||
| val version = parts[2] | ||
|
|
| override fun apply(project: Project) { | ||
| val extension = project.extensions.create("aiDocs", AiDocsExtension::class.java) | ||
| project.configureAiDocsResolving(extension) | ||
| } |
| // Resolved docs are a build artifact: cleaned by `clean` and implicitly gitignored. | ||
| task.outputDirectory.set(rootProject.layout.buildDirectory.dir("ai-docs")) | ||
| } |
| assertTrue(result.output.contains("resolveAiDocs") || !result.output.contains("resolveAiDocs"), | ||
| "Plugin should apply without error even with no dependencies configured") |
- `zipAiDocs`: write ZIP entries with forward slashes (`invariantSeparatorsPath`) so the archive layout is correct on Windows - `resolveAiDocs`: guard unpacking against zip-slip path traversal; reject entries that resolve outside the target directory - `resolveAiDocs`: match artifact `version` (not just group/name) so the right docs are unpacked when multiple versions are present - `resolveAiDocs`: validate the coordinate notation with a clear error instead of an opaque `IndexOutOfBoundsException` - Reuse an existing `aiDocs` extension when present so `com.automattic.android.publish-to-s3` and `com.automattic.android.ai-docs` can be applied together without a name clash - Replace the tautological functional-test assertion with real checks and add coverage for the resolve notation and the both-plugins-applied case
| sourceDir.walkTopDown() | ||
| .filter { it.isFile } | ||
| .forEach { file -> | ||
| // Use forward slashes so the ZIP layout is correct on all platforms. | ||
| val entryPath = file.relativeTo(sourceDir).invariantSeparatorsPath |
| val parts = notation.split(":") | ||
| val group = parts[0] | ||
| val artifact = parts[1] | ||
| val version = if (parts.size >= NOTATION_WITH_VERSION_PARTS) { | ||
| parts[2] | ||
| } else { | ||
| resolveVersionFromDependencyGraph(group, artifact) | ||
| } | ||
|
|
| // Resolved docs are a build artifact: cleaned by `clean` and implicitly gitignored. | ||
| task.outputDirectory.set(rootProject.layout.buildDirectory.dir("ai-docs")) | ||
| } |
| val projectDir = File("build/functionalTest-aiDocs") | ||
| projectDir.mkdirs() | ||
|
|
| val projectDir = File("build/functionalTest-aiDocs-consumer") | ||
| projectDir.mkdirs() | ||
|
|
| val projectDir = File("build/functionalTest-aiDocs-both") | ||
| projectDir.mkdirs() | ||
|
|
| if (!publication.name.endsWith("PluginMarkerMaven")) { | ||
| publication.artifact(zipTask.flatMap { it.outputZip }) { | ||
| it.classifier = AI_DOCS_CLASSIFIER | ||
| it.extension = "zip" | ||
| } |
- `zipAiDocs`: sort files by relative path before zipping for a deterministic ZIP (reproducible output / stable build cache) - `resolveAiDocs`: validate the `resolve(...)` notation at configuration time with a clear error instead of an `IndexOutOfBoundsException` - `resolveAiDocs`: use `maybeCreate` for the `aiDocs` configuration so it does not fail if the configuration already exists - `resolveAiDocs`: write to the current project's build dir instead of `rootProject`, avoiding races when applied to multiple subprojects - Publishing: mark the `ai-docs` artifact `builtBy(zipAiDocs)` so publish never races ahead of the ZIP being produced - Functional tests: delete the fixed project dirs before recreating them so the tests are hermetic
| // Use forward slashes so the ZIP layout is correct on all platforms. | ||
| val entryPath = file.relativeTo(sourceDir).invariantSeparatorsPath | ||
| zos.putNextEntry(ZipEntry(entryPath)) | ||
| file.inputStream().use { it.copyTo(zos) } | ||
| zos.closeEntry() |
| import org.gradle.api.tasks.InputDirectory | ||
| import org.gradle.api.tasks.OutputFile | ||
| import org.gradle.api.tasks.TaskAction | ||
| import java.util.zip.ZipEntry | ||
| import java.util.zip.ZipOutputStream | ||
|
|
||
| abstract class ZipAiDocsTask : DefaultTask() { | ||
| @get:InputDirectory | ||
| abstract val sourceDirectory: DirectoryProperty |
| val deps = extension.dependencies.getOrElse(emptyList()) | ||
| if (deps.isEmpty()) return@afterEvaluate | ||
|
|
||
| val aiDocsConfig = configurations.maybeCreate("aiDocs").apply { | ||
| isTransitive = false | ||
| isCanBeConsumed = false | ||
| } |
| if (isUpToDate(versionDir)) { | ||
| logger.lifecycle("AI docs up-to-date: ${versionDir.absolutePath}") | ||
| return@forEach | ||
| } | ||
|
|
||
| versionDir.mkdirs() | ||
| unpackZip(matchingArtifact.file, versionDir) | ||
| logger.lifecycle("AI docs saved to: ${versionDir.absolutePath}") | ||
| } | ||
| } | ||
|
|
||
| private fun isUpToDate(versionDir: File): Boolean = | ||
| versionDir.exists() && versionDir.list()?.isNotEmpty() == true |
| @TaskAction | ||
| fun resolve() { | ||
| val outputDir = outputDirectory.get().asFile | ||
| outputDir.mkdirs() | ||
|
|
||
| val resolvedArtifacts = aiDocsConfiguration.resolvedConfiguration.resolvedArtifacts | ||
|
|
Address the recurring Gradle-idiom review findings at their root rather than patching each symptom. - Replace the hand-rolled `ZipAiDocsTask` with Gradle's `Zip` task: reproducible archive (stable order + fixed timestamps), forward-slash entries, correct input path-sensitivity, and automatic `builtBy` wiring. The custom task is deleted. - `resolveAiDocs`: extract into a freshly cleaned version directory so an interrupted run can't leave a partial/mixed docs tree. Drops the fragile "non-empty dir means up-to-date" check; Gradle's own up-to-date check still skips the task when inputs/outputs are unchanged. - Add a functional test that publishes an `ai-docs` artifact to a file Maven repo and asserts `resolveAiDocs` unpacks it and removes stale versions. - Document that resolution uses the consumer's repositories on purpose; a plugin-added repository would break `FAIL_ON_PROJECT_REPOS` builds.
- Resolve the `aiDocs` configuration through a lenient artifact view so a coordinate that publishes no `ai-docs` classifier is skipped with a per-module warning instead of failing `resolveAiDocs` - Track resolved artifact files as `@InputFiles` and carry only a coordinate->file map into the task, so re-published content re-extracts and the task is compatible with the configuration cache - Unpack into a freshly cleaned version directory and remove other versions of the same module (a configuration resolves one version per module) - Write the producer ZIP to `build/ai-docs-archive` so it doesn't overlap the consumer's `build/ai-docs` output directory - Set the task description in registration; note the Gradle 7.4+ requirement - Add functional tests for resolve/unpack/cleanup, a missing `ai-docs` artifact, and configuration-cache compatibility
| private fun Project.resolveVersionFromDependencyGraph(group: String, artifact: String): String { | ||
| val matchingDep = configurations | ||
| .flatMap { it.dependencies } | ||
| .find { it.group == group && it.name == artifact } | ||
|
|
||
| return matchingDep?.version | ||
| ?: throw IllegalStateException( | ||
| "Cannot resolve version for '$group:$artifact'. " + | ||
| "Either add it as a dependency or specify the version explicitly: " + | ||
| "resolve(\"$group:$artifact:VERSION\")" | ||
| ) | ||
| } |
| val parts = coordinate.split(":") | ||
| if (parts.size != COORDINATE_PARTS) return@forEach | ||
| val (group, artifact, version) = parts |
- `resolveVersionFromDependencyGraph` collects all declared versions for a module and fails fast when none (e.g. the version comes from a platform/BOM or a constraint) or multiple distinct versions are found, instead of silently picking the first declared dependency - `resolveAiDocs` fails fast on a malformed coordinate instead of silently skipping it, surfacing wiring bugs rather than leaving stale docs - Mark the `aiDocs` configuration `isCanBeResolved = true` so its resolve-only role is explicit under stricter Gradle configuration-role checks
|
@wzieba @ParaskP7 I wanted to note that: because I've been away from the Gradle world for quite some time, I had to heavily rely on Claude for best practices for this PR. This current implementation looks reasonable to me from what I know and remember about implementing custom Gradle plugins/tasks, but I can't be sure, so I'm hoping that you can help me address any issues during the PR review. I was also thinking that even if there are some minor issues, it shouldn't impact any of the existing clients, and I'll find out about them during integration. End-to-end tests I have tried worked out well, but the integration I have is still a POC, so I again can't be sure that it's 100% what we want. Please let me know if you have any questions/concerns! Edit: I forgot to note that all of Copilot's comments should be addressed. I didn't reply to each comment because after its second review, I realized the direction of the PR was wrong and made a big refactor. The three comments from the latest review, which came after the refactor, have all been addressed in 0916199. |
|
Hi @oguzkocer ! Thanks for working on this, it sounds interesting!
I've seen some similar efforts in other projects. E.g. On my AI Enablement program I've experimented with this idea in parsely-android and I've seen solid improvement when asked to integrate the library with the plugin/skill, vs. "just read source" approach. One quick thought I have is:
These are my first thoughts @oguzkocer but I have a question for you: how critical is this for you? Together with @ParaskP7 we're doing RSM 2.0 right now and I try to prioritize my project whenever I can. This PR rather falls into category of non-essentials, so I think I can't work much on it right now. At the same time I don't want to block you with testing it in real life. Maybe we could release this as an After the RSM 2.0 ends, I could take a deeper look and make a post-mortem review, which we could address in another PR, if needed. WDYT? |
|
This is indeed an interesting idea @oguzkocer and what @wzieba said above, +1 for me too! 💯 Btw, I did skim this change real quick, just to understand the basics of it, mainly the e2e flow, so that when I'll find time to properly review it I would be prepared. And, as I was doing so, I want to quickly note something that confused me, and maybe it will others too. It is about this UPDATE: Maybe I need to run the e2e flow with a client using this new |
|
@wzieba @ParaskP7 Thank you both for taking a look and responding very quickly. Really appreciate it 🙇♂️ This PR is a part of my RSM project, so unfortunately it looks like our timelines will not match up. However, it shouldn't take too much time to port it to Just to give you some context: in Once the RSM is over, I don't think I'll have the bandwidth to work on this, so I'll close this PR for now. Please feel free to use or discard the work however you wish. If you end up making it part of the plugin, we can see about using it in To clarify one thing: when I started this project, I considered directly implementing it in Thanks again for your quick responses 🙇♂️ |
|
Gotcha @oguzkocer, thank you for this update and the extra context you provided! 🙇 👍 |
What & why
Adds first-class support for shipping a compact, machine-readable AI docs bundle alongside a library's normal Maven artifacts, plus a consumer-side plugin to fetch and unpack it on demand.
The goal: let tooling (e.g. an AI coding assistant working in a downstream repo) read a small, structured API reference for a dependency instead of crawling large generated sources. The bundle is attached as a standard Maven classifier artifact (
ai-docs, a.zip), so it rides the existing publish/resolve infrastructure — no custom upload/download paths.How it works
Publisher side —
com.automattic.android.publish-to-s3The existing plugin gains an
aiDocs { }extension. Pointing it at a directory zips that directory and attaches the zip to every Maven publication as theai-docsclassifier.plugins { id "com.automattic.android.publish-to-s3" } aiDocs { // A static directory, or (preferably) a docs-generation task's output. from(file("docs/ai-reference")) // from(generateDocs.flatMap { it.outputDirectory }) }zipAiDocstask (a GradleZip, so the archive is reproducible — stable entry order, fixed timestamps, forward-slash entries — andbuiltByis wired automatically).build/ai-docs-archive/and attached to each non–plugin-markerMavenPublicationasclassifier = "ai-docs",extension = "zip".from(...)is a task output,zipAiDocsdepends on it automatically; the source need not exist at configuration time.Consumer side —
com.automattic.android.ai-docs(new plugin)A separate, consumer-only plugin (does not apply
maven-publish).plugins { id "com.automattic.android.ai-docs" } aiDocs { // Version optional — inferred from the dependency graph when omitted. resolve("rs.wordpress.api:kotlin:1.2.3") }resolveAiDocs, which resolvesgroup:artifact:version:ai-docs@zipand unpacks it tobuild/ai-docs/<group>/<artifact>/<version>/.build/, so it is cleaned bycleanand implicitly git-ignored.ai-docsartifact is skipped with a per-module warning instead of failing the task.Key design decisions
dependencyResolutionManagement). The plugin deliberately does not register its own repository, which would break builds usingRepositoriesMode.FAIL_ON_PROJECT_REPOS.Map<String, File>(coordinate → file) and tracks resolved files via@InputFiles, so a re-published SNAPSHOT with new content re-extracts.ArtifactCollection.getResolvedArtifacts()).Files
AiDocsExtension.kt—from(...)/resolve(...)DSL (shared by both plugins).AiDocsPlugin.kt— consumer plugin + the publisher/consumer wiring.ZipAiDocsTask.ktwas removed in favor of Gradle's built-inZiptask.ResolveAiDocsTask.kt— resolve + unpack + stale cleanup + zip-slip guard.PublishToS3Plugin.kt— registers the sharedaiDocsextension and the publishing wiring.AiDocsFunctionalTest.kt— functional coverage.Testing
Automated
Functional tests cover:
zipAiDocsproduces a non-empty archive atbuild/ai-docs-archive/ai-docs.zip.resolveAiDocsdownloads, unpacks, and removes stale versions (publishes a test module with anai-docsclassifier to a file Maven repo).ai-docsartifact resolves cleanly with no output (graceful skip).resolveAiDocsruns with--configuration-cache(guards against carrying non-serializable values into the task).publish-to-s3andai-docstogether shares oneaiDocsextension (no name clash).Manual end-to-end
This plugin is published tag-based, so manual testing uses a local Maven repo. Example using a
kotlinlibrary as producer and any project as consumer.Publish the plugin locally (the
localMaven repo inplugin/build.gradle.kts, or:plugin:publishToMavenLocal):Point the test projects'
pluginManagementat that repo and use the published version.Publish a producer with docs. In a library module:
aiDocs { from(file("docs/ai-reference")) }then publish (
:lib:publishToMavenLocal, orprepareToPublishToS3 ... publish). Verify the published coordinate has a*-ai-docs.zipnext to the.jar/.pom/.module, e.g.:Resolve in a consumer. In another project, ensure the repo hosting the artifact is configured (e.g.
mavenLocal()independencyResolutionManagement), then:plugins { id "com.automattic.android.ai-docs" } aiDocs { resolve("<group>:<artifact>:<version>") }Re-run is incremental / cleanable.
Stale cleanup. Bump the resolved version, re-run
resolveAiDocs, and confirm the previous version directory underbuild/ai-docs/<group>/<artifact>/is gone.Graceful skip. Point
resolve(...)at a dependency that has noai-docsclassifier and confirmresolveAiDocssucceeds with anAI docs artifact not found for ...warning and no output.Notes for reviewers
from(...)accepts aFile,Directory, orProvider<Directory>; prefer a task-output provider sozipAiDocsorders after doc generation.resolve(...), it is inferred from declared dependencies; the build fails fast if it can't be determined unambiguously (e.g. version supplied only via a platform/BOM).