docs: document secrets, custom_runtime_options, and remaining source/sink arguments in README

[david-streamlio]

What

The README's pulsar_source and pulsar_sink Properties tables lag the actual resource schema, so readers relying on the README believe connector credentials can only be supplied through the plaintext configs field — and miss a number of other supported arguments entirely.

This is a documentation gap, not a code gap: the fields are present in pulsar/resource_pulsar_source.go / pulsar/resource_pulsar_sink.go and in the generated docs/resources/*.md; only the README was stale.

Changes (README + examples)

Credential handling (the motivating fix):

• Add the secrets row to both source and sink property tables, plus custom_runtime_options and schema_type to the source table.
• Add secrets usage to the README examples and the runnable examples under examples/sources / examples/sinks, moving credentials (a GSA key / DB password) out of configs.
• Add a credential-handling note: configs is persisted in plaintext (function metadata topic, pulsar-admin ... get output, Terraform state); secrets references are resolved at runtime by the worker's configured SecretsProvider (e.g. KubernetesSecretsProviderConfigurator), so only the reference is persisted.

Remaining undocumented arguments:

• pulsar_source — producer config (max_pending_messages, max_pending_messages_across_partitions, use_thread_local_producers, batch_builder, compression_type) and crypto config (crypto_key_reader_classname, crypto_key_reader_config, encryption_keys, producer_crypto_failure_action, consumer_crypto_failure_action).
• pulsar_sink — classname, dead_letter_topic, max_redeliver_count, negative_ack_redelivery_delay_ms, retain_key_ordering, sink_type.

Why it matters

Embedding API keys / service-account keys / passwords in configs exposes them in the function metadata topic, pulsar-admin sources|sinks get output, audit logs, and Terraform state. Documenting secrets gives users the secure, already-supported path. secrets has been supported since v0.2.0 (PRs #96 / #97).

Notes

• Documentation only; descriptions are copied verbatim from the schema Description strings, so README and the generated docs/resources/*.md agree. No tfplugindocs regeneration required.
• Verified with a schema-key-vs-README cross-check: every top-level argument in both resources is now represented in the README.
• Supersedes docs: document remaining pulsar_source / pulsar_sink arguments in README #205 (consolidated here).

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

🤖 Generated with Claude Code

[github-actions]

@david-streamlio:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[github-actions]

@david-streamlio:Thanks for providing doc info!

[david-streamlio]

@maxsxu @freeznet Friendly ping when you have a moment — this is a docs-only PR (README + examples) documenting secrets and the other already-supported source/sink arguments. All checks are green and the doc label is set. Happy to adjust anything.

[freeznet]

Thanks for this — the table gaps were real and the secrets-over-plaintext-configs guidance is the right thing to document.

I verified the PR's central claim mechanically rather than by eye, and it holds:

• Completeness ✅ — extracted every resourceSource*Key / resourceSink*Key constant from pulsar/resource_pulsar_source.go and pulsar/resource_pulsar_sink.go and diffed against the two README tables on this branch: zero missing, zero extra top-level arguments in either (after excluding the 5 nested input_specs sub-keys).
• Descriptions ✅ — all new rows match the schema Description strings verbatim, so README and generated docs/resources/*.md agree. Agreed no tfplugindocs run needed.
• Required column ✅ — every new row is correctly False. The only Required: true attributes are tenant/namespace/name/archive/destination_topic_name (source) and tenant/namespace/name/cleanup_subscription/archive/auto_ack (sink).
• Secret shape ✅ — the documented { "path": ..., "key": ... } object matches what the acceptance tests already exercise (resource_pulsar_source_test.go:254, resource_pulsar_sink_test.go:237).
• No perpetual-diff risk ✅ — I ran the snippets through terraform validate/apply: jsonencode emits compact key-sorted JSON, which matches what the read path writes back via json.Marshal(map[string]interface{}). Both the README (:) and examples (=) forms parse fine.

Two things I'd like changed before merge, both in the examples rather than the tables:

1. The gsaKey secret in the source example doesn't correspond to anything the connector reads. Both source examples use pulsar-io-file with configs = {"inputDirectory":"opt"} — a filesystem source with no credentials at all. There was no plaintext credential to move, and the file connector never reads a gsaKey. Details inline.
2. examples/ should stay runnable against the repo's own local cluster. hack/pulsar-docker.sh starts a plain Pulsar standalone (default DefaultSecretsProviderConfigurator → ClearTextSecretsProvider, process/thread runtime). The {path, key} indirection is KubernetesSecretsProviderConfigurator-specific — as your README note says. terraform apply still succeeds (Pulsar stores the map without resolving it at submit time), so this fails silently: the JDBC sink just gets no password at runtime. Details inline.

The rest are nits — take or leave.

One follow-up worth a separate issue, since this PR raises the security framing: configs isn't marked Sensitive: true in either schema, so its value is echoed in plan output and stored readably in state regardless of the secrets guidance. Closing that is a code change, and I'd rather not widen a docs PR into it — but the note here would be strictly stronger once it lands.

[freeznet]

Nit — style drift from the convention already in the tree. examples/functions/main.tf:61 writes this as:

  secrets = jsonencode(
  {
    "SECRET1": {
       "path": "sectest"
       "key": "hello"
    }
  })

i.e. the paren on its own line and : separators — which is what your README snippets use. The two new blocks in examples/sinks/main.tf and examples/sources/main.tf use inline jsonencode({ with = instead. Both parse (I checked with terraform validate), so purely cosmetic, but worth matching the functions example so all three read the same.

[freeznet]

Nit, purely cosmetic: the long new keys (max_pending_messages_across_partitions, use_thread_local_producers, crypto_key_reader_classname, and negative_ack_redelivery_delay_ms in the sink table) overflow the padded column widths the rest of the table uses, so the raw source goes ragged. Renders identically on GitHub — only worth a reflow if you want future diffs to stay clean.

[david-streamlio]

Thanks for the thorough review! Addressed in 068acd5: dropped the gsaKey secrets block from the source examples (the file connector reads no credentials), restored the working password in both sink examples so they run against the local standalone cluster, and moved the secrets form to a commented-out alternative with a note that {path, key} references require a resolving SecretsProvider (Kubernetes runtime). Also picked up the type-hint nits and strengthened the credentials note. Skipped the table-padding reflow to keep this diff clean — happy to do it as a follow-up.

Agreed Sensitive: true on configs deserves its own issue.

[david-streamlio]

Filed #213 to track marking configs as Sensitive: true, per the review discussion above.