feat(kinesis) Cross-account Kinesis source support via STS assume-role in JsonKinesisSource - #19383
feat(kinesis) Cross-account Kinesis source support via STS assume-role in JsonKinesisSource#19383suryadanny wants to merge 1 commit into
Conversation
Kinesis has no resource-based policy, so reading a stream in a different AWS account requires IAM role assumption (sts:AssumeRole). Add an optional role-ARN config (hoodie.streamer.source.kinesis.role.arn) that, when set, builds the KinesisClient with an STS assume-role credentials provider (AWS SDK v2). The base STS client uses the default credential chain; static access/secret keys still take precedence. Empty/absent preserves the existing same-account behavior. No external ID is used. The assume-role provider is cached per region|roleArn so a long-lived streaming executor reuses one StsClient per distinct role instead of leaking one per mapPartitions task (AWS SDK v2 does not close a user-supplied credentials provider when the KinesisClient closes). Adds software.amazon.awssdk:sts to hudi-utilities and unit coverage for the config-key contract, ARN serialization to executors, and the credential- provider branch selection.
suryadanny
changed the title
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR adds opt-in cross-account Kinesis support by threading a new hoodie.streamer.source.kinesis.role.arn config through to an STS StsAssumeRoleCredentialsProvider, cached per region|roleArn to avoid leaking an StsClient per micro-batch. The plumbing (config → serializable KinesisReadConfig → executor client build), precedence logic, and caching all look sound after tracing the driver and executor paths. One edge case around credential precedence is worth double-checking in the inline comment. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. One small naming nit on the private helper method; otherwise the code is clean and well-documented.
| public static final ConfigProperty<String> KINESIS_ROLE_ARN = ConfigProperty | ||
| .key(PREFIX + "role.arn") | ||
| .noDefaultValue() | ||
| .sinceVersion("1.2.0") |
There was a problem hiding this comment.
| .sinceVersion("1.2.0") | |
| .sinceVersion("1.3.0") |
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>sts</artifactId> | ||
| <version>${aws.sdk.version}</version> | ||
| </dependency> |
There was a problem hiding this comment.
Should this be shaded in hudi-utilities-bundle?
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19383 +/- ##
============================================
- Coverage 70.49% 67.81% -2.68%
+ Complexity 32775 31636 -1139
============================================
Files 2700 2700
Lines 152896 152915 +19
Branches 19009 19010 +1
============================================
- Hits 107778 103702 -4076
- Misses 36596 40955 +4359
+ Partials 8522 8258 -264
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
5 participants
Tracking issue: #19382
Change Logs
Kinesis has no resource-based policy, so reading a stream in a different AWS account than the application requires IAM role assumption (
sts:AssumeRole). This adds an optional confighoodie.streamer.source.kinesis.role.arn:KinesisSourceConfig— newKINESIS_ROLE_ARNkey (no default). Naming aligns withHoodieAWSConfig.hoodie.aws.role.arn.KinesisOffsetGen.createKinesisClient— when the ARN is set (and no static keys), attach an auto-refreshingStsAssumeRoleCredentialsProvider(AWS SDK v2). The base STS client uses the default credential chain (must be grantedsts:AssumeRoleon the ARN). Static access/secret keys still take precedence; empty/absent ARN preserves the existing same-account behavior. No external ID is used.region|roleArnso a long-lived streaming executor reuses oneStsClientper distinct role instead of leaking one permapPartitionstask (AWS SDK v2 does not close a user-supplied credentials provider when theKinesisClientcloses).KinesisReadConfig— carry the ARN so executors rebuild the client with it (client is built insidemapPartitions, not on the driver).JsonKinesisSource— thread the ARN through on both driver and executor paths.hudi-utilities/pom.xml— addsoftware.amazon.awssdk:sts(StsAssumeRoleCredentialsProvideris not otherwise on the classpath;kinesisdoesn't pull it transitively).New unit tests (no Spark harness): the config-key contract, ARN round-trip +
nullpreservation, ARN surviving Java serialization to executors, and the credential-provider branch selection (assume-role vs default-chain vs static-keys-take-precedence).Impact
Opt-in; no behavior change when the config is unset. Enables reading cross-account Kinesis streams.
Risk level (write none, low medium or high below)
low — additive, gated behind a new no-default config; existing same-account path unchanged.
Documentation Update
Config is self-documenting via
withDocumentation; will surface in the generated configuration docs. No separate doc page needed.Contributor's checklist
🤖 Generated with Claude Code