Add legacy SCEP PKCS7 compatibility

[ArtWachowski]

Name of feature:

Opt-in legacy PKCS#7 compatibility for SCEP clients.

Pain or issue this feature alleviates:

This change addresses two PKCS#7 interoperability problems encountered with legacy SCEP implementations:

Some legacy clients sign authenticated attributes in their original encoded order instead of the DER SET-sorted order normally used during PKCS#7 signature verification. This results in crypto/rsa: verification error even though the original PKCS#7 signature is valid for the encoded attribute order.
SignedData.SetEncryptionAlgorithm() is currently honored by SignWithoutAttr(), but ignored by AddSignerChain(), which always derives the DigestEncryptionAlgorithm from the key and digest algorithm. This prevents callers from generating legacy-compatible SignerInfo values such as rsaEncryption when using RSA/SHA-1.

The changes add an opt-in verification fallback for the first case and make AddSignerChain() honor an explicitly configured encryption algorithm for the second.

Why is this important to the project (if not answered above):

This improves interoperability with older SCEP/PKCS#7 implementations while preserving the existing PKCS#7 behavior by default.

The legacy authenticated-attribute verification path is explicitly disabled unless requested by the caller.

The AddSignerChain() change also makes its behavior consistent with the existing SetEncryptionAlgorithm() API and with SignWithoutAttr().

This forms part of the fix for smallstep/certificates#1723.

Is there documentation on how to use this feature? If so, where?

The authenticated-attribute compatibility behavior is exposed through:

VerifyWithOptions(VerifyOptions{AllowUnsortedAuthenticatedAttributes: true})

and:

VerifyWithChainAndOptions(...)

For signing, the existing API is used:

SignedData.SetEncryptionAlgorithm(...)

before calling AddSigner() / AddSignerChain().

Existing Verify(), VerifyWithChain(), and signing behavior remain unchanged when these options are not used.

In what environments or workflows is this feature supported?

PKCS#7 signature verification where compatibility with legacy authenticated-attribute encoding is explicitly enabled.

PKCS#7 SignedData generation where callers explicitly select a DigestEncryptionAlgorithm through SetEncryptionAlgorithm().

The changes were tested as part of SCEP enrollment interoperability with a legacy client based on strongSwan 5.0.2.

In what environments or workflows is this feature explicitly NOT supported (if any)?

The legacy authenticated-attribute fallback is not enabled globally and does not alter the default verification path.

If AllowUnsortedAuthenticatedAttributes is false or omitted, verification continues to use the existing DER SET-sorted representation only.

No encryption algorithm is forced by this change. If SetEncryptionAlgorithm() is not called, AddSignerChain() continues to infer the algorithm exactly as before.

Supporting links/other PRs/issues:
Part of the fix for smallstep/certificates#1723 — SCEP enrollment failing with crypto/rsa: verification error.
Related legacy SCEP/PKCS#7 interoperability discussion: strongswan/strongswan#2753.
Corresponding smallstep/scep PR adds the SCEP-level opt-in for legacy authenticated-attribute verification.
Corresponding smallstep/certificates PR exposes both compatibility behaviors through SCEP provisioner configuration.

💔Thank you!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Artur Wachowski seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}