CRL config entries cannot be saved due to large size

[jbrandhorst]

Describe the bug
We are running Consul Connect, with Vault in use as the Connect CA. DynamoDB is used for Vault storage. When Consul switches leadership it does a "CA initialization" routine, which generates a new PKI issuer each time. Over time the number of issuers grows, and eventually it reaches a point where the CA initialization fails, because the CRL rebuild step tries saving a CRL config entry so large that it exceeds the DynamoDB max record size, and Consul CA initialization does not complete on the Consul side.

This is the error that Consul reports, which includes the embedded error strings from Vault:

"level":"error"

"message":"Failed to initialize Connect CA","module":"connect.ca","error":"error generating intermediate cert: Error making API request.URL: PUT https://vault/v1/<pki name>/intermediate/set-signed Code: 500. 
Errors:* 1 error occurred:* Rebuilding the CRL failed. While this is indicative of a problem with the imported issuers (perhaps because of their revocation_signature_algorithm), they did import successfully and are now usable. 
It is strongly suggested to fix the CRL building errors before continuing. 
The original error is reproduced below:error building CRLs: unable to persist updated cluster-local CRL config: ValidationException: Item size has exceeded the maximum allowed size status code: 400"

"routine":"CA initialization"

Based on the specific error strings included in the log, the problem is occurring when an internalCRLConfigEntry is written to DynamoDB:

vault/builtin/logical/pki/storage.go

Lines 189 to 197 in 2e30ad5

	type internalCRLConfigEntry struct {
	IssuerIDCRLMap map[issuerID]crlID `json:"issuer_id_crl_map"`
	CRLNumberMap map[crlID]int64 `json:"crl_number_map"`
	LastCompleteNumberMap map[crlID]int64 `json:"last_complete_number_map"`
	CRLExpirationMap map[crlID]time.Time `json:"crl_expiration_map"`
	LastModified time.Time `json:"last_modified"`
	DeltaLastModified time.Time `json:"delta_last_modified"`
	UseGlobalQueue bool `json:"cross_cluster_revocation"`
	}

After analysis of the entries in the DynamoDB table, at the path logical/<secret engine UUID>/crls there is an item with a key config where this config entry is stored. When the CRL build occurs it cannot save the entry to DynamoDB because the size is now larger than DynamoDB's max field size of 400KB.

To Reproduce
Steps to reproduce the behavior:

1. Set up Consul Connect with Vault as the CA
2. Let a number of Consul leadership transitions occur over time
3. The number of issuers in Vault under the path /v1/<pki name>/issuer/<UUID>/ grows
4. Eventually a leadership transition in Consul occurs and the error above will occur because the CRL config cannot be saved when Consul Connect does its ca initialization.

We manually deleted a number of the issuers from DynamoDB in order to get past the condition and get Consul functioning again, as there did not seem to be any operator commands that would help (tidy did not help because the issuers were not expired yet)

Expected behavior
We would like to know:

1. Our CRLs are actually empty, can we simply disable CRL builds to avoid running into this problem?
2. Manually deleting issuer records from DynamoDB seems like a less-than-ideal remediation step, are there any vault operator commands that can address this type of issue?
3. We intend to modify the TTL of the Consul Connect CA intermediate cert to hopefully cut down the number of active issuers, it is currently still set as the Consul default of 1 year (8760h). Are there any other configuration settings we should look into that can reduce the size of the CRL config entry that is stored in DynamoDB?

Environment:

• Vault Server Version (retrieve with vault status): 1.14.1
• Vault CLI Version (retrieve with vault version): v1.12.4
• Server Operating System/Architecture: Alpine Linux v.3.15 (using the Vault docker image from https://hub.docker.com/layers/hashicorp/vault/1.14/images/sha256-7491542ac45b0d76b2eda4504a1b2310a4577a24cab7f7c1c34f274fcd4698f4?context=explore)
• Consul version: 1.15.3

Vault server configuration file(s):

(let me know what specific hcl is helpful here and I can edit the original post)

Additional context

1. It seems that when manually deleting old issuers, the amount of space freed in the CRL config entry (~78 bytes) is less than the amount of space consumed in the storage when a new issuer is added (~240 bytes). So even if you delete old issuers regularly the size of the CRL config will still continue to grow and may eventually run into this problem.

2. When Consul Connect is unable complete it's "CA Initialization" when encountering the CRL max record size issue it tries to reinitialize repeatedly. These attempts to initialize actually generates new issuers in Vault each time, but meanwhile Consul Connect does not actually function when it encounters the error. The repeated creation of more issuers that are unused by Consul compounds the problem, and then requires a lot more manual cleanup of the issuers that were created by Consul when Connect was in a non-functioning state.

[jkirschner-hashicorp]

Hi @jbrandhorst,

I just re-opened a related issue on the Consul repo that your post suggests was not fully closed.

For context, how many issuers were within Vault's /v1/<pki name> when you hit the reported error from Consul?

[jbrandhorst]

Hi @jbrandhorst,

I just re-opened a related issue on the Consul repo that your post suggests was not fully closed.

For context, how many issuers were within Vault's /v1/<pki name> when you hit the reported error from Consul?

Thanks @jkirschner-hashicorp - there were 1850 issuers

[cipherboy]

@jbrandhorst Many thanks for filing this, definitely agree that (1) above is a bug we want to address on the Vault side; that will be fixed in #23007.

[jkirschner-hashicorp]

PR in progress on the Consul side: hashicorp/consul#18773

[ksmiley]

I noticed there's an EncodeJSONAndCompress that seems to be a drop-in replacement for for the uncompressed EncodeJSON that this record uses currently. Could that be used here? There's a decent amount of repetition in the CRL config JSON blob (e.g. each CRL identifier is repeated four times), so it should compress well. It's not exactly a fix, but it would at least increase the number of issuers that it takes to hit the storage limit.

[cipherboy]

\o hey @ksmiley

Interesting thought... In general, simple managed rotation like this without cross signing is the easy case to trigger this. As the number of issuers grow, the automated chain building that occurs when issuers are added will be much slower than storage limitations... I think O(n log n) in the average case but IIRC O(n^3 log n) in the worst case for issuers with lots of small-ish cliques? If Consul instead switched to cross-signing, this would also slow down chain building and likely run into context deadline exceeded errors, rather than failure to write to storage due to size limitations... So I'd probably avoid having hundres or thousands of issuers within a single mount and stick to tens or at most low hundreds of issuers anyways (also, CRL building & certificate->issuer attribution gets slower with this many issuers as well, except if there are no revocations but even that requires writing & updating empty CRLs).

(Also, as an aside, the permissions model also gets a bit whacky IMO... Doable, but hard to manage).

So you're absolutely right that it could help with this case, but I think you might not want a thousand or more in the mount anyways :-)

[cipherboy]

This seems to have been addressed on both sides and will be in the next set of releases. Closing, thanks all! :-)