Increased Latency in Auth Calls after Upgrading to Vault 1.13.1

[Rahul-Manglani]

Describe the bug
A clear and concise description of what the bug is.
After upgrading our HashiCorp Vault to version 1.13.1, we have encountered latency issues with authentication (auth) calls. We have been using AWS authentication in our setup and the below summary is based on the same, although this would apply to other auth methods as well. Upon thorough investigation, we have identified that this latency is a result of a recently introduced feature: "Enable role based quotas for lease-count quotas" (commit: 614cee3) in the Vault GitHub repository.

The root cause of the latency stems from a new method called DetermineRoleFromLoginRequest, which is utilized specifically for determining the role. In our case, this method invokes the Security Token Service (STS) every time it is called. Prior to this feature, the STS was only invoked once for AWS authentication. However, with the implementation of DetermineRoleFromLoginRequest, every auth call now triggers three additional invocations of STS for AWS authentication thus increasing the latency by ~3x.

These three invocations occur from the following points within the code:

Rate limiting handler: util.go#L70
HandleLoginRequest: request_handling.go#L1482
CreateLoginToken: request_handling.go#L1788

This increased frequency of STS calls for AWS authentication has led to noticeable latency in our authentication workflow. We believe it would be beneficial to optimize this implementation to reduce the number of STS invocations and alleviate the resulting latency impact.

To Reproduce
Steps to reproduce the behavior:

1. Enable AWS auth - vault auth enable aws
2. Create a test policy and a test role and associate AWS IAM principal to it
3. vault login -method=aws role=test-role
4. Observe latency difference

Expected behavior
A clear and concise description of what you expected to happen.
There should be no or minimal latency in auth calls

Environment:

• Vault Server Version (retrieve with vault status): 1.13.1
  vault status
  Key Value

---

Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.15.0-beta1
Build Date n/a
Storage Type dynamodb
Cluster Name vault-cluster-cc951ebb
Cluster ID {clusterId}
HA Enabled true
HA Cluster :8200
HA Mode active
Active Since 2023-06-22T22:45:09.977161Z

• Vault CLI Version (retrieve with vault version):
• Server Operating System/Architecture:

Vault server configuration file(s):

# Paste your Vault config here.
# Be sure to scrub any sensitive values

Additional context
Stack traces that might help:

1. DetermineRoleFromLoginRequest call from RateLimitQuotaWrapping:
   awsauth.submitCallerIdentityRequest (path_login.go:1718) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginIamGetRoleNameCallerIdAndEntity (path_login.go:320) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRoleIam (path_login.go:338) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRole (path_login.go:161) github.com/hashicorp/vault/builtin/credential/aws
   :2
   framework.(*Backend).HandleRequest (backend.go:300) github.com/hashicorp/vault/sdk/framework
   :2
   plugin.(*backend).HandleRequest (backend.go:95) github.com/hashicorp/vault/builtin/plugin/v5
   vault.(*Core).DetermineRoleFromLoginRequest (core.go:3795) github.com/hashicorp/vault/vault
   vault.(*Core).DetermineRoleFromLoginRequestFromBytes (core.go:3780) github.com/hashicorp/vault/vault
   http.rateLimitQuotaWrapping.func1 (util.go:70) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapGenericHandler.func1 (handler.go:442) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   cleanhttp.PrintablePathCheckHandler.func1 (handlers.go:42) github.com/hashicorp/go-cleanhttp
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.serverHandler.ServeHTTP (server.go:2936) net/http
   http.(*conn).serve (server.go:1995) net/http
   http.(*Server).Serve.func3 (server.go:3089) net/http
   runtime.goexit (asm_arm64.s:1172) runtime

• Async Stack Trace
  http.(*Server).Serve (server.go:3089) net/http

2. STS call from HandleRequest:
   awsauth.submitCallerIdentityRequest (path_login.go:1718) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginIamGetRoleNameCallerIdAndEntity (path_login.go:320) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginUpdateIam (path_login.go:1334) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginUpdate (path_login.go:588) github.com/hashicorp/vault/builtin/credential/aws
   :2
   framework.(*Backend).HandleRequest (backend.go:300) github.com/hashicorp/vault/sdk/framework
   :2
   plugin.(*backend).HandleRequest (backend.go:95) github.com/hashicorp/vault/builtin/plugin/v5
   vault.(*Router).routeCommon (router.go:782) github.com/hashicorp/vault/vault
   vault.(*Router).Route (router.go:552) github.com/hashicorp/vault/vault
   vault.(*Core).doRouting (request_handling.go:851) github.com/hashicorp/vault/vault
   vault.(*Core).handleLoginRequest (request_handling.go:1413) github.com/hashicorp/vault/vault
   vault.(*Core).handleCancelableRequest (request_handling.go:693) github.com/hashicorp/vault/vault
   vault.(*Core).switchedLockHandleRequest (request_handling.go:501) github.com/hashicorp/vault/vault
   vault.(*Core).HandleRequest (request_handling.go:461) github.com/hashicorp/vault/vault
   http.request (handler.go:923) github.com/hashicorp/vault/http
   http.handleLogicalInternal.func1 (logical.go:378) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.handleRequestForwarding.func1 (handler.go:857) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.(*ServeMux).ServeHTTP (server.go:2500) net/http
   http.wrapHelpHandler.func1 (help.go:28) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapCORSHandler.func1 (cors.go:32) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.rateLimitQuotaWrapping.func1 (util.go:113) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapGenericHandler.func1 (handler.go:442) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   cleanhttp.PrintablePathCheckHandler.func1 (handlers.go:42) github.com/hashicorp/go-cleanhttp
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.serverHandler.ServeHTTP (server.go:2936) net/http
   http.(*conn).serve (server.go:1995) net/http
   http.(*Server).Serve.func3 (server.go:3089) net/http
   runtime.goexit (asm_arm64.s:1172) runtime

• Async Stack Trace
  http.(*Server).Serve (server.go:3089) net/http

3. DetermineRoleFromLoginRequest call from HandleLoginRequest:
   awsauth.submitCallerIdentityRequest (path_login.go:1718) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginIamGetRoleNameCallerIdAndEntity (path_login.go:320) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRoleIam (path_login.go:338) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRole (path_login.go:161) github.com/hashicorp/vault/builtin/credential/aws
   :2
   framework.(*Backend).HandleRequest (backend.go:300) github.com/hashicorp/vault/sdk/framework
   :2
   plugin.(*backend).HandleRequest (backend.go:95) github.com/hashicorp/vault/builtin/plugin/v5
   vault.(*Core).DetermineRoleFromLoginRequest (core.go:3795) github.com/hashicorp/vault/vault
   vault.(*Core).handleLoginRequest (request_handling.go:1495) github.com/hashicorp/vault/vault
   vault.(*Core).handleCancelableRequest (request_handling.go:693) github.com/hashicorp/vault/vault
   vault.(*Core).switchedLockHandleRequest (request_handling.go:501) github.com/hashicorp/vault/vault
   vault.(*Core).HandleRequest (request_handling.go:461) github.com/hashicorp/vault/vault
   http.request (handler.go:923) github.com/hashicorp/vault/http
   http.handleLogicalInternal.func1 (logical.go:378) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.handleRequestForwarding.func1 (handler.go:857) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.(*ServeMux).ServeHTTP (server.go:2500) net/http
   http.wrapHelpHandler.func1 (help.go:28) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapCORSHandler.func1 (cors.go:32) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.rateLimitQuotaWrapping.func1 (util.go:113) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapGenericHandler.func1 (handler.go:442) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   cleanhttp.PrintablePathCheckHandler.func1 (handlers.go:42) github.com/hashicorp/go-cleanhttp
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.serverHandler.ServeHTTP (server.go:2936) net/http
   http.(*conn).serve (server.go:1995) net/http
   http.(*Server).Serve.func3 (server.go:3089) net/http
   runtime.goexit (asm_arm64.s:1172) runtime

• Async Stack Trace
  http.(*Server).Serve (server.go:3089) net/http

4. DetermineRoleFromLoginRequest call from LoginCreateToken:
   awsauth.submitCallerIdentityRequest (path_login.go:1718) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginIamGetRoleNameCallerIdAndEntity (path_login.go:320) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRoleIam (path_login.go:338) github.com/hashicorp/vault/builtin/credential/aws
   awsauth.(*backend).pathLoginResolveRole (path_login.go:161) github.com/hashicorp/vault/builtin/credential/aws
   :2
   framework.(*Backend).HandleRequest (backend.go:300) github.com/hashicorp/vault/sdk/framework
   :2
   plugin.(*backend).HandleRequest (backend.go:95) github.com/hashicorp/vault/builtin/plugin/v5
   vault.(*Core).DetermineRoleFromLoginRequest (core.go:3795) github.com/hashicorp/vault/vault
   vault.(*Core).LoginCreateToken (request_handling.go:1802) github.com/hashicorp/vault/vault
   vault.(*Core).handleLoginRequest (request_handling.go:1687) github.com/hashicorp/vault/vault
   vault.(*Core).handleCancelableRequest (request_handling.go:693) github.com/hashicorp/vault/vault
   vault.(*Core).switchedLockHandleRequest (request_handling.go:501) github.com/hashicorp/vault/vault
   vault.(*Core).HandleRequest (request_handling.go:461) github.com/hashicorp/vault/vault
   http.request (handler.go:923) github.com/hashicorp/vault/http
   http.handleLogicalInternal.func1 (logical.go:378) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.handleRequestForwarding.func1 (handler.go:857) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.(*ServeMux).ServeHTTP (server.go:2500) net/http
   http.wrapHelpHandler.func1 (help.go:28) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapCORSHandler.func1 (cors.go:32) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.rateLimitQuotaWrapping.func1 (util.go:113) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.wrapGenericHandler.func1 (handler.go:442) github.com/hashicorp/vault/http
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   cleanhttp.PrintablePathCheckHandler.func1 (handlers.go:42) github.com/hashicorp/go-cleanhttp
   http.HandlerFunc.ServeHTTP (server.go:2122) net/http
   http.serverHandler.ServeHTTP (server.go:2936) net/http
   http.(*conn).serve (server.go:1995) net/http
   http.(*Server).Serve.func3 (server.go:3089) net/http
   runtime.goexit (asm_arm64.s:1172) runtime

• Async Stack Trace
  http.(*Server).Serve (server.go:3089) net/http

[heatherezell]

Are you able to share before/after latency times? Thanks! :)

[Rahul-Manglani]

Latency numbers with vault main branch:

rmanglani@rmangla-ltmdamt vault % git branch

• main
  v1.11.3
  rmanglani@rmangla-ltmdamt vault % time ./bin/vault login -method=aws role=test-role
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.

Key Value

---

token {{redacted}}
token_accessor {{redacted}}
token_duration 768h
token_renewable true
token_policies {{redacted}}
identity_policies []
policies {{redacted}}
token_meta_role_id {{redacted}}
token_meta_account_id {{redacted}}
token_meta_auth_type iam
./bin/vault login -method=aws role=test-role 0.05s user 0.02s system 2% cpu 2.225 total

rmanglani@rmangla-ltmdamt vault % time ./bin/vault login -method=aws role=test-role
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value

---

token {{redacted}}
token_accessor {{redacted}}
token_duration 768h
token_renewable true
token_policies {{redacted}}
identity_policies []
policies {{redacted}}
token_meta_auth_type iam
token_meta_role_id {{redacted}}
token_meta_account_id {{redacted}}
./bin/vault login -method=aws role=test-role 0.06s user 0.03s system 3% cpu 2.288 total

[Rahul-Manglani]

latency numbers with vault 1.11.3 branch:

rmanglani@rmangla-ltmdamt vault % git branch
main

• v1.11.3

rmanglani@rmangla-ltmdamt vault % time ./bin/vault login -method=aws role=test-role
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value

---

token {{redacted}}
token_accessor {{redacted}}
token_duration 768h
token_renewable true
token_policies {{redacted}}
identity_policies []
policies {{redacted}}
token_meta_auth_type iam
token_meta_role_id {{redacted}}
token_meta_account_id {{redacted}}
./bin/vault login -method=aws role=test-role 0.05s user 0.03s system 8% cpu 0.906 total

rmanglani@rmangla-ltmdamt vault % time ./bin/vault login -method=aws role=test-role
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value

---

token {{redacted}}
token_accessor {{redacted}}
token_duration 768h
token_renewable true
token_policies {{redacted}}
identity_policies []
policies {{redacted}}
token_meta_account_id {{redacted}}
token_meta_auth_type iam
token_meta_role_id {{redacted}}
./bin/vault login -method=aws role=test-role 0.06s user 0.03s system 9% cpu 0.894 total

[heatherezell]

Thank you! I appreciate the quick response. :)

[kschoche]

Hi! Just following up here :)
This issue was resolved in #22583 and back-ported+released in v1.13.7.
Cheers