Cassandra backend does not respect the configured consistency level on initial connection

[dave92082]

While attempting to leverage AWS Keyspaces (Apache Cassandra compatible serverless implementation) as a backend, Vault is unable to establish an initial connection. When the connection is initially established, the consistency level is hard coded to Quorum for the connection and then is updated after the connection is established.

AWS Kesyspaces only supports the following consistency levels:

• ONE
• LOCAL_ONE
• LOCAL_QUORUM

During the setup of the connection, the Cassandra backend verifies that the consistency is set and defaults to LOCAL_QUORUM which is supported by AWS Keyspaces however that default consistency is not passed to the gocql driver prior to starting the session. If the consistency is not passed to gocql prior to initialization, the client defaults to QUORUM which is not supported.

The configured consistency is validated with the cassandra.go physical package:

           if cs, ok := conf["consistency"]; ok {
		switch cs {
		case "ANY":
			consistency = gocql.Any
		case "ONE":
			consistency = gocql.One
		case "TWO":
			consistency = gocql.Two
		case "THREE":
			consistency = gocql.Three
		case "QUORUM":
			consistency = gocql.Quorum
		case "ALL":
			consistency = gocql.All
		case "LOCAL_QUORUM":
			consistency = gocql.LocalQuorum
		case "EACH_QUORUM":
			consistency = gocql.EachQuorum
		case "LOCAL_ONE":
			consistency = gocql.LocalOne
		default:
			return nil, fmt.Errorf("'consistency' must be one of {ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, LOCAL_ONE}")
		}
	}

After validation however, the session is established prior to setting the consistency level within the gocql client:

        connectStart := time.Now()
	cluster := gocql.NewCluster(hosts...)
	cluster.Port = port
	cluster.Keyspace = keyspace

	sess, err := cluster.CreateSession()
	if err != nil {
		return nil, err
	}
	metrics.MeasureSince([]string{"cassandra", "connect"}, connectStart)
	sess.SetConsistency(consistency)

Within the gocql driver used for the underlying connection, the Consitency is defaulted to Quorum if it is not set prior to calling CreateSession:

func NewCluster(hosts ...string) *ClusterConfig {
	cfg := &ClusterConfig{
		Hosts:                  hosts,
		CQLVersion:             "3.0.0",
		Timeout:                600 * time.Millisecond,
		ConnectTimeout:         600 * time.Millisecond,
		Port:                   9042,
		NumConns:               2,
		Consistency:            Quorum,
		MaxPreparedStmts:       defaultMaxPreparedStmts,
		MaxRoutingKeyInfo:      1000,
		PageSize:               5000,
		DefaultTimestamp:       true,
		MaxWaitSchemaAgreement: 60 * time.Second,
		ReconnectInterval:      60 * time.Second,
		ConvictionPolicy:       &SimpleConvictionPolicy{},
		ReconnectionPolicy:     &ConstantReconnectionPolicy{MaxRetries: 3, Interval: 1 * time.Second},
		WriteCoalesceWaitTime:  200 * time.Microsecond,
	}
	return cfg
}

To Reproduce
Steps to reproduce the behavior:

1. Create a basic configuration for Vault that uses an AWS Keyspaces table:

ui            = true
cluster_addr  = "http://127.0.0.1:8201"
api_addr      = "http://127.0.0.1:8200"
disable_mlock = true

storage "cassandra" {
  hosts = "cassandra.us-east-1.amazonaws.com:9142"
  consistency = "LOCAL_QUORUM"
  protocol_version = 3
  initial_connection_timeout = 1000
  tls = 1
  tls_skip_verify = 1
  username = "keyspaces-XXXXXXXXXXXXXXXXX"
  password = "XXXXXXXXXXX-REDACTED-XXXXXXXXX"
}

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_disable = true
}

2. Start Vault server:

vault server --config vault.hcl

3. Examine output of the connection attemp:

2023/12/29 11:11:26 error: failed to connect to 3.234.248.236:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.238.167.216:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.238.167.154:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.234.248.208:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.234.248.227:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.234.248.207:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:26 error: failed to connect to 3.238.167.151:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:27 error: failed to connect to 3.238.167.236:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:27 error: failed to connect to 3.234.248.220:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
2023/12/29 11:11:27 error: failed to connect to 3.234.248.253:9142 due to error: Consistency level QUORUM is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE
Error initializing storage of type cassandra: no connections were made when creating the session
2023-12-29T11:11:25.625-0800 [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""

Expected behavior
Vault should respect the configured consistency level and connect to the database successfully

Environment:

• Vault Server Version (retrieve with vault status): N/A - Cannot Start due to failed connection
• Vault CLI Version (retrieve with vault version): Vault v1.15.4 (9b61934559ba31150860e618cf18e816cbddc630), built 2023-12-04T17:45:28Z
• Server Operating System/Architecture: Darwin/Arm64

Vault server configuration file(s):

ui            = true
cluster_addr  = "http://127.0.0.1:8201"
api_addr      = "http://127.0.0.1:8200"
disable_mlock = true

storage "cassandra" {
    hosts = "cassandra.us-east-1.amazonaws.com:9142"
    consistency = "LOCAL_QUORUM"
    protocol_version = 3
    initial_connection_timeout = 1000
    tls = 1
    tls_skip_verify = 1
    username = "keyspaces-XXXXXXXXXXXXXXXXX"
    password = "XXXXXXXXXXX-REDACTED-XXXXXXXXX"
}

listener "tcp" {
    address       = "127.0.0.1:8200"
    tls_disable = true
}