Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

warming cluster is not removed at new CDS response #13994

Closed
lambdai opened this issue Nov 12, 2020 · 2 comments · Fixed by #13997
Closed

warming cluster is not removed at new CDS response #13994

lambdai opened this issue Nov 12, 2020 · 2 comments · Fixed by #13997

Comments

@lambdai
Copy link
Contributor

lambdai commented Nov 12, 2020

If the cluster "foo" is warming, and the next CDS response doesn't contains "foo",
the cluster "foo" will not be removed.

This doesn't follow the concept of wildcard subscription. Also, LDS api doesn't
act like this.

CC @htuch

@lambdai lambdai added bug triage Issue requires triage labels Nov 12, 2020
@lambdai lambdai changed the title warming cluster is not removed is not removed warming cluster is not removed at new CDS response Nov 12, 2020
@lambdai
Copy link
Contributor Author

lambdai commented Nov 12, 2020

Although this behavior is even guarded by integration test.

// Send the first warming cluster.
sendDiscoveryResponse<envoy::config::cluster::v3::Cluster>(
Config::TypeUrl::get().Cluster, {buildCluster("warming_cluster_1")},
{buildCluster("warming_cluster_1")}, {"cluster_0"}, "2");
test_server_->waitForGaugeEq("cluster_manager.warming_clusters", 1);
EXPECT_TRUE(compareDiscoveryRequest(Config::TypeUrl::get().ClusterLoadAssignment, "1",
{"warming_cluster_1"}, {"warming_cluster_1"}, {"cluster_0"}));
// Send the second warming cluster.
sendDiscoveryResponse<envoy::config::cluster::v3::Cluster>(
Config::TypeUrl::get().Cluster, {buildCluster("warming_cluster_2")},
{buildCluster("warming_cluster_2")}, {}, "3");
test_server_->waitForGaugeEq("cluster_manager.warming_clusters", 2);
// We would've got a Cluster discovery request with version 2 here, had the CDS not been paused.
EXPECT_TRUE(compareDiscoveryRequest(Config::TypeUrl::get().ClusterLoadAssignment, "1",
{"warming_cluster_2", "warming_cluster_1"},
{"warming_cluster_2"}, {}));
// Finish warming the clusters.
sendDiscoveryResponse<envoy::config::endpoint::v3::ClusterLoadAssignment>(
Config::TypeUrl::get().ClusterLoadAssignment,
{buildClusterLoadAssignment("warming_cluster_1"),
buildClusterLoadAssignment("warming_cluster_2")},
{buildClusterLoadAssignment("warming_cluster_1"),
buildClusterLoadAssignment("warming_cluster_2")},
{"cluster_0"}, "2");
// Validate that clusters are warmed.
test_server_->waitForGaugeEq("cluster_manager.warming_clusters", 0);

IMHO it's a bug. Should we create a flag to optionally introduce new behavior.

@htuch
Copy link
Member

htuch commented Nov 13, 2020

Seems broken, probably don't even need a runtime flag.

@mattklein123 mattklein123 added area/cluster_manager and removed bug triage Issue requires triage labels Nov 13, 2020
@mattklein123 mattklein123 added this to the 1.17.0 milestone Nov 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants