Merge branch 'master' into mpuncel/secret-hc-sequence

* master: (22 commits) http: using CONNECT_ERROR for HTTP/2 (envoyproxy#13519) listener: respect address.pipe.mode (it didn't work) (envoyproxy#13493) examples: Fix more deprecations/warnings in configs (envoyproxy#13529) overload: tcp connection refusal overload action (envoyproxy#13311) tcp: towards pluggable upstreams (envoyproxy#13331) conn_pool: fixing comments (envoyproxy#13520) Prevent SEGFAULT when disabling listener (envoyproxy#13515) Convert overload manager config literals to YAML (envoyproxy#13518) Fix runtime feature variable name (envoyproxy#13533) dependencies: refactor repository location schema utils, cleanups. (envoyproxy#13452) router: fix an invalid ASSERT when encoding metadata frames in the router. (envoyproxy#13511) http2: Proactively disconnect connections flooded when resetting stream (envoyproxy#13482) ci use azp to sync filter example (envoyproxy#13501) mongo_proxy: support configurable command list for metrics (envoyproxy#13494) http local rate limit: note token bucket is shared (envoyproxy#13525) wasm/extensions: Wasm extension policy. (envoyproxy#13526) http: removing envoy.reloadable_features.http1_flood_protection (envoyproxy#13508) build: update ppc64le CI build status shield (envoyproxy#13521) dependencies: enforce dependency shepherd sign-off via RepoKitteh. (envoyproxy#13522) Add no_traffic_healthy_interval (envoyproxy#13336) ... Signed-off-by: Michael Puncel <[email protected]>
mpuncel · Oct 14, 2020 · ef3dbab · ef3dbab
2 parents 177e31c + cfaefef
commit ef3dbab
Show file tree

Hide file tree

Showing 112 changed files with 1,751 additions and 796 deletions.
diff --git a/.azure-pipelines/pipelines.yml b/.azure-pipelines/pipelines.yml
@@ -115,6 +115,24 @@ jobs:
           artifactSuffix: ".arm64"
           bazelBuildExtraOptions: "--sandbox_base=/tmp/sandbox_base"
 
+  - job: filter_example
+    displayName: "filter-example sync"
+    dependsOn: []
+    condition: and(succeeded(), eq(variables['PostSubmit'], true))
+    steps:
+      - task: InstallSSHKey@0
+        inputs:
+          hostName: "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
+          sshPublicKey: "$(FilterExamplePublicKey)"
+          sshPassphrase: "$(SshDeployKeyPassphrase)"
+          sshKeySecureFile: "$(FilterExamplePrivateKey)"
+
+      - bash: ci/filter_example_mirror.sh
+        displayName: "Sync envoy-filter-example"
+        workingDirectory: $(Build.SourcesDirectory)
+        env:
+          AZP_BRANCH: $(Build.SourceBranch)
+
   - job: bazel
     displayName: "Linux-x64"
     dependsOn: ["release"]

diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -35,20 +35,9 @@ jobs:
              - "9d:3b:fe:7c:09:3b:ce:a9:6a:de:de:41:fb:6b:52:62"
        - run: ci/go_mirror.sh
 
-   filter_example_mirror:
-     executor: ubuntu-build
-     steps:
-       - run: rm -rf /home/circleci/project/.git # CircleCI git caching is likely broken
-       - checkout
-       - add_ssh_keys:
-           fingerprints:
-             - "f6:f9:df:90:9c:4b:5f:9c:f4:69:fd:42:94:ff:88:24"
-       - run: ci/filter_example_mirror.sh
-
 workflows:
   version: 2
   all:
     jobs:
       - api
       - go_control_plane_mirror
-      - filter_example_mirror
diff --git a/DEPENDENCY_POLICY.md b/DEPENDENCY_POLICY.md
@@ -50,8 +50,8 @@ Dependency declarations must:
 * CPEs are compulsory for all dependencies that are not purely build/test.
   [CPEs](https://en.wikipedia.org/wiki/Common_Platform_Enumeration) provide metadata that allow us
   to correlate with related CVEs in dashboards and other tooling, and also provide a machine
-  consumable join key. You can consult the latest [CPE
-  dictionary](https://nvd.nist.gov/products/cpe) to find a CPE for a dependency.`"N/A"` should only
+  consumable join key. You can consult [CPE
+  search](https://nvd.nist.gov/products/cpe/search) to find a CPE for a dependency.`"N/A"` should only
   be used if no CPE for the project is available in the CPE database. CPEs should be _versionless_
   with a `:*` suffix, since the version can be computed from `version`.
 
@@ -97,6 +97,14 @@ basis:
 Where possible, we prefer the latest release version for external dependencies, rather than master
 branch GitHub SHA tarballs.
 
+## Dependency shepherds
+
+Sign-off from the [dependency
+shepherds](https://github.com/orgs/envoyproxy/teams/dependency-shepherds) is
+required for every PR that modifies external dependencies. The shepherds will
+look to see that the policy in this document is enforced and that metadata is
+kept up-to-date.
+
 ## Dependency patches
 
 Occasionally it is necessary to introduce an Envoy-side patch to a dependency in a `.patch` file.

diff --git a/EXTENSION_POLICY.md b/EXTENSION_POLICY.md
@@ -59,6 +59,19 @@ In the event that the Extension PR author is a sponsoring maintainer and no othe
 is available, another maintainer may be enlisted to perform a minimal review for style and common C++
 anti-patterns. The Extension PR must still be approved by a non-maintainer reviewer.
 
+## Wasm extensions
+
+Wasm extensions are not allowed in the main envoyproxy/envoy repository unless
+part of the Wasm implementation validation. The rationale for this policy:
+* Wasm extensions should not depend upon Envoy implementation specifics as
+  they exist behind a version independent ABI. Hence, there is little value in
+  qualifying Wasm extensions in the main repository.
+* Wasm extensions introduce extensive dependencies via crates, etc. We would
+  prefer to keep the envoyproxy/envoy repository dependencies minimal, easy
+  to reason about and maintain.
+* We do not implement any core extensions in Wasm and do not plan to in the
+  medium term.
+
 ## Extension stability and security posture
 
 Every extension is expected to be tagged with a `status` and `security_posture` in its

diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@ involved and how Envoy plays a role, read the CNCF
 [![Azure Pipelines](https://dev.azure.com/cncf/envoy/_apis/build/status/11?branchName=master)](https://dev.azure.com/cncf/envoy/_build/latest?definitionId=11&branchName=master)
 [![CircleCI](https://circleci.com/gh/envoyproxy/envoy/tree/master.svg?style=shield)](https://circleci.com/gh/envoyproxy/envoy/tree/master)
 [![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/envoy.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:envoy)
-[![Jenkins](https://img.shields.io/jenkins/s/https/powerci.osuosl.org/job/build-envoy-master/badge/icon/.svg?label=ppc64le%20build)](http://powerci.osuosl.org/job/build-envoy-master/)
+[![Jenkins](https://powerci.osuosl.org/buildStatus/icon?job=build-envoy-static-master&subject=ppc64le%20build)](https://powerci.osuosl.org/job/build-envoy-static-master/)
 
 ## Documentation
 

diff --git a/api/bazel/envoy_http_archive.bzl b/api/bazel/envoy_http_archive.bzl
@@ -10,8 +10,7 @@ def envoy_http_archive(name, locations, **kwargs):
         # This repository has already been defined, probably because the user
         # wants to override the version. Do nothing.
         return
-    loc_key = kwargs.pop("repository_key", name)
-    location = locations[loc_key]
+    location = locations[name]
 
     # HTTP tarball at a given URL. Add a BUILD file if requested.
     http_archive(

diff --git a/api/bazel/external_deps.bzl b/api/bazel/external_deps.bzl
@@ -0,0 +1,116 @@
+load("@envoy_api//bazel:repository_locations_utils.bzl", "load_repository_locations_spec")
+
+# Envoy dependencies may be annotated with the following attributes:
+DEPENDENCY_ANNOTATIONS = [
+    # List of the categories describing how the dependency is being used. This attribute is used
+    # for automatic tracking of security posture of Envoy's dependencies.
+    # Possible values are documented in the USE_CATEGORIES list below.
+    # This attribute is mandatory for each dependecy.
+    "use_category",
+
+    # Attribute specifying CPE (Common Platform Enumeration, see https://nvd.nist.gov/products/cpe) ID
+    # of the dependency. The ID may be in v2.3 or v2.2 format, although v2.3 is prefferred. See
+    # https://nvd.nist.gov/products/cpe for CPE format. Use single wildcard '*' for version and vector elements
+    # i.e. 'cpe:2.3:a:nghttp2:nghttp2:*'. Use "N/A" for dependencies without CPE assigned.
+    # This attribute is optional for components with use categories listed in the
+    # USE_CATEGORIES_WITH_CPE_OPTIONAL
+    "cpe",
+]
+
+# NOTE: If a dependency use case is either dataplane or controlplane, the other uses are not needed
+# to be declared.
+USE_CATEGORIES = [
+    # This dependency is used in API protos.
+    "api",
+    # This dependency is used in build process.
+    "build",
+    # This dependency is used to process xDS requests.
+    "controlplane",
+    # This dependency is used in processing downstream or upstream requests (core).
+    "dataplane_core",
+    # This dependency is used in processing downstream or upstream requests (extensions).
+    "dataplane_ext",
+    # This dependecy is used for logging, metrics or tracing (core). It may process unstrusted input.
+    "observability_core",
+    # This dependecy is used for logging, metrics or tracing (extensions). It may process unstrusted input.
+    "observability_ext",
+    # This dependency does not handle untrusted data and is used for various utility purposes.
+    "other",
+    # This dependency is used only in tests.
+    "test_only",
+]
+
+# Components with these use categories are not required to specify the 'cpe'
+# and 'last_updated' annotation.
+USE_CATEGORIES_WITH_CPE_OPTIONAL = ["build", "other", "test_only", "api"]
+
+def _fail_missing_attribute(attr, key):
+    fail("The '%s' attribute must be defined for external dependecy " % attr + key)
+
+# Method for verifying content of the repository location specifications.
+#
+# We also remove repository metadata attributes so that further consumers, e.g.
+# http_archive, are not confused by them.
+def load_repository_locations(repository_locations_spec):
+    locations = {}
+    for key, location in load_repository_locations_spec(repository_locations_spec).items():
+        mutable_location = dict(location)
+        locations[key] = mutable_location
+
+        if "sha256" not in location or len(location["sha256"]) == 0:
+            _fail_missing_attribute("sha256", key)
+
+        if "project_name" not in location:
+            _fail_missing_attribute("project_name", key)
+        mutable_location.pop("project_name")
+
+        if "project_desc" not in location:
+            _fail_missing_attribute("project_desc", key)
+        mutable_location.pop("project_desc")
+
+        if "project_url" not in location:
+            _fail_missing_attribute("project_url", key)
+        project_url = mutable_location.pop("project_url")
+        if not project_url.startswith("https://") and not project_url.startswith("http://"):
+            fail("project_url must start with https:// or http://: " + project_url)
+
+        if "version" not in location:
+            _fail_missing_attribute("version", key)
+        mutable_location.pop("version")
+
+        if "use_category" not in location:
+            _fail_missing_attribute("use_category", key)
+        use_category = mutable_location.pop("use_category")
+
+        if "dataplane_ext" in use_category or "observability_ext" in use_category:
+            if "extensions" not in location:
+                _fail_missing_attribute("extensions", key)
+            mutable_location.pop("extensions")
+
+        if "last_updated" not in location:
+            _fail_missing_attribute("last_updated", key)
+        last_updated = mutable_location.pop("last_updated")
+
+        # Starlark doesn't have regexes.
+        if len(last_updated) != 10 or last_updated[4] != "-" or last_updated[7] != "-":
+            fail("last_updated must match YYYY-DD-MM: " + last_updated)
+
+        if "cpe" in location:
+            cpe = mutable_location.pop("cpe")
+
+            # Starlark doesn't have regexes.
+            cpe_components = len(cpe.split(":"))
+
+            # We allow cpe:2.3:a:foo:* and cpe:2.3.:a:foo:bar:* only.
+            cpe_components_valid = cpe_components in [5, 6]
+            cpe_matches = (cpe == "N/A" or (cpe.startswith("cpe:2.3:a:") and cpe.endswith(":*") and cpe_components_valid))
+            if not cpe_matches:
+                fail("CPE must match cpe:2.3:a:<facet>:<facet>:*: " + cpe)
+        elif not [category for category in USE_CATEGORIES_WITH_CPE_OPTIONAL if category in location["use_category"]]:
+            _fail_missing_attribute("cpe", key)
+
+        for category in location["use_category"]:
+            if category not in USE_CATEGORIES:
+                fail("Unknown use_category value '" + category + "' for dependecy " + key)
+
+    return locations
diff --git a/api/bazel/repositories.bzl b/api/bazel/repositories.bzl
@@ -1,40 +1,43 @@
 load(":envoy_http_archive.bzl", "envoy_http_archive")
-load(":repository_locations.bzl", "REPOSITORY_LOCATIONS")
+load(":external_deps.bzl", "load_repository_locations")
+load(":repository_locations.bzl", "REPOSITORY_LOCATIONS_SPEC")
 
-def api_dependencies():
+REPOSITORY_LOCATIONS = load_repository_locations(REPOSITORY_LOCATIONS_SPEC)
+
+# Use this macro to reference any HTTP archive from bazel/repository_locations.bzl.
+def external_http_archive(name, **kwargs):
     envoy_http_archive(
-        "bazel_skylib",
+        name,
         locations = REPOSITORY_LOCATIONS,
+        **kwargs
     )
-    envoy_http_archive(
-        "com_envoyproxy_protoc_gen_validate",
-        locations = REPOSITORY_LOCATIONS,
+
+def api_dependencies():
+    external_http_archive(
+        name = "bazel_skylib",
     )
-    envoy_http_archive(
+    external_http_archive(
+        name = "com_envoyproxy_protoc_gen_validate",
+    )
+    external_http_archive(
         name = "com_google_googleapis",
-        locations = REPOSITORY_LOCATIONS,
     )
-    envoy_http_archive(
+    external_http_archive(
         name = "com_github_cncf_udpa",
-        locations = REPOSITORY_LOCATIONS,
     )
 
-    envoy_http_archive(
+    external_http_archive(
         name = "prometheus_metrics_model",
-        locations = REPOSITORY_LOCATIONS,
         build_file_content = PROMETHEUSMETRICS_BUILD_CONTENT,
     )
-    envoy_http_archive(
+    external_http_archive(
         name = "opencensus_proto",
-        locations = REPOSITORY_LOCATIONS,
     )
-    envoy_http_archive(
+    external_http_archive(
         name = "rules_proto",
-        locations = REPOSITORY_LOCATIONS,
     )
-    envoy_http_archive(
+    external_http_archive(
         name = "com_github_openzipkin_zipkinapi",
-        locations = REPOSITORY_LOCATIONS,
         build_file_content = ZIPKINAPI_BUILD_CONTENT,
     )
 

diff --git a/api/bazel/repository_locations.bzl b/api/bazel/repository_locations.bzl
@@ -1,4 +1,5 @@
-DEPENDENCY_REPOSITORIES_SPEC = dict(
+# This should match the schema defined in external_deps.bzl.
+REPOSITORY_LOCATIONS_SPEC = dict(
     bazel_skylib = dict(
         project_name = "bazel-skylib",
         project_desc = "Common useful functions and rules for Bazel",
@@ -88,23 +89,3 @@ DEPENDENCY_REPOSITORIES_SPEC = dict(
         use_category = ["api"],
     ),
 )
-
-def _format_version(s, version):
-    return s.format(version = version, dash_version = version.replace(".", "-"), underscore_version = version.replace(".", "_"))
-
-# Interpolate {version} in the above dependency specs. This code should be capable of running in both Python
-# and Starlark.
-def _dependency_repositories():
-    locations = {}
-    for key, location in DEPENDENCY_REPOSITORIES_SPEC.items():
-        mutable_location = dict(location)
-        locations[key] = mutable_location
-
-        # Fixup with version information.
-        if "version" in location:
-            if "strip_prefix" in location:
-                mutable_location["strip_prefix"] = _format_version(location["strip_prefix"], location["version"])
-            mutable_location["urls"] = [_format_version(url, location["version"]) for url in location["urls"]]
-    return locations
-
-REPOSITORY_LOCATIONS = _dependency_repositories()
diff --git a/api/bazel/repository_locations_utils.bzl b/api/bazel/repository_locations_utils.bzl
@@ -0,0 +1,20 @@
+def _format_version(s, version):
+    return s.format(version = version, dash_version = version.replace(".", "-"), underscore_version = version.replace(".", "_"))
+
+# Generate a "repository location specification" from raw repository
+# specification. The information should match the format required by
+# external_deps.bzl. This function mostly does interpolation of {version} in
+# the repository info fields. This code should be capable of running in both
+# Python and Starlark.
+def load_repository_locations_spec(repository_locations_spec):
+    locations = {}
+    for key, location in repository_locations_spec.items():
+        mutable_location = dict(location)
+        locations[key] = mutable_location
+
+        # Fixup with version information.
+        if "version" in location:
+            if "strip_prefix" in location:
+                mutable_location["strip_prefix"] = _format_version(location["strip_prefix"], location["version"])
+            mutable_location["urls"] = [_format_version(url, location["version"]) for url in location["urls"]]
+    return locations
diff --git a/api/envoy/config/core/v3/health_check.proto b/api/envoy/config/core/v3/health_check.proto
@@ -54,7 +54,7 @@ enum HealthStatus {
   DEGRADED = 5;
 }
 
-// [#next-free-field: 24]
+// [#next-free-field: 25]
 message HealthCheck {
   option (udpa.annotations.versioning).previous_message_type = "envoy.api.v2.core.HealthCheck";
 
@@ -284,6 +284,21 @@ message HealthCheck {
   // The default value for "no traffic interval" is 60 seconds.
   google.protobuf.Duration no_traffic_interval = 12 [(validate.rules).duration = {gt {}}];
 
+  // The "no traffic healthy interval" is a special health check interval that
+  // is used for hosts that are currently passing active health checking
+  // (including new hosts) when the cluster has received no traffic.
+  //
+  // This is useful for when we want to send frequent health checks with
+  // `no_traffic_interval` but then revert to lower frequency `no_traffic_healthy_interval` once
+  // a host in the cluster is marked as healthy.
+  //
+  // Once a cluster has been used for traffic routing, Envoy will shift back to using the
+  // standard health check interval that is defined.
+  //
+  // If no_traffic_healthy_interval is not set, it will default to the
+  // no traffic interval and send that interval regardless of health state.
+  google.protobuf.Duration no_traffic_healthy_interval = 24 [(validate.rules).duration = {gt {}}];
+
   // The "unhealthy interval" is a health check interval that is used for hosts that are marked as
   // unhealthy. As soon as the host is marked as healthy, Envoy will shift back to using the
   // standard health check interval that is defined.