Merge remote-tracking branch 'upstream/master' into timeless_watchdog

Signed-off-by: Antonio Vicente <[email protected]>

CODEOWNERS

@@ -130,6 +130,7 @@ extensions/filters/common/original_src @snowp @klarose
 /*/extensions/filters/http/decompressor @rojkov @dio
 # Watchdog Extensions
 /*/extensions/watchdog/profile_action @kbaichoo @htuch
+/*/extensions/watchdog/abort_action @kbaichoo @htuch
 # Core upstream code
 extensions/upstreams/http @alyssawilk @snowp @mattklein123
 extensions/upstreams/http/http @alyssawilk @snowp @mattklein123

STYLE.md

@@ -105,17 +105,18 @@
 A few general notes on our error handling philosophy:
 
 * All error code returns should be checked.
-* At a very high level, our philosophy is that errors that are *likely* to happen should be
-  gracefully handled. Examples of likely errors include any type of network error, disk IO error,
-  bad data returned by an API call, bad data read from runtime files, etc. Errors that are
-  *unlikely* to happen should lead to process death, under the assumption that the additional burden
-  of defensive coding and testing is not an effective use of time for an error that should not happen
-  given proper system setup. Examples of these types of errors include not being able to open the shared
-  memory region, an invalid initial JSON config read from disk, system calls that should not fail
-  assuming correct parameters (which should be validated via tests), etc. Examples of system calls
-  that should not fail when passed valid parameters include most usages of `setsockopt()`,
-  `getsockopt()`, the kernel returning a valid `sockaddr` after a successful call to `accept()`,
-  `pthread_create()`, `pthread_join()`, etc.
+* At a very high level, our philosophy is that errors should be handled gracefully when caused by:
+  - Untrusted network traffic OR
+  - Raised by the Envoy process environment and are *likely* to happen
+* Examples of likely environnmental errors include any type of network error, disk IO error, bad
+  data returned by an API call, bad data read from runtime files, etc. Errors in the Envoy
+  environment that are *unlikely* to happen after process initialization, should lead to process
+  death, under the assumption that the additional burden of defensive coding and testing is not an
+  effective use of time for an error that should not happen given proper system setup. Examples of
+  these types of errors include not being able to open the shared memory region, system calls that
+  should not fail assuming correct parameters (which should be validated via tests), etc. Examples
+  of system calls that should not fail when passed valid parameters include the kernel returning a
+  valid `sockaddr` after a successful call to `accept()`, `pthread_create()`, `pthread_join()`, etc.
 * OOM events (both memory and FDs) are considered fatal crashing errors. An OOM error should never
   silently be ignored and should crash the process either via the C++ allocation error exception, an
   explicit `RELEASE_ASSERT` following a third party library call, or an obvious crash on a subsequent

api/BUILD

@@ -245,6 +245,7 @@ proto_library(
         "//envoy/extensions/upstreams/http/http/v3:pkg",
         "//envoy/extensions/upstreams/http/tcp/v3:pkg",
         "//envoy/extensions/wasm/v3:pkg",
+        "//envoy/extensions/watchdog/abort_action/v3alpha:pkg",
         "//envoy/extensions/watchdog/profile_action/v3alpha:pkg",
         "//envoy/service/accesslog/v3:pkg",
         "//envoy/service/auth/v3:pkg",

api/envoy/config/core/v3/substitution_format_string.proto

@@ -65,4 +65,14 @@ message SubstitutionFormatString {
   //   empty string, so that empty values are omitted entirely.
   // * for ``json_format`` the keys with null values are omitted in the output structure.
   bool omit_empty_values = 3;
+
+  // Specify a *content_type* field.
+  // If this field is not set then ``text/plain`` is used for *text_format* and
+  // ``application/json`` is used for *json_format*.
+  //
+  // .. code-block::
+  //
+  //   content_type: "text/html; charset=UTF-8"
+  //
+  string content_type = 4;
 }

api/envoy/config/route/v3/route_components.proto

@@ -997,7 +997,9 @@ message RouteAction {
   // limits. By default, if the route configured rate limits, the virtual host
   // :ref:`rate_limits <envoy_api_field_config.route.v3.VirtualHost.rate_limits>` are not applied to the
   // request.
-  google.protobuf.BoolValue include_vh_rate_limits = 14;
+  //
+  // This field is deprecated. Please use :ref:`vh_rate_limits <envoy_v3_api_field_extensions.filters.http.ratelimit.v3.RateLimitPerRoute.vh_rate_limits>`
+  google.protobuf.BoolValue include_vh_rate_limits = 14 [deprecated = true];
 
   // Specifies a list of hash policies to use for ring hash load balancing. Each
   // hash policy is evaluated individually and the combined result is used to

api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto

@@ -100,3 +100,19 @@ message RateLimit {
   XRateLimitHeadersRFCVersion enable_x_ratelimit_headers = 8
       [(validate.rules).enum = {defined_only: true}];
 }
+
+message RateLimitPerRoute {
+  enum VhRateLimitsOptions {
+    // Use the virtual host rate limits unless the route has a rate limit policy.
+    OVERRIDE = 0;
+
+    // Use the virtual host rate limits even if the route has a rate limit policy.
+    INCLUDE = 1;
+
+    // Ignore the virtual host rate limits even if the route does not have a rate limit policy.
+    IGNORE = 2;
+  }
+
+  // Specifies if the rate limit filter should include the virtual host rate limits.
+  VhRateLimitsOptions vh_rate_limits = 1 [(validate.rules).enum = {defined_only: true}];
+}

api/envoy/extensions/watchdog/abort_action/v3alpha/BUILD

@@ -0,0 +1,9 @@
+# DO NOT EDIT. This file is generated by tools/proto_format/proto_sync.py.
+
+load("@envoy_api//bazel:api_build_system.bzl", "api_proto_package")
+
+licenses(["notice"])  # Apache 2
+
+api_proto_package(
+    deps = ["@com_github_cncf_udpa//udpa/annotations:pkg"],
+)

api/envoy/extensions/watchdog/abort_action/v3alpha/abort_action.proto

@@ -0,0 +1,32 @@
+syntax = "proto3";
+
+package envoy.extensions.watchdog.abort_action.v3alpha;
+
+import "google/protobuf/duration.proto";
+
+import "udpa/annotations/status.proto";
+import "udpa/annotations/versioning.proto";
+import "validate/validate.proto";
+
+option java_package = "io.envoyproxy.envoy.extensions.watchdog.abort_action.v3alpha";
+option java_outer_classname = "AbortActionProto";
+option java_multiple_files = true;
+option (udpa.annotations.file_status).work_in_progress = true;
+option (udpa.annotations.file_status).package_version_status = ACTIVE;
+
+// [#protodoc-title: Watchdog Action that sends a SIGABRT to kill the process.]
+// [#extension: envoy.watchdog.abort_action]
+
+// A GuardDogAction that will terminate the process by sending SIGABRT to the
+// stuck thread. This would allow easier access to the call stack of the stuck
+// thread since we would run signal handlers on that thread. This would be
+// more useful than the default watchdog kill behaviors since those PANIC
+// from the watchdog's thread.
+
+// This is currently only implemented for systems that support kill to send
+// signals.
+message AbortActionConfig {
+  // How long to wait for the thread to respond to the SIGABRT before killing the
+  // process from this action. This is a blocking action.
+  google.protobuf.Duration wait_duration = 1;
+}

api/versioning/BUILD

@@ -128,6 +128,7 @@ proto_library(
         "//envoy/extensions/upstreams/http/http/v3:pkg",
         "//envoy/extensions/upstreams/http/tcp/v3:pkg",
         "//envoy/extensions/wasm/v3:pkg",
+        "//envoy/extensions/watchdog/abort_action/v3alpha:pkg",
         "//envoy/extensions/watchdog/profile_action/v3alpha:pkg",
         "//envoy/service/accesslog/v3:pkg",
         "//envoy/service/auth/v3:pkg",

bazel/README.md

@@ -127,6 +127,11 @@ for how to update or override dependencies.
     startup --output_base=C:/_eb
     ```
 
+    Bazel also creates file symlinks when building Envoy. It's strongly recommended to enable file symlink support 
+    using [Bazel's instructions](https://docs.bazel.build/versions/master/windows.html#enable-symlink-support).
+    For other common issues, see the 
+    [Using Bazel on Windows](https://docs.bazel.build/versions/master/windows.html) page.
+
     [python3](https://www.python.org/downloads/): Specifically, the Windows-native flavor distributed
     by python.org. The POSIX flavor available via MSYS2, the Windows Store flavor and other distributions
     will not work. Add a symlink for `python3.exe` pointing to the installed `python.exe` for Envoy scripts
@@ -145,7 +150,8 @@ for how to update or override dependencies.
     package. Earlier versions of VC++ Build Tools/Visual Studio are not recommended or supported.
     If installed in a non-standard filesystem location, be sure to set the `BAZEL_VC` environment variable
     to the path of the VC++ package to allow Bazel to find your installation of VC++. NOTE: ensure that
-    the `link.exe` that resolves on your PATH is from VC++ Build Tools and not `/usr/bin/link.exe` from MSYS2.
+    the `link.exe` that resolves on your PATH is from VC++ Build Tools and not `/usr/bin/link.exe` from MSYS2,
+    which is determined by their relative ordering in your PATH.
     ```
     set BAZEL_VC=%USERPROFILE%\VSBT2019\VC
     set PATH=%PATH%;%USERPROFILE%\VSBT2019\VC\Tools\MSVC\14.26.28801\bin\Hostx64\x64
@@ -160,10 +166,11 @@ for how to update or override dependencies.
     set PATH=%PATH%;%USERPROFILE%\VSBT2019\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
     ```
 
-    [MSYS2 shell](https://msys2.github.io/): Set the `BAZEL_SH` environment variable to the path
-    of the installed MSYS2 `bash.exe` executable. Additionally, setting the `MSYS2_ARG_CONV_EXCL` environment
-    variable to a value of `*` is often advisable to ensure argument parsing in the MSYS2 shell
-    behaves as expected.
+    [MSYS2 shell](https://msys2.github.io/): Install to a path with no spaces, e.g. C:\msys32. 
+    
+    Set the `BAZEL_SH` environment variable to the path of the installed MSYS2 `bash.exe` 
+    executable. Additionally, setting the `MSYS2_ARG_CONV_EXCL` environment variable to a value 
+    of `*` is often advisable to ensure argument parsing in the MSYS2 shell behaves as expected.
     ```
     set PATH=%PATH%;%USERPROFILE%\msys64\usr\bin
     set BAZEL_SH=%USERPROFILE%\msys64\usr\bin\bash.exe
@@ -181,7 +188,7 @@ for how to update or override dependencies.
     The TMPDIR path and MSYS2 `mktemp` command are used frequently by the `rules_foreign_cc`
     component of Bazel as well as Envoy's test scripts, causing problems if not set to a path
     accessible to both Windows and msys commands. [Note the `ci/windows_ci_steps.sh` script
-    which builds envoy and run tests in CI) creates this symlink automatically.]
+    which builds envoy and run tests in CI creates this symlink automatically.]
 
     In the MSYS2 shell, install additional packages via pacman:
     ```
@@ -212,7 +219,8 @@ for how to update or override dependencies.
    in your shell for buildifier to work.
 1. `go get -u github.com/bazelbuild/buildtools/buildozer` to install buildozer. You may need to set `BUILDOZER_BIN` to `$GOPATH/bin/buildozer`
    in your shell for buildozer to work.
-1. `bazel build //source/exe:envoy-static` from the Envoy source directory.
+1. `bazel build //source/exe:envoy-static` from the Envoy source directory. Add `-c opt` for an optimized release build or
+   `-c dbg` for an unoptimized, fully instrumented debugging build.
 
 ## Building Envoy with the CI Docker image
 

bazel/foreign_cc/BUILD

@@ -115,8 +115,9 @@ envoy_cmake_external(
         "CMAKE_USE_GSSAPI": "off",
         "HTTP_ONLY": "on",
         "CMAKE_INSTALL_LIBDIR": "lib",
-        # Explicitly enable Unix sockets and disable crypto for Windows
-        "USE_UNIX_SOCKETS": "on",
+        # Explicitly enable Unix sockets, once afunix.h is correctly detected
+        # "USE_UNIX_SOCKETS": "on",
+        # Explicitly disable "Windows" crypto for Windows
         "CURL_DISABLE_CRYPTO_AUTH": "on",
         # C-Ares.
         "ENABLE_ARES": "on",
@@ -138,6 +139,10 @@ envoy_cmake_external(
         "ZLIB_INCLUDE_DIR": "$EXT_BUILD_DEPS/zlib/include",
         "CMAKE_CXX_COMPILER_FORCED": "on",
         "CMAKE_C_FLAGS_BAZEL": "-fPIC",
+        # Note we use Bazel's flags (not _RELEASE/_DEBUG CMake flags), but this toggle
+        # also works around a bug in CMP0091 logic which re-injected a badly placed -M flag.
+        # See https://github.com/bazelbuild/rules_foreign_cc/issues/426
+        "CURL_STATIC_CRT": "on",
     },
     defines = ["CURL_STATICLIB"],
     generate_crosstool_file = True,

bazel/repositories.bzl

@@ -12,6 +12,7 @@ WINDOWS_SKIP_TARGETS = [
     "envoy.tracers.lightstep",
     "envoy.tracers.datadog",
     "envoy.tracers.opencensus",
+    "envoy.watchdog.abort_action",
 ]
 
 # Make all contents of an external repository accessible under a filegroup.  Used for external HTTP
@@ -674,8 +675,6 @@ def _com_github_curl():
         build_file_content = BUILD_ALL_CONTENT + """
 cc_library(name = "curl", visibility = ["//visibility:public"], deps = ["@envoy//bazel/foreign_cc:curl"])
 """,
-        patches = ["@envoy//bazel/foreign_cc:curl-revert-cmake-minreqver.patch"],
-        patch_args = ["-p1"],
         **location
     )
     native.bind(

bazel/repository_locations.bzl

@@ -522,8 +522,8 @@ DEPENDENCY_REPOSITORIES_SPEC = dict(
     com_github_curl = dict(
         project_name = "curl",
         project_url = "https://curl.haxx.se",
-        version = "7.69.1",
-        sha256 = "01ae0c123dee45b01bbaef94c0bc00ed2aec89cb2ee0fd598e0d302a6b5e0a98",
+        version = "7.72.0",
+        sha256 = "d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2",
         strip_prefix = "curl-{version}",
         urls = ["https://github.com/curl/curl/releases/download/curl-{underscore_version}/curl-{version}.tar.gz"],
         use_category = ["dataplane"],

docs/root/api-v3/config/watchdog/watchdog.rst

@@ -6,3 +6,4 @@ Watchdog
   :maxdepth: 2
 
   ../../extensions/watchdog/profile_action/v3alpha/*
+  ../../extensions/watchdog/abort_action/v3alpha/*

docs/root/configuration/http/http_conn_man/local_reply.rst

@@ -49,6 +49,8 @@ The response body content type can be customized. If not specified, the content
 
 Local reply format can be specified as :ref:`SubstitutionFormatString <envoy_v3_api_msg_config.core.v3.SubstitutionFormatString>`. It supports :ref:`text_format <envoy_v3_api_field_config.core.v3.SubstitutionFormatString.text_format>` and :ref:`json_format <envoy_v3_api_field_config.core.v3.SubstitutionFormatString.json_format>`.
 
+Optionally, content-type can be modified further via :ref:`content_type <envoy_v3_api_field_config.core.v3.SubstitutionFormatString.content_type>` field. If not specified, default content-type is `text/plain` for :ref:`text_format <envoy_v3_api_field_config.core.v3.SubstitutionFormatString.text_format>` and `application/json` for :ref:`json_format <envoy_v3_api_field_config.core.v3.SubstitutionFormatString.json_format>`.
+
 Example of a LocalReplyConfig with `body_format` field.
 
 .. code-block::
@@ -63,7 +65,8 @@ Example of a LocalReplyConfig with `body_format` field.
             runtime_key: key_b
     status_code: 401
     body_format_override:
-      text_format: "%LOCAL_REPLY_BODY% %REQ(:path)%"
+      text_format: "<h1>%LOCAL_REPLY_BODY% %REQ(:path)%</h1>"
+      content_type: "text/html; charset=UTF-8"
   - filter:
       status_code_filter:
         comparison:

docs/root/intro/arch_overview/security/ssl.rst

@@ -24,7 +24,7 @@ requirements (TLS1.2, SNI, etc.). Envoy supports the following TLS features:
   across hot restarts and between parallel Envoy instances (typically useful in a front proxy
   configuration).
 * **BoringSSL private key methods**: TLS private key operations (signing and decrypting) can be
-  performed asynchronously from an extension. This allows extending Envoy to support various key
+  performed asynchronously from :ref:`an extension <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.PrivateKeyProvider>`. This allows extending Envoy to support various key
   management schemes (such as TPM) and TLS acceleration. This mechanism uses
   `BoringSSL private key method interface <https://github.com/google/boringssl/blob/c0b4c72b6d4c6f4828a373ec454bd646390017d4/include/openssl/ssl.h#L1169>`_.
 
@@ -182,6 +182,30 @@ infrastructure.
 Client TLS authentication filter :ref:`configuration reference
 <config_network_filters_client_ssl_auth>`.
 
+.. _arch_overview_ssl_custom_handshaker:
+
+Custom handshaker extension
+---------------------------
+
+The :ref:`CommonTlsContext <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CommonTlsContext.custom_handshaker>`
+has a ``custom_handshaker`` extension which can be used to override SSL handshake
+behavior entirely. This is useful for implementing any TLS behavior which is
+difficult to express with callbacks. It is not necessary to write a custom
+handshaker to use private key methods, see the
+:ref:`private key method interface <arch_overview_ssl>` described above.
+
+To avoid reimplementing all of the `Ssl::ConnectionInfo <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/include/envoy/ssl/connection.h#L19>`_ interface, a custom
+implementation might choose to extend
+`Envoy::Extensions::TransportSockets::Tls::SslHandshakerImpl <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/source/extensions/transport_sockets/tls/ssl_handshaker.h#L40>`_.
+
+Custom handshakers need to explicitly declare via `HandshakerCapabilities <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/include/envoy/ssl/handshaker.h#L68-L89>`_
+which TLS features they are responsible for. The default Envoy handshaker will
+manage the remainder.
+
+A useful example handshaker, named ``SslHandshakerImplForTest``, lives in
+`this test <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/test/extensions/transport_sockets/tls/handshaker_test.cc#L174-L184>`_
+and demonstrates special-case ``SSL_ERROR`` handling and callbacks.
+
 .. _arch_overview_ssl_trouble_shooting:
 
 Trouble shooting