Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network: add socket interface factory and config option #11630

Merged
merged 16 commits into from
Jul 17, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions api/envoy/config/bootstrap/v3/bootstrap.proto
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import "envoy/config/core/v3/base.proto";
import "envoy/config/core/v3/config_source.proto";
import "envoy/config/core/v3/event_service_config.proto";
import "envoy/config/core/v3/extension.proto";
import "envoy/config/core/v3/socket_interfaces.proto";
import "envoy/config/core/v3/socket_option.proto";
import "envoy/config/listener/v3/listener.proto";
import "envoy/config/metrics/v3/stats.proto";
Expand Down Expand Up @@ -60,6 +61,9 @@ message Bootstrap {
// These static secrets can be used by :ref:`SdsSecretConfig
// <envoy_api_msg_extensions.transport_sockets.tls.v3.SdsSecretConfig>`
repeated envoy.extensions.transport_sockets.tls.v3.Secret secrets = 3;

// Allows loading of multiple socket interfaces and configuration of the default one
core.v3.SocketInterfacesConfig socket_interfaces = 4;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the thinking that these will one day be dynamically discovered by an xDS? If not, you don't need to put them in static_resources, they can just live top-level.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Actually this was just me not knowing what I was doing, so thanks for the comment!

Another thing that's confusing is the ci/circleci:docs error. Should I add the new *.proto files to some list or is there something wrong with them?

}

message DynamicResources {
Expand Down
4 changes: 4 additions & 0 deletions api/envoy/config/bootstrap/v4alpha/bootstrap.proto

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 20 additions & 0 deletions api/envoy/config/core/v3/default_socket_interface.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
syntax = "proto3";

package envoy.config.core.v3;

import "udpa/annotations/status.proto";
import "udpa/annotations/versioning.proto";
import "validate/validate.proto";

option java_package = "io.envoyproxy.envoy.config.core.v3";
option java_outer_classname = "DefaultSocketInterfaceProto";
option java_multiple_files = true;
option (udpa.annotations.file_status).package_version_status = ACTIVE;

// [#protodoc-title: Default Socket Interface configuration]
// Configuration for default socket interface that relies on OS dependent syscall to create
// sockets.
// [#extension: envoy.config.core]

message DefaultSocketInterface {
}
25 changes: 25 additions & 0 deletions api/envoy/config/core/v3/socket_interfaces.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
syntax = "proto3";

package envoy.config.core.v3;

import "envoy/config/core/v3/extension.proto";

import "udpa/annotations/status.proto";
import "udpa/annotations/versioning.proto";
import "validate/validate.proto";

option java_package = "io.envoyproxy.envoy.config.core.v3";
option java_outer_classname = "SocketInterfacesProto";
option java_multiple_files = true;
option (udpa.annotations.file_status).package_version_status = ACTIVE;

// [#protodoc-title: Socket Interfaces Configuration ]

// Used to select the socket interface implementation to use
message SocketInterfacesConfig {
// Addition socket interfaces to initialize
repeated TypedExtensionConfig custom_types = 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you document how the name or type URL of these extensions is significant and used during resolution?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Let me know if it's not clear.


// Override default socket interface
TypedExtensionConfig custom_default_type = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this just be a string? I.e. the extension config is above, this is just specifying some entry in it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

}
22 changes: 22 additions & 0 deletions api/envoy/config/core/v4alpha/default_socket_interface.proto

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 28 additions & 0 deletions api/envoy/config/core/v4alpha/socket_interfaces.proto

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 25 additions & 0 deletions generated_api_shadow/envoy/config/core/v3/socket_interfaces.proto

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions include/envoy/network/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ envoy_cc_library(
deps = [
":address_interface",
":io_handle_interface",
"//include/envoy/config:typed_config_interface",
"@envoy_api//envoy/config/core/v3:pkg_cc_proto",
],
)
Expand Down
2 changes: 2 additions & 0 deletions include/envoy/network/socket.h
Original file line number Diff line number Diff line change
Expand Up @@ -271,5 +271,7 @@ class SocketInterface {
virtual bool ipFamilySupported(int domain) PURE;
};

using SocketInterfacePtr = std::unique_ptr<SocketInterface>;

} // namespace Network
} // namespace Envoy
12 changes: 12 additions & 0 deletions source/common/network/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,16 @@ envoy_cc_library(
],
)

envoy_cc_library(
name = "socket_interface_factory_lib",
hdrs = ["socket_interface_factory.h"],
deps = [
"//include/envoy/config:typed_config_interface",
"//include/envoy/network:socket_interface",
"//include/envoy/registry",
],
)

envoy_cc_library(
name = "socket_lib",
srcs = [
Expand All @@ -184,9 +194,11 @@ envoy_cc_library(
],
deps = [
":address_lib",
":socket_interface_factory_lib",
"//include/envoy/network:socket_interface",
"//source/common/common:assert_lib",
"//source/common/singleton:threadsafe_singleton",
"@envoy_api//envoy/config/core/v3:pkg_cc_proto",
],
)

Expand Down
41 changes: 41 additions & 0 deletions source/common/network/socket_interface_factory.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#pragma once

#include "envoy/config/typed_config.h"
#include "envoy/network/socket.h"
#include "envoy/registry/registry.h"

#include "common/singleton/threadsafe_singleton.h"

#include "absl/container/flat_hash_map.h"

namespace Envoy {
namespace Network {

class SocketInterfaceFactory : public virtual Config::TypedFactory {
public:
virtual SocketInterfacePtr createSocketInterface(const Protobuf::Message& config) PURE;
std::string category() const override { return "envoy.config.core.socket_interface"; }
};

class DefaultSocketInterfaceFactory : public SocketInterfaceFactory {
public:
SocketInterfacePtr createSocketInterface(const Protobuf::Message& config) override;
ProtobufTypes::MessagePtr createEmptyConfigProto() override;
std::string name() const override { return "envoy.config.core.default_socket_interface"; }
};

DECLARE_FACTORY(DefaultSocketInterfaceFactory);

using SocketInterfacesMap = absl::flat_hash_map<std::string, SocketInterfacePtr>;
using SocketInterfacesSingleton = InjectableSingleton<SocketInterfacesMap>;

static inline const SocketInterface* socketInterface(std::string name) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to be used to find the right SocketInterface implementation when Address::Instance has a non-empty socket_interface field.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't the registry basically provide this? (type_url -> factory) I'm probably missing why we need to redo this type of stuff.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I set this up, and maybe that's not what we want, is: typed_config is handed off to SocketInterfaceFactory registered via REGISTER_FACTORY (eg declaration above at line 27 is registered here), and looked up via typed_config.name(), which then creates a SocketInterface implementation that we add to the hash table stored in the SocketInterfacesSingleton. So this new hash table is for implementations instances as opposed to factories.

If there's a nicer way this could be done with type_url, point me to an example and I'll try to follow it (still have a lot of things to learn :-() .

Copy link
Member

@mattklein123 mattklein123 Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will defer to @lizan and @kyessenov who have more experience with the factory, but I'm wondering if we could change it slightly so the registered factory just spits out sockets and then we would just look up the factory by type_url?

auto it = SocketInterfacesSingleton::get().find(name);
if (it == SocketInterfacesSingleton::get().end()) {
return nullptr;
}
return it->second.get();
}

} // namespace Network
} // namespace Envoy
12 changes: 12 additions & 0 deletions source/common/network/socket_interface_impl.cc
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
#include "common/network/socket_interface_impl.h"

#include "envoy/common/exception.h"
#include "envoy/config/core/v3/default_socket_interface.pb.h"
#include "envoy/network/socket.h"

#include "common/api/os_sys_calls_impl.h"
#include "common/network/address_impl.h"
#include "common/network/io_socket_handle_impl.h"
#include "common/network/socket_interface_factory.h"

namespace Envoy {
namespace Network {
Expand Down Expand Up @@ -80,6 +82,16 @@ bool SocketInterfaceImpl::ipFamilySupported(int domain) {
return SOCKET_VALID(result.rc_);
}

SocketInterfacePtr DefaultSocketInterfaceFactory::createSocketInterface(const Protobuf::Message&) {
return std::make_unique<SocketInterfaceImpl>();
}

ProtobufTypes::MessagePtr DefaultSocketInterfaceFactory::createEmptyConfigProto() {
return std::make_unique<envoy::config::core::v3::DefaultSocketInterface>();
}

REGISTER_FACTORY(DefaultSocketInterfaceFactory, SocketInterfaceFactory);

static SocketInterfaceLoader* socket_interface_ =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we do this in server.cc? i.e. if another socket interface is registered and specified as default, we don't need to instantiate default socket interface?

Copy link
Member Author

@florincoras florincoras Jul 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely a good point but if tests are run without the server being properly initialized, the SocketInterfaceSingleton won't be properly initialized. Just ran a quick test and they do crash.

Should we maybe statically initialize the singleton to the default SocketInterface in some common test file, like test/test_common/environment.cc?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out integration:hotrestart_test, integration:run_envoy_test and test:router_tool_test still fail :-(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe statically initialize the singleton to the default SocketInterface in some common test file, like test/test_common/environment.cc?

Yeah I think that's good way to do test.

OTOH, if this causes a lot churn, we can do static registration here, then let's not do factory registration for default interface. Because even they're configured through bootstrap_extensions, it has nothing different than statically registered one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe statically initialize the singleton to the default SocketInterface in some common test file, like test/test_common/environment.cc?

Yeah I think that's good way to do test.

Because we do some network related things early on, before initializing the server, (e.g., call validateIpv4Supported), we'll have to initialize very early on the SocketInterfaceSingleton to something.

OTOH, if this causes a lot churn, we can do static registration here, then let's not do factory registration for default interface. Because even they're configured through bootstrap_extensions, it has nothing different than statically registered one?

Do you mean to keep the global static registration in socket_interface_impl.cc (which is a bit of a misnomer at this point but that's for another time)? Or do you think we should move that to server.cc (and keep it as a global static)? I'm fine either way.

As for the factory registration, here's the weird use case I came up with: people may want to configure a different default but may still want to configure addresses to use what we now call SocketInterfaceImpl, which is registered with name envoy.config.core.default_socket_interface. So, we still end up needing the factory registration for name lookups, unless we decide we don't support that.

Tying this to your first suggestion: we could maybe do a SocketInterfaceSingleton::initialize(Network::socketInterface("envoy.config.core.default_socket_interface")) very early on, e.g., somewhere in main_common.cc or in PlatformImpl() and that would avoid the need for the static init and allocating a redundant SocketInterfaceImpl unique ptr by reusing the factory registration.

Copy link
Member

@lizan lizan Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean to keep the global static registration in socket_interface_impl.cc (which is a bit of a misnomer at this point but that's for another time)? Or do you think we should move that to server.cc (and keep it as a global static)? I'm fine either way.

Either
a. move initialization to server.cc and fix test
b. keep global static registration without declaring factory

works for me.

In long term we might want a. but to keep PR smaller and easier to review we can do a. as a follow up.

As for the factory registration, here's the weird use case I came up with: people may want to configure a different default but may still want to configure addresses to use what we now call SocketInterfaceImpl, which is registered with name envoy.config.core.default_socket_interface. So, we still end up needing the factory registration for name lookups, unless we decide we don't support that.

I meant we should drop it until we can support it in a meaningful way? i.e. when address can specify socket interface.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's fine with you, let's try to do a. as a follow up after we fix Address. If that's okay, I'll push a quick fix to the PR (SocketInterface lookup needs some casting).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not to forget about it, I pushed the fix now, but that's independent of the changes we've been discussing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Let me know if anything else is needed (the update seems to have verified fine).

new SocketInterfaceLoader(std::make_unique<SocketInterfaceImpl>());

Expand Down
1 change: 1 addition & 0 deletions source/server/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ envoy_cc_library(
"//source/common/config:runtime_utility_lib",
"//source/common/config:utility_lib",
"//source/common/network:resolver_lib",
"//source/common/network:socket_interface_factory_lib",
"//source/common/network:socket_option_factory_lib",
"//source/common/network:utility_lib",
"//source/common/protobuf:utility_lib",
Expand Down
Loading