Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add basic opentracing support #5544

Merged
merged 74 commits into from
Jul 11, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
5afd89a
Configure and initialise tracer
JorikSchellekens Jun 21, 2019
b43e3af
Scope manager using LogContexts
JorikSchellekens Jun 21, 2019
297c5fe
jaeger is a dependency now
JorikSchellekens Jun 21, 2019
45d412e
Carrier inject and extraction for Twisted Headers
JorikSchellekens Jun 21, 2019
fc069fe
Trace federation requests on the way in and out.
JorikSchellekens Jun 21, 2019
227447d
Create logcontext for new scope.
JorikSchellekens Jun 21, 2019
f35bde3
Remove scope from logcontext if logcontext is top level
JorikSchellekens Jun 24, 2019
7d9c3d3
Disable tracer if not configured
JorikSchellekens Jun 24, 2019
37607eb
typo
JorikSchellekens Jun 24, 2019
ee2f21f
Remove dependence on jaeger internals
JorikSchellekens Jun 24, 2019
ed5d00f
bools
JorikSchellekens Jun 24, 2019
a69e6aa
Set service name
JorikSchellekens Jun 24, 2019
6a6f97a
:Explicitely state that the tracer is disabled
JorikSchellekens Jun 24, 2019
ce21e41
Merge branch 'develop' into joriks/opentracing
JorikSchellekens Jun 25, 2019
98c8038
Black is the new black
JorikSchellekens Jun 25, 2019
889afc1
Newsfile
JorikSchellekens Jun 25, 2019
db10eeb
Code style
JorikSchellekens Jun 25, 2019
4e1eaf2
Use the new config setup.
JorikSchellekens Jun 25, 2019
8ca185d
Generate config.
JorikSchellekens Jun 25, 2019
0009fdd
Copyright
JorikSchellekens Jun 25, 2019
e90bfa7
Rename config to opentracing
JorikSchellekens Jun 25, 2019
640381e
Remove user whitelisting
JorikSchellekens Jun 25, 2019
0f57aac
Empty whitelist by default
JorikSchellekens Jun 25, 2019
dd2f06d
User ConfigError instead of RuntimeError
JorikSchellekens Jun 25, 2019
bae3b5b
Use isinstance
JorikSchellekens Jun 25, 2019
9c39f0e
Use tag constants for opentracing.
JorikSchellekens Jun 25, 2019
bae63e9
Remove debug comment and no need to explicitely record error
JorikSchellekens Jun 25, 2019
4925558
Typo?...
JorikSchellekens Jun 25, 2019
8104779
Two errors a "s(c)entry"
JorikSchellekens Jun 25, 2019
609d97b
Docstrings!
JorikSchellekens Jun 25, 2019
04781e9
typos
JorikSchellekens Jun 25, 2019
088176e
Remove debugging brainslip
JorikSchellekens Jun 25, 2019
dde4618
Homeserver Whitlisting
JorikSchellekens Jun 25, 2019
34235a4
Better opentracing config comment
JorikSchellekens Jun 25, 2019
1d591af
linting
JorikSchellekens Jun 25, 2019
37a650c
Inclue worker name in service_name
JorikSchellekens Jun 26, 2019
4bf9e11
Make opentracing an optional dependency
JorikSchellekens Jun 26, 2019
941b655
Neater config retreival
JorikSchellekens Jul 2, 2019
9654277
Typoed
JorikSchellekens Jul 2, 2019
9ce4a3d
Clean up dummy tags
JorikSchellekens Jul 2, 2019
b008aa4
Instantiate tracing as object instead of global class
JorikSchellekens Jul 2, 2019
e27516b
Inlcude opentracing as a homeserver member.
JorikSchellekens Jul 2, 2019
9d2734b
Thread opentracing to the request level
JorikSchellekens Jul 2, 2019
df255d7
Reference opetnracing through hs
JorikSchellekens Jul 2, 2019
64f8510
Instantiate dummy opentracin g for tests.
JorikSchellekens Jul 2, 2019
69561e0
About to revert, just keeping the unfinished changes just in case
JorikSchellekens Jul 2, 2019
4656403
Revert back to global state, commit number:
JorikSchellekens Jul 2, 2019
a60572f
Use class level methods in tracerutils
JorikSchellekens Jul 2, 2019
bc6b236
Start and stop requests spans in a place where we
JorikSchellekens Jul 2, 2019
68e4595
Seen it, isort it
JorikSchellekens Jul 2, 2019
70a501a
Merge remote-tracking branch 'origin/develop' into joriks/opentracing
JorikSchellekens Jul 2, 2019
9bfe90f
Make sure to close the active span.
JorikSchellekens Jul 3, 2019
800e707
I'm getting black and blue from this.
JorikSchellekens Jul 3, 2019
fd37a69
Merge remote-tracking branch 'origin/develop' into joriks/opentracing
JorikSchellekens Jul 3, 2019
b09aa24
Logger formatting
JorikSchellekens Jul 3, 2019
374e81c
Outdated comment
JorikSchellekens Jul 3, 2019
e111dcd
Import opentracing at the top
JorikSchellekens Jul 3, 2019
14376e7
Return a contextmanager
JorikSchellekens Jul 3, 2019
24b3834
Start tracing client requests from the servlet
JorikSchellekens Jul 4, 2019
386285b
Return noop context manager if not tracing
JorikSchellekens Jul 4, 2019
3dde444
Explicitely say that these are federation requests
JorikSchellekens Jul 4, 2019
b5cd7df
Include servlet name in client requests
JorikSchellekens Jul 4, 2019
f93b60d
Use context manager
JorikSchellekens Jul 4, 2019
6d45cd0
Merge remote-tracking branch 'origin/develop' into joriks/opentracing
JorikSchellekens Jul 4, 2019
6ab84bb
Move opentracing to logging/
JorikSchellekens Jul 4, 2019
9a3c98a
Seen it, isort it again!
JorikSchellekens Jul 4, 2019
0c19608
Ignore twisted return exceptions on context exit
JorikSchellekens Jul 5, 2019
7414549
Merge branch 'develop' into joriks/opentracing
JorikSchellekens Jul 8, 2019
18a902f
Escape the scope
JorikSchellekens Jul 10, 2019
49038b1
Scopes should be entered to make them useful.
JorikSchellekens Jul 10, 2019
00e08bd
Nicer decorator names
JorikSchellekens Jul 10, 2019
06baa49
Just one init, init?
JorikSchellekens Jul 10, 2019
b07a4c6
Don't need to close something that isn't open
JorikSchellekens Jul 10, 2019
6f3109e
Docs make you smarter
JorikSchellekens Jul 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/5544.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added opentracing and configuration options.
17 changes: 17 additions & 0 deletions docs/sample_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1395,3 +1395,20 @@ password_config:
# module: "my_custom_project.SuperRulesSet"
# config:
# example_option: 'things'


## Opentracing ##
# These settings enable opentracing which implements distributed tracing
# This allows you to observe the causal chain of events across servers
# including requests, key lookups etc. across any server running
# synapse or any other other services which supports opentracing.
# (specifically those implemented with jaeger)

#opentracing:
# # Enable / disable tracer
# tracer_enabled: false
# # The list of homeservers we wish to expose our current traces to.
# # The list is a list of regexes which are matched against the
# # servername of the homeserver
# homeserver_whitelist:
# - ".*"
3 changes: 3 additions & 0 deletions synapse/app/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,9 @@ def handle_sighup(*args, **kwargs):
# Load the certificate from disk.
refresh_certificate(hs)

# Start the tracer
synapse.logging.opentracing.init_tracer(hs.config)

# It is now safe to start your Synapse.
hs.start_listening(listeners)
hs.get_datastore().start_profiling()
Expand Down
2 changes: 2 additions & 0 deletions synapse/config/homeserver.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
from .stats import StatsConfig
from .third_party_event_rules import ThirdPartyRulesConfig
from .tls import TlsConfig
from .tracer import TracerConfig
from .user_directory import UserDirectoryConfig
from .voip import VoipConfig
from .workers import WorkerConfig
Expand Down Expand Up @@ -75,5 +76,6 @@ class HomeServerConfig(
ServerNoticesConfig,
RoomDirectoryConfig,
ThirdPartyRulesConfig,
TracerConfig,
):
pass
50 changes: 50 additions & 0 deletions synapse/config/tracer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# -*- coding: utf-8 -*-
# Copyright 2019 The Matrix.org Foundation C.I.C.d
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from ._base import Config, ConfigError


class TracerConfig(Config):
def read_config(self, config, **kwargs):
self.tracer_config = config.get("opentracing")

self.tracer_config = config.get("opentracing", {"tracer_enabled": False})

if self.tracer_config.get("tracer_enabled", False):
# The tracer is enabled so sanitize the config
# If no whitelists are given
self.tracer_config.setdefault("homeserver_whitelist", [])

if not isinstance(self.tracer_config.get("homeserver_whitelist"), list):
raise ConfigError("Tracer homesererver_whitelist config is malformed")

def generate_config_section(cls, **kwargs):
return """\
## Opentracing ##
# These settings enable opentracing which implements distributed tracing
# This allows you to observe the causal chain of events across servers
# including requests, key lookups etc. across any server running
# synapse or any other other services which supports opentracing.
# (specifically those implemented with jaeger)

#opentracing:
# # Enable / disable tracer
# tracer_enabled: false
# # The list of homeservers we wish to expose our current traces to.
# # The list is a list of regexes which are matched against the
# # servername of the homeserver
# homeserver_whitelist:
# - ".*"
"""
26 changes: 21 additions & 5 deletions synapse/federation/transport/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from twisted.internet import defer

import synapse
import synapse.logging.opentracing as opentracing
from synapse.api.errors import Codes, FederationDeniedError, SynapseError
from synapse.api.room_versions import RoomVersions
from synapse.api.urls import (
Expand Down Expand Up @@ -288,14 +289,29 @@ def new_func(request, *args, **kwargs):
logger.warn("authenticate_request failed: %s", e)
raise

if origin:
with ratelimiter.ratelimit(origin) as d:
yield d
# Start an opentracing span
with opentracing.start_active_span_from_context(
request.requestHeaders,
"incoming-federation-request",
tags={
"request_id": request.get_request_id(),
opentracing.tags.SPAN_KIND: opentracing.tags.SPAN_KIND_RPC_SERVER,
opentracing.tags.HTTP_METHOD: request.get_method(),
opentracing.tags.HTTP_URL: request.get_redacted_uri(),
opentracing.tags.PEER_HOST_IPV6: request.getClientIP(),
"authenticated_entity": origin,
},
):
if origin:
with ratelimiter.ratelimit(origin) as d:
yield d
response = yield func(
origin, content, request.args, *args, **kwargs
)
else:
response = yield func(
origin, content, request.args, *args, **kwargs
)
else:
response = yield func(origin, content, request.args, *args, **kwargs)

defer.returnValue(response)

Expand Down
28 changes: 24 additions & 4 deletions synapse/http/matrixfederationclient.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
from twisted.web._newclient import ResponseDone
from twisted.web.http_headers import Headers

import synapse.logging.opentracing as opentracing
import synapse.metrics
import synapse.util.retryutils
from synapse.api.errors import (
Expand Down Expand Up @@ -339,9 +340,25 @@ def _send_request(
else:
query_bytes = b""

headers_dict = {b"User-Agent": [self.version_string_bytes]}
# Retreive current span
scope = opentracing.start_active_span(
"outgoing-federation-request",
tags={
opentracing.tags.SPAN_KIND: opentracing.tags.SPAN_KIND_RPC_CLIENT,
opentracing.tags.PEER_ADDRESS: request.destination,
opentracing.tags.HTTP_METHOD: request.method,
opentracing.tags.HTTP_URL: request.path,
},
finish_on_close=True,
)

# Inject the span into the headers
headers_dict = {}
opentracing.inject_active_span_byte_dict(headers_dict, request.destination)

with limiter:
headers_dict[b"User-Agent"] = [self.version_string_bytes]

with limiter, scope:
# XXX: Would be much nicer to retry only at the transaction-layer
# (once we have reliable transactions in place)
if long_retries:
Expand Down Expand Up @@ -419,6 +436,10 @@ def _send_request(
response.phrase.decode("ascii", errors="replace"),
)

opentracing.set_tag(
opentracing.tags.HTTP_STATUS_CODE, response.code
)
JorikSchellekens marked this conversation as resolved.
Show resolved Hide resolved

if 200 <= response.code < 300:
pass
else:
Expand Down Expand Up @@ -499,8 +520,7 @@ def _send_request(
_flatten_response_never_received(e),
)
raise

defer.returnValue(response)
defer.returnValue(response)

def build_auth_headers(
self, destination, method, url_bytes, content=None, destination_is=None
Expand Down
7 changes: 6 additions & 1 deletion synapse/http/servlet.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from canonicaljson import json

from synapse.api.errors import Codes, SynapseError
from synapse.logging.opentracing import trace_servlet

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -290,7 +291,11 @@ def register(self, http_server):
for method in ("GET", "PUT", "POST", "OPTIONS", "DELETE"):
if hasattr(self, "on_%s" % (method,)):
method_handler = getattr(self, "on_%s" % (method,))
http_server.register_paths(method, patterns, method_handler)
http_server.register_paths(
method,
patterns,
trace_servlet(self.__class__.__name__, method_handler),
)

else:
raise NotImplementedError("RestServlet must register something.")
8 changes: 6 additions & 2 deletions synapse/logging/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ class LoggingContext(object):
"alive",
"request",
"tag",
"scope",
]

thread_local = threading.local()
Expand Down Expand Up @@ -238,6 +239,7 @@ def __init__(self, name=None, parent_context=None, request=None):
self.request = None
self.tag = ""
self.alive = True
self.scope = None

self.parent_context = parent_context

Expand Down Expand Up @@ -322,10 +324,12 @@ def copy_to(self, record):
another LoggingContext
"""

# 'request' is the only field we currently use in the logger, so that's
# all we need to copy
# we track the current request
record.request = self.request

# we also track the current scope:
record.scope = self.scope

def start(self):
if get_thread_id() != self.main_thread:
logger.warning("Started logcontext %s on different thread", self)
Expand Down
Loading