Clarify recommended setup

[maxidorius]

EDIT as to provide guidance to new users - Synapse devs official answer #1 and #2

TL;DR

• CLIENT and FEDERATION ports are DIFFERENT, they do not use the same port.
• TCP 8448 (Default, can change): Federation, HTTPS, original generated self-signed certificate, directly exposing port TCP 8448 of synapse (NO reverse proxy, NO replace certificate)
• TCP 443: Clients, HTTPS, regular certificate (e.g. Let's Encrypt), reversed proxy to port TCP 8008 of synapse

The self-signed certificate of synapse SHOULD NOT be replaced and port 8448 should only be used for federation (server traffic) and directly exposed publicly.

For clients connections, a reverse proxy should be reachable publicly with a regular certificate (e.g. Let's Encrypt) on port 443 that goes to the port 8008 of synapse.

This is the last option in the first list below.

---

The community is currently discussing what is the official recommended setup for synapse in terms of ports and reverse proxying. Here are the three visions:

1. Use a regular (e.g. LTE) certificate on port 8448 and expose that for clients and servers, replacing the self-signed generated one.
2. Use a regular certificate on 443 reversed proxy to 8008 and same cert on 8448, replacing the self-signed generated one.
3. Use a regular certificate on 443 reversed proxy to 8008 and keep the self-signed generated one on 8448

My personal believe is to do go for 3) but it seems many users in the community for for 1) or 2), which I believe is due to misleading documentation for synapse.

Why I think 1) and 2) occurs due to misleading doc:
First mention of 8008 and 8448 is at Configuring synapse, last paragraph which gives the first misleading advise.
It says 8448 has a self-signed certificate which is fine to test with clients but should be replaced in production but also state that it's fine for Federation. So if you want to connect clients to 8448, you should put a proper certificate - yet doesn't instruct to connect clients to it outside of a testing scope.

The next bit comes with Connecting to Synapse from a client which says again that the easiest way to try a synapse install is to connect to 8448 but totally ignores the fact it's a self-signed and will be a problem. So you're made to believe you should have changed the certificate.

Then we come to Using a reverse proxy with Synapse which says it will work nicely with clients as you expose 443 and continues by saying clients and servers don't necessarily need to connect to the same port, says and shows a nginx config of 443->8008 for clients.

Finally, several generic points:

• Changing the certificate only matters for clients. Servers don't require anything specific.
• 8448 doesn't work out of the box for clients (due to self-signed) but does for federation
• 443 to a unencrypted 8xxx port is the typical setup with TLS offloading
• 8008 is enabled by default in the config. If 1) was intended, why not just keep 8448?
• if you use 443 for clients, you don't need to give the port when configuring, just put https://example.org - easy for users, easy for admins
• Reverse proxy 443 to 8448 with self-signed certificate might not work out of the box, so you would go for 8008 (and double TLS/SSL would not make much sense)
• synapse README never instructs or even recommends to use 8448 for clients in production, only for testing
• going for 2) means managing two configuration files and two copies of the same certs after each renewal
• The Server-Server Spec talks about caching results of looking up certificates and signing keys, which could cause downtime depending on homeservers implementation/configuiration?

---

Could the Matrix devs please clearly state what is the recommended setup?

[realitygaps]

https://matrix.org/docs/guides/lets-encrypt.html also seems to suggest putting LE on the synapse server directly is a correct approach

[ArchangeGabriel]

I do 3). Excerpt from my config:

listeners:
  -
    port: 8448
    bind_addresses:
      - '::'
    type: http
    tls: true
    x_forwarded: false
    resources:
      - names: [federation]
        compress: false

  - port: 8008
    tls: false
    bind_addresses: ['::1','127.0.0.1']
    type: http
    x_forwarded: true
    resources:
      - names: [client]
        compress: true

[turt2live]

I also use 3, with the difference being I reverse proxy 8448 to point to the federation_reader. It's reverse proxied to make it easier to switch targets more easily, if required.

[richvdh]

My recommendation for a "production" setup is option 3, as it means you can have a proper LetsEncrypt cert on port 443 without having to worry about updating the cert used by synapse when it expires (and how that interacts with other servers in the federation).

Certainly the README isn't clear in this respect. That's partly an artefact of the fact it tries to deal with a number of different scenarios, notably somebody trying to set up a "test" instance on a local machine, where having to mess about with reverse-proxies is somewhat overkill. I'm not sure what the best way to address the confusion would be; possibly the README should be broken up into separate documents on topics such as "setting up a local synapse for development/test purposes" and "setting up a production synapse with optional federation". Either way it could do with the attention of a proper technical author rather than the occasional half-hour to from a software dev trying to patch it up.

[richvdh]

Changing the certificate only matters for clients. Servers don't require anything specific.

Yeeeees, but as you have also noted:

The Server-Server Spec talks about caching results of looking up certificates and signing keys, which could cause downtime depending on homeservers implementation/configuiration?

yes. Synapse is currently a bit lax in this respect (#1362), but you could easily find yourself with outages if you're not quite careful about how you replace your certificates for federation.

[aventrax]

I use the 3) option, and I agree, docs must be coherent..

[maxidorius]

Also, homeserver.yaml talks about replacing the self-signed certificate. That should also be replaced with a proper indication that it should not be in production, only for testing.

# You can replace the self-signed certificate that synapse
# autogenerates on launch with your own SSL certificate + key pair
# if you like.  Any required intermediary certificates can be
# appended after the primary certificate in hierarchical order.

[richvdh]

hopefully improved as of #2468

[amanusk]

Could some one please clarify how to setup the reverse proxy using LetsEncrypt with apache?

Between this issue and the readme I am quite confused what goes where.

I have setup a domain and a have a letsencrypt certificate setup with --apache.

Currently I have a file /etc/apache2/sites-available/<myserver.com>-le-ssl.conf

The content of the file is

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName <myserver.com>
  SSLCertificateFile /etc/letsencrypt/live/<myserver.com>/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/<myserver.com>/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf

  SSLProxyEngine On

  <Location /_matrix>
      ProxyPass http://localhost:8008
      ProxyPassReverse http://localhost:8008
  </Location>

</VirtualHost>
</IfModule>

configuration in homeserver.yaml (no other configuration was changed)

  - port: 8008
    tls: false
    bind_addresses: ['127.0.0.1']
    type: http

    x_forwarded: true

    resources:
      - names: [client, webclient]
        compress: true
      - names: [federation]
        compress: false

When trying to connect with <myserver.com/_matrix> I get

No Such Resource

No such child resource.

[maxidorius]

@amanusk In your apache config, amend as such:

  <Location /_matrix>
      ProxyPass http://localhost:8008/_matrix
      ProxyPassReverse http://localhost:8008/_matrix
  </Location>

[amanusk]

@maxidor, thanks!
After the change I get
Your browser approached me (at /_matrix) with the method "GET". I only allow the method HEAD here.
When accessing <myserver.com>/_matrix
but <myserver.com>/_matrix/client works

[maxidorius]

then it's fine. If you have more issue, you should open a dedicated issue or come in #matrix:matrix.org

[maxidorius]

@richvdh FYI I keep seeing several people a week that still do it the wrong way, but I don't have good ideas on how to improve this further short of a full rewrite of the README. Any ideas on that front?

[ptman]

I think one very concrete improvement would be:

1. Finish dockerfile/image
2. Use said docker image in production (ideally on matrix.org), so that it's kept up to date and free of bugs
3. write docker-compose.yaml that sets up postgres, synapse, reverse proxy (nginx? traefik? caddy?), turn? This would show ports etc. quite clearly and would make it easy to fire up something to test with

[maxidorius]

@ptman This is specifically for those NOT using docker images, so from source or distro packages

[ptman]

many elements remain: database, synapse, reverse proxy, with config that listens on different ports and proxies traffic to synapse appropriately

[ptman]

maybe just add the nginx/caddy configs then?

[amanusk]

Adding an apache2 example config when using Lets Encrypt to the README will definitely help

[g4borg]

Note that in the upcloud tutorial i used to install matrix as homeserver, it also had an optional paragraph about certificate replacement for matrix to directly use LetsEncrypt from the homeserver.yaml.

For this it mentioned to set sth like:
tls_certificate_path: "/etc/letsencrypt/live/<my.domain.name>/cert.pem"

of course, this won't work anyway if your default setup uses root user access only for that folder, which it does on debian and derivatives at least. so you would have to copy the certificate around, which yet again other tutorials suggest, or worse, start synapse as root.
being lazy and paranoid as i am, this is the main reason i stayed with 3)

so yes, indeed a common doc, with clear statements of what is needed and best behaviour is a good idea. i am still - up to this point in time - unsure, whether my 8448 port for federation should have self signed certs, needs renewal, etc. or not.

btw. I knew about this issue also because I found the reverse proxy apache config here.
note on debian systems - you might also want to do

a2enmod proxy
a2enmod proxy_http

[richvdh]

please don't follow the upcloud tutorial, for this reason.

so yes, indeed a common doc

The need for this is tracked at matrix-org/matrix.org#206