From b33d4c3e69e3c5966e829cb9dc390ea59601da93 Mon Sep 17 00:00:00 2001 From: HAProxy Date: Tue, 29 Jan 2019 22:10:22 +0900 Subject: [PATCH] Backport HAProxy v1.8.19 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed commit of the following: commit ebf033b47d58aa04ae9913038c9369dab8740411 Author: Willy Tarreau Date: Mon Feb 11 14:16:19 2019 +0100 [RELEASE] Released version 1.8.19 Released version 1.8.19 with the following main changes : - DOC: ssl: Clarify when pre TLSv1.3 cipher can be used - DOC: ssl: Stop documenting ciphers example to use - BUG/MINOR: spoe: do not assume agent->rt is valid on exit - BUG/MINOR: lua: initialize the correct idle conn lists for the SSL sockets - BUG/MEDIUM: spoe: initialization depending on nbthread must be done last - BUG/MEDIUM: server: initialize the idle conns list after parsing the config - BUG/MAJOR: spoe: Don't try to get agent config during SPOP healthcheck - BUG/MAJOR: stream: avoid double free on unique_id - BUG/MINOR: config: Reinforce validity check when a process number is parsed commit 072b42ca7301399fdefa78988c1da227576ab2f8 Author: Christopher Faulet Date: Thu Feb 7 16:29:41 2019 +0100 BUG/MINOR: config: Reinforce validity check when a process number is parsed Now, in the function parse_process_number(), when a process number or a set of processes is parsed, an error is triggered if an invalid character is found. It means following syntaxes are not forbidden and will emit an alert during the HAProxy startup: 1a 1/2 1-2-3 This bug was reported on Github. See issue #36. This patch may be backported to 1.9 and 1.8. (cherry picked from commit 18cca781f5384f060704ad80018d80bdd4e01e76) [wt: adjusted context, s/max/LONGBITS] Signed-off-by: Willy Tarreau (cherry picked from commit adadecb56f68e173b19b0aa28225812e463d6930) Signed-off-by: Willy Tarreau commit 109b76f51c282ca51d0b6e6c0c9202e3c50ff1db Author: Willy Tarreau Date: Sun Feb 10 18:49:37 2019 +0100 BUG/MAJOR: stream: avoid double free on unique_id Commit 32211a1 ("BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free().") addressed a memory leak but in exchange may cause double-free due to the fact that after freeing s->unique_id it doesn't null it and then calls http_end_txn() which frees it again. Thus the process quickly crashes at runtime. This fix must be backported to all stable branches where the aforementioned patch was backported. (cherry picked from commit 09c4bab41188c13e7a9227f8baaff230ebdd0875) Signed-off-by: Willy Tarreau (cherry picked from commit 451c5a8879a9d59b489ad5117c984044d41c8338) Signed-off-by: Willy Tarreau commit 7cd8fc9eb3dfc89ef7c8d7b958f09552b3ccdc9c Author: Christopher Faulet Date: Thu Feb 7 16:13:26 2019 +0100 BUG/MAJOR: spoe: Don't try to get agent config during SPOP healthcheck During SPOP healthchecks, a dummy appctx is used to create the HAPROXY-HELLO frame and then to parse the AGENT-HELLO frame. No agent are attached to it. So it is important to not rely on an agent during these stages. When HAPROXY-HELLO frame is created, there is no problem, all accesses to an agent are guarded. This is not true during the parsing of the AGENT-HELLO frame. Thus, it is possible to crash HAProxy with a SPOA declaring the async or the pipelining capability during a healthcheck. This patch must be backported to 1.9 and 1.8. (cherry picked from commit 11389018bc92bf7b94533e682af5cb4bbf0e43d9) Signed-off-by: Willy Tarreau (cherry picked from commit 644283b87d183ad3a22c3f3e0145bde8387b4e6e) Signed-off-by: Willy Tarreau commit a2cf951ac907e6cf89cfd5eee414e29e8cd8422a Author: Willy Tarreau Date: Thu Feb 7 14:46:29 2019 +0100 BUG/MEDIUM: server: initialize the idle conns list after parsing the config The idle conns lists are sized according to the number of threads. As such they cannot be initialized during the parsing since nbthread can be set later, as revealed by this simple config which randomly crashes when used. Let's do this at the end instead. listen proxy bind :4445 mode http timeout client 10s timeout server 10s timeout connect 10s http-reuse always server s1 127.0.0.1:8000 global nbthread 8 This fix must be backported to 1.9 and 1.8. (cherry picked from commit 835daa119e75a54689375da9c77e5d0e6bd7362f) Signed-off-by: Willy Tarreau (cherry picked from commit 778e19d8bc056531a40e9eaeee4a37aff0037e68) [wt: also update srv->update_status, and s/srv/newsrv as in 980855b] Signed-off-by: Willy Tarreau commit 419fffe172c438652b06d2be08c70c7876e4f331 Author: Willy Tarreau Date: Thu Feb 7 13:40:33 2019 +0100 BUG/MEDIUM: spoe: initialization depending on nbthread must be done last The agent used to be configured depending on global.nbthread while nbthread may be set after the agent is parsed. Let's move this part to the spoe_check() function to make sure nbthread is always correct and arrays are appropriately sized. This fix must be backported to 1.9 and 1.8. (cherry picked from commit b0769b2ed668119e024e2fbda79eef46ece8511a) Signed-off-by: Willy Tarreau (cherry picked from commit ad254a7e9414d0771d2ba9b20a0d5cceb13fbc32) [wt: a few more fields exist in 1.8: applets_act, applets_idle, sending_rate] Signed-off-by: Willy Tarreau commit c5d303f85a6c5badc65e600a8492e9c89fcc84e7 Author: Willy Tarreau Date: Thu Feb 7 14:48:24 2019 +0100 BUG/MINOR: lua: initialize the correct idle conn lists for the SSL sockets Commit 40a007cf2 ("MEDIUM: threads/server: Make connection list (priv/idle/safe) thread-safe") made a copy-paste error when initializing the Lua sockets, as the TCP one was initialized twice. Fortunately it has no impact because the pointers are set to NULL after a memset(0) and are not changed in between. This must be backported to 1.9 and 1.8. (cherry picked from commit b784b35ce88842c3b7630085775fdd6e5f858bc8) Signed-off-by: Willy Tarreau (cherry picked from commit d10cdf434721a3f5f2453f60d544a56dcf426d66) Signed-off-by: Willy Tarreau commit 3abc8d3c90e7726b0772ee8e700e68620cd01490 Author: Willy Tarreau Date: Thu Feb 7 14:22:52 2019 +0100 BUG/MINOR: spoe: do not assume agent->rt is valid on exit As reported by Christopher, we may call spoe_release_agent() when leaving after an allocation failure or a config parse error. We must not assume agent->rt is valid there as the allocation could have failed. This should be backported to 1.9 and 1.8. (cherry picked from commit 3ddcf7643cfe5b542d72b0f6f815fc302e8e3bc9) Signed-off-by: Willy Tarreau (cherry picked from commit c4b91f291015dc39a7a953df41a13c2c452b50bc) Signed-off-by: Willy Tarreau commit 7410f366b258d44ac8e31290aff8e781cf1d4179 Author: Bertrand Jacquin Date: Sun Feb 3 18:48:49 2019 +0000 DOC: ssl: Stop documenting ciphers example to use Since TLS ciphers are not well understand, it is very common pratice to copy and paste parameters from documentation and use them as-is. Since RC4 should not be used anymore, it is wiser to link users to up to date documnetation from Mozilla to avoid unsafe configuration in the wild. Clarify the location of man pages for OpenSSL when missing. (cherry picked from commit 4f03ab06a90df8e88ba2e347f52465b31392acc4) Signed-off-by: Willy Tarreau (cherry picked from commit 72a3f9a0beaf821a27e2fe36ac8e035cf3713133) Signed-off-by: Willy Tarreau commit 625aa784efe0641d6efd7202301b70ac54771024 Author: Bertrand Jacquin Date: Sun Feb 3 18:35:25 2019 +0000 DOC: ssl: Clarify when pre TLSv1.3 cipher can be used This is mainly driven by the fact TLSv1.3 will have a successor at some point. (cherry picked from commit 8cf7c1eb6123bce935f592844a4638d74b462aae) Signed-off-by: Willy Tarreau (cherry picked from commit feb20b8764524c230c5fb4c364f2341b70129e88) Signed-off-by: Willy Tarreau commit 75ee9fca192ee28b13ba81aa9e956c84b8b4c61d Author: Willy Tarreau Date: Wed Feb 6 15:31:22 2019 +0100 [RELEASE] Released version 1.8.18 Released version 1.8.18 with the following main changes : - DOC: http-request cache-use / http-response cache-store expects cache name - BUG/MAJOR: cache: fix confusion between zero and uninitialized cache key - BUG/MEDIUM: ssl: Disable anti-replay protection and set max data with 0RTT. - DOC: Be a bit more explicit about allow-0rtt security implications. - BUG/MEDIUM: ssl: missing allocation failure checks loading tls key file - BUG/MINOR: backend: don't use url_param_name as a hint for BE_LB_ALGO_PH - BUG/MINOR: backend: balance uri specific options were lost across defaults - BUG/MINOR: backend: BE_LB_LKUP_CHTREE is a value, not a bit - BUG/MINOR: stick_table: Prevent conn_cur from underflowing - BUG/MINOR: server: don't always trust srv_check_health when loading a server state - BUG/MINOR: check: Wake the check task if the check is finished in wake_srv_chk() - BUG/MEDIUM: ssl: Fix handling of TLS 1.3 KeyUpdate messages - DOC: mention the effect of nf_conntrack_tcp_loose on src/dst - MINOR: h2: add a bit-based frame type representation - MINOR: h2: declare new sets of frame types - BUG/MINOR: mux-h2: CONTINUATION in closed state must always return GOAWAY - BUG/MINOR: mux-h2: headers-type frames in HREM are always a connection error - BUG/MINOR: mux-h2: make it possible to set the error code on an already closed stream - BUG/MINOR: hpack: return a compression error on invalid table size updates - DOC: nbthread is no longer experimental. - BUG/MINOR: spoe: corrected fragmentation string size - BUG/MINOR: deinit: tcp_rep.inspect_rules not deinit, add to deinit - SCRIPTS: add the slack channel URL to the announce script - SCRIPTS: add the issue tracker URL to the announce script - BUG/MINOR: stream: don't close the front connection when facing a backend error - MINOR: xref: Add missing barriers. - BUG/MEDIUM: mux-h2: wake up flow-controlled streams on initial window update - BUG/MEDIUM: mux-h2: fix two half-closed to closed transitions - BUG/MEDIUM: mux-h2: make sure never to send GOAWAY on too old streams - BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection - MINOR: stream-int: expand the flags to 32-bit - MINOR: stream-int: add a new flag to mention that we want the connection to be killed - MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection - BUG/MEDIUM: mux-h2: do not close the connection on aborted streams - BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). - BUG/MINOR: config: fix bind line thread mask validation - BUG/MAJOR: config: verify that targets of track-sc and stick rules are present - BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes - BUG/MINOR: config: make sure to count the error on incorrect track-sc/stick rules commit e50b9b2bd00b89e99ac9e5253eb23634336ffead Author: Willy Tarreau Date: Wed Feb 6 10:25:07 2019 +0100 BUG/MINOR: config: make sure to count the error on incorrect track-sc/stick rules When commit 151e1ca98 ("BUG/MAJOR: config: verify that targets of track-sc and stick rules are present") added a check for some process inconsistencies between rules and their stick tables, some errors resulted in a "return 0" statement, which is taken as "no error" in some cases. Let's fix this. This must be backported to all versions using the above commit. (cherry picked from commit 1a0fe3becd99d7860b4eeaccec407325d5a8b8c2) Signed-off-by: Willy Tarreau (cherry picked from commit 6981c190b6aea3d11eadadfd48b2a1fde05e232e) Signed-off-by: Willy Tarreau commit 4f256797faf9bedb890e1aab3b592af3f0dd8948 Author: Willy Tarreau Date: Tue Feb 5 13:37:19 2019 +0100 BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes When a filter is installed on a proxy and references spoe, we must be absolutely certain that the whole chain is valid on a given process when running in multi-process mode. The problem here is that if a proxy 1 runs on process 1, referencing an SPOE agent itself based on a backend running on process 2, this last one will be completely deinited on process 1, and will thus cause random crashes when it gets messages from this proess. This patch makes sure that the whole chain is valid on all of the caller's processes. This fix must be backported to all spoe-enabled maintained versions. It may potentially disrupt configurations which already randomly crash. There hardly is any intermediary solution though, such configurations need to be fixed. (cherry picked from commit 2bdcfde4260ac9115b8a0b7aa916975799273ea9) Signed-off-by: Willy Tarreau (cherry picked from commit b54a86df6f12ac82614921a8c0d4078d781b1ea7) Signed-off-by: Willy Tarreau commit a7f9b5545e13a9222e4912fd746f61336e43806d Author: Willy Tarreau Date: Tue Feb 5 11:38:38 2019 +0100 BUG/MAJOR: config: verify that targets of track-sc and stick rules are present Stick and track-sc rules may optionally designate a table in a different proxy. In this case, a number of verifications are made such as validating that this proxy actually exists. However, in multi-process mode, the target table might indeed exist but not be bound to the set of processes the rules will execute on. This will definitely result in a random behaviour especially if these tables do require peer synchronization, because some tasks will be started to try to synchronize form uninitialized areas. The typical issue looks like this : peers my-peers peer foo ... listen proxy bind-process 1 stick on src table ip ... backend ip bind-process 2 stick-table type ip size 1k peers my-peers While it appears obvious that the example above will not work, there are less obvious situations, such as having bind-process in a defaults section and having a larger set of processes for the referencing proxy than the referenced one. The present patch adds checks for such situations by verifying that all processes from the referencing proxy are present on the other one in all track-sc* and stick-* rules, and in sample fetch / converters referencing another table so that sc_inc_gpc0() and similar are safe as well. This fix must be backported to all maintained versions. It may potentially disrupt configurations which already randomly crash. There hardly is any intermediary solution though, such configurations need to be fixed. (cherry picked from commit 151e1ca989968f5092baa593efd9f485e4947d17) Signed-off-by: Willy Tarreau (cherry picked from commit c294490c4c977e9c1202ef76b2e0449a038caaa1) Signed-off-by: Willy Tarreau commit 4672f5920df461278985aed3e4b70f26fdf61d09 Author: Willy Tarreau Date: Sat Feb 2 17:46:24 2019 +0100 BUG/MINOR: config: fix bind line thread mask validation When no nbproc is specified, a computation leads to reading bind_thread[-1] before checking if the thread mask is valid for a bind conf. It may either report a false warning and compute a wrong mask, or miss some incorrect configs. This must be backported to 1.9 and possibly 1.8. (cherry picked from commit 6b4a39adc4f9f21dec00a118128f179ade698b17) Signed-off-by: Willy Tarreau (cherry picked from commit 2116028dd441165058fc1e89c2914c5c181c5723) Signed-off-by: Willy Tarreau commit 56fd8658819e504782e0580443ebcd351c5414c3 Author: Olivier Houchard Date: Fri Feb 1 18:10:46 2019 +0100 BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). In stream_free(), free s->unique_id. We may still have one, because it's allocated in log.c::strm_log() no matter what, even if it's a TCP connection and thus it won't get free'd by http_end_txn(). Failure to do so leads to a memory leak. This should probably be backported to all maintained branches. (cherry picked from commit 32211a17eb1f1a18d960ec2a451992a928aaaf95) Signed-off-by: Willy Tarreau (cherry picked from commit f49cc4bfbcd052e6dd448a564e3cd9505ae3fbae) Signed-off-by: Willy Tarreau commit b1e0f0f4c1c5d9066bfa06347f33d7891615fbdc Author: Willy Tarreau Date: Thu Jan 31 19:12:48 2019 +0100 BUG/MEDIUM: mux-h2: do not close the connection on aborted streams We used to rely on a hint that a shutw() or shutr() without data is an indication that the upper layer had performed a tcp-request content reject and really wanted to kill the connection, but sadly there is another situation where this happens, which is failed keep-alive request to a server. In this case the upper layer stream silently closes to let the client retry. In our case this had the side effect of killing all the connection. Instead of relying on such hints, let's address the problem differently and rely on information passed by the upper layers about the intent to kill the connection. During shutr/shutw, this is detected because the flag CS_FL_KILL_CONN is set on the connstream. Then only in this case we send a GOAWAY(ENHANCE_YOUR_CALM), otherwise we only send the reset. This makes sure that failed backend requests only fail frontend requests and not the whole connections anymore. This fix relies on the two previous patches adding SI_FL_KILL_CONN and CS_FL_KILL_CONN as well as the fix for the connection close, and it must be backported to 1.9 and 1.8, though the code in 1.8 could slightly differ (cs is always valid) : BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection MINOR: stream-int: add a new flag to mention that we want the connection to be killed MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection (cherry picked from commit 180590409ffd34d4032f89839482ab098aae6f04) Signed-off-by: Willy Tarreau (cherry picked from commit b147fb2fdcfef423e545cb43afdbe1268874f796) [wt: adjusted context] Signed-off-by: Willy Tarreau commit 9f7b1f33ab16eadf9d0055ce0e81ef44a7fb569f Author: Willy Tarreau Date: Thu Jan 31 19:09:59 2019 +0100 MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection This is the equivalent of SI_FL_KILL_CONN but for the connstreams. It will be set by the stream-interface during the various shutdown operations. (cherry picked from commit 51d0a7e54c4d2b1c90cc182a022f3635ac0ebf1c) Signed-off-by: Willy Tarreau (cherry picked from commit d3ee7806176c4f5ae666ae1d5d197a364e191033) [wt: adjusted flag value] Signed-off-by: Willy Tarreau commit fa276b2432de18cde3c12dbf5f8fc550296a8de2 Author: Willy Tarreau Date: Thu Jan 31 19:02:43 2019 +0100 MINOR: stream-int: add a new flag to mention that we want the connection to be killed The new flag SI_FL_KILL_CONN is now set by the rare actions which deliberately want the whole connection (and not just the stream) to be killed. This is only used for "tcp-request content reject", "tcp-response content reject", "tcp-response content close" and "http-request reject". The purpose is to desambiguate the close from a regular shutdown. This will be used by the next patches. (cherry picked from commit 0f9cd7b196073f6d3a3826049b985edcd20c18be) Signed-off-by: Willy Tarreau (cherry picked from commit 3c297e2b0d5f7b88ce7b5b476649e7b2c3bd839c) [wt: adjusted context; http code in proto_http, not http_act] Signed-off-by: Willy Tarreau commit 5f08f61e72828456507788582a937f9e93060e24 Author: Willy Tarreau Date: Wed Nov 14 10:53:42 2018 +0100 MINOR: stream-int: expand the flags to 32-bit We used to have enough of 16 bits, with 3 still available but it's not possible to add the rx/tx blocking bits there. Let's extend the format to 32 bits and slightly reorder the fields to maintain the struct size to 64 bytes. Nothing else was changed. (cherry picked from commit a44e576f62d68d937fefff4004b8d5631ede4f15) [wt: needed for the next fixes] Signed-off-by: Willy Tarreau commit 334191b95a18423227c42bc68992025ec724dcab Author: Willy Tarreau Date: Thu Jan 31 18:48:20 2019 +0100 BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection When finishing to respond on a stream, a shutw() is called (resulting in either an end of stream or RST), then h2_detach() is called, and may decide to kill the connection is a number of conditions are satisfied. Actually one of these conditions is that a GOAWAY frame was already sent or attempted to be sent. This one is wrong, because it can happen in at least these two situations : - a shutw() sends a GOAWAY to obey tcp-request content reject - a graceful shutdown is pending In both cases, the connection will be aborted with the mux buffer holding some data. In case of a strong abort the client will not see the GOAWAY or RST and might want to try again, which is counter-productive. In case of the graceful shutdown, it could result in truncated data. It looks like a valid candidate for the issue reported here : https://www.mail-archive.com/haproxy@formilux.org/msg32433.html A backport to 1.9 and 1.8 is necessary. (cherry picked from commit 4dbda620f2872b33aefc8c87ac34f7c71dbd1701) Signed-off-by: Willy Tarreau (cherry picked from commit ddc68d3cc6a6ce8b4b4ac1853252701c1d984059) [wt: adjusted context] Signed-off-by: Willy Tarreau commit e57bb019b1ee3d902aa530caaa7b8f1b27b38cb0 Author: Willy Tarreau Date: Wed Jan 30 19:20:09 2019 +0100 BUG/MEDIUM: mux-h2: make sure never to send GOAWAY on too old streams The H2 spec requires to send GOAWAY when the client sends a frame after it has already closed using END_STREAM. Here the corresponding case was the fallback of a series of tests on the stream state, but it unfortunately also catches old closed streams which we don't know anymore. Thus any late packet after we've sent an RST_STREAM will trigger this GOAWAY and break other streams on the connection. This can happen when launching two tabs in a browser targetting the same slow page through an H2-to-H2 proxy, and pressing Escape to stop one of them. The other one gets an error when the page finally responds (and it generally retries), and the logs in the middle indicate SD-- flags since the late response was cancelled. This patch takes care to only send GOAWAY on streams we still know. It must be backported to 1.9 and 1.8. (cherry picked from commit 24ff1f834151727cb107995b72a72e9992fd8159) Signed-off-by: Willy Tarreau (cherry picked from commit 27a97e1ac0aadf453fde1811b4a84ca91e3a5847) Signed-off-by: Willy Tarreau commit 899c2f6d518c51524a5c5780439eceab9216769c Author: Willy Tarreau Date: Wed Jan 30 19:28:32 2019 +0100 BUG/MEDIUM: mux-h2: fix two half-closed to closed transitions When receiving a HEADERS or DATA frame with END_STREAM set, we would inconditionally switch to half-closed(remote). This is wrong because we could already have been in half-closed(local) and need to switch to closed. This happens in the following situations : - receipt of the end of a client upload after we've already responded (e.g. redirects to POST requests) - receipt of a response on the backend side after we've already finished sending the request (most common case). This may possibly have caused some streams to stay longer than needed at the end of a transfer, though this is not apparent in tests. This must be backported to 1.9 and 1.8. (cherry picked from commit fc10f599cc5e5606c15be4828848e04ed2c70f9c) Signed-off-by: Willy Tarreau (cherry picked from commit 29922e39eae8e13d3eb7db64b34e6ea1abd90284) [wt: in 1.8 only the DATA frames were affected, for headers we could not be in anything but OPEN first] Signed-off-by: Willy Tarreau commit e16779d6e77e24306e210c66e85e113173ad8a6f Author: Willy Tarreau Date: Wed Jan 30 16:11:20 2019 +0100 BUG/MEDIUM: mux-h2: wake up flow-controlled streams on initial window update When a settings frame updates the initial window, all affected streams's window is updated as well. However the streams are not put back into the send list if they were already blocked on flow control. The effect is that such a stream will only be woken up by a WINDOW_UPDATE message but not by a SETTINGS changing the initial window size. This can be verified with h2spec's test http2/6.9.2/1 which occasionally fails without this patch. It is unclear whether this situation is really met in field, but the fix is trivial, it consists in adding each unblocked streams to the wait list as is done for the window updates. This fix must be backported to 1.9. For 1.8 the patch needs quite a few adaptations. It's better to copy-paste the code block from h2c_handle_window_update() adding the stream to the send_list when its mws is > 0. (cherry picked from commit b1c9edc579aedd608107c4693c17160474b5ae62) Signed-off-by: Willy Tarreau (cherry picked from commit d195a9c7e877423106c315d23184b47f4d30971c) [wt: adapted according to description] Signed-off-by: Willy Tarreau commit 5c63f7dd25d80dcdf34ebbae4178e550ede036be Author: Olivier Houchard Date: Fri Jan 18 17:21:32 2019 +0100 MINOR: xref: Add missing barriers. Add a few missing barriers in the xref code, it's unlikely to be a problem for x86, but may be on architectures with weak memory ordering. (cherry picked from commit ff5dd74e25e1069d74635dba9e8215a6093c481e) Signed-off-by: Willy Tarreau (cherry picked from commit 7fb71841f7b07f3079ee98fd204b23587939c1d5) Signed-off-by: Willy Tarreau commit ec70cf52e9ef8f86f932bcfbfc4c1dd01bb6ad5e Author: Willy Tarreau Date: Thu Jan 31 18:58:06 2019 +0100 BUG/MINOR: stream: don't close the front connection when facing a backend error In 1.5-dev13, a bug was introduced by commit e3224e870 ("BUG/MINOR: session: ensure that we don't retry connection if some data were sent"). If a connection error is reported after some data were sent (and lost), we used to accidently mark the front connection as being in error instead of only the back one because the two direction flags were applied to the same channel. This case is extremely rare with raw connections but can happen a bit more often with multiplexed streams. This will result in the error not being correctly reported to the client. This patch can be backported to all supported versions. (cherry picked from commit 28e581b21c8229aa50b7e45148dd46fa6f43da5e) Signed-off-by: Willy Tarreau (cherry picked from commit 9fae3fc9ba6612180b84a5deb27c5ddee7ac366e) Signed-off-by: Willy Tarreau commit 09e83d2d8c0cb046b561ef826012b5ee6077c409 Author: Willy Tarreau Date: Tue Jan 29 06:51:16 2019 +0100 SCRIPTS: add the issue tracker URL to the announce script This way it's easier for users to follow the status of pending issues with each release. (cherry picked from commit 9589c3bce78a28fc75978b26168e912455d2b525) Signed-off-by: Willy Tarreau (cherry picked from commit e68c1ed442d7294968af061ddac8caf8b6601547) Signed-off-by: Willy Tarreau commit 7f8ca01af0055a21d954025a65bb9f51c0d217b4 Author: Willy Tarreau Date: Wed Dec 19 18:59:51 2018 +0100 SCRIPTS: add the slack channel URL to the announce script It's just to provide the URL in the usual URLs when releasing. (cherry picked from commit d6cad12d1aa4786fcb4f6003df3e9593d7e34c19) Signed-off-by: Willy Tarreau commit b7b08a3d30fc8037a72619da1bb9c902678797a1 Author: Kevin Zhu Date: Wed Jan 30 16:01:21 2019 +0800 BUG/MINOR: deinit: tcp_rep.inspect_rules not deinit, add to deinit It seems like this can be backported as far as 1.5. (cherry picked from commit 13ebef7ecb94168039241bce66c2bdc0a3789c16) Signed-off-by: Willy Tarreau (cherry picked from commit c6358e374dd040c6b4bc5e6d6933f2b9a1b213d2) Signed-off-by: Willy Tarreau commit 57cd186404695078fba61c7c4ffae0eaddb553f4 Author: Miroslav Zagorac Date: Sun Jan 13 16:55:01 2019 +0100 BUG/MINOR: spoe: corrected fragmentation string size This patch must be backported to 1.9 and 1.8. (cherry picked from commit 6b3690bc6ae44c60677d55c6a82b459f76b91e30) Signed-off-by: Willy Tarreau (cherry picked from commit eac5bea548dd07d82af12ef91f47a2e908e52bd7) [wt: adjusted to new buffer API] Signed-off-by: Willy Tarreau commit f97ec4633e8a7d9a795ccf0c8d404139023d5f53 Author: Willy Tarreau Date: Sat Jan 26 14:20:55 2019 +0100 DOC: nbthread is no longer experimental. It was mentioned when releasing 1.8 but early bugs have long been addressed and this comment discourages some users from using threads. This should be backported to 1.9 and 1.8 now. (cherry picked from commit 1f672a8162eda18c404c6784dd749b6e061e2e4d) Signed-off-by: Willy Tarreau (cherry picked from commit 227f473d78d2cacdc01fdab80b9bc337753ec4c0) Signed-off-by: Willy Tarreau commit e264e9aa25c044221ab109777f4ce7f46064347a Author: Willy Tarreau Date: Thu Jan 24 10:47:10 2019 +0100 BUG/MINOR: hpack: return a compression error on invalid table size updates RFC7541#6.3 mandates that an error is reported when a dynamic table size update announces a size larger than the one configured with settings. This is tested by h2spec using test "hpack/6.3/1". This must be backported to 1.9 and possibly 1.8 as well. (cherry picked from commit 1e7d444eec69db192d026a542262891b8de89e0c) Signed-off-by: Willy Tarreau (cherry picked from commit 012a14fe6aa2b3ef42a6677a60fa97870968c938) [wt: adjusted context] Signed-off-by: Willy Tarreau commit dccc35ba3573af267c5316784859ae39b83c7b3e Author: Willy Tarreau Date: Thu Jan 24 10:02:24 2019 +0100 BUG/MINOR: mux-h2: make it possible to set the error code on an already closed stream When sending RST_STREAM in response to a frame delivered on an already closed stream, we used not to be able to update the error code and deliver an RST_STREAM with a wrong code (e.g. H2_ERR_CANCEL). Let's always allow to update the code so that RST_STREAM is always sent with the appropriate error code (most often H2_ERR_STREAM_CLOSED). This should be backported to 1.9 and possibly to 1.8. (cherry picked from commit 175cebb38ad7e06ae207ab947b02a344660f981b) Signed-off-by: Willy Tarreau (cherry picked from commit e199e526a943a5d25fc321308df81b3ac9e70109) [wt: adjusted context. id==0 is OK here as well] Signed-off-by: Willy Tarreau commit 2d18a33b439c9ff95dd9ece67aeef6d23ad073d2 Author: Willy Tarreau Date: Thu Jan 24 09:43:32 2019 +0100 BUG/MINOR: mux-h2: headers-type frames in HREM are always a connection error There are incompatible MUST statements in the HTTP/2 specification. Some require a stream error and others a connection error for the same situation. As discussed in the thread below, let's always apply the connection error when relevant (headers-like frame in half-closed(remote)) : https://mailarchive.ietf.org/arch/msg/httpbisa/pOIWRBRBdQrw5TDHODZXp8iblcE This must be backported to 1.9, possibly to 1.8 as well. (cherry picked from commit 5b4eae33dee01224f0ece9db0891ca7a1fb2805d) Signed-off-by: Willy Tarreau (cherry picked from commit 9db8619e1f7f67cbad46d4c5bee33a9dd4c0d3b6) Signed-off-by: Willy Tarreau commit d3cbaa592d49aa85c64fd8c8b5a0d71ea2517fd4 Author: Willy Tarreau Date: Thu Jan 24 09:36:53 2019 +0100 BUG/MINOR: mux-h2: CONTINUATION in closed state must always return GOAWAY Since we now support CONTINUATION frames, we must take care of properly aborting the connection when they are sent on a closed stream. By default we'd get a stream error which is not sufficient since the compression context is modified and unrecoverable. More info in this discussion : https://mailarchive.ietf.org/arch/msg/httpbisa/azZ1jiOkvM3xrpH4jX-Q72KoH00 This needs to be backported to 1.9 and possibly to 1.8 (less important there). (cherry picked from commit 113c7a2794a86e658faf80b000a5d849f30e299e) Signed-off-by: Willy Tarreau (cherry picked from commit eeac5e76b582cc72864cbd2540925b1ac3b72807) Signed-off-by: Willy Tarreau commit 81981189761dc8a9708af21f3b60f3a74b6d5901 Author: Willy Tarreau Date: Thu Jan 24 09:31:40 2019 +0100 MINOR: h2: declare new sets of frame types This patch adds H2_FT_HDR_MASK to group all frame types carrying headers information, and H2_FT_LATE_MASK to group frame types allowed to arrive after a stream was closed. (cherry picked from commit 71c3811589b2e8d8e28f91c7e47bd05594a739ab) Signed-off-by: Willy Tarreau (cherry picked from commit 0ec0583ce38cca41ae06384d7960b338eac6a84e) Signed-off-by: Willy Tarreau commit 7c84b83f4d1e4be861874fb2f18ba245cca18cc8 Author: Willy Tarreau Date: Fri Dec 21 14:56:57 2018 +0100 MINOR: h2: add a bit-based frame type representation This will ease checks among sets of frames. (cherry picked from commit deab244dc150032e1c57368289391af73c0b8aee) Signed-off-by: Willy Tarreau (cherry picked from commit 53f59371852419247083aac450e087485a2f5889) Signed-off-by: Willy Tarreau commit 2e405726a0c6be6617905522bde9038f75e623c4 Author: Willy Tarreau Date: Wed Jan 23 10:02:15 2019 +0100 DOC: mention the effect of nf_conntrack_tcp_loose on src/dst On rare occasions the logs may report inverted src/dst when using conntrack with this sysctl. Add a mention for it in the doc. More info here : https://www.spinics.net/lists/netdev/msg544878.html (cherry picked from commit 64ded3db2c686bad582cf9bb9fcabf21cb4becb7) Signed-off-by: Willy Tarreau (cherry picked from commit 037f9ac4a2cc4b344859af1cff7b30d5ecabe9e0) Signed-off-by: William Lallemand commit b68a427a236e7b9b0cf8b1c4a5360d960cdf9458 Author: Dirkjan Bussink Date: Mon Jan 21 09:35:03 2019 -0800 BUG/MEDIUM: ssl: Fix handling of TLS 1.3 KeyUpdate messages In OpenSSL 1.1.1 TLS 1.3 KeyUpdate messages will trigger the callback that is used to verify renegotiation is disabled. This means that these KeyUpdate messages fail. In OpenSSL 1.1.1 a better mechanism is available with the SSL_OP_NO_RENEGOTIATION flag that disables any TLS 1.2 and earlier negotiation. So if this SSL_OP_NO_RENEGOTIATION flag is available, instead of having a manual check, trust OpenSSL and disable the check. This means that TLS 1.3 KeyUpdate messages will work properly. Reported-By: Adam Langley (cherry picked from commit 526894ff3925d272c13e57926aa6b5d9d8ed5ee3) [wt: gh issue #24; Needs to be backported till 1.8] Signed-off-by: Willy Tarreau (cherry picked from commit 062c5a190d50c4aa9c5bde88c8c5c85c5f15fc7b) Signed-off-by: William Lallemand commit 7a74ffef9f356304b46ab862858cead85d451b5f Author: Christopher Faulet Date: Mon Jan 21 14:15:50 2019 +0100 BUG/MINOR: check: Wake the check task if the check is finished in wake_srv_chk() With tcp-check, the result of the check is set by the function tcpcheck_main() from the I/O layer. So it is important to wake up the check task to handle the result and finish the check. Otherwise, we will wait the task timeout to handle the result of a tcp-check, delaying the next check by as much. This patch also fixes a problem about email alerts reported by PiBa-NL (Pieter) on the ML [1] on all versions since the 1.6. So this patch must be backported from 1.9 to 1.6. [1] https://www.mail-archive.com/haproxy@formilux.org/msg32190.html (cherry picked from commit 774c486cece942570b6a9d16afe236a16ee12079) Signed-off-by: Willy Tarreau (cherry picked from commit 3722dfbbfadf8f83f82feb3e67fbe482a5c94840) Signed-off-by: William Lallemand commit 1c95076d881b7508a8d0819b1cfd642e364b255c Author: Jérôme Magnin Date: Sun Jan 20 11:27:40 2019 +0100 BUG/MINOR: server: don't always trust srv_check_health when loading a server state When we load health values from a server state file, make sure what we assign to srv->check.health actually matches the state we restore. This should be backported as far as 1.6. (cherry picked from commit f57afa453a685cfd92b7a27ef6e6035cb384ff57) Signed-off-by: Willy Tarreau (cherry picked from commit 75455a0b78ce4ac723698df26c014b38467843b1) Signed-off-by: William Lallemand commit 93b3994091b5bd17b43c9d91ecae470d33157e25 Author: Tim Duesterhus Date: Fri Jan 4 00:11:59 2019 +0100 BUG/MINOR: stick_table: Prevent conn_cur from underflowing When using the peers feature a race condition could prevent a connection from being properly counted. When this connection exits it is being "uncounted" nonetheless, leading to a possible underflow (-1) of the conn_curr stick table entry in the following scenario : - Connect to peer A (A=1, B=0) - Peer A sends 1 to B (A=1, B=1) - Kill connection to A (A=0, B=1) - Connect to peer B (A=0, B=2) - Peer A sends 0 to B (A=0, B=0) - Peer B sends 0/2 to A (A=?, B=0) - Kill connection to B (A=?, B=-1) - Peer B sends -1 to A (A=-1, B=-1) This fix may be backported to all supported branches. (cherry picked from commit 8b87c01c4d59247d9fb51a38cd12d5d94324b6a4) Signed-off-by: Willy Tarreau (cherry picked from commit 4ceecc8a4ee6f46f20c7729056e14af5a8757121) Signed-off-by: William Lallemand commit 7c6a6149a91d2e240a5a63f981c5d07d681df725 Author: Willy Tarreau Date: Mon Jan 14 17:07:39 2019 +0100 BUG/MINOR: backend: BE_LB_LKUP_CHTREE is a value, not a bit There are a few instances where the lookup algo is tested against BE_LB_LKUP_CHTREE using a binary "AND" operation while this macro is a value among a set, and not a bit. The test happens to work because the value is exactly 4 and no bit overlaps with the other possible values but this is a latent bug waiting for a new LB algo to appear to strike. At the moment the only other algo sharing a bit with it is the "first" algo which is never supported in the same code places. This fix should be backported to maintained versions for safety if it passes easily, otherwise it's not important as it will not fix any visible issue. (cherry picked from commit 6c30be52da3d949a8dd6fb5e2de7319c031e656e) Signed-off-by: Willy Tarreau (cherry picked from commit 48147c424680b7e887fb176662d58d87baa16098) Signed-off-by: William Lallemand commit a5027f804144536f79829443b33e6c19c32b690a Author: Willy Tarreau Date: Mon Jan 14 16:29:52 2019 +0100 BUG/MINOR: backend: balance uri specific options were lost across defaults The "balance uri" options "whole", "len" and "depth" were not properly inherited from the defaults sections. In addition, "whole" and "len" were not even reset when parsing "uri", meaning that 2 subsequent "balance uri" statements would not have the expected effect as the options from the first one would remain for the second one. This may be backported to all maintained versions. (cherry picked from commit 602a499da5e81d6b4cfe8410f0fc6d53c1e06745) Signed-off-by: Willy Tarreau (cherry picked from commit f00758fde5961e3bebc508852faeee4d9d80b0e0) [wla: cfg_parse_listen() is still in cfgparse.c in 1.8] Signed-off-by: William Lallemand commit 98f9549fa466e3b73a04f17dbc05fd88427c72f4 Author: Willy Tarreau Date: Mon Jan 14 15:17:46 2019 +0100 BUG/MINOR: backend: don't use url_param_name as a hint for BE_LB_ALGO_PH At a few places in the code we used to rely on this variable to guess what LB algo was in place. This is wrong because if the defaults section presets "balance url_param foo" and a backend uses "balance roundrobin", these locations will still see this url_param_name set and consider it. The harm is limited, as this only causes the beginning of the request body to be buffered. And in general this is a bad practice which prevents us from cleaning the lbprm stuff. Let's explicitly check the LB algo instead. This may be backported to all currently maintained versions. (cherry picked from commit 089eaa0ba73913187e93d52c3ea34faa01fd8f9c) Signed-off-by: Willy Tarreau (cherry picked from commit 70d1744bb41daab4110071e4855504b6dc47bda9) [wla: no htx in 1.8] Signed-off-by: William Lallemand commit 30cd01cbfd40201f3abe246216a85c69352aa79c Author: Emeric Brun Date: Thu Jan 10 10:51:13 2019 +0100 BUG/MEDIUM: ssl: missing allocation failure checks loading tls key file This patch fixes missing allocation checks loading tls key file and avoid memory leak in some error cases. This patch should be backport on branches 1.9 and 1.8 (cherry picked from commit 09852f70e0ed0f23cf9287b1ce55bb6a60112f32) Signed-off-by: Willy Tarreau (cherry picked from commit a1dc55a63cfbc8f440b72b6def3957bf1fad12b2) Signed-off-by: William Lallemand commit aca7e5aed7e036489ccc83d925103e94653b8670 Author: Olivier Houchard Date: Tue Jan 8 15:35:32 2019 +0100 DOC: Be a bit more explicit about allow-0rtt security implications. Document a bit better than allow-0rtt can trivially be used for replay attacks, and so should only be used when it's safe to replay a request. This should probably be backported to 1.8 and 1.9. (cherry picked from commit 69752964944ef9c8dc03477ee95bc7d149a72089) Signed-off-by: Willy Tarreau (cherry picked from commit bb0df71201ad5b2d0cec514773d244275e5240df) Signed-off-by: William Lallemand commit 9f01534cd68de78c74b50d7b8def07a72c2a3b49 Author: Olivier Houchard Date: Wed Jan 2 18:46:41 2019 +0100 BUG/MEDIUM: ssl: Disable anti-replay protection and set max data with 0RTT. When using early data, disable the OpenSSL anti-replay protection, and set the max amount of early data we're ready to accept, based on the size of buffers, or early data won't work with the released OpenSSL 1.1.1. This should be backported to 1.8. (cherry picked from commit 51088ce68fee0bae52118d6823873417046f9efe) Signed-off-by: Willy Tarreau (cherry picked from commit 6703b633078b6bae12395ee3e310427b37965d68) Signed-off-by: William Lallemand commit a64e5574e40e3e0819c82e35a7e3d2fa65febc73 Author: Willy Tarreau Date: Fri Jan 11 19:38:25 2019 +0100 BUG/MAJOR: cache: fix confusion between zero and uninitialized cache key The cache uses the first 32 bits of the uri's hash as the key to reference the object in the cache. It makes a special case of the value zero to mean that the object is not in the cache anymore. The problem is that when an object hashes as zero, it's still inserted but the eb32_delete() call is skipped, resulting in the object still being chained in the memory area while the block has been reclaimed and used for something else. Then when objects which were chained below it (techically any object since zero is at the root) are deleted, the walk through the upper object may encounter corrupted values where valid pointers were expected. But while this should only happen statically once on 4 billion, the problem gets worse when the cache-use conditions don't match the cache-store ones, because cache-store runs with an uninitialized key, which can create objects that will never be found by the lookup code, or worse, entries with a zero key preventing eviction of the tree node and resulting in a crash. It's easy to accidently end up on such a config because the request rules generally can't be used to decide on the response : http-request cache-use cache if { path_beg /images } http-response cache-store cache In this test, mixing traffic with /images/$RANDOM and /foo/$RANDOM will result in random keys being inserted, some of them possibly being zero, and crashes will quickly happen. The fix consists in 1) always initializing the transaction's cache_hash to zero, and 2) never storing a response for which the hash has not been calculated, as indicated by the value zero. It is worth noting that objects hashing as value zero will never be cached, but given that there's only one chance among 4 billion that this happens, this is totally harmless. This fix must be backported to 1.9 and 1.8. (cherry picked from commit c9036c00044a8d81561113886ecec9a9ce71bd3b) Signed-off-by: Willy Tarreau (cherry picked from commit 5a6279fcc16da479304bcabc1705e8653f274337) Signed-off-by: William Lallemand commit 6648ff0cccee04a6a0c0e64050151b5d6c5bac51 Author: Jarno Huuskonen Date: Fri Jan 4 14:05:02 2019 +0200 DOC: http-request cache-use / http-response cache-store expects cache name Adds missing cache name option to http-request cache-use and http-response cache-store documentation. Also adds optional if/unless condition to 10.2.2. Proxy section: http-request cache-use / http-response cache-store (cherry picked from commit 251a6b72a8b6f0a4b167f6a2960e422d682aed80) Signed-off-by: Willy Tarreau (cherry picked from commit 5376f6af9239fdf8a79b6c912387de12e3c9d6cd) [wla: no http-request/response section in 1.8] Signed-off-by: William Lallemand commit e89d25b22da1eefa88ef5aa8ad6fa21e1bd4c801 Author: Willy Tarreau Date: Tue Jan 8 14:11:02 2019 +0100 [RELEASE] Released version 1.8.17 Released version 1.8.17 with the following main changes : - BUG/MAJOR: stream-int: Update the stream expiration date in stream_int_notify() - MINOR: mux-h2: only increase the connection window with the first update - BUG/MEDIUM: mux-h2: mark that we have too many CS once we have more than the max - BUG/MEDIUM: server: Also copy "check-sni" for server templates. - MINOR: lb: allow redispatch when using consistent hash - MINOR: stream/cli: fix the location of the waiting flag in "show sess all" - MINOR: stream/cli: report more info about the HTTP messages on "show sess all" - BUG/MEDIUM: cli: make "show sess" really thread-safe - BUG/MINOR: lua: Return an error if a legacy HTTP applet doesn't send anything - BUG/MINOR: lua: bad args are returned for Lua actions - BUG/MEDIUM: lua: dead lock when Lua tasks are trigerred - BUG/CRITICAL: mux-h2: re-check the frame length when PRIORITY is used commit e1fb3ad02889d8063eace879171cddb7edf477f3 Author: Willy Tarreau Date: Mon Dec 31 07:41:24 2018 +0100 BUG/CRITICAL: mux-h2: re-check the frame length when PRIORITY is used Tim Düsterhus reported a possible crash in the H2 HEADERS frame decoder when the PRIORITY flag is present. A check is missing to ensure the 5 extra bytes needed with this flag are actually part of the frame. As per RFC7540#4.2, let's return a connection error with code FRAME_SIZE_ERROR. Many thanks to Tim for responsibly reporting this issue with a working config and reproducer. This issue was assigned CVE-2018-20615. This fix must be backported to 1.9 and 1.8. (cherry picked from commit a01f45e3ced23c799f6e78b5efdbd32198a75354) Signed-off-by: Willy Tarreau (cherry picked from commit ce376ea771ad5484cf0c7559c59e7ea807733df6) Signed-off-by: Willy Tarreau commit d94c44ef9874040e5809369aaa0648896400dec0 Author: Thierry FOURNIER Date: Sun Jan 6 19:04:24 2019 +0100 BUG/MEDIUM: lua: dead lock when Lua tasks are trigerred When a task is created from Lua context out of initialisation, the hlua_ctx_init() function can be called from safe environement, so we must not initialise it. While the support of threads appear, the safe environment set a lock to ensure only one Lua execution at a time. If we initialize safe environment in another safe environmenet, we have a dead lock. this patch adds the support of the idicator "already_safe" whoch indicates if the context is initialized form safe Lua fonction. thank to Flakebi for the report This patch must be backported to haproxy-1.9 and haproxy-1.8 (cherry picked from commit bf90ce12aaf71f7a18a1ad63d3a0b6909be97512) Signed-off-by: Willy Tarreau (cherry picked from commit 842cebf77d45664f73f5917cf48c4a062b18a57f) Signed-off-by: Willy Tarreau commit 174c4a5cf0febef88041faebc6430f66bae4adca Author: Thierry FOURNIER Date: Sun Jan 6 19:38:49 2019 +0100 BUG/MINOR: lua: bad args are returned for Lua actions In tcp actions case, the argument n - 1 is returned. For example: http-request lua.script stuff display "stuff" as first arg tcp-request content lua.script stuff display "lua.script" as first arg The action parser doesn't use the *cur_arg value. Thanks to Andy Franks for the bug report. This patch mist be backported in haproxy-1.8 and haproxy-1.9 (cherry picked from commit 1725c2e3951d4eeae136125f417c620fa0ed3847) Signed-off-by: Willy Tarreau (cherry picked from commit 10bf073b93422ef32daa0a2c23f708987269a1a3) Signed-off-by: Willy Tarreau commit e24e5fd2b510b81fb802df41067f36b904420e90 Author: Christopher Faulet Date: Tue Dec 18 21:20:57 2018 +0100 BUG/MINOR: lua: Return an error if a legacy HTTP applet doesn't send anything In legacy mode, if an HTTP applet does not send any response, an error 500 is returned. (cherry picked from commit cc26b13ea51982edccce7bd5aec7b58f395acd4c) Signed-off-by: Willy Tarreau commit 4b57858a43dd11c9257b91079f6ce256a6fe38d8 Author: Willy Tarreau Date: Fri Jan 4 17:42:57 2019 +0100 BUG/MEDIUM: cli: make "show sess" really thread-safe This one used to rely on a few spin locks around lists manipulations only but 1) there were still a few races (e.g. when aborting, or between STAT_ST_INIT and STAT_ST_LIST), and 2) after last commit which dumps htx info it became obvious that dereferencing the buffer contents is not safe at all. This patch uses the thread isolation from the rendez-vous point instead, to guarantee that nothing moves during the dump. It may make the dump a bit slower but it will be 100% safe. This fix must be backported to 1.9, and possibly to 1.8 which likely suffers from the short races above, eventhough they're extremely hard to trigger. (cherry picked from commit e6e52366c185db13ccaf0d2d669909c157d55923) Signed-off-by: Willy Tarreau (cherry picked from commit 53ce8fc062c9a80273698412badb1837290718aa) [wt: s/si_rx_room_blk/si_applet_cant_put/] Signed-off-by: Willy Tarreau commit 784260e63dbb3a994c046bf65bbf7eedcec296a8 Author: Willy Tarreau Date: Mon Jan 7 10:38:10 2019 +0100 MINOR: stream/cli: report more info about the HTTP messages on "show sess all" The "show sess all" command didn't allow to detect whether compression is in use for a given stream, which is sometimes annoying. Let's add a few more info about the HTTP messages, namely the flags, body len, chunk len and the "next" pointer. (cherry picked from commit 7778b59be1d444c7e0cb5b2fd6c10d9aa54f773d) Signed-off-by: Willy Tarreau (cherry picked from commit e9686e78b0e1df98122fe95b8e7129312cdc3bf7) Signed-off-by: Willy Tarreau commit 6d9b1b7235176d28e12d2c92e97dadd5e31884f5 Author: Willy Tarreau Date: Mon Jan 7 10:10:07 2019 +0100 MINOR: stream/cli: fix the location of the waiting flag in "show sess all" The "waiting" flag indicates if the stream is waiting for some memory, and was placed on the same output line as the txn for ease of reading. But since 1.6 the txn is not part of the stream anymore so this output was placed under a condition, resulting in "waiting" to appear only when a txn is present. Let's move it upper, closer to the stream's flags to fix this. This may safely be backported though it has little value for older versions. (cherry picked from commit adf7a15bd1d41c45a214410745479e3381ef45de) Signed-off-by: Willy Tarreau (cherry picked from commit f8b90fb6967d10428ef545ccb277c2013db44394) Signed-off-by: Willy Tarreau commit 5f768a2eab35e7ac16f49cd2c0b495e3daae2e81 Author: Willy Tarreau Date: Wed Jan 2 14:48:31 2019 +0100 MINOR: lb: allow redispatch when using consistent hash Redispatch traditionally only worked for cookie based persistence. Adding redispatch support for consistent hash based persistence - also update docs. Reported by Oskar Stenman on discourse: https://discourse.haproxy.org/t/balance-uri-consistent-hashing-redispatch-3-not-redispatching/3344 Should be backported to 1.8. Cc: Lukas Tribus (cherry picked from commit 59884a646c046411a9080e72d2266ba6a4d4d166) Signed-off-by: Willy Tarreau (cherry picked from commit 97320b5d4147277d0a685fd9c29e4dcf17310769) Signed-off-by: Willy Tarreau commit 5b9c962725e8352189911f2bdac7e3fa14f73846 Author: Olivier Houchard Date: Fri Dec 21 19:42:01 2018 +0100 BUG/MEDIUM: server: Also copy "check-sni" for server templates. When using server templates, if "check-sni" is used, make sure it shows up in all the created servers. This should be backported to 1.8 and 1.9. (cherry picked from commit 21944019cabcb46ceb95b7fd925528b9dace4e35) Signed-off-by: Willy Tarreau (cherry picked from commit c1446f2079d25d9224b1d8b88510ac020c8c47fc) Signed-off-by: Willy Tarreau commit 384253481ee25dd4bb271f71cd2d47d732926d16 Author: Willy Tarreau Date: Sun Dec 23 20:43:58 2018 +0100 BUG/MEDIUM: mux-h2: mark that we have too many CS once we have more than the max Since commit f210191 ("BUG/MEDIUM: h2: don't accept new streams if conn_streams are still in excess") we're refraining from reading input frames if we've reached the limit of number of CS. The problem is that it prevents such situations from working fine. The initial purpose was in fact to prevent from reading new HEADERS frames when this happens, and causes some occasional transfer hiccups and pauses with large concurrencies. Given that we now properly reject extraneous streams before checking this value, we can be sure never to have too many streams, and that any higher value is only caused by a scheduling reason and will go down after the scheduler calls the code. This fix must be backported to 1.9 and possibly to 1.8. It may be tested using h2spec this way with an h2spec config : while :; do h2spec -o 5 -v -t -S -k -h 127.0.0.1 -p 4443 http2/5.1.2 done (cherry picked from commit a87546624369ef94546907d90a17da9c985399fd) Signed-off-by: Willy Tarreau (cherry picked from commit 69a87adfea4090b218d8239de6b81d4376e243fc) Signed-off-by: Willy Tarreau commit c26ae3e6bd739b5936ba2da2e821064a060f156e Author: Willy Tarreau Date: Sun Dec 23 09:49:04 2018 +0100 MINOR: mux-h2: only increase the connection window with the first update Commit dc57236 ("BUG/MINOR: mux-h2: advertise a larger connection window size") caused a WINDOW_UPDATE message to be sent early with the connection to increase the connection's window size. It turns out that it causes some minor trouble that need to be worked around : - varnishtest cannot transparently cope with the WU frames during the handshake, forcing all tests to explicitly declare the handshake sequence ; - some vtc scripts randomly fail if the WU frame is sent after another expected response frame, adding uncertainty to some tests ; - h2spec doesn't correctly identify these WU at the connection level that it believes are the responses to some purposely erroneous frames it sends, resulting in some errors being reported None of these are a problem with real clients but they add some confusion during troubleshooting. Since the fix above was intended to increase the upload bandwidth, we have another option which is to increase the window size with the first WU frame sent for the connection. This way, no WU frame is sent until one is really needed, and this first frame will adjust the window to the maximum value. It will make the window increase slightly later, so the client will experience the first round trip when uploading data, but this should not be perceptible, and is not worth the extra hassle needed to maintain our debugging abilities. As an extra bonus, a few extra bytes are saved for each connection until the first attempt to upload data. This should possibly be backported to 1.9 and 1.8. (cherry picked from commit 97aaa6765870d9fd32900ab83d124e58fec6d09b) Signed-off-by: Willy Tarreau (cherry picked from commit f000410146683dd516e65bd0445e9dfa8b172115) Signed-off-by: Willy Tarreau commit ca3a8768ddf3766db6b4b9e261c891c7d12ecb09 Author: Christopher Faulet Date: Thu Jan 3 16:24:54 2019 +0100 BUG/MAJOR: stream-int: Update the stream expiration date in stream_int_notify() Since a long time, the expiration date of a stream is only updated in process_stream(). It is calculated, among others, using the channels expiration dates for reads and writes (.rex and .wex values). But these values are updated by the stream-interface. So when this happens at the connection layer, the update is only done if the stream's task is woken up. Otherwise, the stream expiration date is not immediatly updated. This leads to unexpected behaviours. Time to time, users reported that the wrong timeout was hitted or the wrong termination state was reported. This is partly because of this bug. Recently, we observed some blocked sessions for a while when big objects are served from the cache applet. It seems only concern the clients not reading the response. Because delivered objects are big, not all data can be sent. And because delivered objects are big, data are fast forwarded (from the input to the output with no stream wakeup). So in such situation, the stream expiration date is never updated and no timeout is hitted. The session remains blocked while the client remains connected. This bug exists at least since HAProxy 1.5. But recent changes on the connection layer make it more visible. It must be backported from 1.9 to 1.6. And with more pain it should be backported to 1.5. (cherry picked from commit d7607de06574f00e999434051e082615908939a6) Signed-off-by: Willy Tarreau (cherry picked from commit dcb673d23ec09af5b4cad93b34b30bc31a3495f4) Signed-off-by: Willy Tarreau commit 5c3f23783032298a602127e554a5eef0dfdf357e Author: William Lallemand Date: Fri Dec 21 16:17:53 2018 +0100 [RELEASE] Released version 1.8.16 Released version 1.8.16 with the following main changes : - BUG/MINOR: logs: leave startup-logs global and not per-thread - BUG/MEDIUM: dns: Don't prevent reading the last byte of the payload in dns_validate_response() - BUG/MEDIUM: dns: overflowed dns name start position causing invalid dns error commit 8794496bd6cd3a90082a232a69f961aa94ae87af Author: Nikhil Agrawal Date: Thu Dec 20 10:50:59 2018 +0530 BUG/MEDIUM: dns: overflowed dns name start position causing invalid dns error In dns_read_name() when dns name is used with compression and start position of name is greater than 255 name read is incorrect and causes invalid dns error. eg: 0xc11b c specifies name compression being used. 11b represent the start position of name but currently we are using only 1b for start position. This should be backported as far as 1.7. (cherry picked from commit 2fa66c3b9348d179e478d3d584471ee8989c3f6e) Signed-off-by: William Lallemand commit fe7b9f0e397e69e78dfe76ce364f409833bc7d54 Author: Jérôme Magnin Date: Thu Dec 20 16:47:31 2018 +0100 BUG/MEDIUM: dns: Don't prevent reading the last byte of the payload in dns_validate_response() A regression was introduced with efbbdf72 BUG: dns: Prevent out-of-bounds read in dns_validate_dns_response() as it prevented from taking into account the last byte of the payload. this patch aims at fixing it. this must be backported in 1.8. (cherry picked from commit 8d4e7dc880d2094658fead50dedd9c22c95c556a) Signed-off-by: William Lallemand commit 929b756c2d67f78a783890ec1ca38a83c7b064d9 Author: Willy Tarreau Date: Sat Dec 15 16:55:36 2018 +0100 BUG/MINOR: logs: leave startup-logs global and not per-thread Commit f8188c6 ("MEDIUM: threads/logs: Make logs thread-safe") made logs thread-local but it also made the copy of the startup-logs thread-local, meaning that when threads are configured, upon startup the list of startup logs appears to be empty. Let's just remove the THEAD_LOCAL directive there, as the check for the startup period is already present. This fix should be backported to 1.8. (cherry picked from commit a648399c901485a4985f786075535756946113cc) Signed-off-by: William Lallemand commit 6b6a350afe3b08a1a60c80fe9120a1c9d10448ef Author: Willy Tarreau Date: Thu Dec 13 00:59:21 2018 +0100 [RELEASE] Released version 1.8.15 Released version 1.8.15 with the following main changes : - MINOR: threads: Make sure threads_sync_pipe is initialized before using it. - DOC: clarify force-private-cache is an option - BUG/MINOR: connection: avoid null pointer dereference in send-proxy-v2 - BUG/MINOR: backend: check that the mux installed properly - BUG/MEDIUM: buffers: Make sure we don't wrap in buffer_insert_line2/replace2. - MEDIUM: ssl: add support for ciphersuites option for TLSv1.3 - BUG/MEDIUM: Cur/CumSslConns counters not threadsafe. - BUG/MINOR: checks: queues null-deref - BUG/MEDIUM: mworker: segfault receiving SIGUSR1 followed by SIGTERM. - BUG/MEDIUM: stream: don't crash on out-of-memory - BUILD: ssl: fix null-deref warning in ssl_fc_cipherlist_str sample fetch - BUILD: ssl: fix another null-deref warning in ssl_sock_switchctx_cbk() - BUILD: stick-table: make sure not to fail on task_new() during initialization - BUILD: peers: check allocation error during peers_init_sync() - DOC: Fix a few typos - BUG/MEDIUM: threads: fix thread_release() at the end of the rendez-vous point - BUG/MEDIUM: threads: make sure threads_want_sync is marked volatile - BUILD: compiler: add a new statement "__unreachable()" - MINOR: lua: all functions calling lua_yieldk() may return - BUILD: lua: silence some compiler warnings about potential null derefs (#2) - BUILD: lua: silence some compiler warnings after WILL_LJMP - CLEANUP: stick-tables: Remove unneeded double (()) around conditional clause - BUILD: Makefile: add a "make opts" target to simply show the build options - BUILD: Makefile: speed up compiler options detection - BUILD: Makefile: silence an option conflict warning with clang - MINOR: server: Use memcpy() instead of strncpy(). - MINOR: cfgparse: Write 130 as 128 as 0x82 and 0x80. - MINOR: peers: use defines instead of enums to appease clang. - DOC: fix reference to map files in MAINTAINERS - BUILD: compiler: rename __unreachable() to my_unreachable() - BUG/MEDIUM: pools: Fix the usage of mmap()) with DEBUG_UAF. - BUG/MEDIUM: h2: Close connection if no stream is left an GOAWAY was sent. - BUILD: Makefile: add the new ERR variable to force -Werror - BUG/MINOR: cache: Crashes with "total-max-size" > 2047(MB). - BUG/MINOR: cache: Wrong usage of shctx_init(). - BUG/MINOR: ssl: Wrong usage of shctx_init(). - DOC: cache: Missing information about "total-max-size" - BUG/MINOR: only mark connections private if NTLM is detected - BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic - BUG/MAJOR: http: http_txn_get_path() may deference an inexisting buffer - BUG/MEDIUM: auth/threads: use of crypt() is not thread-safe - BUG/MINOR: config: better detect the presence of the h2 pattern in npn/alpn - BUG/MEDIUM: Make sure stksess is properly aligned. - BUG/MINOR: config: Copy default error messages when parsing of a backend starts - BUG/MEDIUM: hpack: fix encoding of "accept-ranges" field - BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id - BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name - BUG/MINOR: cfgparse: Fix the call to post parser of the last sections parsed - BUG/MINOR: lb-map: fix unprotected update to server's score - BUG/MEDIUM: sample: Don't treat SMP_T_METH as SMP_T_STR. - BUG/MINOR: hpack: fix off-by-one in header name encoding length calculation - BUG/MINOR: mux-h2: refrain from muxing during the preface - BUG/MINOR: mux-h2: advertise a larger connection window size - BUILD: compression: fix build error with DEFAULT_MAXZLIBMEM - BUILD: threads: fix minor build warnings when threads are disabled - MINOR: stats: report the number of active jobs and listeners in "show info" - MINOR: servers: Free [idle|safe|priv]_conns on exit. - DOC: clarify that check-sni needs an argument. - DOC: refer to check-sni in the documentation of sni - BUG: dns: Prevent stack-exhaustion via recursion loop in dns_read_name - BUG: dns: Prevent out-of-bounds read in dns_read_name() - BUG: dns: Prevent out-of-bounds read in dns_validate_dns_response() - BUG: dns: Fix out-of-bounds read via signedness error in dns_validate_dns_response() - BUG: dns: Fix off-by-one write in dns_validate_dns_response() - DOC: Update configuration doc about the maximum number of stick counters. - DOC: restore note about "independant" typo - DOC: Fix typos in README and CONTRIBUTING - DOC: Fix typos in different subsections of the documentation - DOC: fix a few typos in the documentation commit be7326883fe16e2a302fb5869a2e45ba16777359 Author: Joseph Herlant Date: Tue Nov 13 16:55:16 2018 -0800 DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation. (cherry picked from commit 71b4b150f21a1cc0dd75f022baf80a0d2cbb00ff) [wt: dropped a few non-existing entries] Signed-off-by: Willy Tarreau commit 00e719a9032e1259fd805f09e98f21742513425b Author: Joseph Herlant Date: Tue Nov 13 19:45:17 2018 -0800 DOC: Fix typos in different subsections of the documentation Fix typos found in the design-thoughts, internals and lua-api subsections of the documentation. (cherry picked from commit 02cedc48d323d76f976b86d385e3a106f98a6a9b) [wt: dropped the notes-layers part] Signed-off-by: Willy Tarreau commit 811afc54a2b450cec750464e8579b9b93002be1a Author: Joseph Herlant Date: Fri Nov 9 17:44:10 2018 -0800 DOC: Fix typos in README and CONTRIBUTING Few typos detected by misspell in the README and CONTRIBUTING. Even if one of them is on a listing of commits. I'm assuming that if we want to enforce less typos in the commits, having one in the contributing guide is not the best example. (cherry picked from commit e07bc14e350a32a9ccf9e3d415e7d60c4d5bef09) Signed-off-by: Willy Tarreau commit e50782d97a2f5d4797593f06ebf3fd7c490600ce Author: Lukas Tribus Date: Thu Nov 8 12:41:42 2018 +0100 DOC: restore note about "independant" typo The independant -> independent error was fixed in 801a0a35 ("DOC: fix name for "option independant-streams"), but the note about the wrong name was erroneously fixed in 0e82b92a ("DOC: fix a few config typos"). Restore the "wrong" name so that when reasearching this option people can actually find it. Could be backported to 1.8. (cherry picked from commit 745f15eba97535c1b7894f81f6bc252e1a41e5d0) Signed-off-by: Willy Tarreau commit 443d0083bd3e470228e8046a053564e18bcea9b0 Author: Moemen MHEDHBI Date: Tue Sep 25 17:50:53 2018 +0200 DOC: Update configuration doc about the maximum number of stick counters. Previous patches added support to tracking up to MAX_SESS_STKCTR stick counters in the same connection, but without updating the DOC, it is done here. (cherry picked from commit 9cf4634a2b895bfe92ae47a1e4433aae1aa76065) Signed-off-by: Willy Tarreau commit 694f585a671a1dea12fdbaa3a2be2737fdd58f3a Author: Remi Gacogne Date: Wed Dec 5 17:59:56 2018 +0100 BUG: dns: Fix off-by-one write in dns_validate_dns_response() The maximum number of bytes in a DNS name is indeed 255, but we need to allocate one more byte for the NULL-terminating byte. Otherwise dns_read_name() might return 255 for a very long name, causing dns_validate_dns_response() to write a NULL value one byte after the end of the buffer: dns_answer_record->name[len] = 0; The next fields in the struct being filled from the content of the query, it might have been possible to fill them with non-0 values, causing for example a strlen() of the name to read past the end of the struct and access unintended parts of the memory, possibly leading to a crash. To be backported to 1.8, probably also 1.7. (cherry picked from commit 00488ddef5497026465b65b1cafcc1b37b753b83) Signed-off-by: Willy Tarreau commit 12e27845513f87fe2df88e5795d0273f0b992a91 Author: Remi Gacogne Date: Wed Dec 5 17:57:49 2018 +0100 BUG: dns: Fix out-of-bounds read via signedness error in dns_validate_dns_response() Since the data_len field of the dns_answer_item struct was an int16_t, record length values larger than 2^15-1 were causing an integer overflow and thus may have been interpreted as negative, making us read well before the beginning of the buffer. This might have led to information disclosure or a crash. To be backported to 1.8, probably also 1.7. (cherry picked from commit bc552102ad0ba14eaf83a93a5119f316fa6481f5) Signed-off-by: Willy Tarreau commit 2e53fe850be462dab2c1141f044a94d248d68bfe Author: Remi Gacogne Date: Wed Dec 5 17:56:29 2018 +0100 BUG: dns: Prevent out-of-bounds read in dns_validate_dns_response() We need to make sure that the record length is not making us read past the end of the data we received. Before this patch we could for example read the 16 bytes corresponding to an AAAA record from the non-initialized part of the buffer, possibly accessing anything that was left on the stack, or even past the end of the 8193-byte buffer, depending on the value of accepted_payload_size. To be backported to 1.8, probably also 1.7. (cherry picked from commit efbbdf72992cd20458259962346044cafd9331c0) Signed-off-by: Willy Tarreau commit 0de9052d0fa8d49401c7eb68d94d3616c996c076 Author: Remi Gacogne Date: Wed Dec 5 17:55:10 2018 +0100 BUG: dns: Prevent out-of-bounds read in dns_read_name() Some callers of dns_read_name() do not make sure that we can read the first byte, holding the length of the next label, without going past our buffer, so we need to make sure of that. In addition, if the label is a compressed one we need to make sure that we can read the following byte to compute the target offset. To be backported to 1.8, probably also 1.7. (cherry picked from commit 2d19fbcab21356f5aee1f2a52591a38665360bb6) Signed-off-by: Willy Tarreau commit 2b514b24f71af8ff8c6593636850b9a312a05278 Author: Remi Gacogne Date: Wed Dec 5 17:52:54 2018 +0100 BUG: dns: Prevent stack-exhaustion via recursion loop in dns_read_name When a compressed pointer is encountered, dns_read_name() will call itself with the pointed-to offset in the packet. With a specially crafted packet, it was possible to trigger an infinite-loop recursion by making the pointer points to itself. While it would be possible to handle that particular case differently by making sure that the target is different from the current offset, it would still be possible to craft a packet with a very long chain of valid pointers, always pointing backwards. To prevent a stack exhaustion in that case, this patch restricts the number of recursive calls to 100, which should be more than enough. To be backported to 1.8, probably also 1.7. (cherry picked from commit 58df5aea0a0c926b2238f65908f5e9f83d1cca25) Signed-off-by: Willy Tarreau commit 853e92f51e1a84cd255b475a82c6bb33a03b5395 Author: Jérôme Magnin Date: Sun Dec 9 16:03:40 2018 +0100 DOC: refer to check-sni in the documentation of sni Make it obvious in the description of the sni directive that it can not be used for health checks, and refer to the appropriate directive. This can be backported to 1.8 as check-sni appeared in 1.8. (cherry picked from commit b36a6d21db63dcdaf479939a5ff862cf71795175) Signed-off-by: Willy Tarreau commit 020c8f5d90913e7321518de38478422336f437e6 Author: Jérôme Magnin Date: Sun Dec 9 16:08:26 2018 +0100 DOC: clarify that check-sni needs an argument. Make it more obvious that check-sni requires an argument, and that it can only be a string. Also refer to sni for proxied traffic. This can be backported to 1.8 as check-sni appeared in 1.8. (cherry picked from commit ae9bb7600136d540f3a3ccd8c4bc5059861899bf) Signed-off-by: Willy Tarreau commit 75a68dbab4c963fae91c7aa0e1ccdfc9d881db6e Author: Olivier Houchard Date: Thu Nov 22 18:50:54 2018 +0100 MINOR: servers: Free [idle|safe|priv]_conns on exit. Don't forget to free idle_conns, safe_conns and priv_conns on exit. This can be backported to 1.8. (cherry picked from commit 7fc3be76c776533637f8493eb9c0d84704589006) Signed-off-by: Willy Tarreau commit cd6a9f06b02f4dd04d1d2ac496f2fc8dfcbba494 Author: Willy Tarreau Date: Mon Nov 5 14:38:13 2018 +0100 MINOR: stats: report the number of active jobs and listeners in "show info" When an haproxy process doesn't stop after a reload, it's because it still has some active "jobs", which mainly are active sessions, listeners, peers or other specific activities. Sometimes it's difficult to troubleshoot the cause of these issues (which generally are the result of a bug) only because some indicators are missing. This patch add the number of listeners, the number of jobs, and the stopping status to the output of "show info". This way it becomes a bit easier to try to narrow down the cause of such an issue should it happen. A typical use case is to connect to the CLI before reloading, then issuing the "show info" command to see what happens. In the normal situation, stopping should equal 1, jobs should equal 1 (meaning only the CLI is still active) and listeners should equal zero. The patch is so trivial that it could make sense to backport it to 1.8 in order to help with troubleshooting. (cherry picked from commit 00098ea034d4e3a548916cac7f50058a7c1b5256) Signed-off-by: Willy Tarreau commit 0539df4a07caa798a948e5d4cb4452de4207f3e3 Author: Willy Tarreau Date: Sun Dec 2 19:28:41 2018 +0100 BUILD: threads: fix minor build warnings when threads are disabled These potential null-deref warnings are emitted on gcc 7 and above when threads are disabled due to the use of objt_server() after an existing validity test. Let's switch to __objt_server() since we know the pointer is valid, it will not confuse the compiler. Some of these may be backported to 1.8. (cherry picked from commit b54c40ac0b19075d50d2e1a3aae2b3cf49591604) [wt: dropped pcli and htx] Signed-off-by: Willy Tarreau commit bce94072706fa4045992f847cda3fdd2448071d3 Author: Willy Tarreau Date: Mon Nov 26 10:24:45 2018 +0100 BUILD: compression: fix build error with DEFAULT_MAXZLIBMEM The tune.maxzlibmem setting was moved with commit 368780334 ("MEDIUM: compression: move the zlib-specific stuff from global.h to compression.c") but the preset value using DEFAULT_MAXZLIBMEM was incorrectly moved : - the field is in "global" and not "global.tune" - the trailing comma instead of semi-colon will make it either zero (threads enabled), break (threads enabled with debugging), or cast the memprintf's return pointer to int (threads disabled) It simply proves that nobody ever used DEFAULT_MAXZLIBMEM since 1.8! This needs to be backported to 1.8. (cherry picked from commit 3bfcd10218f96acfdbbf17a898df07703b73e06d) Signed-off-by: Willy Tarreau commit b13827a3c10f4f47f472cd2e920bee18a3155f46 Author: Willy Tarreau Date: Wed Dec 12 08:08:05 2018 +0100 BUG/MINOR: mux-h2: advertise a larger connection window size By default H2 uses a 65535 bytes window for the connection, and changing it requires sending a WINDOW_UPDATE message. We only used to update the window when receiving data, thus never increasing it further. As reported by user klzgrad on the mailing list, this seriously limits the upload bitrate, and will have an even higher impact on the backend H2 connections to origin servers. There is no technical reason for keeping this window so low, so let's increase it to the maximum possible value (2G-1). We do this by pretending we've already received that many data minus the maximum data the client might already send (65535), so that an early WINDOW_UPDATE message is sent right after the SETTINGS frame. This should be backported to 1.8. This patch depends on previous patch "BUG/MINOR: mux-h2: refrain from muxing during the preface". (cherry picked from commit dc572364c66967e16e1e029b49ad8d9d13d6bb3d) [wt: adjust context] Signed-off-by: Willy Tarreau commit 2a6c688b722875ccccccc7aa4aafd4d38d9e656b Author: Willy Tarreau Date: Wed Dec 12 08:03:58 2018 +0100 BUG/MINOR: mux-h2: refrain from muxing during the preface The condition to refrain from processing the mux was insufficient as it would only handle the outgoing connections. In essence it is not that much of a problem since we don't have streams yet on an incoming connetion. But it prevents waiting for the end of the preface before sending an early WINDOW_UPDATE message, thus causing the connections to fail in this case. This must be backported to 1.8 with a few minor adaptations. (cherry picked from commit 75a930affb363de213a325a65515a95efeb2ff65) [wt: adjust context] Signed-off-by: Willy Tarreau commit e5996c6b51364f18f8fbf80b16636c3f00c015f8 Author: Willy Tarreau Date: Tue Dec 11 06:46:03 2018 +0100 BUG/MINOR: hpack: fix off-by-one in header name encoding length calculation In hpack_encode_header() there is a length check to verify that a literal header name fits in the buffer, but there it an off-by-one in this length check, which forgets the byte required to mark the encoding type (literal without indexing). It should be harmless though as it cannot be triggered since response headers passing through haproxy are limited by the reserve, which is not the case of the output buffer. This fix should be backported to 1.8. (cherry picked from commit 7571015939eb0ab560df0fefc434e273ec9d93ff) [wt: adjust context] Signed-off-by: Willy Tarreau commit a6d094a3cb36f0d7e94b3b78d1820a8ad1160543 Author: Olivier Houchard Date: Fri Dec 7 15:23:41 2018 +0100 BUG/MEDIUM: sample: Don't treat SMP_T_METH as SMP_T_STR. In smp_dup(), don't consider a SMP_T_METH with an unknown method the same as SMP_T_STR. The string and string length aren't stored at the same place. This should be backported to 1.8. (cherry picked from commit 4468f1cacbe833a04701fcd846be959bc07803f5) [wt: adjusted buffer field names] Signed-off-by: Willy Tarreau commit d4674bbcddb783e98b7f5c691838f8fff31f01ac Author: Willy Tarreau Date: Sun Dec 2 19:22:55 2018 +0100 BUG/MINOR: lb-map: fix unprotected update to server's score The loop trying to figure the best server is theorically capable of finishing the loop with best == NULL, causing the HA_ATOMIC_SUB() to fail there. However for this to happen the list should be empty, which is avoided at the beginning of the function. As it is, the function still remains at risk so better address this now. This patch should be backported to 1.8. (cherry picked from commit c8b476d9521e4355825ae89c24c12ffbbb1791c7) Signed-off-by: Willy Tarreau commit 38539b8f06b7ee9deae53040e5b560329ba8ee24 Author: Christopher Faulet Date: Sun Dec 2 09:37:38 2018 +0100 BUG/MINOR: cfgparse: Fix the call to post parser of the last sections parsed Wrong variable was used to know if we need to call the callback post_section_parser() or not. We must use 'cs' and not 'pcs'. This patch must be backported in 1.8 with the commit 7805e2b ("BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name"). (cherry picked from commit bf7a9597e2e27f999562043eca6b95763d166130) Signed-off-by: Willy Tarreau commit 2df0bd4fe7c51b45530a39970ce17e01b2cceaa6 Author: Christopher Faulet Date: Fri Nov 30 13:50:47 2018 +0100 BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name When a section's parser is registered, it can also define a post section callback, called at the end of the section parsing. But when 2 sections with the same name followed each other, the transition between them was missed. This induced 2 bugs. First, the call to the post section callback was skipped. Then, the parsing of the second section was mixed with the first one. This patch must be backported in 1.8. (cherry picked from commit 7805e2bc1faf04169866c801087fd794535ecbb2) Signed-off-by: Willy Tarreau commit 2810ac1dd11cc7909d101ece1bb3e38a777273d2 Author: Baptiste Assmann Date: Wed Nov 28 15:20:25 2018 +0100 BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id In ssl_sock_parse_clienthello(), the code considers that SSL Sessionid size is '1', and then considers that the SSL cipher suite is availble right after the session id size information. This actually works in a single case, when the client does not send a session id. This patch fixes this issue by introducing the a propoer way to parse the session id and move forward the cursor by the session id length when required. Need to be backported to 1.8. (cherry picked from commit 6be139f86735156384d373fb5633fc4140250f60) Signed-off-by: Willy Tarreau commit 64a7cbf4ea3276b72ffb121acab676542cebde1b Author: Willy Tarreau Date: Tue Nov 20 04:47:38 2018 +0100 BUG/MEDIUM: hpack: fix encoding of "accept-ranges" field James Brown reported that when an "accept-ranges" header field is sent through haproxy and converted from HTTP/1.1 to H2, it's mis-encoded as "accept-language". It happens that it's one of the few very common header fields encoded using its index value and that this index value was misread in the spec as 17 instead of 18, resulting in the wrong name being sent. Thanks to Lukas for spotting the issue in the HPACK encoder itself. This fix must be backported to 1.8. (cherry picked from commit 4bf194cbdbcda8ec4ce83d7f12d2fe9b08483c94) [wla: buffer API edit] Signed-off-by: William Lallemand commit 54824cf35520b11ff97e0937beeb429dcdc55fd0 Author: Christopher Faulet Date: Mon Nov 12 11:57:31 2018 +0100 BUG/MINOR: config: Copy default error messages when parsing of a backend starts To be used, error messages declared in a default section must be copied when the parsing of a proxy section starts. But this was only done for frontends. This patch may be backported to older versions. (cherry picked from commit 6b44975fbd2f7d81074e20bc07fc0e01466cc9c9) Signed-off-by: William Lallemand commit 5b259db1160fa12820cc5ba6399e4dbcefa6ab22 Author: Olivier Houchard Date: Wed Nov 14 17:54:36 2018 +0100 BUG/MEDIUM: Make sure stksess is properly aligned. When we allocate struct stksess, we also allocate memory to store the associated data before the struct itself. As the data can be of different types, they can have different size. However, we need the struct stksess to be properly aligned, as it can do 64bits load/store (including atomic load/stores) on 64bits platforms, and some of them doesn't support unaligned access. So, when allocating the struct stksess, round the size up to the next multiple of sizeof(void *), and make sure the struct stksess itself is properly aligned. Many thanks to Paul Martin for investigating and reporting that bug. This should be backported to earlier releases. (cherry picked from commit 52dabbc4fad338233c7f0c96f977a43f8f81452a) Signed-off-by: William Lallemand commit c990c7fe448248c4e2a34b84b593cc1b3536b328 Author: Willy Tarreau Date: Sun Nov 11 10:36:25 2018 +0100 BUG/MINOR: config: better detect the presence of the h2 pattern in npn/alpn In 1.8, commit 45a66cc ("MEDIUM: config: ensure that tune.bufsize is at least 16384 when using HTTP/2") tried to avoid an annoying issue making H2 fail when haproxy is built with default buffer sizes smaller than 16kB, which used to be the case for a very long time. Sadly, the test only sees when NPN/ALPN exactly match "h2" and not when it's combined like "h2,http/1.1" nor "http/1.1,h2". We can safely use strstr() there because the string is prefixed by the token's length (0x02) which is unambiguous as it cannot be part of any other token. This fix should be backported to 1.8 as a safety guard against bad configurations. (cherry picked from commit 4db49c0704898e51892a176505299de3e022c5ea) Signed-off-by: William Lallemand commit a873c161d251abd025008034c0ddef8cd7f39511 Author: Willy Tarreau Date: Mon Oct 29 18:02:54 2018 +0100 BUG/MEDIUM: auth/threads: use of crypt() is not thread-safe It was reported here that authentication may fail when threads are enabled : https://bugzilla.redhat.com/show_bug.cgi?id=1643941 While I couldn't reproduce the issue, it's obvious that there is a problem with the use of the non-reentrant crypt() function there. On Linux systems there's crypt_r() but not on the vast majority of other ones. Thus a first approach consists in placing a lock around this crypt() call. Another patch may relax it when crypt_r() is available. This fix must be backported to 1.8. Thanks to Ryan O'Hara for the quick notification. (cherry picked from commit 34d4b525a129baa6f52a930ae629ddb1ba4255c2) Signed-off-by: Willy Tarreau commit 69d4ddf919fc4bc6d296a743baeccdd44fb89be6 Author: Willy Tarreau Date: Sun Oct 28 20:13:12 2018 +0100 BUG/MAJOR: http: http_txn_get_path() may deference an inexisting buffer When the "path" sample fetch function is called without any path, the function doesn't check that the request buffer is allocated. While this doesn't happen with the request during processing, it can definitely happen when mistakenly trying to reference a path from the response since the request channel is not allocated anymore. It's certain that this bug was emphasized by the buffer changes that went in 1.9 and the HTTP refactoring, but at first glance, 1.8 doesn't seem 100% safe either so it's possible that older version are affected as well. Thanks to PiBa-NL for reporting this bug with a reproducer. (cherry picked from commit 9d9ccdbf8b1178fefa2843c83bc6612733f9eca6) [wt: minor adaptation to older buffer API. There are some call places which don't look structurally safe though in their context the buffer always ought to be there] Signed-off-by: Willy Tarreau commit a100980f50f92e588c2b60f20571e84bf749f3e3 Author: Lukas Tribus Date: Sat Oct 27 20:07:40 2018 +0200 BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic While "option prefer-last-server" only applies to non-deterministic load balancing algorithms, 401/407 responses actually caused haproxy to prefer the last server unconditionally. As this breaks deterministic load balancing algorithms like uri, this patch applies the same condition here. Should be backported to 1.8 (together with "BUG/MINOR: only mark connections private if NTLM is detected"). (cherry picked from commit 80512b186fd7f4ef3bc7d9c92b281c549d72aa8a) Signed-off-by: Willy Tarreau commit 7b728d616e417f0a8cd25375f70b8a332ad23a71 Author: Lukas Tribus Date: Sat Oct 27 20:06:59 2018 +0200 BUG/MINOR: only mark connections private if NTLM is detected Instead of marking all connections that see a 401/407 response private (for connection reuse), this patch detects a RFC4559/NTLM authentication scheme and restricts the private setting to those connections. This is so we can reuse connections with 401/407 responses with deterministic load balancing algorithms later (which requires another fix). This fixes the problem reported here by Elliot Barlas : https://discourse.haproxy.org/t/unable-to-configure-load-balancing-per-request-over-persistent-connection/3144 Should be backported to 1.8. (cherry picked from commit fd9b68c48ecdba5e7971899f4eec315c8e3a3cfe) Signed-off-by: Willy Tarreau commit 9c416cf3cc449fd46880d5a7c3fdd1bb98447b68 Author: Frédéric Lécaille Date: Thu Oct 25 10:46:40 2018 +0200 DOC: cache: Missing information about "total-max-size" (cherry picked from commit e3c83d80e3aadb7b2641b861725c9d1dd7dc6713) [wt: this only retrieves from the original patch the part related to the max configurable size for total-max-size] Signed-off-by: Willy Tarreau commit 49f82640bf3c9a9c808568344bfa94d279c95b7e Author: Frédéric Lécaille Date: Thu Oct 25 20:22:46 2018 +0200 BUG/MINOR: ssl: Wrong usage of shctx_init(). With this patch we check that shctx_init() does not return 0. Must be backported to 1.8. (cherry picked from commit 4c8aa117f9bda3b5253f03ad5a7135a9165060f5) Signed-off-by: Willy Tarreau commit 68c23dedaaae8f29d26c4791b30d138ed1411548 Author: Frédéric Lécaille Date: Thu Oct 25 20:18:59 2018 +0200 BUG/MINOR: cache: Wrong usage of shctx_init(). With this patch we check that shctx_init() does not returns 0. This is possible if the maxblocks argument, which is passed as an int, is negative due to an implicit conversion. Must be backported to 1.8. (cherry picked from commit bc584494e625983f16f35982aa6dd6889e8dd222) Signed-off-by: Willy Tarreau commit bf7b382e528ab62a9f695b07e659d2f77545e93d Author: Frédéric Lécaille Date: Thu Oct 25 20:17:45 2018 +0200 BUG/MINOR: cache: Crashes with "total-max-size" > 2047(MB). With this patch we support cache size larger than 2047 (MB) and prevent haproxy from crashing when "total-max-size" is parsed as negative values by atoi(). The limit at parsing time is 4095 MB (UINT_MAX >> 20). May be backported to 1.8. (cherry picked from commit b9b8b6b6beb84b6b942d24eda56bfbe3812cc294) Signed-off-by: Willy Tarreau commit e1b3aa5613a5edbb52a44d69b3e6007d9d631981 Author: Willy Tarreau Date: Mon Oct 22 06:22:46 2018 +0200 BUILD: Makefile: add the new ERR variable to force -Werror Instead of having to fiddle with the CFLAGS, let's have ERR=1 to enable -Werror. (cherry picked from commit 23cd43e2d6fa2b6892a786a1a720c5f24e657f10) Signed-off-by: Willy Tarreau commit 105abe2f7a2e518afda9eb3bda5cceb60f6fd1b2 Author: Olivier Houchard Date: Sun Oct 21 03:01:20 2018 +0200 BUG/MEDIUM: h2: Close connection if no stream is left an GOAWAY was sent. When we're closing a stream, is there's no stream left and a goaway was sent, close the connection, there's no reason to keep it open. [wt: it's likely that this is needed in 1.8 as well, though it's unclear how to trigger this issue, some tests are needed] (cherry picked from commit 52b946686c28891a4359e9361676dc62af4fffad) Signed-off-by: Willy Tarreau commit 7e751a3c24a7021075fb298025c4a1ce98a5b049 Author: Olivier Houchard Date: Sun Oct 21 01:33:11 2018 +0200 BUG/MEDIUM: pools: Fix the usage of mmap()) with DEBUG_UAF. When mapping memory with mmap(), we should use a fd of -1, not 0. 0 may work on linux, but it doesn't work on FreeBSD, and probably other OSes. It would be nice to backport this to 1.8 to help debugging there. (cherry picked from commit 62975a7740cba4bdaf1c096dd246feba854d2410) Signed-off-by: Willy Tarreau commit 9011ff6c9fb5128dd50b4210e05199ea6337b82b Author: Willy Tarreau Date: Sat Oct 20 17:45:48 2018 +0200 BUILD: compiler: rename __unreachable() to my_unreachable() Olivier reported that on FreeBSD __unreachable is already defined and causes build warnings. Let's rename it then. (cherry picked from commit 4e7cc3381b27e3971b02b73a113ecc13916e1f20) Signed-off-by: Willy Tarreau commit c1ef9f5389e5debb132b7e2ab40f178ed413a978 Author: Lukas Tribus Date: Wed Oct 17 01:40:11 2018 +0200 DOC: fix reference to map files in MAINTAINERS s/maps/map (cherry picked from commit b75e828b298c958beb10c830a1ccb3df0840c30c) Signed-off-by: Willy Tarreau commit 0d31b8e1dae2bd0ad73c90748a03f9cfeed837d8 Author: Olivier Houchard Date: Tue Oct 16 18:49:26 2018 +0200 MINOR: peers: use defines instead of enums to appease clang. Clang (rightfully) warns that we're trying to set chars to values >= 128. Use defines with hex values instead of an enum to address this. (cherry picked from commit 33992267aac00d7e8ae67e0703bf7fffc9cf9b54) Signed-off-by: Willy Tarreau commit 1993e23d59e37ee7befbc64bf1535640a16354bc Author: Olivier Houchard Date: Tue Oct 16 18:39:38 2018 +0200 MINOR: cfgparse: Write 130 as 128 as 0x82 and 0x80. Write 130 and 128 as 8x82 and 0x80, to avoid warnings about casting from int to size. "check_req" should probably be unsigned, but it's hard to do so. (cherry picked from commit 3332090a2d3e9e84bac67af79fb03be111359429) Signed-off-by: Willy Tarreau commit 541e3b40b394fb6bde563ff8ce4c882dafca4eb1 Author: Olivier Houchard Date: Tue Oct 16 18:35:01 2018 +0200 MINOR: server: Use memcpy() instead of strncpy(). Use memcpy instead of strncpy, strncpy buys us nothing, and gcc is being annoying. (cherry picked from commit 17f8b90736d811ac9a04af198a3aee34e9935cec) Signed-off-by: Willy Tarreau commit a7e9853db925b12b1d040be8b04bafc11d84d685 Author: Willy Tarreau Date: Tue Oct 16 18:11:34 2018 +0200 BUILD: Makefile: silence an option conflict warning with clang clang complains that -fno-strict-overflow is not used when -fwrapv is used, which breaks the build when -Werror is used. Let's introduce a cc-opt-alt function to emit the former only then the latter is not supported (since it implies the former). (cherry picked from commit 0d7a2ae4f5199ec37ead6914fa24d40ec0989a4d) Signed-off-by: Willy Tarreau commit 5df1480da4c4e58830d108f4f0f3347598c55ab3 Author: Willy Tarreau Date: Wed Oct 3 09:52:51 2018 +0200 BUILD: Makefile: speed up compiler options detection Commits b78016649 and d3a7f4035 brought the ability to detect the build options and warnings that the compiler supports. However, they're detected using "$(CC) -c", which is 50% slower than "$(CC) -E" for the same result, just because it starts the assembler at the end. Given that we're starting to check for a number of warnings, this detection alone starts to become visible, taking a bit more than 300 ms on the build time. Let's switch to -E instead to shrink this incompressible time by roughly 100 ms. (cherry picked from commit f11ca5e7a43c772637018ec2ad981a9fd7d3816f) [wt: only backported for context and consistency with next patch] Signed-off-by: Willy Tarreau commit 0820ab24974cd2bad84c8ec5a90f7ce0e1681cf0 Author: Willy Tarreau Date: Wed Oct 3 09:40:22 2018 +0200 BUILD: Makefile: add a "make opts" target to simply show the build options We're often missing an easy way to map input variables to output ones. The "opts" build target will simply show the input variables and the ones passed to the compiler and linker. This way it's easier to quickly see what a given build script or package will use, or the detected warnings supported by the compiler. (cherry picked from commit a8b12c6bb73b924f6429c3ae4d20b96992e92c2e) [wt: this is not needed but significantly helps for packaging] Signed-off-by: Willy Tarreau commit b884ba5222a765b395e8ac93971639a0452d6422 Author: Dirkjan Bussink Date: Fri Sep 14 14:31:22 2018 +0200 CLEANUP: stick-tables: Remove unneeded double (()) around conditional clause In the past this conditional had multiple conditionals which is why the additional parentheses were needed. The conditional was simplified but the duplicate parentheses were not cleaned up. (cherry picked from commit ff57f1bbcf8af1e6389520aa845df5aa97ef55b6) [wt: fixes build warnings with clang] Signed-off-by: Willy Tarreau commit 3f39e1d4b5ca37e57247034421c69bc301d996b2 Author: Willy Tarreau Date: Tue Oct 16 17:57:36 2018 +0200 BUILD: lua: silence some compiler warnings after WILL_LJMP These ones are on error paths that are properly handled by luaL_error() which does a longjmp() but the compiler cannot know it. By adding an __unreachable() statement in WILL_LJMP(), there is no ambiguity anymore. This may be backported to 1.8 but these previous patches are needed first : - BUILD: compiler: add a new statement "__unreachable()" - MINOR: lua: all functions calling lua_yieldk() may return - BUILD: lua: silence some compiler warnings about potential null derefs (#2) (cherry picked from commit b059b894cdf795f134b6e53ff95ea7f907feb846) Signed-off-by: Willy Tarreau commit 8019e88dd1ac73a3baa71e9acfbc1b7a3fbc7442 Author: Willy Tarreau Date: Tue Oct 16 17:37:12 2018 +0200 BUILD: lua: silence some compiler warnings about potential null derefs (#2) Here we make sure that appctx is always taken from the unchecked value since we know it's an appctx, which explains why it's immediately dereferenced. A missing test was added to ensure that task_new() does not return a NULL. This may be backported to 1.8. (cherry picked from commit e09101e8d92b0c0ef8674fbc791e309112ab7f1c) Signed-off-by: Willy Tarreau commit 330e08dfc588dc9b0ad42203123fab6c191ca2f8 Author: Willy Tarreau Date: Tue Oct 16 17:52:55 2018 +0200 MINOR: lua: all functions calling lua_yieldk() may return There was a mistake when tagging functions which always use longjmp and those which may use it in that all those supposed to call lua_yieldk() may return without calling longjmp. Thus they must not use WILL_LJMP() but MAY_LJMP(). It has zero impact on the code emitted as such, but prevents other fixes from being properly implemented : this was the cause of the previous failure with the __unreachable() calls. This may be backported to older versions. It may or may not apply well depending on the context, though the change simply consists in replacing "WILL_LJMP(hlua_yieldk" with "MAY_LJMP(hlua_yieldk", and same with the single call to lua_yieldk() in hlua_yieldk(). (cherry picked from commit 9635e03c41e95dff38731f67cc9d8b00e3731d2a) Signed-off-by: Willy Tarreau commit d26a40412197ba61a72368c71e8a8582d686d28c Author: Willy Tarreau Date: Mon Oct 15 11:53:34 2018 +0200 BUILD: compiler: add a new statement "__unreachable()" This statement is used as a hint for the compiler so that it knows that the location where it's placed cannot be reached. It will mostly be used after longjmp() or equivalent statements that deal with error processing and that the compiler doesn't know will not return on certain conditions, so that it doesn't complain about null dereferences on error paths. (cherry picked from commit 8d26f02e693121764bfa0cb48c9a7ab31e17225d) Signed-off-by: Willy Tarreau commit 4805c249aabc45cd59386694f962e19ab50e8ca9 Author: Willy Tarreau Date: Tue Oct 16 16:57:40 2018 +0200 BUG/MEDIUM: threads: make sure threads_want_sync is marked volatile The threads_want_sync variable is not volatile, which allows the compiler to cache old copies of it for long parts of code and possibly optimize some tests away. This could result in deadlocks when using heavy queue activity or health check state changes. There is no upstream commit for this fix because the sync point was completely removed from 1.9. This fix is exclusively for 1.8. Signed-off-by: Willy Tarreau commit 75795017480da0f0a1157e945043249fe625f92f Author: Willy Tarreau Date: Tue Oct 16 16:11:56 2018 +0200 BUG/MEDIUM: threads: fix thread_release() at the end of the rendez-vous point There is a bug in this function used to release other threads. It leaves the current thread marked as harmless. If after this another thread does a thread_isolate(), but before the first one reaches poll(), the second thread will believe it's alone while it's not. This must be backported to 1.8 since the rendez-vous point was merged into 1.8.14. (cherry picked from commit a9c0252b2e8ff7bb728b84d977ac6e9581ea12f8) Signed-off-by: Willy Tarreau commit c6eb147201c1d05afaadc5fd248b17be91f97331 Author: Bertrand Jacquin Date: Sat Oct 13 16:06:18 2018 +0100 DOC: Fix a few typos these are mostly spelling mistakes, some of them might be candidate for backporting as well. (cherry picked from commit d5e4de8e5f99108e31dc7a23a0e91c4231e37974) Signed-off-by: Willy Tarreau commit d28afe3631e20a9fcca47efde031d62e501eff48 Author: Willy Tarreau Date: Mon Oct 15 11:18:03 2018 +0200 BUILD: peers: check allocation error during peers_init_sync() peers_init_sync() doesn't check task_new()'s return value and doesn't return any result to indicate success or failure. Let's make it return an int and check it from the caller. This can be backported as far as 1.6. (cherry picked from commit d944344f01d9ea914d94c45f6ac7c224c6143fc9) Signed-off-by: Willy Tarreau commit eb72c1faedc39c68fb1246ea8a97d1f96831756c Author: Willy Tarreau Date: Mon Oct 15 11:12:15 2018 +0200 BUILD: stick-table: make sure not to fail on task_new() during initialization Gcc reports a potential null-deref error in the stick-table init code. While not critical there, it's trivial to fix. This check has been missing since 1.4 so this fix can be backported to all supported versions. (cherry picked from commit 848522f05df9e60eea9274e11f1e9fcd19594a5c) Signed-off-by: Willy Tarreau commit 892c21240adb9ac230d4bd27cc8be4767b4902aa Author: Willy Tarreau Date: Mon Oct 15 13:20:07 2018 +0200 BUILD: ssl: fix another null-deref warning in ssl_sock_switchctx_cbk() This null-deref cannot happen either as there necesarily is a listener where this function is called. Let's use __objt_listener() to address this. This may be backported to 1.8. (cherry picked from commit a8825520b785d592467c45e183ad8213cb7bf891) Signed-off-by: Willy Tarreau commit d332b12b262ad7df1c8bdda52dad100f40399d24 Author: Willy Tarreau Date: Mon Oct 15 11:01:59 2018 +0200 BUILD: ssl: fix null-deref warning in ssl_fc_cipherlist_str sample fetch Gcc 6.4 detects a potential null-deref warning in smp_fetch_ssl_fc_cl_str(). This one is not real since already addressed a few lines above. Let's use __objt_conn() instead of objt_conn() to avoid the extra test that confuses it. This could be backported to 1.8. (cherry picked from commit b729077710b14c75936909409e27a4fa0badcb54) Signed-off-by: Willy Tarreau commit 4bf6d76a22b9b601fd57df4aa0f4fba62733cb07 Author: Willy Tarreau Date: Mon Oct 15 11:08:55 2018 +0200 BUG/MEDIUM: stream: don't crash on out-of-memory In case pool_alloc() fails in stream_new(), we try to detach the stream from the list before it has been added, dereferencing a NULL. In order to fix it, simply move the LIST_DEL call upwards. This must be backported to 1.8. (cherry picked from commit e5f229e6392fd54aaba7fe58f457723c16b9d15f) Signed-off-by: Willy Tarreau commit df4822ea169adc5c7c987fa077438f0ded1ac39b Author: Emeric Brun Date: Thu Oct 11 15:27:07 2018 +0200 BUG/MEDIUM: mworker: segfault receiving SIGUSR1 followed by SIGTERM. This bug appeared only if nbthread > 1. Handling the pipe with the master, multiple threads of the same worker could process the deinit(). In addition, deinit() was called while some other threads were still performing some tasks. This patch assign the handler of the pipe with master to only the first thread and removes the call to deinit() before exiting with an error. This patch should be backported in v1.8. (cherry picked from commit c8c0ed91cb4436491efd2ce2c4b4b1694aeeccca) [wt: adjusted context] Signed-off-by: Willy Tarreau commit 8a6c4ff3f407b916bc08da4e76ed7813768ac937 Author: mildis Date: Tue Oct 2 16:46:34 2018 +0200 BUG/MINOR: checks: queues null-deref queues can be null if calloc() failed. Bypass free* calls when calloc did fail. (cherry picked from commit 5ab01cb01114065a3573570a48e84815e751bf14) Signed-off-by: Willy Tarreau commit 30ba96df349ace825749a57490defeb50001a550 Author: Emeric Brun Date: Wed Oct 10 14:51:02 2018 +0200 BUG/MEDIUM: Cur/CumSslConns counters not threadsafe. CurSslConns inc/dec operations are not threadsafe. The unsigned CurSslConns counter can wrap to a negative value. So we could notice connection rejects because of MaxSslConns limit artificially exceeded. CumSslConns inc operation are also not threadsafe so we could miss some connections and show inconsistenties values compared to CumConns. This fix should be backported to v1.8. (cherry picked from commit 7ad43e7928c9a61b40332e4d5e9a7ccc33e6b65b) Signed-off-by: Willy Tarreau commit 4be76416751aa22992a44f2f5cfdba506809fd89 Author: Dirkjan Bussink Date: Fri Sep 14 11:14:21 2018 +0200 MEDIUM: ssl: add support for ciphersuites option for TLSv1.3 OpenSSL released support for TLSv1.3. It also added a separate function SSL_CTX_set_ciphersuites that is used to set the ciphers used in the TLS 1.3 handshake. This change adds support for that new configuration option by adding a ciphersuites configuration variable that works essentially the same as the existing ciphers setting. Note that it should likely be backported to 1.8 in order to ease usage of the now released openssl-1.1.1. (cherry picked from commit 415150f7640b06740fa832363d186c5c6565338e) Signed-off-by: Willy Tarreau commit 45e9f3c660c872e93588cf1c0b74c192f2c8c3d5 Author: Olivier Houchard Date: Wed Sep 26 15:09:58 2018 +0200 BUG/MEDIUM: buffers: Make sure we don't wrap in buffer_insert_line2/replace2. In buffer_insert_line2() and buffer_replace2(), we can't afford to wrap, so don't use b_tail to check if we do, directly use b->p + b->i instead. This should be backported to previous versions. (cherry picked from commit 363c745569b6ffd8f095d2b7758131d08aa27219) Signed-off-by: Christopher Faulet [cf: This patch was adapted and its commit message too. Because of the refactoring of the buffer's API in 1.9, the original patch fixes same bug in ci_insert_line2/b_rep_blk.] commit e725a7f9bfd8b7fe2e74c62c7c6bf2b9ebf83772 Author: Willy Tarreau Date: Wed Oct 3 10:20:19 2018 +0200 BUG/MINOR: backend: check that the mux installed properly The return value from conn_install_mux() was not checked, so if an inconsistency happens in the code, or a memory allocation fails while initializing the mux, we can crash while using an uninitialized mux. In practice the code inconsistency does not really happen since we cannot configure such a situation, except during development, but the out of memory condition could definitely happen. This should be backported to 1.8 (the code is a bit different there, there are two calls to conn_install_mux()). (cherry picked from commit 33dd4ef81245bb868b22f99b9be45d0791131eec) Signed-off-by: Christopher Faulet commit f6d20e718131aa2b468ff0a6c42e20c0b900e58b Author: Ilya Shipitsin Date: Sat Sep 15 00:50:05 2018 +0500 BUG/MINOR: connection: avoid null pointer dereference in send-proxy-v2 found by coverity. [wt: this bug was introduced by commit 404d978 ("MINOR: add ALPN information to send-proxy-v2"). It might be triggered by a health check on a server using ppv2 or by an applet making use of such a server, if at all configurable]. This needs to be backported to 1.8. (cherry picked from commit ca56fce8bd271928b18d38b439bd35bd273fe8d4) Signed-off-by: Christopher Faulet commit 18aff2297ce844362f28ea5317c289ba154bd33d Author: Lukas Tribus Date: Mon Oct 1 02:00:16 2018 +0200 DOC: clarify force-private-cache is an option "boolean" may confuse users into thinking they need to provide additional arguments, like false or true. This is a simple option like many others, so lets not confuse the users with internals. Also fixes an additional typo. Should be backported to 1.8 and 1.7. (cherry picked from commit 2793578eaf934bbf28f742a35f3a1ae656280324) Signed-off-by: Christopher Faulet commit 14844e448b637fea2770bcb03a43a010c4c8176d Author: Olivier Houchard Date: Thu Sep 27 14:55:34 2018 +0200 MINOR: threads: Make sure threads_sync_pipe is initialized before using it. thread_want_sync() might be called before thread_sync_init() was called, at least when reading the server state file, as apply_server_state() is called before thread_sync_init(). So make sure the threads_sync_pipe was initialized before writing to it, if it was not, there's no thread, so no need to sync anything anyway, and if we don't check it we'll end up writing a 'S' on stdin. this only applies to 1.8. commit 52e4d43ba395c950c9d2121ca55b105ed54a85a4 Author: Willy Tarreau Date: Thu Sep 20 14:09:31 2018 +0200 [RELEASE] Released version 1.8.14 Released version 1.8.14 with the following main changes : - BUG/MEDIUM: servers: check the queues once enabling a server - BUG/MEDIUM: queue: prevent a backup server from draining the proxy's connections - MINOR: dns: fix wrong score computation in dns_get_ip_from_response - MINOR: dns: new DNS options to allow/prevent IP address duplication - BUG/MEDIUM: lua: possible CLOSE-WAIT state with '\n' headers - MINOR: threads: Introduce double-width CAS on x86_64 and arm. - BUG/MEDIUM: threads: fix the double CAS implementation for ARMv7 - MINOR: threads: add more consistency between certain variables in no-thread case - BUG/MEDIUM: threads: fix the no-thread case after the change to the sync point - MEDIUM: hathreads: implement a more flexible rendez-vous point - BUG/MEDIUM: cli: make "show fd" thread-safe - BUG/MINOR: ssl: empty connections reported as errors. - BUG/MEDIUM: ssl: fix missing error loading a keytype cert from a bundle. - BUG/MEDIUM: ssl: loading dh param from certifile causes unpredictable error. - BUG/MINOR: map: fix map_regm with backref - DOC: dns: explain set server ... fqdn requires resolver - DOC: ssl: Use consistent naming for TLS protocols - BUG/MEDIUM: lua: socket timeouts are not applied - BUG/MEDIUM: cli/threads: protect all "proxy" commands against concurrent updates - BUG/MEDIUM: cli/threads: protect some server commands against concurrent operations - DOC: Fix spelling error in configuration doc - BUG/MEDIUM: unix: provide a ->drain() function - BUG/MINOR: lua: Bad HTTP client request duration. - BUG/MEDIUM: mux_pt: dereference the connection with care in mux_pt_wake() - BUG/MEDIUM: lua: reset lua transaction between http requests - BUG/MEDIUM: hlua: Make sure we drain the output buffer when done. - BUG/MAJOR: thread: lua: Wrong SSL context initialization. - BUG/MEDIUM: hlua: Don't call RESET_SAFE_LJMP if SET_SAFE_LJMP returns 0. - BUG/MEDIUM: dns/server: fix incomatibility between SRV resolution and server state file - BUG/MEDIUM: ECC cert should work with TLS < v1.2 and openssl >= 1.1.1 - MINOR: thread: implement HA_ATOMIC_XADD() - BUG/MINOR: stream: use atomic increments for the request counter - BUG/MEDIUM: session: fix reporting of handshake processing time in the logs - BUG/MEDIUM: h2: fix risk of memory leak on malformated wrapped frames - BUG/MINOR: dns: check and link servers' resolvers right after config parsing - BUG/MINOR: http/threads: atomically increment the error snapshot ID - BUG/MEDIUM: snapshot: take the proxy's lock while dumping errors - BUG/MAJOR: kqueue: Don't reset the changes number by accident. - BUG/MINOR: server: Crash when setting FQDN via CLI. - DOC: Fix typos in lua documentation - BUG/MEDIUM: patterns: fix possible double free when reloading a pattern list - BUG/MINOR: tools: fix set_net_port() / set_host_port() on IPv4 - BUG/MINOR: cli: make sure the "getsock" command is only called on connections - BUG/CRITICAL: hpack: fix improper sign check on the header index value commit b4e05a3daa30f657db01ec144a0e48850c48f813 Author: Willy Tarreau Date: Mon Sep 17 14:07:33 2018 +0200 BUG/CRITICAL: hpack: fix improper sign check on the header index value Tim Düsterhus found using afl-fuzz that some parts of the HPACK decoder use incorrect bounds checking which do not catch negative values after a type cast. The first culprit is hpack_valid_idx() which takes a signed int and is fed with an unsigned one, but a few others are affected as well due to being designed to work with an uint16_t as in the table header, thus not being able to detect the high offset bits, though they are not exposed if hpack_valid_idx() is fixed. The impact is that the HPACK decoder can be crashed by an out-of-bounds read. The only work-around without this patch is to disable H2 in the configuration. CVE-2018-14645 was assigned to this bug. This patch addresses all of these issues at once. It must be backported to 1.8. (cherry picked from commit 7f2a44d319283b5d8e551640bb63c15a2df20358) Signed-off-by: Willy Tarreau commit f7377194f0f1249bf47a743963819263fe0a523b Author: Willy Tarreau Date: Thu Sep 20 11:22:29 2018 +0200 BUG/MINOR: cli: make sure the "getsock" command is only called on connections Theorically nothing would prevent a front applet form connecting to a stats socket, and if a "getsock" command was issued, it would cause a crash. Right now nothing in the code does this so in its current form there is no impact. It may or may not be backported to 1.8. (cherry picked from commit c2b7f80a91ed01f78aa22b3e2afeda0672501e0c) Signed-off-by: Willy Tarreau commit 88c166870a779985d50f6a2cf840c844bc9b64da Author: Willy Tarreau Date: Thu Sep 20 10:48:35 2018 +0200 BUG/MINOR: tools: fix set_net_port() / set_host_port() on IPv4 These two functions were apparently written on the same model as their parents when added by commit 11bcb6c4f ("[MEDIUM] IPv6 support for syslog") except that they perform an assignment instead of a return, and as a result fall through the next case where the assigned value may possibly be partially overwritten. At least under Linux the port offset is the same in both sockaddr_in and sockaddr_in6 so the value is written twice without side effects. This needs to be backported as far as 1.5. (cherry picked from commit 4c0fcc2314f746b05bd5a613f188784761dc9ac9) Signed-off-by: Willy Tarreau commit 43dd794b56bc25f6454271c92a02a41debcc48f3 Author: Dragan Dosen Date: Tue Sep 18 20:18:09 2018 +0200 BUG/MEDIUM: patterns: fix possible double free when reloading a pattern list A null pointer assignment was missing after free() in function pat_ref_reload() which can lead to segfault. This bug was introduced in commit b5997f7 ("MAJOR: threads/map: Make acls/maps thread safe"). Must be backported to 1.8. (cherry picked from commit f147479bd56bfeb442b25c458dab95b70d6e1c8b) Signed-off-by: Willy Tarreau commit 466bb48aad6e37b960072c44e0835f61896c940e Author: Bertrand Jacquin Date: Mon Sep 10 21:26:07 2018 +0100 DOC: Fix typos in lua documentation (cherry picked from commit 874a35cb55022d55786d14fa7b7ea7b5aaba7424) [wt: Socket.settimeout() takes an integer in 1.8] Signed-off-by: Willy Tarreau commit 0521120f45f069c58a870bace43bd7d8f3673a49 Author: Frédéric Lécaille Date: Tue Aug 21 15:04:23 2018 +0200 BUG/MINOR: server: Crash when setting FQDN via CLI. This patch ensures that a DNS resolution may be launched before setting a server FQDN via the CLI. Especially, it checks that resolvers was set. A LEVEL 4 reg testing file is provided. Thanks to Lukas Tribus for having reported this issue. Must be backported to 1.8. (cherry picked from commit 5afb3cfbcc9e690ee104d64b80aadfe5fd829af0) Signed-off-by: Willy Tarreau commit 8e5b0923a604685e5483cb28a51de3fb261929bb Author: Olivier Houchard Date: Tue Sep 11 14:44:51 2018 +0200 BUG/MAJOR: kqueue: Don't reset the changes number by accident. In _update_fd(), if the fd wasn't polled, and we don't want it to be polled, we just returned 0, however, we should return changes instead, or all previous changes will be lost. This should be backported to 1.8. (cherry picked from commit 5ab33944cd7a978c9a30ee6823bdf58e90c0abdb) Signed-off-by: Willy Tarreau commit 3e986846e2c36455ebadbb54997602567bd48926 Author: Willy Tarreau Date: Fri Sep 7 19:55:44 2018 +0200 BUG/MEDIUM: snapshot: take the proxy's lock while dumping errors The proxy's lock it held while filling the error but not while dumping it, so it's possible to dereference pointers being replaced, typically server pointers. The risk is very low and unlikely but not inexistent. Since "show errors" is rarely used in parallel, let's simply grab the proxy's lock while dumping. Ideally we should use an R/W lock here but it will not make any difference. This patch must be backported to 1.8, but the code is in proto_http.c there, though mostly similar. (cherry picked from commit 36b2736a6952c2853be5d729b5ca53fcc86eb2f2) [wt: applied by hand to proto_http.c] Signed-off-by: Willy Tarreau commit 0e14ba77e518c8d5ae94bdca1802d468ebb1827b Author: Willy Tarreau Date: Fri Sep 7 11:29:59 2018 +0200 BUG/MINOR: http/threads: atomically increment the error snapshot ID Let's use an atomic increment for the error snapshot, as we'd rather not assign the same ID to two errors happening in parallel. It's very unlikely that it will ever happen though. This patch must be backported to 1.8 with the other one it relies on ("MINOR: thread: implement HA_ATOMIC_XADD()"). (cherry picked from commit e9e878a05665dc1f4dd238ead6c01c295879726c) Signed-off-by: Willy Tarreau commit e39fd1511663510d88006500b84b770beb3390eb Author: Baptiste Assmann Date: Fri Aug 10 10:56:38 2018 +0200 BUG/MINOR: dns: check and link servers' resolvers right after config parsing On the Mailing list, Marcos Moreno reported that haproxy configuration validation (through "haproxy -c cfgfile") does not detect when a resolvers section does not exist for a server. That said, this checking is done after HAProxy has started up. The problem is that this can create production issue, since init script can't detect the problem before starting / reloading HAProxy. To fix this issue, this patch registers the function which validates DNS configuration validity and run it right after configuration parsing is finished (through cfg_register_postparser()). Thanks to it, now "haproxy -c cfgfile" will fail when a server points to a non-existing resolvers section (or any other validation made by the function above). Backport status: 1.8 (cherry picked from commit 044fd5bc2c37cf53550ad75a9bee8be30de242df) Signed-off-by: Willy Tarreau commit 52338a1455eb6d2361ee84eeee5b783bcc011441 Author: Willy Tarreau Date: Wed Sep 5 18:30:05 2018 +0200 BUG/MEDIUM: h2: fix risk of memory leak on malformated wrapped frames While parsing a headers frame, if the frame is wrapped in the buffer and needs to be unwrapped, it will be duplicated before being processed. But if it contains certain combinations of invalid flags, the parser returns without releasing the temporary buffer leading to a memory leak. This fix needs to be backported to 1.8. (cherry picked from commit a0d11b6fd5b6cec3af9a70e38895a665a666ae80) Signed-off-by: Willy Tarreau commit 8342ef90968f1989dd72dc7d1edeaef0579b7a00 Author: Willy Tarreau Date: Wed Sep 5 11:56:48 2018 +0200 BUG/MEDIUM: session: fix reporting of handshake processing time in the logs The handshake processing time used to be stored per stream, which was valid when there was exactly one stream per session. With H2 and multiplexing it's not the case anymore and the reported handshake times are wrong in the logs as it's computed between the TCP accept() and the stream creation. Let's first move the handshake where it belongs, which is the session. However, this is not enough because we don't want to report an excessive idle time either for H2 (since many requests use the connection). So the solution used here is to have the stream retrieve sess->tv_accept and the handshake duration when the stream is created, and let the mux immediately reset them. This way, the handshake time becomes zero for the second and subsequent requests in H2 (which was already the case in H1), and the idle time exactly counts how long the connection remained unused while it could be used, so in H1 it runs from the end of the previous response and in H2 it runs from the end of the previous request since the channel is already available. This patch will need to be backported to 1.8. (cherry picked from commit 590a0514f2d36e2e35704654a6588ff7c82871f6) Signed-off-by: Willy Tarreau commit 9e175481659198481222958b469946e42b31c6f2 Author: Willy Tarreau Date: Wed Sep 5 16:21:29 2018 +0200 BUG/MINOR: stream: use atomic increments for the request counter The request counter is incremented when creating a new stream and when resetting a stream, preparing for a new request. Unfortunately during the thread migration this was missed, leading to non-atomic increments in case threads are in use. The most visible side effect is that two requests may have the same ID from time to time in the logs. However the SPOE also uses this ID to route responses back to the stream so it may also lead to occasional spurious SPOE timeouts. Note that it still doesn't guarantee temporal unicity in the stream identifiers since a long and a short connection could technically use the same ID. The likeliness that this happens at the same time is almost null (roughly threads*runqueue_depth/2^32 that it happens in the same poll loop), but it will have to be addressed later anyway. This patch must be backported to 1.8 with the other one it relies on ("MINOR: thread: implement HA_ATOMIC_XADD()"). (cherry picked from commit 90a7c03ec0bc405fd9514a0a65dccc15b8c4a993) Signed-off-by: Willy Tarreau commit aef23d73eb4d09f916fcea574c8b5bc4aafad4f6 Author: Willy Tarreau Date: Wed Sep 5 16:11:03 2018 +0200 MINOR: thread: implement HA_ATOMIC_XADD() We've been missing it several times and now we'll need it to increment a request counter. Let's do it once for all. This patch will need to be backported to 1.8 with the associated fix. (cherry picked from commit 9378df89f65b2fa8b6effb6dfbe7a4de70b53702) Signed-off-by: Willy Tarreau commit d309a74aadbc6f37c8d97448a6f9416754c6db4c Author: Emmanuel Hocdet Date: Mon Sep 3 16:29:16 2018 +0200 BUG/MEDIUM: ECC cert should work with TLS < v1.2 and openssl >= 1.1.1 With openssl >= 1.1.1 and boringssl multi-cert is natively supported. ECDSA/RSA selection is done and work correctly with TLS >= v1.2. TLS < v1.2 have no TLSEXT_TYPE_signature_algorithms extension: ECC certificate can't be selected, and handshake fail if no RSA cert is present. Safe ECC certificate selection without client announcement can be very tricky (browser compatibilty). The safer approach is to select ECDSA certificate if no other certificate matches, like it is with openssl < 1.1.1: certificate selection is only done via the SNI. Thanks to Lukas Tribus for reporting this and analysing the problem. This patch should be backported to 1.8 (cherry picked from commit 9f9b0c6a7f34021ca8ce1e7e174f1fce33c5a703) Signed-off-by: Willy Tarreau commit 0a64551f8c03cba97a5896a9567f7246b941dea7 Author: Baptiste Assmann Date: Tue Sep 4 09:57:17 2018 +0200 BUG/MEDIUM: dns/server: fix incomatibility between SRV resolution and server state file Server state file has no indication that a server is currently managed by a DNS SRV resolution. And thus, both feature (DNS SRV resolution and server state), when used together, does not provide the expected behavior: a smooth experience... This patch introduce the "SRV record name" in the server state file and loads and applies it if found and wherever required. (cherry picked from commit 6d0f38f00dd7191f8502cfd0b6c96399370b786b) [This patch applies to HAProxy 1.8 branch only] Signed-off-by: Willy Tarreau commit db4d7d6f4e341dd60149cd1bf0310e6831299f32 Author: Olivier Houchard Date: Mon Aug 27 12:59:14 2018 +0200 BUG/MEDIUM: hlua: Don't call RESET_SAFE_LJMP if SET_SAFE_LJMP returns 0. If SET_SAFE_LJMP returns 0, the spinlock is already unlocked, and lua_atpanic is already set back to hlua_panic_safe, so there's no need to call RESET_SAFE_LJMP. This should be MFC'd into 1.8. (cherry picked from commit 9e643ea17257432c6079a8644401271a2a80619a) Signed-off-by: Willy Tarreau commit 5877e9b880fd0cebc8fe157474d590e97084c242 Author: Frédéric Lécaille Date: Wed Aug 29 13:46:24 2018 +0200 BUG/MAJOR: thread: lua: Wrong SSL context initialization. When calling ->prepare_srv() callback for SSL server which depends on global "nbthread" value, this latter was not already parsed, so equal to 1 default value. This lead to bad memory accesses. Thank you to Pieter (PiBa-NL) for having reported this issue and for having provided a very helpful reg testing file to reproduce this issue (reg-test/lua/b00002.*). Must be backported to 1.8. (cherry picked from commit 54f2bcf22bbf91090c4a1949fab76e9d7b8fd4ee) Signed-off-by: Willy Tarreau commit 77c28f10787ece6fae9ab4e8af5ca15a85d0cb06 Author: Olivier Houchard Date: Tue Aug 28 14:41:31 2018 +0200 BUG/MEDIUM: hlua: Make sure we drain the output buffer when done. In hlua_applet_tcp_fct(), drain the output buffer when the applet is done running, every time we're called. Overwise, there's a race condition, and the output buffer could be filled after the applet ran, and as it is never cleared, the stream interface will never be destroyed. This should be backported to 1.8 and 1.7. (cherry picked from commit 594c8c5015abb0d0c0a8d41fea573a2bcd63a351) [wt: s / co_data(si_oc(si)) / si_ob(si)->o] Signed-off-by: Willy Tarreau commit 5e772266a3682b723ee7522e74e4ef9eacec6218 Author: Patrick Hemmer Date: Wed Aug 22 10:02:00 2018 -0400 BUG/MEDIUM: lua: reset lua transaction between http requests Previously LUA code would maintain the transaction state between http requests, resulting in things like txn:get_priv() retrieving data from a previous request. This addresses the issue by ensuring the LUA state is reset between requests. Co-authored-by: Tim Düsterhus (cherry picked from commit e3faf02581b36c1c2d58d147bf053ff1257cc319) Signed-off-by: Willy Tarreau commit d9a130e1962c2a5352f33088c563f4248a102c48 Author: Willy Tarreau Date: Fri Aug 24 15:48:59 2018 +0200 BUG/MEDIUM: mux_pt: dereference the connection with care in mux_pt_wake() mux_pt_wake() calls data->wake() which can return -1 indicating that the connection was just destroyed. We need to check for this condition and immediately exit in this case otherwise we dereference a just freed connection. Note that this mainly happens on idle connections between two HTTP requests. It can have random implications between requests as it may lead a wrong connection's polling to be re-enabled or disabled for example, especially with threads. This patch must be backported to 1.8. (cherry picked from commit ad7f0ad1c3c9c541a4c315b24d4500405d1383ee) Signed-off-by: Willy Tarreau commit 5b58c92dc9357a87aa3fe94c8121f683feb9c80e Author: Frédéric Lécaille Date: Wed Jul 18 14:25:26 2018 +0200 BUG/MINOR: lua: Bad HTTP client request duration. HTTP LUA applet callback should not update the date on which the HTTP client requests arrive. This was done just after the LUA applet has completed its job. This patch simply removes the affected statement. The same fixe has been applied to TCP LUA applet callback. To reproduce this issue, as reported by Patrick Hemmer, implement an HTTP LUA applet which sleeps a bit before replying: core.register_service("foo", "http", function(applet) core.msleep(100) applet:set_status(200) applet:start_response() end) This had as a consequence to log %TR field with approximatively the same value as the LUA sleep time. Thank you to Patrick Hemmer for having reported this issue. Must be backported to 1.8, 1.7 and 1.6. (cherry picked from commit 83ed5d58d2c767d03ce97aef484863a6e1c37a94) Signed-off-by: Willy Tarreau commit f87ea7d2fdcfa3ccd5d605b3ce96642d28f20f6b Author: Willy Tarreau Date: Fri Aug 24 14:31:53 2018 +0200 BUG/MEDIUM: unix: provide a ->drain() function Right now conn_sock_drain() calls the protocol's ->drain() function if it exists, otherwise it simply tries to disable polling for receiving on the connection. This doesn't work well anymore since we've implemented the muxes in 1.8, and it has a side effect with keep-alive backend connections established over unix sockets. What happens is that if during the idle time after a request, a connection reports some data, si_idle_conn_null_cb() is called, which will call conn_sock_drain(). This one sees there's no drain() on unix sockets and will simply disable polling for data on the connection. But it doesn't do anything on the conn_stream. Thus while leaving the conn_fd_handler, the mux's polling is updated and recomputed based on the conn_stream's polling state, which is still enabled, and nothing changes, so we see the process use 100% CPU in this case because the FD remains active in the cache. There are several issues that need to be addressed here. The first and most important is that we cannot expect some protocols to simply stop reading data when asked to drain pending data. So this patch make the unix sockets rely on tcp_drain() since the functions are the same. This solution is appropriate for backporting, but a better one is desired for the long term. The second issue is that si_idle_conn_null_cb() shouldn't drain the connection but the conn_stream. At the moment we don't have any way to drain a conn_stream, though a flag on rcv_buf() will do it well. Until we support muxes on the server side it is not a problem so this part can be addressed later. This fix must be backported to 1.8. (cherry picked from commit fe5d2ac65fd58a8320e8dc725219c1bce5839592) Signed-off-by: Willy Tarreau commit d13cb1516cb5ae4cb8322ed630e1d4e1f584fd77 Author: Jens Bissinger Date: Thu Aug 23 14:11:27 2018 +0200 DOC: Fix spelling error in configuration doc Fix spelling error in logging section of configuration doc. (cherry picked from commit 15c64ff4fb9f1f64b31306ac53b38fc4d5fb1538) Signed-off-by: Willy Tarreau commit 0dbaa252df906cc9c1d0dc7a075c16e039ab1c5b Author: Willy Tarreau Date: Tue Aug 21 15:35:31 2018 +0200 BUG/MEDIUM: cli/threads: protect some server commands against concurrent operations The server-specific CLI commands "set weight", "set maxconn", "disable agent", "enable agent", "disable health", "enable health", "disable server" and "enable server" were not protected against concurrent accesses. Now they take the server lock around the sensitive part. This patch must be backported to 1.8. (cherry picked from commit 3bcc2699ba08dd3971ae7a56631994b2524d2acb) Signed-off-by: Willy Tarreau commit 3c42f13badd149c9c3152d7b2e653bde5da7c17a Author: Willy Tarreau Date: Tue Aug 21 14:50:44 2018 +0200 BUG/MEDIUM: cli/threads: protect all "proxy" commands against concurrent updates The proxy-related commands like "{enable|disable|shutdown} frontend", "{enable|disable} dynamic-cookie", "set dynamic-cookie-key" were not protected against concurrent accesses making their use dangerous with threads. This patch must be backported to 1.8. (cherry picked from commit a275a3710eaa365150fe89e2e7a8fbdce87bb30e) Signed-off-by: Willy Tarreau commit 947a3f71ad9733dde6645edb91b6becb3ee51e7c Author: Cyril Bonté Date: Fri Aug 17 23:51:02 2018 +0200 BUG/MEDIUM: lua: socket timeouts are not applied Sachin Shetty reported that socket timeouts set in LUA code have no effect. Indeed, connect timeout is never modified and is always set to its default, set to 5 seconds. Currently, this patch will apply the specified timeout value to the connect timeout. For the read and write timeouts, the issue is that the timeout is updated but the expiration dates were not updated. This patch should be backported up to the 1.6 branch. (cherry picked from commit 7bb634549794298fc701d33efd93c7289dcf9cb7) Signed-off-by: Willy Tarreau commit 54aecf18aeabe09bccf8db5e34b99bc36d468088 Author: Bertrand Jacquin Date: Tue Aug 14 00:56:13 2018 +0100 DOC: ssl: Use consistent naming for TLS protocols In most cases, "TLSv1.x" naming is used across and documentation, lazy people tend to grep too much and may not find what they are looking for. Fixing people is hard. (cherry picked from commit a25282bb399bfad8ed04b494b567fe97f0a58d65) Signed-off-by: Willy Tarreau commit 29a43c20faa06100ade61fc24a5ee3bddfa3021a Author: Lukas Tribus Date: Tue Aug 14 11:39:35 2018 +0200 DOC: dns: explain set server ... fqdn requires resolver Abhishek Gupta reported on discourse that set server [...] fqdn always fails. Further investigation showed that this requires the internal DNS resolver to be configured. Add this requirement to the docs. Must be backported to 1.8. (cherry picked from commit c5dd5a500a237780eb9ab6e7069949cb19b6ff7d) Signed-off-by: Willy Tarreau commit a1110e24e5be53ba5fe9ab82372c02a60da06cf9 Author: Emeric Brun Date: Tue Jul 17 09:47:07 2018 -0400 BUG/MINOR: map: fix map_regm with backref Due to a cascade of get_trash_chunk calls the sample is corrupted when we want to read it. The fix consist to use a temporary chunk to copy the sample value and use it. (cherry picked from commit 271022150d7961b9aa39dbfd88e0c6a4bc48c3ee) Signed-off-by: Willy Tarreau commit 399714287a04d6b453ba95e4a3904a7644827d0b Author: Emeric Brun Date: Thu Aug 16 15:14:12 2018 +0200 BUG/MEDIUM: ssl: loading dh param from certifile causes unpredictable error. If the dh parameter is not found, the openssl's error global stack was not correctly cleared causing unpredictable error during the following parsing (chain cert parsing for instance). This patch should be backported in 1.8 (and perhaps 1.7) (cherry picked from commit e1b4ed4352619f985d7d65f5d95a830ef5775c46) Signed-off-by: Willy Tarreau commit 9923082f94e0af83d03e030f4274d3db374b248c Author: Emeric Brun Date: Thu Aug 16 15:11:12 2018 +0200 BUG/MEDIUM: ssl: fix missing error loading a keytype cert from a bundle. If there was an issue loading a keytype's part of a bundle, the bundle was implicitly ignored without errors. This patch should be backported in 1.8 (and perhaps 1.7) (cherry picked from commit eb155b6ca6c1a8aaffa30285d453909b97979f5f) Signed-off-by: Willy Tarreau commit 8c2e3b35a951c41b80efe4c3368d1244bab2bea4 Author: Emeric Brun Date: Thu Aug 16 11:36:40 2018 +0200 BUG/MINOR: ssl: empty connections reported as errors. Empty connection is reported as handshake error even if dont-log-null is specified. This bug affect is a regression du to: BUILD: ssl: fix to build (again) with boringssl New openssl 1.1.1 defines OPENSSL_NO_HEARTBEATS as boring ssl so the test was replaced by OPENSSL_IS_BORINGSSL This fix should be backported on 1.8 (cherry picked from commit 77e8919fc6f382f3a7facdc814b8618b8987200f) Signed-off-by: Willy Tarreau commit f41ca2546e3c35cc389f45428341ec03dade314d Author: Willy Tarreau Date: Thu Aug 2 11:05:48 2018 +0200 BUG/MEDIUM: cli: make "show fd" thread-safe The "show fd" command was implemented as a debugging aid but it's not thread safe. Its features have grown, it can now dump some mux-specific parts and is being used in production to capture some useful debugging traces. But it will quickly crash the process when used during an H2 load test for example, especially when haproxy is built with the DEBUG_UAF option. It cannot afford not to be thread safe anymore. Let's make use of the new rendez-vous point using thread_isolate() / thread_release() to ensure that the data being dumped are not changing under us. The dump becomes slightly slower under load but now it's safe. This should be backported to 1.8 along with the rendez-vous point code once considered stable enough. (cherry picked from commit bf9fd650883b23604b7cd4aabf04fc0c4c8fe7c7) Signed-off-by: Willy Tarreau commit b505a8d719c208073959eff07f4af202ef49a8a1 Author: Willy Tarreau Date: Thu Aug 2 10:16:17 2018 +0200 MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release(). (cherry picked from commit 60b639ccbe919b86790267d7e45a39b75434acbe) [wt: this will be needed to fix the "show fd" command with threads] Signed-off-by: Willy Tarreau commit 7607ec0917c33ce511d46b791dfa5550451dd538 Author: Willy Tarreau Date: Tue Aug 7 10:07:15 2018 +0200 BUG/MEDIUM: threads: fix the no-thread case after the change to the sync point In commit 0c026f4 ("MINOR: threads: add more consistency between certain variables in no-thread case"), we ensured that we don't have all_threads_mask zeroed anymore. But one test was missed for the write() to the sync pipe. This results in a situation where when running single-threaded, once a server status changes, a wake-up message is written to the pipe and never consumed, showing a 100% CPU usage. No backport is needed. (cherry picked from commit ab657ce2511c4e19b0191fbe1c98cfd823a3c5d6) [wt: the offending patch was just backported as the previous one] Signed-off-by: Willy Tarreau commit ece550d98e1c10017fb91ecfa0d19ae9d2dc45da Author: Willy Tarreau Date: Wed Aug 1 19:12:20 2018 +0200 MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore. (cherry picked from commit 0c026f49e7348bce5b3c74be896ae208ae6e26a4) [wt: the thread code feels safer with this, especially with the small updates needed for the rdv point; missed one occurrence fixed by next patch] Signed-off-by: Willy Tarreau commit ad84851746243d85f9be59703e9bee0f5c5f8eba Author: Willy Tarreau Date: Wed Feb 14 14:16:28 2018 +0100 BUG/MEDIUM: threads: fix the double CAS implementation for ARMv7 Commit f61f0cb ("MINOR: threads: Introduce double-width CAS on x86_64 and arm.") introduced the double CAS. But the ARMv7 version is bogus, it uses the value of the pointers instead of dereferencing them. When lucky, it simply doesn't build due to impossible registers combinations. Otherwise it will immediately crash at run time when facing traffic. No backport is needed, this bug was introduced in 1.9-dev. (cherry picked from commit 41ccb194d1d14669e0592e5373ef5776f099e82a) [wt: backported only to keep safe code eventhough we don't use this function in 1.8] Signed-off-by: Willy Tarreau commit cd753064396f9563640fef940ce2a89e192042b1 Author: Olivier Houchard Date: Thu Dec 21 17:13:05 2017 +0100 MINOR: threads: Introduce double-width CAS on x86_64 and arm. Introduce double-width compare-and-swap on arches that support it, right now x86_64, arm, and aarch64. Also introduce functions to do memory barriers. (cherry picked from commit f61f0cb95ffbfe403219226d427cd292ca79965a) [wt: this is backported only to have the barriers for the new rdv point] Signed-off-by: Willy Tarreau commit d804e5e6b76bfd34576305ff33fe32aacb1fa5b7 Author: Thierry FOURNIER Date: Sat Jun 30 10:37:33 2018 +0200 BUG/MEDIUM: lua: possible CLOSE-WAIT state with '\n' headers The Lua parser doesn't takes in account end-of-headers containing only '\n'. It expects always '\r\n'. If a '\n' is processes the Lua parser considers it miss 1 byte, and wait indefinitely for new data. When the client reaches their timeout, it closes the connection. This close is not detected and the connection keep in CLOSE-WAIT state. I guess that this patch fix only a visible part of the problem. If the Lua HTTP parser wait for data, the timeout server or the connectio closed by the client may stop the applet. How reproduce the problem: HAProxy conf: global lua-load bug38.lua frontend frt timeout client 2s timeout server 2s mode http bind *:8080 http-request use-service lua.donothing Lua conf core.register_service("donothing", "http", function(applet) end) Client request: echo -ne 'GET / HTTP/1.1\n\n' | nc 127.0.0.1 8080 Look for CLOSE-WAIT in the connection with "netstat" or "ss". I use this script: while sleep 1; do ss | grep CLOSE-WAIT; done This patch must be backported in 1.6, 1.7 and 1.8 Workaround: enable the "hard-stop-after" directive, and perform periodic reload. (cherry picked from commit 70d318ccb760ee25f166a75d163f38545f074ff1) Signed-off-by: Willy Tarreau commit c1bfcd002f54d1d84a99282d13f875c2649f3d70 Author: Baptiste Assmann Date: Fri Jun 22 15:04:43 2018 +0200 MINOR: dns: new DNS options to allow/prevent IP address duplication By default, HAProxy's DNS resolution at runtime ensure that there is no IP address duplication in a backend (for servers being resolved by the same hostname). There are a few cases where people want, on purpose, to disable this feature. This patch introduces a couple of new server side options for this purpose: "resolve-opts allow-dup-ip" or "resolve-opts prevent-dup-ip". (cherry picked from commit 8e2d9430c0562ed74276d7f58e92706c384c0a36) [wt: this is backported to 1.8 upon request from Baptiste because it offers the option to revert to 1.7 behaviour, which some people depend on. The address deduplication used on 1.8 apparently is not suited to everyone] Signed-off-by: Willy Tarreau commit 7d395954136c45e1533f355068399fec5e606db1 Author: Baptiste Assmann Date: Fri Jun 22 13:03:50 2018 +0200 MINOR: dns: fix wrong score computation in dns_get_ip_from_response dns_get_ip_from_response() is used to compare the caller current IP to the IP available in the records returned by the DNS server. A scoring system is in place to get the best IP address available. That said, in the current implementation, there are a couple of issues: 1. a comment does not match what the code does 2. the code does not match what the commet says (score value is not incremented with '2') This patch fixes both issues. Backport status: 1.8 (cherry picked from commit 84221b4e9010810cf93b7ad7a31d825fa9fc26bf) [wt: Baptiste explicitly asked for this one to be backported to stable] Cc: Baptiste Signed-off-by: Willy Tarreau commit 5550143cd6de58c6e733e389c6946e3dd26e89c0 Author: Willy Tarreau Date: Tue Aug 7 10:44:58 2018 +0200 BUG/MEDIUM: queue: prevent a backup server from draining the proxy's connections When switching back from a backup to an active server, the backup server currently continues to drain the proxy's connections, which is a problem because it's not expected to be able to pick them. This patch ensures that a backup server will only pick backend connections if there is no active server and it is the selected backup server or all backup servers are supposed to be used. This issue seems to have existed forever, so this fix should be backported to all stable versions. (cherry picked from commit a8694654ba021bf1e0e560a98ab5e70dc44d212e) Signed-off-by: Willy Tarreau commit ef9b56022c656df34044103a317b7b890ced6628 Author: Willy Tarreau Date: Tue Aug 7 10:14:53 2018 +0200 BUG/MEDIUM: servers: check the queues once enabling a server Commit 64cc49c ("MAJOR: servers: propagate server status changes asynchronously.") heavily changed the way the server states are updated since they became asynchronous. During this change, some code was lost, which is used to shut down some sessions from a backup server and to pick pending connections from a proxy once a server is turned back from maintenance to ready state. The effect is that when temporarily disabling a server, connections stay in the backend's queue, and when re-enabling it, they are not picked and they expire in the backend's queue. Now they're properly picked again. This fix must be backported to 1.8. (cherry picked from commit 6a78e61694d69beb49c0e8486be9550f5e8b7d08) Signed-off-by: Willy Tarreau commit ada31afbc1e9095d494973cad91a4e507c4c1d9b Author: Willy Tarreau Date: Mon Jul 30 17:32:17 2018 +0200 [RELEASE] Released version 1.8.13 Released version 1.8.13 with the following main changes : - MINOR: systemd: consider exit status 143 as successful - BUG/MINOR: ssl: properly ref-count the tls_keys entries - MINOR: mux: add a "show_fd" function to dump debugging information for "show fd" - MINOR: h2: implement a basic "show_fd" function - BUG/MINOR: h2: remove accidental debug code introduced with show_fd function - MINOR: h2: keep a count of the number of conn_streams attached to the mux - MINOR: h2: add the mux and demux buffer lengths on "show fd" - BUG/MEDIUM: h2: don't accept new streams if conn_streams are still in excess - BUG/MEDIUM: h2: never leave pending data in the output buffer on close - BUG/MEDIUM: h2: make sure the last stream closes the connection after a timeout - BUG/MINOR: http: Set brackets for the unlikely macro at the right place - BUILD: Generate sha256 checksums in publish-release - MINOR: debug: Add check for CO_FL_WILL_UPDATE - MINOR: debug: Add checks for conn_stream flags - BUG/MEDIUM: threads: Fix the exit condition of the thread barrier - MINOR: h2: add the error code and the max/last stream IDs to "show fd" - BUG/MEDIUM: stream-int: don't immediately enable reading when the buffer was reportedly full - BUG/MEDIUM: stats: don't ask for more data as long as we're responding - BUG/MINOR: servers: Don't make "server" in a frontend fatal. - BUG/MEDIUM: threads/sync: use sched_yield when available - BUG/MEDIUM: h2: prevent orphaned streams from blocking a connection forever - BUG/MINOR: config: stick-table is not supported in defaults section - BUG/MINOR: threads: Handle nbthread == MAX_THREADS. - BUG/MEDIUM: threads: properly fix nbthreads == MAX_THREADS - MINOR: threads: move "nbthread" parsing to hathreads.c - BUG/MEDIUM: threads: unbreak "bind" referencing an incorrect thread number - MEDIUM: proxy_protocol: Convert IPs to v6 when protocols are mixed - SCRIPTS: git-show-backports: add missing quotes to "echo" commit 8dcf95775bd7b58e767adedaf8b5edb30cd302d2 Author: Willy Tarreau Date: Mon Jul 30 14:45:55 2018 +0200 SCRIPTS: git-show-backports: add missing quotes to "echo" Two places were not properly protected, resulting in '*' in commit subjects being interpreted! (cherry picked from commit 48d92ee3f4739e86d14486081fe0117446e91c2d) Signed-off-by: Willy Tarreau commit 9621edf0d9f072cd3cc41c83ac00c5c64a197da4 Author: Tim Duesterhus Date: Fri Jul 27 18:46:13 2018 +0200 MEDIUM: proxy_protocol: Convert IPs to v6 when protocols are mixed http-request set-src possibly creates a situation where src and dst are from different address families. Convert both addresses to IPv6 to avoid a PROXY UNKNOWN. This patch should be backported to haproxy 1.8. (cherry picked from commit 7fec02153712e5d823ec7a7593b7af0963111a6f) [wt: this is not exactly a bug fix but it addresses a situation not covered by the proxy protocol design which can easily arise when chaining proxies and which causes real trouble] Signed-off-by: Willy Tarreau commit 41b518a987bcc2b8e8cb9c17d097d301848d6b5f Author: Willy Tarreau Date: Fri Jul 27 18:07:41 2018 +0200 BUG/MEDIUM: threads: unbreak "bind" referencing an incorrect thread number The "process" directive on "bind" lines supports process references and thread references. No check is performed on the thread number validity, so that if a listener is only bound to non-existent threads, the traffic will never be processed. It easily happens when setting one bind line per thread with an incorrect (or reduced) thread count. No warning appears and some random connections are never served. It also happens when setting thread references with threads support disabled at build time. This patch makes use of the all_threads_mask variable to detect if some referenced threads don't exist, to emit a warning and fix this. This patch needs to be backported to 1.8, just like the previous one which it depends on (MINOR: threads: move "nbthread" parsing to hathreads.c). (cherry picked from commit c477b6fcc91c382f7edc2caca692c4059487cd6f) Signed-off-by: Willy Tarreau commit cbc8840268af7c6d67c742d31f1c866e9b0f4009 Author: Willy Tarreau Date: Mon Jul 30 10:34:35 2018 +0200 MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c. (cherry picked from commit 0ccd32285fbe8a1a394f7b8fb9f7e6282ec9a32d) Signed-off-by: Willy Tarreau commit 5b38c47413f001f775f0db7fa85ce5063c6dfa97 Author: Willy Tarreau Date: Mon Jul 30 10:44:51 2018 +0200 BUG/MEDIUM: threads: properly fix nbthreads == MAX_THREADS While moving Olivier's patch for nbthread==MAX_THREADS in commit 3e12304 ("BUG/MINOR: threads: Handle nbthread == MAX_THREADS.") to hathreads.c, I missed one place resulting in the computed thread mask being used as the thread count, which is worse than the initial bug. Let's fix it properly this time. This fix must be backported to 1.8 just like the other one. (cherry picked from commit 5e954e1f27729f85eabc4fe4159a3460774f3abc) Signed-off-by: Willy Tarreau commit da3af98da2065262ded73f40d4de80e0aefeba91 Author: Olivier Houchard Date: Fri Jul 27 17:06:59 2018 +0200 BUG/MINOR: threads: Handle nbthread == MAX_THREADS. If nbthread is MAX_THREADS, the shift operation needed to compute all_threads_mask fails in thread_sync_init(). Instead pass a number of threads to this function and let it compute the mask without overflowing. This should be backported to 1.8. (cherry picked from commit 3e12304ae004284e3186b489f398908e59a8574d) Signed-off-by: Willy Tarreau commit f933eb7229b5298a3f2ec33886cb5e5354fa9ba5 Author: Willy Tarreau Date: Fri Jul 27 10:26:22 2018 +0200 BUG/MINOR: config: stick-table is not supported in defaults section Thierry discovered that the following config crashes haproxy while parsing the config (it's probably the smallest crasher) : defaults stick-table type ip size 1M And indeed it does because it looks for the current proxy's name which it does not have as it's the default one. This affects all versions since 1.6. This fix must be backported to all versions back to 1.6. (cherry picked from commit c786768dbaa178a2b9ad82eebf004eb3e541201d) Signed-off-by: Willy Tarreau commit 6fb0ed7544c18e4a91469ab8d70fe95bfeb65222 Author: Willy Tarreau Date: Fri Jul 27 09:55:14 2018 +0200 BUG/MEDIUM: h2: prevent orphaned streams from blocking a connection forever Some h2 connections remaining in CLOSE_WAIT state forever have been reported for a while. Thanks to detailed captures provided by Milan Petruzelka, the sequence where this happens became clearer : 1) multiple streams compete for the mux and are queued in the send_list 2) at this point the mux has to emit a GOAWAY for any reason (for example because it received a bad message) 3) the streams are woken up, notified about the error 4) h2_detach() is called for each of them 5) the CS they are detached from the H2S 6) since the streams are marked as blocked for some room, they are orphaned and nothing more is done on them. 7) at this point, any activity on the connection goes through h2_wake() which sees the conneciton in ERROR2 state, tries again to release the streams, cannot, and stops polling (thus even connection errors cannot be detected anymore). => from this point, no more events can be received on the connection, and the streams remain orphaned forever. This patch makes sure that we never return without doing anything once an error was met. It has to act both on the h2_detach() side (for h2 streams being detached after the error was emitted) and on the h2_wake() side (for errors reported after h2s have already been orphaned). Many thanks to Milan Petruzelka and Janusz Dziemidowicz for their awesome work on this issue, collecting traces and testing patches, and to Olivier Doucet for extra testing and confirming the fix. This fix must be backported to 1.8. (cherry picked from commit a2b5181e7a29ac50645b89909c73ca031c7b1d8b) Signed-off-by: Willy Tarreau commit 3aa7e5a7c3112be2906f41f49ffa57b0c071b600 Author: Willy Tarreau Date: Fri Jul 27 07:47:24 2018 +0200 BUG/MEDIUM: threads/sync: use sched_yield when available There is a corner case with the sync point which can significantly degrade performance. The reason is that it forces all threads to busy spin there, and that if there are less CPUs available than threads, this busy activity from some threads will force others to wait longer in epoll() or to simply be scheduled out while doing something else, and will increase the time needed to reach the sync point. Given that the sync point is not expected to be stressed *that* much, better call sched_yield() while waiting there to release the CPU and offer it to waiting threads. On a simple test with 4 threads bound to two cores using "maxconn 1" on the server line, the performance was erratic before the recent scheduler changes (between 40 and 200 conn/s with hundreds of ms response time), and it jumped to 7200 with 12ms response time with this fix applied. It should be backported to 1.8 since 1.8 is affected as well. (cherry picked from commit 3ea2490b4885250d932e537d2cbe0394a228f8f3) Signed-off-by: Willy Tarreau commit f8a0ef4e31e79682eda1c55bc7d259d59c497df0 Author: Olivier Houchard Date: Tue Jul 24 16:48:59 2018 +0200 BUG/MINOR: servers: Don't make "server" in a frontend fatal. When parsing the configuration, if "server", "default-server" or "server-template" are found in a frontend, we first warn that it will be ignored, only to be considered a fatal error later. Be true to our word, and just ignore it. This should be backported to 1.8 and 1.7. (cherry picked from commit 306e653331f107902f6211f03e09779cfe0171d0) Signed-off-by: Willy Tarreau commit 604ce198e0b17dda5ef34bfb818bbe19875b6b73 Author: Willy Tarreau Date: Tue Jul 24 17:05:54 2018 +0200 BUG/MEDIUM: stats: don't ask for more data as long as we're responding The stats applet is still a bit hackish. It uses the HTTP txn to parse the POST contents. Due to this it pretends not having parsed the request from the buffer so that the HTTP parser continues to work fine on these data. This comes with a side effect : the request lies pending in the channel's buffer, and because of this, stream_int_update_applet() always wakes the applet up. It's very visible when retrieving a large stats page over a slow link as haproxy eats 100% of the CPU waiting for the data to leave. While the proper long term solution definitely is to consume these data and parse the body from the applet, changing this is not suitable for a fix. What this patch does instead is to disable request polling as long as there are pending data in the response buffer. Given that for almost all cases, the applet remains busy sending data, this is at least enough to ensure that we don't wake up for the pending request data while we're waiting for the client to receive these data. Now a 5k backend stats page is dumped at 1% CPU over a 10 Mbps link instead of 100%, using 1500 epoll_wait() calls instead of 80000. Note that the previous fix (BUG/MEDIUM: stream-int: don't immediately enable reading when the buffer was reportedly full) is necessary for the effects of the fix to be noticed since both bugs have the exact same effect. This fix must be backported at least as far as 1.5. (cherry picked from commit 055ba4f505e5117f59d3b843afc4c737dd2e21d5) Signed-off-by: Willy Tarreau commit 2683b3af2dcecd703df9e9460a3cf161980d8fd1 Author: Willy Tarreau Date: Tue Jul 24 16:56:34 2018 +0200 BUG/MEDIUM: stream-int: don't immediately enable reading when the buffer was reportedly full There is a long-time issue which affects some applets, at least the stats applet. If a large stats page is read over a slow link, regularly the channel's buffer contains too many response data to allow another round of ci_putblk() to copy a new message. In this case the applet calls si_applet_cant_put() to mention that it failed to emit data into the channel's buffer, and wants to be called only once some room is made. The problem is that stream_int_update(), which is called from process_stream(), will clear this flag whenever it sees there's some spare room in the channel's buffer. It causes the applet to be woken again immediately. This is very visible when reading a large stats page over a slow link, because in this case haproxy will run at 100% CPU and strace shows mostly epoll_wait(0). It is very likely that some other applets like CLI, Lua, peers or SPOE have also been affected but that the effect were less noticeable because it was mixed with traffic. Ideally stream_int_update() should not touch these flags, but changing this would require a very careful auditing of all users. Instead here what we do is that we respect the flag if the channel still has output data. This way the flag will automatically disappear once the buffer is empty, and the applet function will be called only when input data remains, if at all. This patch alone is not enough to observe the behaviour change on the stats page because another bug takes over, addressed by next patch (BUG/MEDIUM: stats: don't ask for more data as long as we're responding). When both are applied, dumping stats for 5k backends over a 10 Mbps link take 1% CPU instead of 100%, with 1.5k epoll_wait() calls instead of 80k. This fix should be backported at least as far as 1.5. (cherry picked from commit 171d5f203a7091f271fee2c513a1e00d5639ee38) [wt: s/co_data(ic)/ic->buf->o/] Signed-off-by: Willy Tarreau commit 12a4b5c69d3eb0cc01bf92df215c7cc8d70ee8bc Author: Willy Tarreau Date: Tue Jul 24 14:12:42 2018 +0200 MINOR: h2: add the error code and the max/last stream IDs to "show fd" This is intented to help debugging H2 in field. (cherry picked from commit 616ac81dec5759990ee600047d8ad900f6eba6e8) [wt: adapted context a little bit] Signed-off-by: Willy Tarreau commit 5e100b4ae521ac7cf1539de33d8a703c16c141a5 Author: Christopher Faulet Date: Fri Jul 20 09:31:53 2018 +0200 BUG/MEDIUM: threads: Fix the exit condition of the thread barrier In thread_sync_barrier, we exit when all threads have set their own bit in the barrier mask. It is done by comparing it to all_threads_mask. But we must not use a simple equality to do so, becaue all_threads_mask may change. Since commit ba86c6c25 ("MINOR: threads: Be sure to remove threads from all_threads_mask on exit"), when a thread exit, its bit is removed from all_threads_mask. Instead, we must use a bitwise AND to test is all bits of all_threads_mask are set. This also requires that all_threads_mask is set to volatile if we want to catch changes. This patch must be backported in 1.8. (cherry picked from commit ddb6c16576e92b75db699c265645431cc916d4b8) Signed-off-by: William Lallemand commit 402630e4fcda9ab5a4e333eace36a3c585d07198 Author: Christopher Faulet Date: Thu Mar 1 08:55:21 2018 +0100 MINOR: debug: Add checks for conn_stream flags This may be carefully backported to 1.8 (a few flags don't exist there). (cherry picked from commit 5f8ef13d5d090b0e52c71186192d5ed3632c2f00) [wla: removed CS_FL_WAIT_FOR_HS and CS_FL_REOS] Signed-off-by: William Lallemand commit 18d01553b0fa0b1a23f70572b8731dd4c1be7f12 Author: Christopher Faulet Date: Tue Feb 27 15:41:10 2018 +0100 MINOR: debug: Add check for CO_FL_WILL_UPDATE This could be backported to 1.8. (cherry picked from commit aff9328739efaabf26a39987722ce036058318d2) Signed-off-by: William Lallemand commit 1d933e509855134da61f19faf8d2358c37b7a3d0 Author: Tim Duesterhus Date: Thu Jul 19 23:57:56 2018 +0200 BUILD: Generate sha256 checksums in publish-release Currently only md5 signatures are generated. While md5 still is not broken with regard to preimage attacks, sha256 clearly is the current secure solution. This patch should be backported to all supported branches. (cherry picked from commit 3ce3811a9c510bc5c33047be058c1e3601f9e036) Signed-off-by: William Lallemand commit f0025e85ee2b5087bb9dd4e923e9a5bc7fffffa3 Author: Christopher Faulet Date: Fri Jul 20 09:54:26 2018 +0200 BUG/MINOR: http: Set brackets for the unlikely macro at the right place When test on the header "Early-Data" is made, the unlikely macro must encompass the condition. This patch must be backported in 1.8. (cherry picked from commit 005e79e5dd7e6bb915ebba793e030c1fcc2dc4f6) Signed-off-by: William Lallemand commit 99c3e049c0888f3c7382a583a20db273d0c768c5 Author: Willy Tarreau Date: Wed Jun 13 14:24:56 2018 +0200 BUG/MEDIUM: h2: make sure the last stream closes the connection after a timeout If a timeout strikes on the connection side with some active streams, there is a corner case which can sometimes cause the following sequence to happen : - There are active streams but there are data in the mux buffer (eg: a client suddenly disconnected during a download with pending requests). The timeout is active. - The timeout strikes, h2_timeout_task() is called, kills the task and doesn't close the connection since there are streams left ; The connection is marked in H2_CS_ERROR ; - the streams are woken up and closed ; - when the last stream closes, calling h2_detach(), it sees the tree list is empty, but there is no condition allowing the connection to be closed (mbuf->o > 0), thus it does nothing ; - since the task is dead, there's no more hope to clear this situation later For now we can take care of this by adding a test for the presence of H2_CS_ERROR and !task, implying the timeout task triggered already and will not be able to handle this again. Over the long term it seems like a more reliable test on should be made, so that it is possible to know whether or not someone is still able to close this connection. A big thanks to Janusz Dziemidowicz and Milan Petruzelka for providing many details helping in figuring this bug. (cherry picked from commit 42d55b9b6a0426252d4795a88cc85c9859db79f5) Signed-off-by: Willy Tarreau commit 718680651d64975b545df2af8fa29d53b99a5746 Author: Willy Tarreau Date: Thu Jul 19 10:58:28 2018 +0200 BUG/MEDIUM: h2: never leave pending data in the output buffer on close We currently don't process trailers on H2, but this has an impact : on chunked HTTP/1 responses, we decide to emit the ES bit once we see the 0CRLF. From this point the stream switches to the CLOSED state, which aborts processing of the remaining bytes. Thus the extra CRLF which ends trailers is not processed and remains in the buffer. This prevents the stream from being notified about end of transmission, which in turn keeps the mux busy and prevents the connection from quitting. The case of the trailers is not the root cause of this issue, though it is what triggers it. The root cause is that upon error and/or close, once we know we're not going to process any more data, we must absolutely flush any remaining bytes from the output buffer, otherwise there is no way the stream can quit. This is what this patch does. It looks very likely related to the issues reported and debugged by Janusz Dziemidowicz and Milan Petruzelka. One way to reproduce it is to chain two proxies with the last one emitting chunked data (typically using the stats page) : global stats socket /tmp/sock1 mode 666 level admin stats timeout 1h tune.ssl.default-dh-param 1024 tune.bufsize 16384 defaults mode http timeout connect 4s timeout client 10s timeout server 20s listen px1 bind :4443 ssl crt rsa+dh2048.pem npn h2 alpn h2 server s1 127.0.0.1:4445 listen px2 bind :4444 ssl crt rsa+dh2048.pem npn h2 alpn h2 bind :4445 stats uri / Then use curl to fetch the stats through px1 : curl --http2 -k "https://127.0.0.1:4443/" When curl is sent to the first one, "show sess" issued to the CLI will show a remaining session during the client timeout. When curl is aimed at port 4444 (px2), there is no such remaining session. This fix needs to be backported to 1.8. (cherry picked from commit 00610960a196f01b6e6b549e29eb1cf2426d253a) Signed-off-by: Willy Tarreau commit 431b94f96a26bdd3ac2eabee93036f307b85eea4 Author: Willy Tarreau Date: Thu Jul 19 10:11:38 2018 +0200 BUG/MEDIUM: h2: don't accept new streams if conn_streams are still in excess The streams bookkeeping made in H2 is used for protocol compliance only but it doesn't consider the number of conn_streams still attached to the mux. It causes an issue when http-request set-nice rules are applied on H2 requests processed on a saturated machine. Indeed, in this case, the requests are accepted and assigned a default nice value of zero. When they are processed, their nice value changes to a higher one (say 1024). The response is sent through the H2 mux, which detects the end of stream and decrements the protocol-level stream count (h2c->nb_streams). The client may then send a new request. But the conn_stream is still attached and will require a new call to process_stream() to finish, which is made through the scheduler. Given that the machine is saturated, it is assumed that many tasks are present in the scheduler. Thus the closing tasks holding a higher nice value will pass after the new stream creations. If the client is fast enough with a low latency link, it may add a lot of new stream creations before the stream terminations have a chance to disappear due to their high nice value, resulting in a huge amount of memory being used. The solution consists in letting a mux always monitor its conn_streams and refrain from creating new ones when it is full. Here the H2 mux checks the nb_cs counter and sets a new blocked flag (H2_CF_DEM_TOOMANY) if the limit was reached, so that the frame parser requests a pause in the new stream creation, leaving some time for the pending conn_streams to vanish. Several experiments were made using varying thresholds to see if overbooking would provide any benefit here but it turned out not to be the case, so the conn_stream limit remains set to the exact streams limit. Interestingly various performance measurements showed that the code tends to be slightly faster now than without the limit, probably due to the smoother memory usage. This commit requires previous patch ("MINOR: h2: keep a count of the number of conn_streams attached to the mux"). It needs to be backported to 1.8. (cherry picked from commit f210191dcdf32a2cb263c5bd22b7fc98698ce59a) Signed-off-by: Willy Tarreau commit a8402295abec2e9413d309bea0f5a83d55b21366 Author: Willy Tarreau Date: Thu Jul 19 10:54:43 2018 +0200 MINOR: h2: add the mux and demux buffer lengths on "show fd" It is convenient during debugging sessions to know if the mux and demux buffers are empty/full/other. Let's report this on "show fd" output. (cherry picked from commit c65edac804573799fff0aaaae3ace5febb15c39e) Signed-off-by: Willy Tarreau commit bb8c3d514a51dcc872c5e326a923344aba7b750e Author: Willy Tarreau Date: Thu Jul 19 09:04:05 2018 +0200 MINOR: h2: keep a count of the number of conn_streams attached to the mux The h2 mux only knows about the number of H2 streams which are not in a CLOSED state. This is used for protocol compliance. But it doesn't hold the number of really attached streams. It is a problem because depending on scheduling, it is possible that more streams are attached to the mux than the ones seen at the protocol level, due to some streams taking some time to be detached. Let's add this count based on the conn_streams. Note: this patch is part of a series of fixes which will have to be backported to 1.8. (cherry picked from commit 7ac60e836ad7c0dae3412c289b34c5affbec7d0a) Signed-off-by: Willy Tarreau commit 0bdd3fdf91a2e55c17aad2004057f728d82055c1 Author: Willy Tarreau Date: Fri Mar 30 17:41:19 2018 +0200 BUG/MINOR: h2: remove accidental debug code introduced with show_fd function Commit e3f36cd ("MINOR: h2: implement a basic "show_fd" function") accidently brought one surrounding debugging part that was in the same context. No backport needed. (cherry picked from commit 1a1dd6066fb3b3ea88969d277c6f5030b5b380ab) Signed-off-by: Willy Tarreau commit 7c3eee33f92999b42a8bc270bde65bcc57cc4d64 Author: Willy Tarreau Date: Fri Mar 30 14:43:13 2018 +0200 MINOR: h2: implement a basic "show_fd" function The purpose here is to dump some information regarding an H2 connection, and a few statistics about its streams. The output looks like this : 35 : st=0x55(R:PrA W:PrA) ev=0x00(heopi) [lc] cache=0 owner=0x7ff49ee15e80 iocb=0x588a61(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00201366 fe=decrypt mux=H2 mux_ctx=0x7ff49ee16f30 st0=2 flg=0x00000002 fctl_cnt=0 send_cnt=33 tree_cnt=33 orph_cnt=0 - st0 is the connection's state (FRAME_H here) - flg is the connection's flags (MUX_MFULL here) - fctl_cnt is the number of streams in the fctl_list - send_cnt is the number of streams in the send_list - tree_cnt is the number of streams in the streams_by_id tree - orph_cnt is the number of orphaned streams (cs==0) in the tree (cherry picked from commit e3f36cd479738cf80c2d83575722bea7ee0f8e0b) Signed-off-by: Willy Tarreau commit 9980c528077936a2dda2f445a082fbd7ad4987ee Author: Willy Tarreau Date: Fri Mar 30 14:41:19 2018 +0200 MINOR: mux: add a "show_fd" function to dump debugging information for "show fd" This function will be called from the CLI's "show fd" command to append some extra mux-specific information that only the mux handler can decode. This is supposed to help collect various hints about what is happening when facing certain anomalies. (cherry picked from commit b011d8f4c4ae87f4874b636866bfc69fa44a0e49) Signed-off-by: Willy Tarreau commit 37b815c49429a0be900bc86543a5ff3901d305c5 Author: Willy Tarreau Date: Tue Jul 17 10:05:32 2018 +0200 BUG/MINOR: ssl: properly ref-count the tls_keys entries Commit 200b0fa ("MEDIUM: Add support for updating TLS ticket keys via socket") introduced support for updating TLS ticket keys from the CLI, but missed a small corner case : if multiple bind lines reference the same tls_keys file, the same reference is used (as expected), but during the clean shutdown, it will lead to a double free when destroying the bind_conf contexts since none of the lines knows if others still use it. The impact is very low however, mostly a core and/or a message in the system's log upon old process termination. Let's introduce some basic refcounting to prevent this from happening, so that only the last bind_conf frees it. Thanks to Janusz Dziemidowicz and Thierry Fournier for both reporting the same issue with an easy reproducer. This fix needs to be backported from 1.6 to 1.8. (cherry picked from commit 17b4aa1adc88987f411ae007b7865c59cdf37c1b) Signed-off-by: Willy Tarreau commit fdc6c62dbebf4b646b4f80c383e3b00f34b0440f Author: Vincent Bernat Date: Fri Jun 22 20:57:03 2018 +0200 MINOR: systemd: consider exit status 143 as successful The master process will exit with the status of the last worker. When the worker is killed with SIGTERM, it is expected to get 143 as an exit status. Therefore, we consider this exit status as normal from a systemd point of view. If it happens when not stopping, the systemd unit is configured to always restart, so it has no adverse effect. This has mostly a cosmetic effect. Without the patch, stopping HAProxy leads to the following status: ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2018-06-22 20:35:42 CEST; 8min ago Docs: man:haproxy(1) file:/usr/share/doc/haproxy/configuration.txt.gz Process: 32715 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143) Process: 32714 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS) Main PID: 32715 (code=exited, status=143) After the patch: ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:haproxy(1) file:/usr/share/doc/haproxy/configuration.txt.gz (cherry picked from commit 3b479bd5f5f50ce91cabed32bb26556313552d23) Signed-off-by: William Lallemand commit 8a200c71bd0848752b71a1aed5727563962b3a1a Author: William Lallemand Date: Wed Jun 27 16:27:32 2018 +0200 [RELEASE] Released version 1.8.12 Released version 1.8.12 with the following main changes : - BUG/MAJOR: stick_table: Complete incomplete SEGV fix - MINOR: stick-tables: make stktable_release() do nothing on NULL commit 8488b7ce261e6b7fe2435b36afe2a59edd213c82 Author: Willy Tarreau Date: Wed Jun 27 06:25:57 2018 +0200 MINOR: stick-tables: make stktable_release() do nothing on NULL stktable_release() has been involved in two recent crashes by being used without enough care. Just like any free() function this one is often called on an exit path with a possibly unsafe argument. Given that there is another case (smp_fetch_sc_trackers()) which theorically could call it with an unchecked NULL, though it cannot happen since the function doesn't support being called with src_* hence cannot make use of tmpstkctr, let's rather move the check into the function itself to make it safer for the long term. This patch could be backported to 1.8 as a strengthening measure. (cherry picked from commit 43e903553edc94bb9b33e965f37d8d218f7d1482) Signed-off-by: William Lallemand commit c28c2bfba25f5420b778cfcc703c6f27a3e5ec7d Author: Tim Duesterhus Date: Tue Jun 26 15:57:29 2018 +0200 BUG/MAJOR: stick_table: Complete incomplete SEGV fix This commit completes the incomplete segmentation fault fix in commit ac1f3ed64b58bd178865c6f2cc8f6f306d9e1e15. Likewise it must be backported to haproxy 1.8. (cherry picked from commit 65189c17c694b0b44e0d324d63c055f5329e61c9) Signed-off-by: William Lallemand commit 1d6ef58dda87728b5e8d49c752e9aff74d22be4d Author: William Lallemand Date: Tue Jun 26 14:33:12 2018 +0200 [RELEASE] Released version 1.8.11 Released version 1.8.11 with the following main changes : - BUG/MAJOR: Stick-tables crash with segfault when the key is not in the stick-table - BUG/BUILD: threads: unbreak build without threads commit 6155f15d8aeb13c4eb014ff54d6a08828102e8c5 Author: William Lallemand Date: Sun Jun 24 09:37:03 2018 +0200 BUG/BUILD: threads: unbreak build without threads The build without threads was once again broken. This issue was introduced in commit ba86c6c ("MINOR: threads: Be sure to remove threads from all_threads_mask on exit"). This is exactly the same problem as last time it happened, because of all_threads_mask not being defined with USE_THREAD= This must be backported in 1.8 (cherry picked from commit 091d827e096ea97b95af99d3366bc0115d0a24a8) Signed-off-by: William Lallemand commit de9d4c677b9b4b355d5cb850ec6e5456dd84b700 Author: Thierry FOURNIER Date: Mon Jun 25 22:35:20 2018 +0200 BUG/MAJOR: Stick-tables crash with segfault when the key is not in the stick-table When a lookup is done on a key not present in the stick-table the "st" pointer is NULL and it is used to return the converter result, but it is used untested with stktable_release(). This regression was introduced in 1.8.10 here: BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters commit d7bd88009d88dd413e01bc0baa90d6662a3d7718 Author: Daniel Corbett Date: Sun May 27 09:47:12 2018 -0400 Minimal conf for reproducong the problem: frontend test mode http stick-table type ip size 1m expire 1h store gpc0 bind *:8080 http-request redirect location /a if { src,in_table(test) } The segfault is triggered using: curl -i http://127.0.0.1:8080/ This patch must be backported in 1.8 (cherry picked from commit ac1f3ed64b58bd178865c6f2cc8f6f306d9e1e15) Signed-off-by: William Lallemand commit ec17d7a98f30326918219ba876fcfc56f6ad6823 Author: William Lallemand Date: Fri Jun 22 15:58:22 2018 +0200 [RELEASE] Released version 1.8.10 Released version 1.8.10 with the following main changes : - BUG/MINOR: lua: Socket.send threw runtime error: 'close' needs 1 arguments. - BUG/MEDIUM: spoe: Flags are not encoded in network order - BUG/MEDIUM: contrib/mod_defender: Use network order to encode/decode flags - BUG/MEDIUM: contrib/modsecurity: Use network order to encode/decode flags - BUG/MINOR: ssl/lua: prevent lua from affecting automatic maxconn computation - BUG/MEDIUM: cache: don't cache when an Authorization header is present - BUG/MEDIUM: dns: Delay the attempt to run a DNS resolution on check failure. - BUG/BUILD: threads: unbreak build without threads - BUG/BUILD: fd: fix typo causing a warning when threads are disabled - BUG/MEDIUM: fd: Only check update_mask against all_threads_mask. - BUG/MEDIUM: servers: Add srv_addr default placeholder to the state file - BUG/MEDIUM: lua/socket: Length required read doesn't work - BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters - BUG/MEDIUM: spoe: Return an error when the wrong ACK is received in sync mode - MINOR: task/notification: Is notifications registered ? - BUG/MEDIUM: lua/socket: wrong scheduling for sockets - BUG/MAJOR: lua: Dead lock with sockets - BUG/MEDIUM: lua/socket: Notification error - BUG/MEDIUM: lua/socket: Sheduling error on write: may dead-lock - BUG/MEDIUM: lua/socket: Buffer error, may segfault - MAJOR: spoe: upgrade the SPOP version to 2.0 and remove the support for 1.0 - BUG/MINOR: contrib/spoa_example: Don't reset the status code during disconnect - BUG/MINOR: contrib/mod_defender: Don't reset the status code during disconnect - BUG/MINOR: contrib/modsecurity: Don't reset the status code during disconnect - BUG/MINOR: contrib/mod_defender: update pointer on the end of the frame - BUG/MINOR: contrib/modsecurity: update pointer on the end of the frame - DOC: SPOE.txt: fix a typo - DOC: contrib/modsecurity: few typo fixes - BUG/MINOR: unix: Make sure we can transfer abns sockets on seamless reload. - BUG/MEDIUM: threads: handle signal queue only in thread 0 - BUG/MINOR: don't ignore SIG{BUS,FPE,ILL,SEGV} during signal processing - BUG/MINOR: signals: ha_sigmask macro for multithreading - MINOR: lua: Increase debug information - BUG/MAJOR: map: fix a segfault when using http-request set-map - BUG/MINOR: lua: Segfaults with wrong usage of types. - BUG/MAJOR: ssl: Random crash with cipherlist capture - BUG/MAJOR: ssl: OpenSSL context is stored in non-reserved memory slot - BUG/MEDIUM: fd: Don't modify the update_mask in fd_dodelete(). - BUG/MEDIUM: threads: Use the sync point to check active jobs and exit - MINOR: threads: Be sure to remove threads from all_threads_mask on exit commit 7ee1c53acac2d5dd311e391e02c296a5ed5ca047 Author: Christopher Faulet Date: Thu Jun 21 09:57:39 2018 +0200 MINOR: threads: Be sure to remove threads from all_threads_mask on exit When HAProxy is started with several threads, Each running thread holds a bit in the bitfiled all_threads_mask. This bitfield is used here and there to check which threads are registered to take part in a specific processing. So when a thread exits, it seems normal to remove it from all_threads_mask. No direct impact could be identified with this right now but it would be better to backport it to 1.8 as a preventive measure to avoid complex situations like the one in previous bug. (cherry picked from commit ba86c6c25bf252e44589ae2b4d51a67c4f47d244) Signed-off-by: William Lallemand commit 1373ad5a528a3bf84d3ccbf73622d8355ad7c118 Author: Christopher Faulet Date: Wed Jun 20 16:22:03 2018 +0200 BUG/MEDIUM: threads: Use the sync point to check active jobs and exit When HAProxy is shutting down, it exits the polling loop when there is no jobs anymore (jobs == 0). When there is no thread, it works pretty well, but when HAProxy is started with several threads, a thread can decide to exit because jobs variable reached 0 while another one is processing a task (e.g. a health-check). At this stage, the running thread could decide to request a synchronization. But because at least one of them has already gone, the others will wait infinitly in the sync point and the process will never die. To fix the bug, when the first thread (and only this one) detects there is no active jobs anymore, it requests a synchronization. And in the sync point, all threads will check if jobs variable reached 0 to exit the polling loop. This patch must be backported in 1.8. (cherry picked from commit d8fd2af8826c2d33b23c4ef64b88ecd57910e330) Signed-off-by: William Lallemand commit 8b8b597117a799324c27e431d5820b847ad6a41c Author: Olivier Houchard Date: Tue Jun 19 19:18:43 2018 +0200 BUG/MEDIUM: fd: Don't modify the update_mask in fd_dodelete(). Only the pollers should remove bits in the update_mask. Removing it will mean if the fd is currently in the global update list, it will never be removed, and while it's mostly harmless in 1.9, in 1.8, only update_mask is checked to know if the fd is already in the list or not, so we can end up trying to add a fd that is already in the list, and corrupt it, which means some fd may not be added to the poller. This should be backported to 1.8. (cherry picked from commit d0e60d852a2896d1811f10cbac1c7145f0c2159f) Signed-off-by: William Lallemand commit 30b2448182dcd4eebe26cfa3a1cae942ac729134 Author: Thierry FOURNIER Date: Sun Jun 17 21:37:05 2018 +0200 BUG/MAJOR: ssl: OpenSSL context is stored in non-reserved memory slot We never saw unexplicated crash with SSL, so I suppose that we are luck, or the slot 0 is always reserved. Anyway the usage of the macro SSL_get_app_data() and SSL_set_app_data() seem wrong. This patch change the deprecated functions SSL_get_app_data() and SSL_set_app_data() by the new functions SSL_get_ex_data() and SSL_set_ex_data(), and it reserves the slot in the SSL memory space. For information, this is the two declaration which seems wrong or incomplete in the OpenSSL ssl.h file. We can see the usage of the slot 0 whoch is hardcoded, but never reserved. #define SSL_set_app_data(s,arg) (SSL_set_ex_data(s,0,(char *)arg)) #define SSL_get_app_data(s) (SSL_get_ex_data(s,0)) This patch must be backported at least in 1.8, maybe in other versions. (cherry picked from commit 28962c994131d0fa15e2a9f948492eefc68a0d92) Signed-off-by: William Lallemand commit ade2721ed5f254cc3cb5b4d43abaa2d1516378c9 Author: Thierry FOURNIER Date: Sun Jun 17 21:33:01 2018 +0200 BUG/MAJOR: ssl: Random crash with cipherlist capture The cipher list capture struct is stored in the SSL memory space, but the slot is reserved in the SSL_CTX memory space. This causes ramdom crashes. This patch should be backported to 1.8 (cherry picked from commit 16ff050478ab94d5147a4603fbd942a7f931c023) Signed-off-by: William Lallemand commit a301a102a6969e69383dae3883422405e4b30f9e Author: Frédéric Lécaille Date: Fri Jun 15 13:56:04 2018 +0200 BUG/MINOR: lua: Segfaults with wrong usage of types. Patrick reported that this simple configuration made haproxy segfaults: global lua-load /tmp/haproxy.lua frontend f1 mode http bind :8000 default_backend b1 http-request lua.foo backend b1 mode http server s1 127.0.0.1:8080 with this '/tmp/haproxy.lua' script: core.register_action("foo", { "http-req" }, function(txn) txn.sc:ipmask(txn.f:src(), 24, 112) end) This is due to missing initialization of the array of arguments passed to hlua_lua2arg_check() which makes it enter code with corrupted arguments. Thanks a lot to Patrick Hemmer for having reported this issue. Must be backported to 1.8, 1.7 and 1.6. (cherry picked from commit f874a83b57acce760b0e500bbee47115390f777d) Signed-off-by: William Lallemand commit 2b5ef62fc5a9fe0f96a9a85761502159f5e94598 Author: William Lallemand Date: Mon Jun 11 10:53:46 2018 +0200 BUG/MAJOR: map: fix a segfault when using http-request set-map The bug happens with an existing entry, when you try to overwrite the value with wrong data, for example, a string when the type is INT. The code path was not secure and tried to set *err and *merr while err = merr = NULL when performing an http action. Must be backported in 1.6, 1.7, 1.8. (cherry picked from commit 579fb25b6215d64a2e064550c1d186547414cc68) Signed-off-by: William Lallemand commit aec5cbaf52212e41eb3fe7823d19bae5274bb8b0 Author: Thierry FOURNIER Date: Thu Jun 7 14:40:48 2018 +0200 MINOR: lua: Increase debug information When an unrecoverable error raises, the user receive poor information for the trouble shooting. For example: [ALERT] 157/143755 (21212) : Lua function 'hello-world': runtime error: memory allocation error: block too big. Unfortunately, the memory allocation error can be throwed by many function, and we have no informatio to reach the original cause. This patch add the list of function called from the entry point to the function in error, like this: [ALERT] 157/143755 (21212) : Lua function 'hello-world': runtime error: memory allocation error: block too big from [C] method 'req_get_headers', bug35.lua:2 global 'ee', bug35.lua:6 global 'ff', bug35.lua:10 C function line 9. (cherry picked from commit fc044c98e43ca5373c535e1e5f00e47ce8e5acb4) Signed-off-by: William Lallemand commit 1e3c84931becfc0e4fd421d64a2b10568acc2734 Author: William Lallemand Date: Thu Jun 7 11:23:40 2018 +0200 BUG/MINOR: signals: ha_sigmask macro for multithreading The behavior of sigprocmask in an multithreaded environment is undefined. The new macro ha_sigmask() calls either pthreads_sigmask() or sigprocmask() if haproxy was built with thread support or not. This should be backported to 1.8. (cherry picked from commit 6e1796e85d5351535416018112fc19ccacc7fc5e) Signed-off-by: William Lallemand commit ec828c9f437f6d38f1e846da84f30f2115d29f06 Author: William Lallemand Date: Thu Jun 7 09:49:04 2018 +0200 BUG/MINOR: don't ignore SIG{BUS,FPE,ILL,SEGV} during signal processing We don't have any reason of blocking those signals. If SIGBUS, SIGFPE, SIGILL, or SIGSEGV are generated while they are blocked, the result is undefined, unless the signal was generated by kill(2), sigqueue(3), or raise(3). This should be backported to 1.8. (cherry picked from commit 933642c6ef7c3015222cba84c20f13a1ca0eafd6) Signed-off-by: William Lallemand commit 71b97fc7230c588af51fea2ffac92ba5200316ce Author: William Lallemand Date: Thu Jun 7 09:46:01 2018 +0200 BUG/MEDIUM: threads: handle signal queue only in thread 0 Signals were handled in all threads which caused some signals to be lost from time to time. To avoid complicated lock system (threads+signals), we prefer handling the signals in one thread avoiding concurrent access. The side effect of this bug was that some process were not leaving from time to time during a reload. This patch must be backported in 1.8. (cherry picked from commit 1aab50bb4aaf59f0dd6f0e98ccbf023209f92a69) Signed-off-by: William Lallemand commit 25739157bc24c3a7fa670573d81eaaa91baeaed2 Author: Olivier Houchard Date: Wed Jun 6 18:34:34 2018 +0200 BUG/MINOR: unix: Make sure we can transfer abns sockets on seamless reload. When checking if a socket we got from the parent is suitable for a listener, we just checked that the path matched sockname.tmp, however this is unsuitable for abns sockets, where we don't have to create a temporary file and rename it later. To detect that, check that the first character of the sun_path is 0 for both, and if so, that &sun_path[1] is the same too. This should be backported to 1.8. (cherry picked from commit b4dd15bd6fc264cfcbae9b84c0775f25987c0f78) Signed-off-by: William Lallemand commit 6f6d65fb2ddc640bf989fad37698e5c8f0608cf4 Author: David Carlier Date: Thu May 31 16:42:03 2018 +0100 DOC: contrib/modsecurity: few typo fixes Few typo fixes. (cherry picked from commit 53d31594bf0bad8f327c2530755118e3d401d881) Signed-off-by: William Lallemand commit 99f8ec23ab0686d97ea1c18887a28b4ad87f8cc3 Author: Kevin Zhu Date: Fri Jun 1 05:38:00 2018 +0200 DOC: SPOE.txt: fix a typo (cherry picked from commit 730323e56e3816a0c17f3d21857ad276847294db) Signed-off-by: William Lallemand commit bc00260a1f3215a02ae9b60013f7e96058ff2339 Author: Dragan Dosen Date: Fri Jun 1 15:50:57 2018 +0200 BUG/MINOR: contrib/modsecurity: update pointer on the end of the frame Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded"). This patch should be backported to 1.8. (cherry picked from commit ee07dbeca515fed3e6b71791daa2a2d8bd0ea599) Signed-off-by: William Lallemand commit 49c2ac39a3fef511ce13379c73bac37d302a130b Author: Dragan Dosen Date: Fri Jun 1 15:42:12 2018 +0200 BUG/MINOR: contrib/mod_defender: update pointer on the end of the frame Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded"). This patch should be backported to 1.8. (cherry picked from commit f21185f2363484a909ce1d8ef69b9ceab279c04f) Signed-off-by: William Lallemand commit b1b8f73b35d6db07b1ca3f194d69ab9713d71126 Author: Christopher Faulet Date: Thu May 31 16:05:21 2018 +0200 BUG/MINOR: contrib/modsecurity: Don't reset the status code during disconnect When the connection is closed by HAProxy, the status code provided in the DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure to have it in the SPOE logs. This patch may be backported in 1.8. (cherry picked from commit 58b8e492a311b5dfec533734c1e22ea76d352aed) Signed-off-by: William Lallemand commit 80aabe2becff62951f1060b21117a52c09d265ed Author: Christopher Faulet Date: Thu May 31 16:04:53 2018 +0200 BUG/MINOR: contrib/mod_defender: Don't reset the status code during disconnect When the connection is closed by HAProxy, the status code provided in the DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure to have it in the SPOE logs. This patch may be backported in 1.8. (cherry picked from commit 72de8eb7aa4c2e890ed1a5c89d40b8b041c71b0c) Signed-off-by: William Lallemand commit f16a6f57f879486cf9f5c9833155950a4909049e Author: Christopher Faulet Date: Thu May 31 15:59:32 2018 +0200 BUG/MINOR: contrib/spoa_example: Don't reset the status code during disconnect When the connection is closed by HAProxy, the status code provided in the DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure to have it in the SPOE logs. This patch may be backported in 1.8. (cherry picked from commit b47e4385933b645cf6cac846599feb8a823822d1) Signed-off-by: William Lallemand commit 293225b758e35976dd4c4690b644d22b253f3919 Author: Christopher Faulet Date: Thu May 31 14:56:42 2018 +0200 MAJOR: spoe: upgrade the SPOP version to 2.0 and remove the support for 1.0 The commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order") introduced an incompatibility with older agents. So the major version of the SPOP is increased to make the situation unambiguous. And because before the fix, the protocol is buggy, the support of the version 1.0 is removed to be sure to not continue to support buggy agents. The agents in the contrib folder (spoa_example, modsecurity and mod_defender) are also updated to announce the SPOP version 2.0. So, to be clear, from the patch, connections to agents announcing the SPOP version 1.0 will be rejected. This patch must be backported in 1.8. (cherry picked from commit 6381650516b3c3f9c272da66d33d9f9e23139594) Signed-off-by: William Lallemand commit ad782b812065b220422757c3e982105016580a93 Author: Thierry FOURNIER Date: Sun May 27 01:14:47 2018 +0200 BUG/MEDIUM: lua/socket: Buffer error, may segfault The buffer pointer is already updated. It is again updated when it is given to the function ci_putblk(). This patch must be backported in 1.6, 1.7 and 1.8 (cherry picked from commit 66b8919b10d99b022a9105d4908974615c4bba3d) Signed-off-by: William Lallemand commit d36387f93ff009cc1aa6175749a42ca0fb3e9dd6 Author: Thierry FOURNIER Date: Sun May 27 01:27:40 2018 +0200 BUG/MEDIUM: lua/socket: Sheduling error on write: may dead-lock When we write data, we risk to encounter a dead-loack. The function "stream_int_notify()" cannot be called the the cosocket because the caller acquire a lock and when the socket is closed, the cleanup function try to acquire the same lock., so a dead-lock raises. In other way, the function stream_int_update_applet() can't be called because it schedumes the applet only if some activity in the buffers were detected. It is not always the case. We replace this function by appctx_wakeup() which wake up the applet inconditionnaly. The last part of the fix is setting right signals. the applet call the stream_int_update() function if the output buffer si not empty, and ask for put data if some rite signals are registered. This patch must be backported in 1.6, 1.7 and 1.8. Note that it requires patch "MINOR: task/notification: Is notifications registered" to be applied. (cherry picked from commit 101b97619a5d0b45484a37395f1b22d064c806cb) Signed-off-by: William Lallemand commit 631100bd0b5258049a430ea937a1060c0aaa531e Author: Thierry FOURNIER Date: Sun May 27 00:59:48 2018 +0200 BUG/MEDIUM: lua/socket: Notification error Each time the send function yields, a notification must be registered. Without this notification, the task is never wakeup when data arrives. Today, the notification is registered only if the buffer is not available. Other cases like the buffer is too small for all data are not processed. This patch must be backported in 1.6, 1.7 and 1.8 (cherry picked from commit ba42fcd064c3f371610126dee6f25c2d7f1e4040) Signed-off-by: William Lallemand commit de3b6d5dbdccc85fe4ce6b6ab4147a127a34c10c Author: Thierry FOURNIER Date: Fri May 25 15:03:50 2018 +0200 BUG/MAJOR: lua: Dead lock with sockets In some cases, when we are waiting for data and the socket timeout expires, we have a dead lock. The Lua socket locks the applet socket, and call for a notify. The notify immediately executes code and try to acquire the same lock, so ... dead lock. stream_int_notify() cant be used because it wakeup the applet task only if the stream have changes. The changes are forces by Lua, but not repported on the stream. stream_int_update_applet() cant be used because the deadlock. So, I inconditionnaly wakeup the applet. This wake is performed asynchronously, and will call a stream_int_notify(). This patch must be backported in 1.6, 1.7 and 1.8 (cherry picked from commit 7e4ee47accdb93925ddfc18cf6f181ff334abcdc) Signed-off-by: William Lallemand commit 46450c937abcf645944c3d95fe9b6dae37b92f78 Author: Thierry FOURNIER Date: Fri May 25 14:38:57 2018 +0200 BUG/MEDIUM: lua/socket: wrong scheduling for sockets The appctx pointer is given from any variable which are wrong. This implies the wakeup of wrong applet, and the socket are no longer responsive. This behavior is hidden by another inherited error which is fixed in the next patch. This patch remove all wrong appctx affectations. This patch must be backported in 1.6, 1.7 and 1.8 (cherry picked from commit af4bd0867a6ef4ca219bf97fee5205fa928be4ab) Signed-off-by: William Lallemand commit 7618f21f3bd16c2cb46c30c5e95c49f105f370ab Author: Thierry FOURNIER Date: Wed May 30 11:40:08 2018 +0200 MINOR: task/notification: Is notifications registered ? This function returns true is some notifications are registered. This function is usefull for the following patch BUG/MEDIUM: lua/socket: Sheduling error on write: may dead-lock It should be backported in 1.6, 1.7 and 1.8 (cherry picked from commit 9d5422a4b7e71294cf7ce16d60490d907cc92f48) Signed-off-by: William Lallemand commit 720e8e854797ff080eb548531ee89c0033fc6cd5 Author: Christopher Faulet Date: Fri May 25 10:42:37 2018 +0200 BUG/MEDIUM: spoe: Return an error when the wrong ACK is received in sync mode This is required to let a message processing timed out. Because, when it happens, there is no more context attached to the SPOE applet that sent the NOTIFY frame. So when the ACK is received, it is too late. This is the same situation when we receive the wrong ACK. It is invalid in sync mode. Otherwise, the SPOE applet remains in the state "WAITING_SYNC_ACK" until the idle timeout is reached. In such case, the applet is seen as busy and it is unusable. If this happens too often, more and more applets will be created because some others are blocked. If there is a maxconn on the SPOE backend, all processings will be drastically slowdown. Returning an error in such cases, in sync mode, allow us to terminate the SPOE applet. Because it means the agent is unresponsive or too slow. Note this bug exists only if the sync mode is used. This patch must be backported in 1.8. (cherry picked from commit 3a47e5e25c0e5621616b75ab0dd4a56c2f1d0608) Signed-off-by: William Lallemand commit d7bd88009d88dd413e01bc0baa90d6662a3d7718 Author: Daniel Corbett Date: Sun May 27 09:47:12 2018 -0400 BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters When using table_* converters ref_cnt was incremented and never decremented causing entries to not expire. The root cause appears to be that stktable_lookup_key() was called within all sample_conv_table_* functions which was incrementing ref_cnt and not decrementing after completion. Added stktable_release() to the end of each sample_conv_table_* function and reworked the end logic to ensure that ref_cnt is always decremented after use. This should be backported to 1.8 (cherry picked from commit 3e60b11100cbc812b77029ca142b83ac7a314db1) Signed-off-by: William Lallemand commit a96eead1e4d5b2fd97852ea9c53d099de53a1f2c Author: Thierry FOURNIER Date: Fri May 25 16:27:44 2018 +0200 BUG/MEDIUM: lua/socket: Length required read doesn't work The limit of data read works only if all the data is in the input buffer. Otherwise (if the data arrive in chunks), the total amount of data is not taken in acount. Only the current read data are compared to the expected amout of data. This patch must be backported from 1.9 to 1.6 (cherry picked from commit 8c126c7235ced98986a57175a62503d5166ffe19) Signed-off-by: William Lallemand commit be68ac7047ea08ba29e86c42c5dc3c9b33bd98c9 Author: Daniel Corbett Date: Sat May 19 19:43:24 2018 -0400 BUG/MEDIUM: servers: Add srv_addr default placeholder to the state file When creating a state file using "show servers state" an empty field is created in the srv_addr column if the server is from the socket family AF_UNIX. This leads to a warning on start up when using "load-server-state-from-file". This patch defaults srv_addr to "-" if the socket family is not covered. This patch should be backported to 1.8. (cherry picked from commit 9215ffa6b294374e99e5ab027bea9b34642d7277) Signed-off-by: William Lallemand commit 09dbc884b10a9b30dc500f45f32d1610d463a056 Author: Olivier Houchard Date: Tue May 29 14:42:22 2018 +0200 BUG/MEDIUM: fd: Only check update_mask against all_threads_mask. To decide if we have to insert a fd into the update_list, we check its update_mask. We set the update_mask to fdtab[fd].thread_mask when adding it or updating, and when each thread will remove its bit once it has been taken care of. However, even if all threads did their job, update_mask may be non-zero, because for fds that want all threads, the mask will be 0xffffffffffffffff, so except when we run with 64 threads, there will be bits left over. So, to decide if nobody had any update for that fd, we have to check (update_mask & all_threads_mask) == 0, instead of update_mask == 0. Not doing so means fd events can be missed, or fd not removed from the poller, making poller waking with no need, and using 100% of the CPU. This should only be applied on 1.8. commit 6d82e6114f393a764aa5cf423bf3782e36cebe54 Author: Willy Tarreau Date: Wed May 23 20:02:04 2018 +0200 BUG/BUILD: fd: fix typo causing a warning when threads are disabled A warning is issued since commit 954db1d ("BUG/MEDIUM: pollers: Use a global list for fd shared between threads.") when threads are disabled due to a typo. It has no effect beyond this warning. There is no mainline commit ID because this bug is specific to the backport above. commit fbf09c441a4e72c4a690bc7ef25d3374767fe5c5 Author: Willy Tarreau Date: Wed May 23 19:54:43 2018 +0200 BUG/BUILD: threads: unbreak build without threads A few users reported that building without threads was accidently broken after commit 6b96f72 ("BUG/MEDIUM: pollers: Use a global list for fd shared between threads.") due to all_threads_mask not being defined. It's OK to set it to zero as other code parts do when threads are enabled but only one thread is used. This needs to be backported to 1.8. (cherry picked from commit 0cd82e883ef034508ebeef04fd25ffc946e80cfc) Signed-off-by: Willy Tarreau commit 3157ef219c493f3b01192f1b809a086a5b119a1e Author: Olivier Houchard Date: Tue May 22 18:40:07 2018 +0200 BUG/MEDIUM: dns: Delay the attempt to run a DNS resolution on check failure. When checks fail, the code tries to run a dns resolution, in case the IP changed. The old way of doing that was to check, in case the last dns resolution hadn't expired yet, if there were an applicable IP, which should be useless, because it has already be done when the resolution was first done, or to run a new resolution. Both are a locking nightmare, and lead to deadlocks, so instead, just wake the resolvers task, that should do the trick. This should be backported to 1.8. (cherry picked from commit f3d9e608d73fae1a7f00341cb52371e4b2181c66) Signed-off-by: Willy Tarreau commit 17514045e5d934dede62116216c1b016fe23dd06 Author: William Lallemand Date: Tue May 22 11:04:33 2018 +0200 BUG/MEDIUM: cache: don't cache when an Authorization header is present RFC 7234 says: A cache MUST NOT store a response to any request, unless: [...] the Authorization header field (see Section 4.2 of [RFC7235]) does not appear in the request, if the cache is shared, unless the response explicitly allows it (see Section 3.2), [...] In this patch we completely disable the cache upon the receipt of an Authorization header in the request. In this case it's not possible to either use the cache or store into the cache anymore. Thanks to Adam Eijdenberg of Digital Transformation Agency for raising this issue. This patch must be backported to 1.8. (cherry picked from commit 8a16fe0d053b93c00a8bcf86159135f98ca1377e) Signed-off-by: William Lallemand commit dd08a40b58f0dd3e0ed0b954fc2f3f3984cd3ec6 Author: Willy Tarreau Date: Fri May 18 17:08:28 2018 +0200 BUG/MINOR: ssl/lua: prevent lua from affecting automatic maxconn computation Since commit 36d1374 ("BUG/MINOR: lua: Fix SSL initialisation") in 1.6, the Lua code always initializes an SSL server. It caused a small visible side effect which is that by calling ssl_sock_prepare_srv_ctx(), it forces global.ssl_used_backend to 1 and makes the initialization code believe that there are some SSL servers in certain backends. This detection is used to figure how to set the global maxconn value when only the memory usage is limited. As such, even a configuration with no SSL at all will have a very conservative maxconn. The configuration below exhibits this : global ssl-server-verify none stats socket /tmp/sock1 mode 666 level admin tune.bufsize 16384 listen px timeout client 5s timeout server 5s timeout connect 5s bind :4445 #bind :4443 ssl crt rsa+dh2048.pem #server s1 127.0.0.1:8003 ssl Starting it with "-m 200" to limit it to 200 MB of RAM reports 1500 for Maxconn, the same when uncommenting the "server" line, and 1300 when uncommenting the "bind" line, regardless of the "server" line's status. In practice it doesn't make sense to consider that Lua's server template counts for one regular SSL server, because even if used for SSL, it will not take large connection counts, compared to a backend relaying traffic. Thus the solution consists in resetting the ssl_used_backend to its previous value after creating the server_ctx from the Lua code. With the fix, the same config with the same parameters now show : - maxconn=5700 when neither side uses SSL - maxconn=1500 when only one side uses SSL - maxconn=1300 when both sides use SSL This fix can be backported to versions 1.6 and beyond. (cherry picked from commit cbe6da5eb0e1c24b4757fea8ddbf6183fc9dcf43) Signed-off-by: William Lallemand commit 0323f843029746350cec4d197c89e13e987a5f87 Author: Christopher Faulet Date: Fri May 18 14:46:32 2018 +0200 BUG/MEDIUM: contrib/modsecurity: Use network order to encode/decode flags A recent fix on the SPOE revealed a mismatch between the SPOE specification and the modsecurity implementation on the way flags are encoded or decoded. They must be exchanged using the network bytes order and not the host one. Be careful though, this patch breaks the compatiblity with HAProxy SPOE before commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order"). (cherry picked from commit 633f3bffedd7c48b921ad10e5164ee6b0e1d79a2) Signed-off-by: William Lallemand commit becca9decfc7d980218ccf1c9ce933ef9871705a Author: Christopher Faulet Date: Fri May 18 14:38:56 2018 +0200 BUG/MEDIUM: contrib/mod_defender: Use network order to encode/decode flags A recent fix on the SPOE revealed a mismatch between the SPOE specification and the mod_defender implementation on the way flags are encoded or decoded. They must be exchanged using the network bytes order and not the host one. Be careful though, this patch breaks the compatiblity with HAProxy SPOE before commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order"). (cherry picked from commit 48d02d0d21b7ec85de2a6bbeb85757add52e3dfc) Signed-off-by: William Lallemand commit 4060b58c4abfb431953a74db2f54ef249082097b Author: Thierry FOURNIER Date: Fri May 18 12:25:39 2018 +0200 BUG/MEDIUM: spoe: Flags are not encoded in network order The flags are direct copy of the "unsigned int" in the network stream, so the stream contains a 32 bits field encoded with the host endian. - This is not reliable for stream betwen different architecture host - For x86, the bits doesn't correspond to the documentation. This patch add some precision in the documentation and put the bitfield in the stream usig network butes order. Warning: this patch can break compatibility with existing agents. This patch should be backported in all version supporing SPOE Original network capture: 12:28:16.181343 IP 127.0.0.1.46782 > 127.0.0.1.12345: Flags [P.], seq 134:168, ack 59, win 342, options [nop,nop,TS val 2855241281 ecr 2855241281], length 34 0x0000: 4500 0056 6b94 4000 4006 d10b 7f00 0001 E..Vk.@.@....... 0x0010: 7f00 0001 b6be 3039 a3d1 ee54 7d61 d6f7 ......09...T}a.. 0x0020: 8018 0156 fe4a 0000 0101 080a aa2f 8641 ...V.J......./.A 0x0030: aa2f 8641 0000 001e 0301 0000 0000 010f ./.A............ ^^^^^^^^^^ 0x0040: 6368 6563 6b2d 636c 6965 6e74 2d69 7001 check-client-ip. 0x0050: 0006 7f00 0001 ...... Fixed network capture: 12:24:26.948165 IP 127.0.0.1.46706 > 127.0.0.1.12345: Flags [P.], seq 4066280627:4066280661, ack 3148908096, win 342, options [nop,nop,TS val 2855183972 ecr 2855177690], length 34 0x0000: 4500 0056 0538 4000 4006 3768 7f00 0001 E..V.8@.@.7h.... 0x0010: 7f00 0001 b672 3039 f25e 84b3 bbb0 8640 .....r09.^.....@ 0x0020: 8018 0156 fe4a 0000 0101 080a aa2e a664 ...V.J.........d 0x0030: aa2e 8dda 0000 001e 0300 0000 0114 010f ................ ^^^^^^^^^^ 0x0040: 6368 6563 6b2d 636c 6965 6e74 2d69 7001 check-client-ip. 0x0050: 0006 7f00 0001 ...... (cherry picked from commit c4dcaff3f01cdd774d15767962c76910f9d3e39a) Signed-off-by: William Lallemand commit 43bf0bf5f5f83655df2bdb9089d410e49717971f Author: sada Date: Fri May 11 11:48:18 2018 -0700 BUG/MINOR: lua: Socket.send threw runtime error: 'close' needs 1 arguments. Function `hlua_socket_close` expected exactly one argument on the Lua stack. But when `hlua_socket_close` was called from `hlua_socket_write_yield`, Lua stack had 3 arguments. So `hlua_socket_close` threw the exception with message "'close' needs 1 arguments". Introduced new helper function `hlua_socket_close_helper`, which removed the Lua stack argument count check and only checked if the first argument was a socket. This fix should be backported to 1.8, 1.7 and 1.6. (cherry picked from commit 05ed330d72329b74f5488eb102e0c5862b00543b) Signed-off-by: William Lallemand commit 83616ec453f5dc4cddd7c5205943763dc8781f76 Author: William Lallemand Date: Fri May 18 15:10:29 2018 +0200 [RELEASE] Released version 1.8.9 Released version 1.8.9 with the following main changes : - BUG/MINOR: pattern: Add a missing HA_SPIN_INIT() in pat_ref_newid() - BUG/MAJOR: channel: Fix crash when trying to read from a closed socket - BUG/MINOR: log: t_idle (%Ti) is not set for some requests - BUG/MEDIUM: lua: Fix segmentation fault if a Lua task exits - MINOR: h2: detect presence of CONNECT and/or content-length - BUG/MEDIUM: h2: implement missing support for chunked encoded uploads - BUG/MINOR: lua/threads: Make lua's tasks sticky to the current thread - BUG/MINOR: config: disable http-reuse on TCP proxies - BUG/MINOR: checks: Fix check->health computation for flapping servers - BUG/MEDIUM: threads: Fix the sync point for more than 32 threads - BUG/MINOR: lua: Put tasks to sleep when waiting for data - DOC/MINOR: clean up LUA documentation re: servers & array/table. - BUG/MINOR: map: correctly track reference to the last ref_elt being dumped - BUG/MEDIUM: task: Don't free a task that is about to be run. - BUG/MINOR: lua: schedule socket task upon lua connect() - BUG/MINOR: lua: ensure large proxy IDs can be represented - BUG/MEDIUM: http: don't always abort transfers on CF_SHUTR - BUG/MEDIUM: pollers: Use a global list for fd shared between threads. - BUG/MEDIUM: ssl: properly protect SSL cert generation - BUG/MINOR: spoe: Mistake in error message about SPOE configuration commit 604d0eab590a352e0487395ce3785cd88ee75752 Author: Thierry FOURNIER Date: Thu May 10 16:41:26 2018 +0200 BUG/MINOR: spoe: Mistake in error message about SPOE configuration The announced accepted chars are "[a-zA-Z_-.]", but the real accepted alphabet is "[a-zA-Z0-9_.]". Numbers are supported and "-" is not supported. This patch should be backported to 1.8 and 1.7 (cherry picked from commit 01a3f2074044eb623fb350dc8202830181af169a) Signed-off-by: William Lallemand commit f571613244e4c02ca7aada30c89a6244d09d58d4 Author: Willy Tarreau Date: Thu May 17 10:56:47 2018 +0200 BUG/MEDIUM: ssl: properly protect SSL cert generation Commit 821bb9b ("MAJOR: threads/ssl: Make SSL part thread-safe") added insufficient locking to the cert lookup and generation code : it uses lru64_lookup(), which will automatically remove and add a list element to the LRU list. It cannot be simply read-locked. A long-term improvement should consist in using a lockless mechanism in lru64_lookup() to safely move the list element at the head. For now let's simply use a write lock during the lookup. The effect will be minimal since it's used only in conjunction with automatically generated certificates, which are much more expensive and rarely used. This fix must be backported to 1.8. (cherry picked from commit 03f4ec47d9ffff629b07dcba9f0f134a7c7e44b2) Signed-off-by: William Lallemand commit 954db1d01a3d706d4cacd288f28e8517a635d36e Author: Olivier Houchard Date: Thu May 17 18:34:02 2018 +0200 BUG/MEDIUM: pollers: Use a global list for fd shared between threads. With the old model, any fd shared by multiple threads, such as listeners or dns sockets, would only be updated on one threads, so that could lead to missed event, or spurious wakeups. To avoid this, add a global list for fd that are shared, and only remove entries from this list when every thread as updated its poller. This subtly changes the semantics of updt_fd_polling(), as it now unlocks the FD_LOCK on exit. This is similar in spirit to commit 6b96f7289c2f401deef4bdc6e20792360807dde4 (with the bugfix from c55b88ece616afe0b28dc81eb39bad37b5f9c33f) applied, but had to be rewrote, because of the differences between 1.8 and master. This should only be applied to 1.8. commit 1c10e5b1b95142bb3ac385be1e60d8b180b2e99e Author: Willy Tarreau Date: Wed May 16 11:35:05 2018 +0200 BUG/MEDIUM: http: don't always abort transfers on CF_SHUTR Pawel Karoluk reported on Discourse[1] that HTTP/2 breaks url_param. Christopher managed to track it down to the HTTP_MSGF_WAIT_CONN flag which is set there to ensure the connection is validated before sending the headers, as we may need to rewind the stream and hash again upon redispatch. What happens is that in the forwarding code we refrain from forwarding when this flag is set and the connection is not yet established, and for this we go through the missing_data_or_waiting path. This exit path was initially designed only to wait for data from the client, so it rightfully checks whether or not the client has already closed since in that case it must not wait for more data. But it also has the side effect of aborting such a transfer if the client has closed after the request, which is exactly what happens in H2. A study on the code reveals that this whole combined check should be revisited : while it used to be true that waiting had the same error conditions as missing data, it's not true anymore. Some other corner cases were identified, such as the risk to report a server close instead of a client timeout when waiting for the client to read the last chunk of data if the shutr is already present, or the risk to fail a redispatch when a client uploads some data and closes before the connection establishes. The compression seems to be at risk of rare issues there if a write to a full buffer is not yet possible but a shutr is already queued. At the moment these risks are extremely unlikely but they do exist, and their impact is very minor since it mostly concerns an issue not being optimally handled, and the fixes risk to cause more serious issues. Thus this patch only focuses on how the HTTP_MSGF_WAIT_CONN is handled and leaves the rest untouched. This patch needs to be backported to 1.8, and could be backported to earlier versions to properly take care of HTTP/1 requests passing via url_param which are closed immediately after the headers, though this is unlikely as this behaviour is only exhibited by scripts. [1] https://discourse.haproxy.org/t/haproxy-1-8-x-url-param-issue-in-http2/2482/13 (cherry picked from commit ba20dfc50161ba705a746d54ebc1a0a45c46beab) Signed-off-by: Willy Tarreau commit edb4427ab7c070a16cb9a23460f68b3fc3c041bb Author: Willy Tarreau Date: Sun May 6 14:50:09 2018 +0200 BUG/MINOR: lua: ensure large proxy IDs can be represented In function hlua_fcn_new_proxy() too small a buffer was passed to snprintf(), resulting in large proxy or listener IDs to make snprintf() fail. It is unlikely to meet this case but let's fix it anyway. This fix must be backported to all stable branches where it applies. (cherry picked from commit 29d698040d6bb56b29c036aeba05f0d52d8ce94b) Signed-off-by: Willy Tarreau commit 52ec3578c3ddc688ae14da3cd3e7e351494603d8 Author: PiBa-NL Date: Sat May 5 23:51:42 2018 +0200 BUG/MINOR: lua: schedule socket task upon lua connect() The parameters like server-address, port and timeout should be set before process_stream task is called to avoid the stream being 'closed' before it got initialized properly. This is most clearly visible when running with tune.lua.forced-yield=1.. So scheduling the task should not be done when creating the lua socket, but when connect is called. The error "socket: not yet initialised, you can't set timeouts." would then appear. Below code for example also shows this issue, as the sleep will yield the lua code: local con = core.tcp() core.sleep(1) con:settimeout(10) (cherry picked from commit 706d5ee0c366787536213ccd6dea264d20b76a22) [wt: must be backported to 1.7 and 1.6 as well with a different patch, see https://www.mail-archive.com/haproxy@formilux.org/msg29924.html] Signed-off-by: Willy Tarreau commit a0f0db361978154474d76028183647d5991f3b5c Author: Olivier Houchard Date: Fri May 4 15:46:16 2018 +0200 BUG/MEDIUM: task: Don't free a task that is about to be run. While running a task, we may try to delete and free a task that is about to be run, because it's part of the local tasks list, or because rq_next points to it. So flag any task that is in the local tasks list to be deleted, instead of run, by setting t->process to NULL, and re-make rq_next a global, thread-local variable, that is modified if we attempt to delete that task. Many thanks to PiBa-NL for reporting this and analysing the problem. This should be backported to 1.8. (cherry picked from commit 9b36cb4a414c22e13d344afbbe70684e9f2f1d49) Signed-off-by: Willy Tarreau commit b2219ae216a141acdf0e2a3f67d2c85aee2a2bc2 Author: Dragan Dosen Date: Fri May 4 16:27:15 2018 +0200 BUG/MINOR: map: correctly track reference to the last ref_elt being dumped The bug was introduced in the commit 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show map/acl' on cli"). This patch should be backported to 1.8, 1.7 and 1.6. (cherry picked from commit 336a11f75571ad46f74a7c6247c13ed44f95da93) Signed-off-by: Willy Tarreau commit 016feef5483397491af3242162934d9e9dbc6263 Author: Patrick Hemmer Date: Tue May 1 21:30:41 2018 -0400 DOC/MINOR: clean up LUA documentation re: servers & array/table. * A few typos * Fix definitions of values which are tables, not arrays. * Consistent US English naming for "server" instead of "serveur". [tfo: should be backported to 1.6 and higher] (cherry picked from commit c6a1d711a4d47d68611aa28adecdadba96221bde) Signed-off-by: Willy Tarreau commit 335bc7b74eee84f0a3bcb615cadd23fe01d1336c Author: PiBa-NL Date: Wed May 2 22:27:14 2018 +0200 BUG/MINOR: lua: Put tasks to sleep when waiting for data If a lua socket is waiting for data it currently spins at 100% cpu usage. This because the TICK_ETERNITY returned by the socket is ignored when setting the 'expire' time of the task. Fixed by removing the check for yields that return TICK_ETERNITY. This should be backported to at least 1.8. (cherry picked from commit fe971b35aeca9994f3823112c783aa796e74075a) Signed-off-by: Willy Tarreau commit 830324444e57c042666b17ac4584352cca85dafd Author: Christopher Faulet Date: Wed May 2 16:58:40 2018 +0200 BUG/MEDIUM: threads: Fix the sync point for more than 32 threads In the sync point, to know if a thread has requested a synchronization, we call the function thread_need_sync(). It should return 1 if yes, otherwise it should return 0. It is intended to return a signed integer. But internally, instead of returning 0 or 1, it returns 0 or tid_bit (threads_want_sync & tid_bit). So, tid_bit is casted in integer. For the first 32 threads, it's ok, because we always check if thread_need_sync() returns something else than 0. But this is a problem if HAProxy is started with more than 32 threads, because for threads 33 to 64 (so for tid 32 to 63), their tid_bit casted to integer are evaluated to 0. So the sync point does not work for more than 32 threads. Now, the function thread_need_sync() respects its contract, returning 0 or 1. the function thread_no_sync() has also been updated to avoid any ambiguities. This patch must be backported in HAProxy 1.8. (cherry picked from commit 148b16e1ceb819dfcef4c45828121d9cd7474b35) Signed-off-by: Willy Tarreau commit edb5a1efd22eb9918574d962640cd2ae3bb45ad3 Author: Christopher Faulet Date: Wed May 2 12:12:45 2018 +0200 BUG/MINOR: checks: Fix check->health computation for flapping servers This patch fixes an old bug introduced in the commit 7b1d47ce ("MAJOR: checks: move health checks changes to set_server_check_status()"). When a DOWN server is flapping, everytime a check succeds, check->health is incremented. But when a check fails, it is decremented only when it is higher than the rise value. So if only one check succeds for a DOWN server, check->health will remain set to 1 for all subsequent failing checks. So, at first glance, it seems not that terrible because the server remains DOWN. But it is reported in the transitional state "DOWN server, going up". And it will remain in this state until it is UP again. And there is also an insidious side effect. If a DOWN server is flapping time to time, It will end to be considered UP after a uniq successful check, , regardless the rise threshold, because check->health will be increased slowly and never decreased. To fix the bug, we just need to reset check->health to 0 when a check fails for a DOWN server. To do so, we just need to relax the condition to handle a failure in the function set_server_check_status. This patch must be backported to haproxy 1.5 and newer. (cherry picked from commit b119a79fc336f2b6074de1c3113b1682c717985c) Signed-off-by: Willy Tarreau commit 80e179128cfd78d95cdebf7195fd21299e7931b6 Author: Willy Tarreau Date: Sat Apr 28 07:18:15 2018 +0200 BUG/MINOR: config: disable http-reuse on TCP proxies Louis Chanouha reported an inappropriate warning when http-reuse is present in a defaults section while a TCP proxy accidently inherits it and finds a conflict with other options like the use of the PROXY protocol. To fix this patch removes the http-reuse option for TCP proxies. This fix needs to be backported to 1.8, 1.7 and possibly 1.6. (cherry picked from commit 46deab6e64bfda7211b7c3199ad01f136141c86f) Signed-off-by: Christopher Faulet commit 8b8d55be7e94ee3d758d41a21fa86a036e91a264 Author: Christopher Faulet Date: Wed Apr 25 10:34:45 2018 +0200 BUG/MINOR: lua/threads: Make lua's tasks sticky to the current thread PiBa-NL reported a bug with tasks registered in lua when HAProxy is started with serveral threads. These tasks have not specific affinity with threads so they can be woken up on any threads. So, it is impossbile for these tasks to handled cosockets or applets, because cosockets and applets are sticky on the thread which created them. It is forbbiden to manipulate a cosocket from another thread. So to fix the bug, tasks registered in lua are now sticky to the current thread. Because these tasks can be registered before threads creation, the affinity is set the first time a lua's task is processed. This patch must be backported in HAProxy 1.8. (cherry picked from commit 5bc9972ed836517924eea91954d255d317a53418) Signed-off-by: Christopher Faulet commit 05657bd24ebaf20e5c508a435be9a0830591f033 Author: Willy Tarreau Date: Wed Apr 25 20:44:22 2018 +0200 BUG/MEDIUM: h2: implement missing support for chunked encoded uploads Upload requests not carrying a content-length nor tunnelling data must be sent chunked-encoded over HTTP/1. The code was planned but for some reason forgotten during the implementation, leading to such payloads to be sent as tunnelled data. Browsers always emit a content length in uploads so this problem doesn't happen for most sites. However some applications may send data frames after a request without indicating it earlier. The only way to detect that a client will need to send data is that the HEADERS frame doesn't hold the ES bit. In this case it's wise to look for the content-length header. If it's not there, either we're in tunnel (CONNECT method) or chunked-encoding (other methods). This patch implements this. The following request is sent using content-length : curl --http2 -sk https://127.0.0.1:4443/s2 -XPOST -T /large/file and these ones using chunked-encoding : curl --http2 -sk https://127.0.0.1:4443/s2 -XPUT -T /large/file curl --http2 -sk https://127.0.0.1:4443/s2 -XPUT -T - < /dev/urandom Thanks to Robert Samuel Newson for raising this issue with details. This fix must be backported to 1.8. (cherry picked from commit eba10f24b7da27cde60d2db24aeb1147e1657579) Signed-off-by: Christopher Faulet commit a8bcc7dd3fe5aa615f21e795375ff9225f004498 Author: Willy Tarreau Date: Wed Apr 25 18:13:58 2018 +0200 MINOR: h2: detect presence of CONNECT and/or content-length We'll need this in order to support uploading chunks. The h2 to h1 converter checks for the presence of the content-length header field as well as the CONNECT method and returns these information to the caller. The caller indicates whether or not a body is detected for the message (presence of END_STREAM or not). No transfer-encoding header is emitted yet. (cherry picked from commit 174b06a572ef141f15d8b7ea64eb6b34ec4c9af1) Signed-off-by: Christopher Faulet commit 17f3e16826e5b1a3f79b7421d69bb85be09a4ad9 Author: Tim Duesterhus Date: Tue Apr 24 13:56:01 2018 +0200 BUG/MEDIUM: lua: Fix segmentation fault if a Lua task exits PiBa-NL reported that haproxy crashes with a segmentation fault if a function registered using `core.register_task` returns. An example Lua script that reproduces the bug is: mytask = function() core.Info("Stopping task") end core.register_task(mytask) The Valgrind output is as follows: ==6759== Process terminating with default action of signal 11 (SIGSEGV) ==6759== Access not within mapped region at address 0x20 ==6759== at 0x5B60AA9: lua_sethook (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==6759== by 0x430264: hlua_ctx_resume (hlua.c:1009) ==6759== by 0x43BB68: hlua_process_task (hlua.c:5525) ==6759== by 0x4FED0A: process_runnable_tasks (task.c:231) ==6759== by 0x4B2256: run_poll_loop (haproxy.c:2397) ==6759== by 0x4B2256: run_thread_poll_loop (haproxy.c:2459) ==6759== by 0x41A7E4: main (haproxy.c:3049) Add the missing `task = NULL` for the `HLUA_E_OK` case. The error cases have been fixed as of 253e53e661c49fb9723535319cf511152bf09bc7 which first was included in haproxy v1.8-dev3. This bugfix should be backported to haproxy 1.8. (cherry picked from commit cd235c60425dbe66c9015a357369afacc4880211) Signed-off-by: Christopher Faulet commit 0e645ba57ddff9163a3d9b5626f189e974e671bd Author: Rian McGuire Date: Tue Apr 24 11:19:21 2018 -0300 BUG/MINOR: log: t_idle (%Ti) is not set for some requests If TCP content inspection is used, msg_state can be >= HTTP_MSG_ERROR the first time http_wait_for_request is called. t_idle was being left unset in that case. In the example below : stick-table type string len 64 size 100k expire 60s tcp-request inspect-delay 1s tcp-request content track-sc1 hdr(X-Session) %Ti will always be -1, because the msg_state is already at HTTP_MSG_BODY when http_wait_for_request is called for the first time. This patch should backported to 1.8 and 1.7. (cherry picked from commit 89fcb7d929283e904cabad58de495d62fc753da2) Signed-off-by: Christopher Faulet commit e0f6d4a4e8696140d1fcff812fb287d534d702e9 Author: Tim Duesterhus Date: Tue Apr 24 19:20:43 2018 +0200 BUG/MAJOR: channel: Fix crash when trying to read from a closed socket When haproxy is compiled using GCC <= 3.x or >= 5.x the `unlikely` macro performs a comparison with zero: `(x) != 0`, thus returning either 0 or 1. In `int co_getline_nc()` this macro was accidentally applied to the variable `retcode` itself, instead of the result of the comparison `retcode <= 0`. As a result any negative `retcode` is converted to `1` for purposes of the comparison. Thus never taking the branch (and exiting the function) for negative values. This in turn leads to reads of uninitialized memory in the for-loop below: ==12141== Conditional jump or move depends on uninitialised value(s) ==12141== at 0x4EB6B4: co_getline_nc (channel.c:346) ==12141== by 0x421CA4: hlua_socket_receive_yield (hlua.c:1713) ==12141== by 0x421F6F: hlua_socket_receive (hlua.c:1896) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B497: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529711A: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52ABDF0: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529A9F1: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B523: lua_resume (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== ==12141== Use of uninitialised value of size 8 ==12141== at 0x4EB6B9: co_getline_nc (channel.c:346) ==12141== by 0x421CA4: hlua_socket_receive_yield (hlua.c:1713) ==12141== by 0x421F6F: hlua_socket_receive (hlua.c:1896) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B497: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529711A: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52ABDF0: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529A9F1: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B523: lua_resume (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== ==12141== Invalid read of size 1 ==12141== at 0x4EB6B9: co_getline_nc (channel.c:346) ==12141== by 0x421CA4: hlua_socket_receive_yield (hlua.c:1713) ==12141== by 0x421F6F: hlua_socket_receive (hlua.c:1896) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B497: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529711A: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52ABDF0: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B08F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x52A7EFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529A9F1: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== by 0x529B523: lua_resume (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==12141== Address 0x8637171e928bb500 is not stack'd, malloc'd or (recently) free'd Fix this bug by correctly applying the `unlikely` macro to the result of the comparison. This bug exists as of commit ca16b038132444dea06e6d83953034128a812bce which is the first commit adding this function. v1.6-dev1 is the first tag containing this commit, the fix should be backported to haproxy 1.6 and newer. (cherry picked from commit 45be38c9c7ba2b20806f2b887876db4fb5b9457c) Signed-off-by: Christopher Faulet commit 6c9efc8219e35f4eb17e94b364f4c371cfb56cca Author: Aurélien Nephtali Date: Thu Apr 19 16:56:07 2018 +0200 BUG/MINOR: pattern: Add a missing HA_SPIN_INIT() in pat_ref_newid() pat_ref_newid() is lacking a spinlock init. It was probably forgotten in b5997f740b ("MAJOR: threads/map: Make acls/maps thread safe"). Signed-off-by: Aurélien Nephtali (cherry picked from commit 564d15a71ecb3ae3372767866335cfbc068c4b48) Signed-off-by: Christopher Faulet commit cd117685f0cff4f2f5577ef6a21eaae96ebd9f28 Author: Willy Tarreau Date: Thu Apr 19 17:20:31 2018 +0200 [RELEASE] Released version 1.8.8 Released version 1.8.8 with the following main changes : - BUG/MEDIUM: threads: Fix the max/min calculation because of name clashes - BUG/MEDIUM: connection: Make sure we have a mux before calling detach(). - BUG/MINOR: http: Return an error in proxy mode when url2sa fails - BUG/MEDIUM: kqueue: When adding new events, provide an output to get errors. - BUG/MINOR: cli: Guard against NULL messages when using CLI_ST_PRINT_FREE - MINOR: cli: Ensure the CLI always outputs an error when it should - DOC: lua: update the links to the config and Lua API - BUG/CRITICAL: h2: fix incorrect frame length check commit 16a7bc3037d9b50a45d49a8d5a6afb510e1217d6 Author: Willy Tarreau Date: Tue Apr 17 10:28:27 2018 +0200 BUG/CRITICAL: h2: fix incorrect frame length check The incoming H2 frame length was checked against the max_frame_size setting instead of being checked against the bufsize. The max_frame_size only applies to outgoing traffic and not to incoming one, so if a large enough frame size is advertised in the SETTINGS frame, a wrapped frame will be defragmented into a temporary allocated buffer where the second fragment my overflow the heap by up to 16 kB. It is very unlikely that this can be exploited for code execution given that buffers are very short lived and their address not realistically predictable in production, but the likeliness of an immediate crash is absolutely certain. This fix must be backported to 1.8. Many thanks to Jordan Zebor from F5 Networks for reporting this issue in a responsible way. (cherry picked from commit 3f0e1ec70173593f4c2b3681b26c04a4ed5fc588) Signed-off-by: Willy Tarreau commit 555b8b76e81925c28f0bc1a65d5eb4838b71d9d3 Author: Willy Tarreau Date: Thu Apr 19 15:12:26 2018 +0200 DOC: lua: update the links to the config and Lua API The links were still stuck to version 1.6. Let's update them. The patch needs to be carefully backported to 1.8 and 1.7 after editing the respective version (replace 1.9dev with 1.8 or 1.7). (cherry picked from commit daac1e4c79f635272cfca371c21a739589596882) Signed-off-by: Willy Tarreau commit dd9749f5190100f5545c75df103519539c3bf87d Author: Aurélien Nephtali Date: Mon Apr 16 19:02:42 2018 +0200 MINOR: cli: Ensure the CLI always outputs an error when it should When using the CLI_ST_PRINT_FREE state, always output something back if the faulty function did not fill the 'err' variable. The map/acl code could lead to a crash whereas the SSL code was silently failing. Signed-off-by: Aurélien Nephtali (cherry picked from commit 9a4da683a6a0ce312c7e302245d6e327858d7ee8) Signed-off-by: Willy Tarreau commit 5cf39b385c079ccd81af3e16a84dcb2e509c773a Author: Aurélien Nephtali Date: Mon Apr 16 18:50:19 2018 +0200 BUG/MINOR: cli: Guard against NULL messages when using CLI_ST_PRINT_FREE Some error paths (especially those followed when running out of memory) can set the error message to NULL. In order to avoid a crash, use a generic message ("Out of memory") when this case arises. It should be backported to 1.8. Signed-off-by: Aurélien Nephtali (cherry picked from commit c511b7cc97482e7193aa19b8e571fa2ac851ea46) Signed-off-by: Willy Tarreau commit 70633ac0e10934e90e2317ebcc4977d2fa5d4b48 Author: Olivier Houchard Date: Mon Apr 16 13:24:48 2018 +0200 BUG/MEDIUM: kqueue: When adding new events, provide an output to get errors. When adding new events using kevent(), if there's an error, because we're trying to delete an event that wasn't there, or because the fd has already been closed, kevent() will either add an event in the eventlist array if there's enough room for it, and keep on handling other events, or stop and return -1. We want it to process all the events, so give it a large-enough array to store any error. Special thanks to PiBa-NL for diagnosing the root cause of this bug. This should be backported to 1.8. (cherry picked from commit ebaba754297f39b86959fbdc13f66e4534aadeae) Signed-off-by: Willy Tarreau commit 5fb409acaddfda2b13aa6e9e17a437dd687d8ca6 Author: Christopher Faulet Date: Fri Apr 13 15:53:12 2018 +0200 BUG/MINOR: http: Return an error in proxy mode when url2sa fails In proxy mode, the result of url2sa is never checked. So when the function fails to resolve the destination server from the URL, we continue. Depending on the internal state of the connection, we get different behaviours. With a newly allocated connection, the field is not set. So we will get a HTTP error. The status code is 503 instead of 400, but it's not really critical. But, if it's a recycled connection, we will reuse the previous value of , opening a connection on an unexpected server. To fix the bug, we return an error when url2sa fails. This patch should be backported in all version from 1.5. (cherry picked from commit 11ebb2080eb0394f0a1c2fc70f44da2b0107bb56) Signed-off-by: Willy Tarreau commit 478d0657ac5106c1c7d80435c0c721d03c4ad558 Author: Olivier Houchard Date: Fri Apr 13 15:50:27 2018 +0200 BUG/MEDIUM: connection: Make sure we have a mux before calling detach(). In some cases, we call cs_destroy() very early, so early the connection doesn't yet have a mux, so we can't call mux->detach(). In this case, just destroy the associated connection. This should be backported to 1.8. (cherry picked from commit 302f9ef0556b6795bda8882edf56611ee252fceb) Signed-off-by: Willy Tarreau commit 32822368854a29c465d5fdac16b61f44b9ec1eca Author: Christopher Faulet Date: Fri Apr 6 15:33:30 2018 +0200 BUG/MEDIUM: threads: Fix the max/min calculation because of name clashes With gcc < 4.7, when HAProxy is built with threads, the macros HA_ATOMIC_CAS/XCHG/STORE relies on the legacy __sync builtins. These macros are slightly complicated than the versions relying on the '_atomic' builtins. Internally, some local variables are defined, prefixed with '__' to avoid name clashes with the caller. On the other hand, the macros HA_ATOMIC_UPDATE_MIN/MAX call HA_ATOMIC_CAS. Some local variables are also definied in these macros, following the same naming rule as below. The problem is that '__new' variable is used in HA_ATOMIC_MIN/_MAX and in HA_ATOMIC_CAS. Obviously, the behaviour is undefined because '__new' in HA_ATOMIC_CAS is left uninitialized. Unfortunatly gcc fails to detect this error. To fix the problem, all internal variables to macros are now suffixed with name of the macros to avoid clashes (for instance, '__new_cas' in HA_ATOMIC_CAS). This patch must be backported in 1.8. (cherry picked from commit 48aa13f286dde83a88dd8459ee91ebee027c04a6) Signed-off-by: Willy Tarreau commit 02ff0174937c78e43197ec6390f45bcdf7a9d40f Author: Willy Tarreau Date: Sat Apr 7 00:30:56 2018 +0200 [RELEASE] Released version 1.8.7 Released version 1.8.7 with the following main changes : - BUG/MAJOR: cache: always initialize newly created objects - MINOR: servers: Support alphanumeric characters for the server templates names commit bd8c9fb7a40ef5f15ca69852066a1b3ff23dfb96 Author: Thierry Fournier Date: Mon Mar 26 11:54:39 2018 +0200 MINOR: servers: Support alphanumeric characters for the server templates names 'server-template' directive doesn't support the same name alphabet as the 'server' directive. This patch allows the usage of chars [0-9]. [wt: let's backport this to 1.8 to apply the principle of least surprize to people migrating to server templates] (cherry picked from commit f7b7c3e2f2d0bdf835a204cbccdff8005292ccaf) Signed-off-by: Willy Tarreau commit 1c3614dff8c22d94c0f24d4bfc9eb8ab357e6d6f Author: Willy Tarreau Date: Fri Apr 6 19:02:25 2018 +0200 BUG/MAJOR: cache: always initialize newly created objects Recent commit 5bd37fa ("BUG/MAJOR: cache: fix random crashes caused by incorrect delete() on non-first blocks") addressed an issue where dangling objects could be deleted in the cache, but even after this fix some similar segfaults were reported at the same place (cache_free_blocks()). The tree was always corrupted as well. Placing some traces revealed that this time it's caused by a missing initialization in http_action_store_cache() : while object->eb.key is used to note that the object is not in the tree, the first retrieved block may contain random data and is not initialized. Further, this entry can be updated later without the object being inserted into the tree. Thus, if at the end the object is not stored and the blocks are put back to the avail list, the next attempt to use them will find eb.key != 0 and will try to delete the uninitialized block, will see that eb.node.leaf_p is not NULL (random data), and will dereference it as well as a few other uninitialized pointers. It was harder to trigger than the previous one, despite being very closely related. This time the following config was used : listen l1 mode http bind :8888 http-request cache-use c1 http-response cache-store c1 server s1 127.0.0.1:8000 cache c1 total-max-size 4 max-age 10 Httpterm was running on port 8000. And it was stressed this way : $ inject -o 1 -u 500 -P 1 -G '127.0.0.1:8888/?s=4097&p=1&x=%s' ... wait 5 seconds then Ctrl-C ... # wait 3 seconds doing nothing $ inject -o 1 -u 500 -P 1 -G '127.0.0.1:8888/?s=4097&p=1&x=%s' => segfault Other values don't work well. The size and the small pieces in the responses (p=1) are critical to make it work. Here the fix consists in pre-zeroing object->eb.key AND object->eb.leaf_p just after the object is allocated so as to stay consistent with other locations. Ideally this could be simplified later by only relying on eb->node.leaf_p everywhere since in the end the key alone is not a reliable indicator, so that we use only one indicator of being part of the tree or not. This fix needs to be backported to 1.8. (cherry picked from commit 1093a4586c58f3d4eb327c768472990e8b95fd95) Signed-off-by: Willy Tarreau commit f673d031ef2431ab86b314bb24984be0d31dbe14 Author: Willy Tarreau Date: Thu Apr 5 15:35:44 2018 +0200 [RELEASE] Released version 1.8.6 Released version 1.8.6 with the following main changes : - BUG/MINOR: lua: the function returns anything - BUG/MINOR: lua funtion hlua_socket_settimeout don't check negative values - BUILD/MINOR: fix build when USE_THREAD is not defined - MINOR: cli/threads: make "show fd" report thread_sync_io_handler instead of "unknown" - MINOR: cli: make "show fd" report the mux and mux_ctx pointers when available - BUILD/MINOR: cli: fix a build warning introduced by last commit - BUG/MINOR: hpack: fix harmless use of uninitialized value in hpack_dht_insert - CLEANUP: h2: rename misleading h2c_stream_close() to h2s_close() - MINOR: h2: provide and use h2s_detach() and h2s_free() - BUG/MAJOR: h2: remove orphaned streams from the send list before closing - MINOR: h2: always call h2s_detach() in h2_detach() - MINOR: h2: fuse h2s_detach() and h2s_free() into h2s_destroy() - BUG/MEDIUM: h2/threads: never release the task outside of the task handler - BUG/MEDIUM: h2: don't consider pending data on detach if connection is in error - BUILD/MINOR: threads: always export thread_sync_io_handler() - BUG/MEDIUM: h2: always add a stream to the send or fctl list when blocked - BUG/MINOR: checks: check the conn_stream's readiness and not the connection - BUG/MINOR: email-alert: Set the mailer port during alert initialization - BUG/MINOR: cache: fix "show cache" output - BUG/MINOR: fd: Don't clear the update_mask in fd_insert. - BUG/MAJOR: cache: fix random crashes caused by incorrect delete() on non-first blocks - BUG/MINOR: spoe: Initialize variables used during conf parsing before any check - BUG/MINOR: spoe: Don't release the context buffer in .check_timeouts callbaclk commit ad706d0df7d75021e49bdc78c1d08522deb82ac3 Author: Christopher Faulet Date: Tue Mar 20 16:09:20 2018 +0100 BUG/MINOR: spoe: Don't release the context buffer in .check_timeouts callbaclk It is better to let spoe_stop_processing release this buffer because, in .check_timeouts callback, we lack information to know if it should be release or not. For instance, if the processing timeout is reached while the SPOE applet receives the reply, it is preferable to ignore the timeout and process the result. This patch should be backported in 1.8. (cherry picked from commit ac580608d712e1e8c351d84dfda6e5ac773812df) Signed-off-by: Willy Tarreau commit 18380151c1411702a3a90bcbf7c185fece560dc5 Author: Christopher Faulet Date: Fri Mar 23 14:37:14 2018 +0100 BUG/MINOR: spoe: Initialize variables used during conf parsing before any check Some initializations must be done at the beginning of parse_spoe_flt to avoid segmentaion fault when first errors are catched, when the "filter spoe" line is parsed. This patch must be backported in 1.8. [cf: the variable "curvars" doesn't exist in 1.8. So the patch must be adapted.] (cherry picked from commit 84c844eb12b250aa86f2aadaff77c42dfc3cb619) Signed-off-by: Willy Tarreau commit 1ad20b7953ee349ad665f702c5b8122dc912241d Author: Willy Tarreau Date: Wed Apr 4 20:17:03 2018 +0200 BUG/MAJOR: cache: fix random crashes caused by incorrect delete() on non-first blocks Several segfaults were reported in the cache, each time in eb_delete() called from cache_free_blocks() itself called from shctx_row_reserve_hot(). Each time the tree node was corrupted with random cached data (often JS or HTML contents). The problem comes from an incompatibility between the cache's expectations and the recycling algorithm used in the shctx. The shctx allocates and releases a chain of blocks at once. And when it needs to allocate N blocks from the avail list while a chain of M>N is found, it picks the first N from the list, moves them to the hot list, and marks all remaining M-N blocks as isolated blocks (chains of 1). For each such released block, the shctx->free_block() callback is used and passed a pointer to the first and current block of the chain. For the cache, it's cache_free_blocks(). What this function does is check that the current block is the first one, and in this case delete the object from the tree and mark it as not in tree by setting key to zero. The problem this causes is that the tail blocks when M>N become first blocks for the next call to shctx_row_reserve_hot(), these ones will be passed to cache_free_blocks() as list heads, and will be sent to eb_delete() despite containing only cached data. The simplest solution for now is to mark each block as holding no cache object by setting key to zero all the time. It keeps the principle used elsewhere in the code. The SSL code is not subject to this problem because it relies on the block's len not being null, which happens immediately after a block was released. It was uncertain however whether this method is suitable for the cache. It is not critical though since this code is going to change soon in 1.9 to dynamically allocate only the number of required blocks. This fix must be backported to 1.8. Thanks to Thierry for providing exploitable cores. (cherry picked from commit 5bd37fa625e95130b4f576983943ef9a4b04d73c) Signed-off-by: Willy Tarreau commit e86db14628c086920f1848b255b3bfcd477bf211 Author: Olivier Houchard Date: Tue Apr 3 19:06:18 2018 +0200 BUG/MINOR: fd: Don't clear the update_mask in fd_insert. Clearing the update_mask bit in fd_insert may lead to duplicate insertion of fd in fd_updt, that could lead to a write past the end of the array. Instead, make sure the update_mask bit is cleared by the pollers no matter what. This should be backported to 1.8. [wt: 1.8 doesn't have the lockless fdcache changes so here the fdtab.owner is tested under the lock. It also closes a very unlikely race that remained unnoticed since 1.8] (cherry picked from commit 8ef1a6b0d8c2ea1d8075e0b757007f05527952b8) Signed-off-by: Willy Tarreau commit 9fde6e23b7c2ad61c1f2bceb739766eed10edd27 Author: Willy Tarreau Date: Wed Apr 4 11:56:43 2018 +0200 BUG/MINOR: cache: fix "show cache" output The "show cache" command used to dump the header for each entry into into the handler loop, making it repeated every ~16kB of output data. Additionally chunk_appendf() was used instead of chunk_printf(), causing the output to repeat already emitted lines, and the output size to grow in O(n^2). It used to take several minutes to report tens of millions of objects from a small cache containing only a few thousands. There was no more impact though. This fix must be backported to 1.8. (cherry picked from commit afe1de5d98f2f4eb3d1fa495f090a1f99551d61a) Signed-off-by: Willy Tarreau commit 12db95e9962b2e2902a341e49754f27f20fe3ace Author: Christopher Faulet Date: Tue Mar 27 15:35:35 2018 +0200 BUG/MINOR: email-alert: Set the mailer port during alert initialization Since the commit 2f3a56b4f ("BUG/MINOR: tcp-check: use the server's service port as a fallback"), email alerts stopped working because the mailer's port was overriden by the server's port. Remember, email alerts are defined as checks with specific tcp-check rules and triggered on demand to send alerts. So to send an email, a check is executed. Because no specific port's was defined, the server's one was used. To fix the bug, the ports used for checks attached an email alert are explicitly set using the mailer's port. So this port will be used instead of the server's one. In this patch, the assignement to a default port (587) when an email alert is defined has been removed. Indeed, when a mailer is defined, the port must be defined. So the default port was never used. This patch must be backported in 1.8. (cherry picked from commit b797ae1f15b0fb97d1ae8550d37c3fd58a24f5b3) Signed-off-by: Willy Tarreau commit bee4105c3dfe0ff08eac482de5d2abf3bc6c3e70 Author: Willy Tarreau Date: Tue Apr 3 19:31:38 2018 +0200 BUG/MINOR: checks: check the conn_stream's readiness and not the connection Since commit 9aaf778 ("MAJOR: connection : Split struct connection into struct connection and struct conn_stream."), the checks use a conn_stream and not directly the connection anymore. However wake_srv_chk() still used to verify the connection's readiness instead of the conn_stream's. Due to the existence of a mux, the connection is always waiting for receiving something, and doesn't reflect the changes made in event_srv_chk_{r,w}(), causing the connection appear as not ready yet, and the check to be validated only after its timeout. The difference is only visible when sending pure TCP checks, and simply adding a "tcp-check connect" line is enough to work around it. This fix must be backported to 1.8. (cherry picked from commit 2500fc2c3443f69bc05de53d0516da373d2f0cc1) Signed-off-by: Willy Tarreau commit b33d95f93467e81ab16c345a9407f3c1db03b884 Author: Willy Tarreau Date: Fri Mar 30 17:35:38 2018 +0200 BUG/MEDIUM: h2: always add a stream to the send or fctl list when blocked When a stream blocks on a mux buffer full/unallocated or on connection flow control, a flag among H2_SF_MUX_M* is set, but the stream is not always added to the connection's list. It's properly done when the operations are performed from the connection handler but not always when done from the stream handler. For instance, a simple shutr or shutw may fail by lack of room. If it's immediately followed by a call to h2_detach(), the stream remains lying around in no list at all, and prevents the connection from ending. This problem is actually quite difficult to trigger and seems to require some large objects and low server-side timeouts. This patch covers all identified paths. Some are redundant but since the code will change and will be simplified in 1.9, it's better to stay on the safe side here for now. It must be backported to 1.8. (cherry picked from commit b2e290acb68fd5b7387ba74d0a601934e3a6f5a0) Signed-off-by: Willy Tarreau commit 5b18f6f882c9127bcd5b4af525fa69bf8d4f82c1 Author: Willy Tarreau Date: Thu Mar 29 18:54:33 2018 +0200 BUILD/MINOR: threads: always export thread_sync_io_handler() Otherwise it doesn't build again without threads. (cherry picked from commit e96e61cadc85b12e26663c1b62be8a1ae60b909f) commit 58cef63f20cc40248cd1cd113571cae588943d06 Author: Willy Tarreau Date: Thu Mar 29 15:41:32 2018 +0200 BUG/MEDIUM: h2: don't consider pending data on detach if connection is in error Interrupting an h2load test shows that some connections remain active till the client timeout. This is due to the fact that h2_detach() immediately returns if the h2s flags indicate that the h2s is still waiting for some buffer room in the output mux (possibly to emit a response or to send some window updates). If the connection is broken, these data will never leave and must not prevent the stream from being terminated nor the connection from being released. This fix must be backported to 1.8. (cherry picked from commit 3041fcc2fde3f3f33418c9f579b657d993b0006d) Signed-off-by: Willy Tarreau commit 4f2bd42ed3870dbaf143701f0cfbd64966d44252 Author: Willy Tarreau Date: Thu Mar 29 15:22:59 2018 +0200 BUG/MEDIUM: h2/threads: never release the task outside of the task handler Currently, h2_release() will release all resources assigned to the h2 connection, including the timeout task if any. But since the multi-threaded scheduler, the timeout task could very well be queued in the thread-local list of running tasks without any way to remove it, so task_delete() will have no effect and task_free() will cause this undefined object to be dereferenced. In order to prevent this from happening, we never release the task in h2_release(), instead we wake it up after marking its context NULL so that the task handler can release the task. Future improvements could consist in modifying the scheduler so that a task_wakeup() has to be done on any task having to be killed, letting the scheduler take care of it. This fix must be backported to 1.8. This bug was apparently not reported so far. (cherry picked from commit 0975f11d554baf30602ce4be3faf0b9741711a80) Signed-off-by: Willy Tarreau commit 5c9f9a6763177bc6bda9f20954d58135faea3075 Author: Willy Tarreau Date: Wed Mar 28 13:56:39 2018 +0200 MINOR: h2: fuse h2s_detach() and h2s_free() into h2s_destroy() Since these two functions are always used together, let's simplify the code by having a single one for both operations. It also ensures we don't leave wandering elements that risk to leak later. (cherry picked from commit 71049cce3f43a5a0c08af923f5fc9599dc0c453f) [wt: this is only needed for next patches] Signed-off-by: Willy Tarreau commit fc480c32e25ac9c6dd8abf48c346d3f74c9b90bf Author: Willy Tarreau Date: Wed Mar 28 13:51:45 2018 +0200 MINOR: h2: always call h2s_detach() in h2_detach() The code is safer and more robust this way, it avoids multiple paths. This is possible due to the idempotence of LIST_DEL() and eb32_delete() that are called in h2s_detach(). (cherry picked from commit e323f3458c89f025aa9658c63066a13b64c9150c) [wt: this is only needed for next patches] Signed-off-by: Willy Tarreau commit cf2ab4d22d977b172cf155e14060cf0f785f8404 Author: Willy Tarreau Date: Wed Mar 28 11:29:04 2018 +0200 BUG/MAJOR: h2: remove orphaned streams from the send list before closing Several people reported very strange occasional crashes when using H2. Every time it appeared that either an h2s or a task was corrupted. The outcome is that a missing LIST_DEL() when removing an orphaned stream from the list in h2_wake_some_streams() can cause this stream to remain present in the send list after it was freed. This may happen when receiving a GOAWAY frame for example. In the mean time the send list may be processed due to pending streams, and the just released stream is still found. If due to a buffer full condition we left the h2_process_demux() loop before being able to process the pending stream, the pool entry may be reassigned somewhere else. Either another h2 connection will get it, or a task, since they are the same size and are shared. Then upon next pass in h2_process_mux(), the stream is processed again. Either it crashes here due to modifications, or the contents are harmless to it and its last changes affect the other object reasigned to this area (typically a struct task). In the case of a collision with struct task, the LIST_DEL operation performed on h2s corrupts the task's wait queue's leaf_p pointer, thus all the wait queue's structure. The fix consists in always performing the LIST_DEL in h2s_detach(). It will also make h2s_stream_new() more robust against a possible future situation where stream_create_from_cs() could have sent data before failing. Many thanks to all the reporters who provided extremely valuable information, traces and/or cores, namely Thierry Fournier, Yves Lafon, Holger Amann, Peter Lindegaard Hansen, and discourse user "slawekc". This fix must be backported to 1.8. It is probably better to also backport the following code cleanups with it as well to limit the divergence between master and 1.8-stable : 00dd078 CLEANUP: h2: rename misleading h2c_stream_close() to h2s_close() 0a10de6 MINOR: h2: provide and use h2s_detach() and h2s_free() (cherry picked from commit 4a333d3d53af786fe09df2f83b4e5db38cfef004) Signed-off-by: Willy Tarreau commit 518db3f8602fae9caa816ec373855cf0f8c6c45d Author: Willy Tarreau Date: Thu Mar 1 16:27:53 2018 +0100 MINOR: h2: provide and use h2s_detach() and h2s_free() These ones save us from open-coding the cleanup functions on each and every error path. The code was updated to use them with no functional change. (cherry picked from commit 0a10de606685ed4e65d4cc84237c6a09dd6fe27c) Signed-off-by: Willy Tarreau commit 27b2c5ead5cf85626d4169ab46b3246d65033b58 Author: Willy Tarreau Date: Thu Mar 1 16:31:34 2018 +0100 CLEANUP: h2: rename misleading h2c_stream_close() to h2s_close() This function takes an h2c and an h2s but it never uses the h2c, which is a bit confusing at some places in the code. Let's make it clear that it only operates on the h2s instead by renaming it and removing the unused h2c argument. (cherry picked from commit 00dd07895a6ee856c811c6d60a8e3d4c7d973c63) Signed-off-by: Willy Tarreau commit 329da69f6082297aa1b58702b2ac146fca0cb104 Author: Willy Tarreau Date: Tue Mar 27 15:06:02 2018 +0200 BUG/MINOR: hpack: fix harmless use of uninitialized value in hpack_dht_insert A warning is reported here by valgrind on first pass in hpack_dht_insert(). The cause is that the not-yet-initialized dht->head is checked in hpack_dht_get_tail(), though the result is not used, making it have no impact. At the very least it confuses valgrind, and maybe it makes it harder for gcc to optimize the code path. Let's move the variable initialization around to shut it up. Thanks to Olivier for reporting this one. This fix may be backported to 1.8 at least to make valgrind usage less painful. (cherry picked from commit a7394e1b72706abcd5bdda254dfd86877fee400b) Signed-off-by: Willy Tarreau commit f9580c7f9f5c7d401ab6bc953e162eea3448dd15 Author: Willy Tarreau Date: Thu Mar 29 13:19:37 2018 +0200 BUILD/MINOR: cli: fix a build warning introduced by last commit Commit 35b1b48 ("MINOR: cli: make "show fd" report the mux and mux_ctx pointers when available") introduced an accidental build warning due to a missing const statement. Signed-off-by: Willy Tarreau commit 5e11cd160dd0559d9dfa636276ce1a5c7a8f6604 Author: Willy Tarreau Date: Wed Mar 28 18:41:30 2018 +0200 MINOR: cli: make "show fd" report the mux and mux_ctx pointers when available This is handy to quickly distinguish H2 connections as well as to easily access the h2c context. It could be backported to 1.8 to help during troubleshooting sessions. (cherry picked from commit 35b1b48c75d0589a289a15f14bf6dfa89adf6073) Signed-off-by: Willy Tarreau commit d53922fc2b6e9e78599325558b10143d46621d47 Author: Willy Tarreau Date: Wed Mar 28 18:06:47 2018 +0200 MINOR: cli/threads: make "show fd" report thread_sync_io_handler instead of "unknown" The output was confusing when the sync point's dummy handler was shown. This patch should be backported to 1.8 to help with troubleshooting. (cherry picked from commit 4037a3f904b62b32fa3c298cf40e4de54fc83585) Signed-off-by: Willy Tarreau commit 76ac73d679a94e7864a0e2ec47194edb2dc1879d Author: Ilya Shipitsin Date: Sat Mar 24 17:17:32 2018 +0500 BUILD/MINOR: fix build when USE_THREAD is not defined src/queue.o: In function `pendconn_redistribute': /home/ilia/haproxy/src/queue.c:272: undefined reference to `thread_want_sync' src/queue.o: In function `pendconn_grab_from_px': /home/ilia/haproxy/src/queue.c:311: undefined reference to `thread_want_sync' src/queue.o: In function `process_srv_queue': /home/ilia/haproxy/src/queue.c:184: undefined reference to `thread_want_sync' collect2: error: ld returned 1 exit status make: *** [Makefile:900: haproxy] Error 1 To be backported to 1.8. (cherry picked from commit 7741c854cd908dd4947325c36a6feb8203748d16) Signed-off-by: Willy Tarreau commit e0a0443e70c2fe26286cda54652849f0aa785bc1 Author: Thierry Fournier Date: Thu Mar 8 09:59:02 2018 +0100 BUG/MINOR: lua funtion hlua_socket_settimeout don't check negative values Negatives timeouts doesn't have sense. A negative timeout doesn't cause a crash, but the connection expires before the system try to extablish it. This patch should be backported in all versions from 1.6 (cherry picked from commit 17a921b799631661803065f1873326b522e6bc3a) Signed-off-by: Willy Tarreau commit c9d4464e74ac9f4cab9a20be179f14a27f5bd989 Author: Thierry Fournier Date: Thu Mar 8 09:54:32 2018 +0100 BUG/MINOR: lua: the function returns anything The output of these function indicates that one element is pushed in the stack, but no element is set in the stack. Actually, if anyone read the value returned by this function, is gets "something" present in the stack. This patch is a complement of these one: 119a5f10e47f3507e58116 The LuaSocket documentation tell anything about the returned value, but the effective code set an integer of value one. https://github.com/diegonehab/luasocket/blob/316a9455b9cb4637fe6e62b20fbe05f5141fec54/src/timeout.c#L172 Thanks to Tim for the bug report. This patch should be backported in all version from 1.6 (cherry picked from commit e9636f192a5c312424b7598ca182b0425827b08b) Signed-off-by: Willy Tarreau commit 9a083d1428b655c0079b4355d764cc08d66757f2 Author: Willy Tarreau Date: Fri Mar 23 16:58:34 2018 +0100 [RELEASE] Released version 1.8.5 Released version 1.8.5 with the following main changes : - BUG/MINOR: threads: fix missing thread lock labels for 1.8 - BUG/MEDIUM: ssl: Don't always treat SSL_ERROR_SYSCALL as unrecovarable. - BUG/MEDIUM: ssl: Shutdown the connection for reading on SSL_ERROR_SYSCALL - BUG/MINOR: init: Add missing brackets in the code parsing -sf/-st - BUG/MINOR: ssl/threads: Make management of the TLS ticket keys files thread-safe - BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as earlier as possible - BUG/MEDIUM: ssl/sample: ssl_bc_* fetch keywords are broken. - DOC: lua: new prototype for function "register_action()" - DOC: cfgparse: Warn on option (tcp|http)log in backend - BUG/MINOR: debug/pools: properly handle out-of-memory when building with DEBUG_UAF - MINOR: debug/pools: make DEBUG_UAF also detect underflows - BUG/MINOR: h2: Set the target of dbuf_wait to h2c - MINOR: stats: display the number of threads in the statistics. - BUG/MEDIUM: h2: always consume any trailing data after end of output buffers - BUG/MEDIUM: buffer: Fix the wrapping case in bo_putblk - BUG/MEDIUM: buffer: Fix the wrapping case in bi_putblk - Revert "BUG/MINOR: send-proxy-v2: string size must include ('\0')" - MINOR: systemd: Add section for SystemD sandboxing to unit file - MINOR: systemd: Add SystemD's Protect*= options to the unit file - MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file - MINOR/BUILD: fix Lua build on Mac OS X - BUILD/MINOR: fix Lua build on Mac OS X (again) - BUG/MINOR: session: Fix tcp-request session failure if handshake. - CLEANUP: .gitignore: Ignore binaries from the contrib directory - BUG/MINOR: unix: Don't mess up when removing the socket from the xfer_sock_list. - BUG/MEDIUM: h2: also arm the h2 timeout when sending - BUG/MINOR: cli: Fix a crash when passing a negative or too large value to "show fd" - CLEANUP: ssl: Remove a duplicated #include - CLEANUP: cli: Remove a leftover debug message - BUG/MINOR: cli: Fix a typo in the 'set rate-limit' usage - BUG/MEDIUM: fix a 100% cpu usage with cpu-map and nbthread/nbproc - BUG/MINOR: force-persist and ignore-persist only apply to backends - BUG/MEDIUM: spoe: Remove idle applets from idle list when HAProxy is stopping - BUG/MEDIUM: threads/unix: Fix a deadlock when a listener is temporarily disabled - BUG/MAJOR: threads/queue: Fix thread-safety issues on the queues management - BUG/MINOR: dns: don't downgrade DNS accepted payload size automatically - BUG/MINOR: seemless reload: Fix crash when an interface is specified. - BUG/MINOR: cli: Fix a crash when sending a command with too many arguments - BUILD: ssl: Fix build with OpenSSL without NPN capability - BUG/MINOR: spoa-example: unexpected behavior for more than 127 args - BUG/MINOR: lua: return bad error messages - BUG/MEDIUM: tcp-check: single connect rule can't detect DOWN servers - BUG/MINOR: tcp-check: use the server's service port as a fallback - BUG/MEDIUM: threads/queue: wake up other threads upon dequeue - MINOR: log: stop emitting alerts when it's not possible to write on the socket - BUILD/BUG: enable -fno-strict-overflow by default - DOC: log: more than 2 log servers are allowed - DOC: don't suggest using http-server-close - BUG/MEDIUM: h2: properly account for DATA padding in flow control - BUG/MINOR: h2: ensure we can never send an RST_STREAM in response to an RST_STREAM - BUG/MINOR: listener: Don't decrease actconn twice when a new session is rejected commit f4bae5e29b1df8a6c0ae4ea657f8bf6ba3160423 Author: Christopher Faulet Date: Fri Mar 23 15:11:55 2018 +0100 BUG/MINOR: listener: Don't decrease actconn twice when a new session is rejected When a freshly created session is rejected, for any reason, during the accept in the function "session_accept_fd", the variable "actconn" is decreased twice. The first time when the rejected session is released, then in the function "listener_accpect", because of the failure. So it is possible to have an negative value for actconn. Note that, in this case, we will also have a negatve value for the current number of connections on the listener rejecting the session (actconn and l->nbconn are in/decreased in same time). It is easy to reproduce the bug with this small configuration: global stats socket /tmp/haproxy listen test bind *:12345 tcp-request connection reject if TRUE A "show info" on the stat socket, after a connection attempt, will show a very high value (the unsigned representation of -1). To fix the bug, if the function "session_accept_fd" returns an error, it decrements the right counters and "listener_accpect" leaves them untouched. This patch must be backported in 1.8. (cherry picked from commit fe234281d6cf0de8d16a630092a89c25b9fd27f7) Signed-off-by: Willy Tarreau commit 253c6a5d4cfee6a311ce551e8faaa14b1e48ffc1 Author: Willy Tarreau Date: Thu Mar 22 17:37:05 2018 +0100 BUG/MINOR: h2: ensure we can never send an RST_STREAM in response to an RST_STREAM There are some corner cases where this could happen by accident. Since the spec explicitly forbids this (RFC7540#5.4.2), let's add a test in the two only functions which make the RST to avoid this. Thanks to user klzgrad for reporting this problem. Usually it is expected to be harmless but may result in browsers issuing a warning. This fix must be backported to 1.8. (cherry picked from commit 8adae7c15ffb5df8fcbb41496735fef25d6c4df3) Signed-off-by: Willy Tarreau commit da9e9ecf2c082564296589d1e34e9ffc5505f232 Author: Willy Tarreau Date: Thu Mar 22 16:53:12 2018 +0100 BUG/MEDIUM: h2: properly account for DATA padding in flow control Recent fixes made to process partial frames broke the flow control on DATA frames, as the padding is not considered anymore, only the actual data is. Let's simply take account of the padding once the transfer ends. The probability to meet this bug is low because, when used, padding is small and it can require a large number of padded transfers before the window is completely depleted. Thanks to user klzgrad for reporting this bug and confirming the fix. This fix must be backported to 1.8. (cherry picked from commit d1023bbab3985091a9968192bb984d7724a438a3) Signed-off-by: Willy Tarreau commit 5d5caf6c1332a694fb03d0f674b9301f91e06c23 Author: Lukas Tribus Date: Thu Feb 1 23:58:59 2018 +0100 DOC: don't suggest using http-server-close Remove the old suggestion to use http-server-close mode, from the beginnings of keep-alive mode in commit 16bfb021 "MINOR: config: add option http-keep-alive"). We made http-keep-alive default in commit 70dffdaa "MAJOR: http: switch to keep-alive mode by default". (cherry picked from commit 6fd698f4e6410177284ce70926f5c1137a176e6a) Signed-off-by: Willy Tarreau commit 407c97df2146816613803ae215c39089b3852061 Author: Cyril Bonté Date: Tue Mar 20 23:30:27 2018 +0100 DOC: log: more than 2 log servers are allowed Since commit 0f99e3497, loggers are not limited to 2 instances anymore. (cherry picked from commit 3e9548777e72ffc60a029d54673fd21c7178aec8) Signed-off-by: Willy Tarreau commit 992a3815fbcf3bfa9376ae0074e6047261313d46 Author: Willy Tarreau Date: Tue Mar 20 16:46:46 2018 +0100 BUILD/BUG: enable -fno-strict-overflow by default Some time ago, integer overflows detection stopped working in the timer code on recent compliers and were addressed by commit 73bdb32 ("BUG/MAJOR: Use -fwrapv."). By then it was thought that -fno-strict-overflow was not needed as implied, but it resulted from a misinterpretation of the doc, as this one is still needed to disable pointer overflow optimization that is automatically enabled at -O2/-O3/-Os. Unfortunately the compiler happily removes overflow checks without the slightest warning so it's not trivial to guess the extent of this issue without comparing the emitted asm code. By checking the emitted assembly code with and without the option, it was found that the only affected location was the reported one, in ssl_sock_parse_clienthello(), where the test can never fail on any system where the highest userland pointer is at least 64kB away from wrapping (ie all 32/64 bit OS in field), so there it is harmless. This patch must be backported to all maintained versions. Special thanks to Ilya Shipitsin for reporting this issue. (cherry picked from commit cde05c85ef3da3c54b79d7ad3570d0e13ebd620c) Signed-off-by: Willy Tarreau commit 519375417ad9f2983fabcc94eb788dc562e18946 Author: Willy Tarreau Date: Tue Mar 20 11:17:29 2018 +0100 MINOR: log: stop emitting alerts when it's not possible to write on the socket This is a recurring pain when using certain unix domain sockets or when sending to temporarily unroutable addresses, if the process remains in the foreground, the console is full of error which it's impossible to do anything about. It's even worse when the process is remote, or when run from a serial console which will slow the whole process down. Let's send them only once now to warn about a possible config issue, and not pollute the system nor slow everything down. (cherry picked from commit c98aebcdb8f1d21a1a3501bab91e1d702171395e) Signed-off-by: Willy Tarreau commit 6144927128cba8fbba090b9e3faeac6dbf66610c Author: Christopher Faulet Date: Mon Mar 19 15:22:09 2018 +0100 BUG/MEDIUM: threads/queue: wake up other threads upon dequeue The previous patch about queues (5cd4bbd7a "BUG/MAJOR: threads/queue: Fix thread-safety issues on the queues management") revealed a performance drop when multithreading is enabled (nbthread > 1). This happens when pending connections handled by other theads are dequeued. If these other threads are blocked in the poller, we have to wait the poller's timeout (or any I/O event) to process the dequeued connections. To fix the problem, at least temporarly, we "wake up" the threads by requesting a synchronization. This may seem a bit overkill to use the sync point to do a wakeup on threads, but it fixes this performance issue. So we can now think calmly on the good way to address this kind of issues. This patch should be backported in 1.8 with the commit 5cd4bbd7a ("BUG/MAJOR: threads/queue: Fix thread-safety issues on the queues management"). (cherry picked from commit fd83f0bfa414ba7463450b99f86858cf33fa285b) Signed-off-by: Willy Tarreau commit 113bf133502e06e249d8006b4e4bc9cfdb985888 Author: Baptiste Assmann Date: Mon Mar 19 12:22:41 2018 +0100 BUG/MINOR: tcp-check: use the server's service port as a fallback When running tcp-check scripts, one must ensure we can establish a tcp connection first. When doing this action, HAProxy needs a TCP port configured either on the server or on the check itself or on the connect rule itself. For some reasons, the connect code did not evaluate the service port on the server structure... this patch fixes this error. Backport status: 1.8 (cherry picked from commit 2f3a56b4ff207de37bdd944993a88bca781ad243) Signed-off-by: Willy Tarreau commit 69ea5b29b63c3d891844e94b37d8cca94d04dedf Author: Baptiste Assmann Date: Thu Mar 1 21:49:01 2018 +0100 BUG/MEDIUM: tcp-check: single connect rule can't detect DOWN servers When tcpcheck is used to do TCP port monitoring only and the script is composed by a single "tcp-check connect" rule (whatever port and ssl options enabled), then the server can't be seen as DOWN. Simple configuration to reproduce: backend b [...] option tcp-check tcp-check connect server s1 127.0.0.1:22 check The main reason for this issue is that the piece of code which validates that we're not at the end of the chained list (of rules) prevents executing the validation of the establishment of the TCP connection. Since validation is not executed, the rule is terminated and the report says no errors were encountered, hence the server is UP all the time. The workaround is simple: move the connection validation outsied the CONNECT rule processing loop, into the main function. That way, if the connection status is not CONNECTED, then HAProxy will now add more time to wait for it. If the time is expired, an error is now well reported. Backport status: 1.8 (cherry picked from commit 248f1173f272b10bf7ed2aa7061bfafd34160f70) Signed-off-by: Willy Tarreau commit fbe0afa1ea285f95b318071a39e5a69284cd125d Author: Thierry FOURNIER Date: Fri Feb 23 18:41:18 2018 +0100 BUG/MINOR: lua: return bad error messages The returned type is the type of the top of stack value and not the type of the checked argument. [wt: this can be backported to 1.8, 1.7 and 1.6] (cherry picked from commit fd1e955a564f88226c335477d706dd94f25c8081) Signed-off-by: Willy Tarreau commit 1b806013f69bf3ccfebd49ff9ff4566b277688ec Author: Thierry FOURNIER Date: Sun Feb 25 21:33:38 2018 +0100 BUG/MINOR: spoa-example: unexpected behavior for more than 127 args Buf is unsigned, so nbargs will be negative for more then 127 args. Note that I cant test this bug because I cant put sufficient args on the configuration line. It is just detected reading code. [wt: this can be backported to 1.8 & 1.7] (cherry picked from commit 29a05c13d1f43e4f05b4211a9ccc65205b2e1cf0) Signed-off-by: Willy Tarreau commit e8e1cf4b989c19bcc93cbfba908c92c52337d51a Author: Bernard Spil Date: Thu Feb 15 13:34:58 2018 +0100 BUILD: ssl: Fix build with OpenSSL without NPN capability OpenSSL can be built without NEXTPROTONEG support by passing -no-npn to the configure script. This sets the OPENSSL_NO_NEXTPROTONEG flag in opensslconf.h Since NEXTPROTONEG is now considered deprecated, it is superseeded by ALPN (Application Layer Protocol Next), HAProxy should allow building withough NPN support. (cherry picked from commit 13c53f8cc2a4d18eed8ad086465fbf668901e63d) Signed-off-by: Willy Tarreau commit 439ff9feb3a103461dcc29d11c11dbdd0ac814cb Author: Aurélien Nephtali Date: Fri Mar 16 10:11:06 2018 +0100 BUG/MINOR: cli: Fix a crash when sending a command with too many arguments This bug was introduced in 48bcfdab2 ("MEDIUM: dumpstat: make the CLI parser understand the backslash as an escape char"). This should be backported to 1.8. Signed-off-by: Aurélien Nephtali (cherry picked from commit 6a61e968ac43345ecc7d53b1ccaea77fdb5ad042) Signed-off-by: Willy Tarreau commit 55b0b2197547dafcf1513bc4f6e37b48e532c1b3 Author: Olivier Houchard Date: Thu Mar 15 17:48:49 2018 +0100 BUG/MINOR: seemless reload: Fix crash when an interface is specified. When doing a seemless reload, while receiving the sockets from the old process the new process will die if the socket has been bound to a specific interface. This happens because the code that tries to parse the informations bogusly try to set xfer_sock->namespace, while it should be setting wfer_sock->iface. This should be backported to 1.8. (cherry picked from commit 33e083c92e883963beed72ceaea8d12562a7a7ff) Signed-off-by: Willy Tarreau commit 1269501a118272cdf898f060cefb782b8d7ab058 Author: Baptiste Assmann Date: Thu Mar 1 22:26:57 2018 +0100 BUG/MINOR: dns: don't downgrade DNS accepted payload size automatically Automatic downgrade of DNS accepted payload size may have undesired side effect, which could make a backend with all servers DOWN. After talking with Lukas on the ML, I realized this "feature" introduces more issues that it fixes problem. The "best" way to handle properly big responses will be to implement DNS over TCP. To be backported to 1.8. (cherry picked from commit 1fa7d2acce0fcbb4f8aec0199e932556a80e5bc7) Signed-off-by: Willy Tarreau commit 7679698617a6e7ba4f5a6b30d62587337488fb1c Author: Christopher Faulet Date: Wed Mar 14 16:18:06 2018 +0100 BUG/MAJOR: threads/queue: Fix thread-safety issues on the queues management The management of the servers and the proxies queues was not thread-safe at all. First, the accesses to ->pend_pos were not protected. So it was possible to release it on a thread (for instance because the stream is released) and to use it in same time on another one (because we redispatch pending connections for a server). Then, the accesses to stream's information (flags and target) from anywhere is forbidden. To be safe, The stream's state must always be updated in the context of process_stream. So to fix these issues, the queue module has been refactored. A lock has been added in the pendconn structure. And now, when we try to dequeue a pending connection, we start by unlinking it from the server/proxy queue and we wake up the stream. Then, it is the stream reponsibility to really dequeue it (or release it). This way, we are sure that only the stream can create and release its field. However, be careful. This new implementation should be thread-safe (hopefully...). But it is not optimal and in some situations, it could be really slower in multi-threaded mode than in single-threaded one. The problem is that, when we try to dequeue pending connections, we process it from the older one to the newer one independently to the thread's affinity. So we need to wait the other threads' wakeup to really process them. If threads are blocked in the poller, this will add a significant latency. This problem happens when maxconn values are very low. This patch must be backported in 1.8. (cherry picked from commit 5cd4bbd7ab06e341a29a4d2135e80d2e6cf3834a) Signed-off-by: Willy Tarreau commit ae31a2961e63c838e95e2e34026ec17a9ccefa6b Author: Christopher Faulet Date: Fri Mar 16 10:04:47 2018 +0100 BUG/MEDIUM: threads/unix: Fix a deadlock when a listener is temporarily disabled When a listener is temporarily disabled, we start by locking it and then we call .pause callback of the underlying protocol (tcp/unix). For TCP listeners, this is not a problem. But listeners bound on an unix socket are in fact closed instead. So .pause callback relies on unbind_listener function to do its job. Unfortunatly, unbind_listener hold the listener's lock and then call an internal function to unbind it. So, there is a deadlock here. This happens during a reload. To fix the problemn, the function do_unbind_listener, which is lockless, is now exported and is called when a listener bound on an unix socket is temporarily disabled. This patch must be backported in 1.8. (cherry picked from commit 510c0d67ef8c44172b63be1a3d69be5f03ef14c3) Signed-off-by: Willy Tarreau commit 7034083b5063d28276b986d645d18071aba5f4d5 Author: Christopher Faulet Date: Wed Feb 28 13:33:26 2018 +0100 BUG/MEDIUM: spoe: Remove idle applets from idle list when HAProxy is stopping In the SPOE applet's handler, when an applet is switched from the state IDLE to PROCESSING, it is removed for the list of idle applets. But when HAProxy is stopping, this applet can be switched to DISCONNECT. In this case, we also need to remove it from the list of idle applets. Else the applet is removed but still present in the list. It could lead to a segmentation fault or an infinite loop, depending the code path. (cherry picked from commit 7d9f1ba246055046eed547fa35aa546683021dce) [wt: adapted context for 1.8] Signed-off-by: Willy Tarreau commit 069e6bbe3f11fe52f429e65c222d9d24d6c612e5 Author: Cyril Bonté Date: Mon Mar 12 22:02:59 2018 +0100 BUG/MINOR: force-persist and ignore-persist only apply to backends >From the very first day of force-persist and ignore-persist features, they only applied to backends, except that the documentation stated it could also be applied to frontends. In order to make it clear, the documentation is updated and the parser will raise a warning if the keywords are used in a frontend section. This patch should be backported up to the 1.5 branch. (cherry picked from commit 4288c5a9d81f89f65c02574a434b02e98eea943c) Signed-off-by: Willy Tarreau commit 5149cd3c7abad68ddb19a0a5b3b604786d5f1b95 Author: Cyril Bonté Date: Mon Mar 12 21:47:39 2018 +0100 BUG/MEDIUM: fix a 100% cpu usage with cpu-map and nbthread/nbproc Krishna Kumar reported a 100% cpu usage with a configuration using cpu-map and a high number of threads, Indeed, this minimal configuration to reproduce the issue : global nbthread 40 cpu-map auto:1/1-40 0-39 frontend test bind :8000 This is due to a wrong type in a shift operator (int vs unsigned long int), causing an endless loop while applying the cpu affinity on threads. The same issue may also occur with nbproc under FreeBSD. This commit addresses both cases. This patch must be backported to 1.8. (cherry picked from commit d400ab3a369523538c426cb70e059954c76b69c3) Signed-off-by: Willy Tarreau commit f4b00e8416e1981a9100b4904778184ba5ad46ae Author: Aurélien Nephtali Date: Sun Mar 11 16:55:02 2018 +0100 BUG/MINOR: cli: Fix a typo in the 'set rate-limit' usage The correct keyword is 'ssl-sessions' (vs. 'ssl-session'). The typo was introduced in 45c742be05 ('REORG: cli: move the "set rate-limit" functions to their own parser'). Signed-off-by: Aurélien Nephtali (cherry picked from commit b53e20826e8e12a74a73634e7986a06e3c712deb) Signed-off-by: Willy Tarreau commit 303661f72ee2058fbd760b5a82b7a9a695e2f5b4 Author: Aurélien Nephtali Date: Sat Mar 10 20:59:56 2018 +0100 CLEANUP: cli: Remove a leftover debug message This printf() was added in f886e3478d ("MINOR: cli: Add a command to send listening sockets."). Signed-off-by: Aurélien Nephtali (cherry picked from commit bca08762d2f4813ec0db2af69f269e75fcce706d) Signed-off-by: Willy Tarreau commit 55b77ded53ce89fa14356cafd4f71256880b0869 Author: Aurélien Nephtali Date: Tue Feb 20 19:23:07 2018 +0100 CLEANUP: ssl: Remove a duplicated #include openssl/x509.h is included twice since commit fc0421fde ("MEDIUM: ssl: add support for SNI and wildcard certificates"). Signed-off-by: Aurélien Nephtali (cherry picked from commit 76de95a4c0355ae72eb69b04fcf8ce025d3dd703) Signed-off-by: Willy Tarreau commit c770131f579ef694983625e746c04007b9696d58 Author: Aurélien Nephtali Date: Fri Mar 9 18:51:16 2018 +0100 BUG/MINOR: cli: Fix a crash when passing a negative or too large value to "show fd" This bug is present since 7a4a0ac71d ("MINOR: cli: add a new "show fd" command"). This should be backported to 1.8. Signed-off-by: Aurélien Nephtali (cherry picked from commit 498a1157279c3dcd5d83d0fcb80eecd7f108d028) Signed-off-by: Willy Tarreau commit ccfb5d755f1708f890b197375d962d8c938e78bd Author: Willy Tarreau Date: Mon Mar 5 16:10:54 2018 +0100 BUG/MEDIUM: h2: also arm the h2 timeout when sending Right now the h2 idle timeout is only set when there is no stream. If we fail to send because the socket buffers are full (generally indicating the client has left), we also need to arm it so that we can properly expire such connections, otherwise some failed transfers might leave H2 connections pending forever. Thanks to Thierry Fournier for the diag and the traces. This patch needs to be backported to 1.8. (cherry picked from commit 84b118f3120b3c61156f0ada12ae6456bd1a0b5a) Signed-off-by: Willy Tarreau commit 602383576e6420ef6a228feb451ebe2e19465f1b Author: Olivier Houchard Date: Thu Mar 8 18:25:49 2018 +0100 BUG/MINOR: unix: Don't mess up when removing the socket from the xfer_sock_list. When removing the socket from the xfer_sock_list, we want to set next->prev to prev, not to next->prev, which is useless. This should be backported to 1.8. (cherry picked from commit ec9516a6dc6df12299f94cea3c20b8333cac569f) Signed-off-by: Willy Tarreau commit 29dbaa7e6c9f95e85dc09e6324657e5ba87af05e Author: Christopher Faulet Date: Wed Feb 28 11:22:15 2018 +0100 CLEANUP: .gitignore: Ignore binaries from the contrib directory Some binaries were not ignored and polluted the "git status" output. (cherry picked from commit f9f6ed0a518c05902985164dd5f4b99c91f6c384) Signed-off-by: Willy Tarreau commit 2a2110b34550376c794c75a514de6752313dee18 Author: Emeric Brun Date: Mon Mar 5 17:46:16 2018 +0100 BUG/MINOR: session: Fix tcp-request session failure if handshake. Some sample fetches check if session is established using the flag CO_FL_CONNECTED. But in some cases, when a handshake is performed this flag is set too late, after the process of the tcp-request session rules. This fix move the raising of the flag at the beginning of the conn_complete_session function which processes the tcp-request session rules. This fix must be backported to 1.8 (and perhaps 1.7) (cherry picked from commit 1738e86771cce2567483cde0ac23459be8e51ed8) Signed-off-by: Willy Tarreau commit 366af2f5ead73f1b1ee049a66be07d4ac1adc8c0 Author: Willy Tarreau Date: Mon Mar 5 15:37:04 2018 +0100 BUILD/MINOR: fix Lua build on Mac OS X (again) Previous commit (13113d6 "MINOR/BUILD: fix Lua build on Mac OS X") contains a typo, it uses "-export-dynamic" instead of "-export_dynamic" (dash instead of underscore), despite what the commit message suggests, and it obviously doesn't work. Thanks to Kirill A. Korinsky for reporting it. This patch should be backported on each version from 1.6 like the aforementionned one above. (cherry picked from commit b684e7a52cd3733bce2404bcac1b26ea92c7f38d) Signed-off-by: Willy Tarreau commit 5b2e254d8349ce00657a8527e4296b0cfd5a30e3 Author: Thierry Fournier Date: Mon Feb 26 09:25:56 2018 +0100 MINOR/BUILD: fix Lua build on Mac OS X Change gcc option syntax for Mac. -Wl,--export-dynamic is not supported, use -Wl,-export_dynamic. Thanks to Kirill A. Korinsky for the report. This patch should be backported on each version from 1.6 (cherry picked from commit 13113d6abbbe0662d1de0b842bbc4fa8d1907890) Signed-off-by: Willy Tarreau commit 6dc3f846bbef0e0ea320f4279c4d6ae13201f02f Author: Tim Duesterhus Date: Tue Feb 27 20:19:05 2018 +0100 MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file This option takes away system calls that are unneeded for haproxy's operation and thus is a good defense in depth measure. (cherry picked from commit 2788a39c07621e7af0d2efa34b4adabe8a01ad31) Signed-off-by: Willy Tarreau commit 97a97a01bbcb833aee94c76c8d097a4959658b42 Author: Tim Duesterhus Date: Tue Feb 27 20:19:04 2018 +0100 MINOR: systemd: Add SystemD's Protect*= options to the unit file While the haproxy workers usually are running chrooted the master process is not. This patch is a pretty safe defense in depth measure to ensure haproxy cannot touch sensitive parts of the file system. ProtectSystem takes non-boolean arguments in newer SystemD versions, but setting those would leave older systems such as Ubuntu Xenial unprotected. Distro maintainers and system administrators could adapt the ProtectSystem value to the SystemD version they ship. (cherry picked from commit 8a9659212e4d491e4195de72aee4149db6359e09) Signed-off-by: Willy Tarreau commit 593fe17ce417ef001e625b16cdf92968c508f812 Author: Tim Duesterhus Date: Tue Feb 27 20:19:03 2018 +0100 MINOR: systemd: Add section for SystemD sandboxing to unit file This commit adds a warning for settings that possibly provide better sandboxing and explains their tradeoffs. (cherry picked from commit 1ce8de2d93066d29e57ca2bb9cc0e0ff321f2043) Signed-off-by: Willy Tarreau commit dd210612da664b9f95ab5c79711d554a2a7e9ec9 Author: Emmanuel Hocdet Date: Wed Feb 28 12:02:14 2018 +0100 Revert "BUG/MINOR: send-proxy-v2: string size must include ('\0')" This reverts commit 82913e4f79a1f1fb25aec84a2ce2f5f0e5ce1959. TLV string value should not be null-terminated. This should be backported to 1.8. (cherry picked from commit 8c0c34b6e731e3163c0644f50863deb9b7634fdc) Signed-off-by: Willy Tarreau commit 14f325000b91649b9d117c4d53d6b194ed3c7b11 Author: Christopher Faulet Date: Mon Feb 26 10:51:28 2018 +0100 BUG/MEDIUM: buffer: Fix the wrapping case in bi_putblk When the block of data need to be split to support the wrapping, the start of the second block of data was wrong. We must be sure to skup data copied during the first memcpy. This patch must be backported to 1.8. (cherry picked from commit ca6ef506610e9d78f99b7ab2095ce0f8a47e18df) Signed-off-by: Willy Tarreau commit fefb8592821ff0fa56f435c581d6e92e563e7ad7 Author: Christopher Faulet Date: Mon Feb 26 10:47:03 2018 +0100 BUG/MEDIUM: buffer: Fix the wrapping case in bo_putblk When the block of data need to be split to support the wrapping, the start of the second block of data was wrong. We must be sure to skip data copied during the first memcpy. This patch must be backported to 1.8, 1.7, 1.6 and 1.5. (cherry picked from commit b2b279464c5c0f3dfadf02333e06eb0ae8ae8793) Signed-off-by: Willy Tarreau commit 6fc36785addd45cc76a029a023296def53cff135 Author: Willy Tarreau Date: Tue Feb 27 15:37:25 2018 +0100 BUG/MEDIUM: h2: always consume any trailing data after end of output buffers In case a stream tries to emit more data than advertised by the chunks or content-length headers, the extra data remains in the channel's output buffer until the channel's timeout expires. It can easily happen when sending malformed error files making use of a wrong content-length or having extra CRLFs after the empty chunk. It may also be possible to forge such a bad response using Lua. The H1 to H2 encoder must protect itself against this by marking the data presented to it as consumed if it decides to discard them, so that the sending stream doesn't wait for the timeout to trigger. The visible effect of this problem is a huge memory usage and a high concurrent connection count during benchmarks when using such bad data (a typical place where this easily happens). This fix must be backported to 1.8. (cherry picked from commit 35a62705df65632e2717ae0d20a93e0cb3f8f163) Signed-off-by: Willy Tarreau commit 5a1e8fe4c8bcf98a156b6989017f0ceca8696c94 Author: Yves Lafon Date: Mon Feb 26 11:10:37 2018 +0100 MINOR: stats: display the number of threads in the statistics. Add the nbthread global variable to the output, matching nbproc. This may be backported to 1.8 (cherry picked from commit 95317289e9194c90778367227f945e40b14b96fe) [wt: while not a fix, it appears to be the only way to know if threads are being used at run time so this will help for troubleshooting] Signed-off-by: Willy Tarreau commit de425f6c857da26237798b90a646a323e8651a11 Author: Christopher Faulet Date: Mon Feb 26 13:43:38 2018 +0100 BUG/MINOR: h2: Set the target of dbuf_wait to h2c In h2_get_dbuf, when the buffer allocation was failing, dbuf_wait.target was errornously set to the connection (h2c->conn) instead of the h2 connection descriptor (h2c). This patch must be backported to 1.8. (cherry picked from commit 929b52d8a1a5d3271fa8b92b4a6e0f148e3a1d42) Signed-off-by: Willy Tarreau commit e7f8988511029310b7c6371598c6c797858eb0db Author: Willy Tarreau Date: Thu Feb 22 14:14:23 2018 +0100 MINOR: debug/pools: make DEBUG_UAF also detect underflows Since we use padding before the allocated page, it's trivial to place the allocated address there and see if it gets mangled once we release it. This may be backported to stable releases already using DEBUG_UAF. (cherry picked from commit 364d7451061f6ca6df09f9b9c15525483c3f7f07) Signed-off-by: Willy Tarreau commit 8e515267ed63f8a0eb6f2f376773d1769521ede9 Author: Willy Tarreau Date: Thu Feb 22 11:39:23 2018 +0100 BUG/MINOR: debug/pools: properly handle out-of-memory when building with DEBUG_UAF Commit 158fa75 ("MINOR: pools: implement DEBUG_UAF to detect use after free") implemented pool use-after-free detection, but the mmap() return value isn't properly checked, preventing the call to pool_alloc_area() from returning NULL. So on out-of-memory a mangled pointer is returned, causing a crash on the pool_alloc() site instead of forcing a GC. It doesn't affect regular operations however, just complicates complex bug investigations. This fix should be backported to 1.8 and to 1.7. (cherry picked from commit 5a9cce46531ceea9fa4338c064e90f7a514b0ce1) Signed-off-by: Willy Tarreau commit 15c7ee6669a8d08fbe43d0ddbbdcd6d6a8caae4f Author: Tim Duesterhus Date: Mon Feb 5 20:52:27 2018 +0100 DOC: cfgparse: Warn on option (tcp|http)log in backend The option does not seem to have any effect since at least haproxy 1.3. Also the `log-format` directive already warns when being used in a backend. (cherry picked from commit 9ad9f3517ee9e4289a003941a49cf52bd06314e3) Signed-off-by: Willy Tarreau commit e68ada0ca7772d800b77a9c4ab401d85d0ecdb17 Author: Thierry FOURNIER Date: Mon Feb 12 14:46:54 2018 +0100 DOC: lua: new prototype for function "register_action()" This patch should be backported to version 1.8. (cherry picked from commit c5d11c6b3392d548c45d617f132a1269d4d021fa) Signed-off-by: Willy Tarreau commit 7ccf7c9791f2b2329f3940d1347618af3a77bebc Author: Emeric Brun Date: Mon Feb 19 15:59:48 2018 +0100 BUG/MEDIUM: ssl/sample: ssl_bc_* fetch keywords are broken. Since the split between connections and conn-stream objects, this keywords are broken. This patch must be backported in 1.8 (cherry picked from commit eb8def9f34c37537d56a69fcd211d4c4c8006bea) Signed-off-by: Willy Tarreau commit 8a5949f2d74c3a3a6c6da25449992c312b183ef3 Author: Christopher Faulet Date: Fri Feb 2 15:54:15 2018 +0100 BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as earlier as possible When the body length is undefined (no Content-Length or Transfer-Encoding headers), The reponse remains in ending mode, waiting the request is done. So, most of time this is not a problem because the resquest is done before the response. But when a client sends data to a server that replies without waiting all the data, it is really not desirable to wait the end of the request to finish the response. This bug was introduced when the tunneling of the request and the reponse was refactored, in commit 4be980391 ("MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flag"). This patch should be backported in 1.8 and 1.7. (cherry picked from commit fd04fcf5edb0a24cd29ce8f4d4dc2aa3a0e2e82c) Signed-off-by: Willy Tarreau commit d001f3a5a197b74625c8d8ceeec0c3b7397d668c Author: Christopher Faulet Date: Fri Feb 16 11:23:49 2018 +0100 BUG/MINOR: ssl/threads: Make management of the TLS ticket keys files thread-safe A TLS ticket keys file can be updated on the CLI and used in same time. So we need to protect it to be sure all accesses are thread-safe. Because updates are infrequent, a R/W lock has been used. This patch must be backported in 1.8 (cherry picked from commit 16f45c87d5f9bf57fa4e7e71546bce352a727425) Signed-off-by: Willy Tarreau commit 65959091afddae793ff258254f32de62c3f66240 Author: Aurélien Nephtali Date: Sat Feb 17 20:53:11 2018 +0100 BUG/MINOR: init: Add missing brackets in the code parsing -sf/-st The codes tries to strip trailing spaces of arguments but due to missing brackets, it will always exit. It can be reproduced with this (silly) example: $ haproxy -f /etc/haproxy/haproxy.cfg -sf 1234 "1235 " 1236 $ echo $? 1 This was introduced in commit 236062f7c ("MINOR: init: emit warning when -sf/-sd cannot parse argument") Signed-off-by: Aurélien Nephtali (cherry picked from commit 39b89889e73baab618860d9f5a56e684c096850f) Signed-off-by: Willy Tarreau commit f7fa1d461aa71bbc8a6c23fdcfc305f2e52ce5dd Author: Christopher Faulet Date: Mon Feb 19 14:25:15 2018 +0100 BUG/MEDIUM: ssl: Shutdown the connection for reading on SSL_ERROR_SYSCALL When SSL_read returns SSL_ERROR_SYSCALL and errno is unset or set to EAGAIN, the connection must be shut down for reading. Else, the connection loops infinitly, consuming all the CPU. The bug was introduced in the commit 7e2e50500 ("BUG/MEDIUM: ssl: Don't always treat SSL_ERROR_SYSCALL as unrecovarable."). This patch must be backported in 1.8 too. (cherry picked from commit 4ac77a98cda3d0f9b1d9de7bbbda2c91357f0767) Signed-off-by: Willy Tarreau commit 2fcd544272a5498ffa49544e9f06b51bc93e55d1 Author: Olivier Houchard Date: Tue Feb 13 15:17:23 2018 +0100 BUG/MEDIUM: ssl: Don't always treat SSL_ERROR_SYSCALL as unrecovarable. Bart Geesink reported some random errors appearing under the form of termination flags SD in the logs for connections involving SSL traffic to reach the servers. Tomek Gacek and Mateusz Malek finally narrowed down the problem to commit c2aae74 ("MEDIUM: ssl: Handle early data with OpenSSL 1.1.1"). It happens that the special case of SSL_ERROR_SYSCALL isn't handled anymore since this commit. SSL_read() might return <= 0, and SSL_get_erro() return SSL_ERROR_SYSCALL, without meaning the connection is gone. Before flagging the connection as in error, check the errno value. This should be backported to 1.8. (cherry picked from commit 7e2e505006feb8f3b4a7f9e0ac5e89b5a8c4895e) Signed-off-by: Willy Tarreau commit a31856ce7d33ba8c12bf27d5303c7a1bf1a9fe6a Author: Willy Tarreau Date: Mon Feb 19 07:20:02 2018 +0100 BUG/MINOR: threads: fix missing thread lock labels for 1.8 Commit 9e53b19 ("BUG/MINOR: threads: Update labels array because of changes in lock_label enum") was a backport of mainline f51bac2b which ensured completeness of the lock labels when threads are build with debugging enabled. But it lacked the FDTAB_LOCK and POLL_LOCK that were removed in 1.9-dev, causing this warning to appear when building with gcc 5 : include/common/hathreads.h: In function 'lock_label': include/common/hathreads.h:324:2: warning: enumeration value 'FDTAB_LOCK' not handled in switch [-Wswitch] switch (label) { ^ include/common/hathreads.h:324:2: warning: enumeration value 'POLL_LOCK' not handled in switch [-Wswitch] Thread debugging will likely abort when trying to dump these locks. It's harmless since only developers build with thread debugging though. 1.8 is the only version affected. commit 1deb90d5243a5cfa5da7592978592eb9ab2c8c6f Author: Willy Tarreau Date: Thu Feb 8 14:05:15 2018 +0100 [RELEASE] Released version 1.8.4 Released version 1.8.4 with the following main changes : - BUG/MEDIUM: h2: properly handle the END_STREAM flag on empty DATA frames - BUILD: ssl: silence a warning when building without NPN nor ALPN support - BUG/MEDIUM: ssl: cache doesn't release shctx blocks - BUG/MINOR: lua: Fix default value for pattern in Socket.receive - DOC: lua: Fix typos in comments of hlua_socket_receive - BUG/MEDIUM: lua: Fix IPv6 with separate port support for Socket.connect - BUG/MINOR: lua: Fix return value of Socket.settimeout - MINOR: dns: Handle SRV record weight correctly. - BUG/MEDIUM: mworker: execvp failure depending on argv[0] - MINOR: hathreads: add support for gcc < 4.7 - BUILD/MINOR: ancient gcc versions atomic fix - BUG/MEDIUM: stream: properly handle client aborts during redispatch - DOC: clarify the scope of ssl_fc_is_resumed - CONTRIB: debug: fix a few flags definitions - BUG/MINOR: poll: too large size allocation for FD events - BUG/MEDIUM: peers: fix expire date wasn't updated if entry is modified remotely. - MINOR: servers: Don't report duplicate dyncookies for disabled servers. - MINOR: global/threads: move cpu_map at the end of the global struct - MINOR: threads: add a MAX_THREADS define instead of LONGBITS - MINOR: global: add some global activity counters to help debugging - MINOR: threads/fd: Use a bitfield to know if there are FDs for a thread in the FD cache - BUG/MEDIUM: threads/polling: Use fd_cache_mask instead of fd_cache_num - BUG/MEDIUM: fd: maintain a per-thread update mask - MINOR: fd: add a bitmask to indicate that an FD is known by the poller - BUG/MEDIUM: epoll/threads: use one epoll_fd per thread - BUG/MEDIUM: kqueue/threads: use one kqueue_fd per thread - BUG/MEDIUM: threads/mworker: fix a race on startup - BUG/MINOR: mworker: only write to pidfile if it exists - MINOR: threads: Fix build when we're not compiling with threads. - BUG/MINOR: threads: always set an owner to the thread_sync pipe - BUG/MEDIUM: threads/server: Fix deadlock in srv_set_stopping/srv_set_admin_flag - BUG/MEDIUM: checks: Don't try to release undefined conn_stream when a check is freed - BUG/MINOR: kqueue/threads: Don't forget to close kqueue_fd[tid] on each thread - MINOR: threads: Use __decl_hathreads instead of #ifdef/#endif - BUILD: epoll/threads: Add test on MAX_THREADS to avoid warnings when complied without threads - BUILD: kqueue/threads: Add test on MAX_THREADS to avoid warnings when complied without threads - CLEANUP: sample: Fix comment encoding of sample.c - CLEANUP: sample: Fix outdated comment about sample casts functions - BUG/MINOR: sample: Fix output type of c_ipv62ip - CLEANUP: Fix typo in ARGT_MSK6 comment - BUG/MINOR: cli: use global.maxsock and not maxfd to list all FDs - BUG/MINOR: threads: Update labels array because of changes in lock_label enum - BUG/MINOR: epoll/threads: only call epoll_ctl(DEL) on polled FDs - BUG/MEDIUM: spoe: Always try to receive or send the frame to detect shutdowns - BUG/MEDIUM: spoe: Allow producer to read and to forward shutdown on request side - BUG/MINOR: time/threads: ensure the adjusted time is always correct - BUG/MEDIUM: standard: Fix memory leak in str2ip2() - MINOR: init: emit warning when -sf/-sd cannot parse argument - DOC: Describe routing impact of using interface keyword on bind lines - DOC: Mention -Ws in the list of available options - BUG/MINOR: config: don't emit a warning when global stats is incompletely configured commit 021fa041313c45913e59846ffbb366e1405116ad Author: Willy Tarreau Date: Thu Feb 8 09:55:09 2018 +0100 BUG/MINOR: config: don't emit a warning when global stats is incompletely configured Martin Brauer reported an unexpected warning when some parts of the global stats are defined but not the listening address, like below : global #stats socket run/admin.sock mode 660 level admin stats timeout 30s Then haproxy complains : [WARNING] 334/150131 (23086) : config : frontend 'GLOBAL' has no 'bind' directive. Please declare it as a backend if this was intended. This is because of the check for a bind-less frontend (the global section creates a frontend for the stats). There's no clean fix for this one, so here we're simply checking that the frontend is not the global stats one before emitting the warning. This patch should be backported to all stable versions. (cherry picked from commit 58aa5ccd7675e5c960b045e96fdf69845d6449b4) Signed-off-by: Willy Tarreau commit 27e631c46b19228a401572c9478a6515da46d3d2 Author: Pavlos Parissis Date: Wed Feb 7 21:42:16 2018 +0100 DOC: Mention -Ws in the list of available options (cherry picked from commit f65f257871907f831558b1fab626d871fd48e984) Signed-off-by: Willy Tarreau commit ec91da75e8ee9785551d2eb31e7c84cc8e6217a9 Author: Jérôme Magnin Date: Wed Feb 7 11:39:58 2018 +0100 DOC: Describe routing impact of using interface keyword on bind lines (cherry picked from commit 61275198b30a505df7824ba9081d45d5d6d07f81) Signed-off-by: Willy Tarreau commit b6649ffcfcea9ee1f623ddd6d4c485532a450eb1 Author: Chris Lane Date: Mon Feb 5 23:15:44 2018 +0000 MINOR: init: emit warning when -sf/-sd cannot parse argument Previously, -sf and -sd command line parsing used atol which cannot detect errors. I had a problem where I was doing -sf "$pid1 $pid2 $pid" and it was sending the gracefully terminate signal only to the first pid. The change uses strtol and checks endptr and errno to see if the parsing worked. It will exit when the pid list is not parsed. [wt: this should be backported to 1.8] (cherry picked from commit 236062f7cea355bae9bbb6f5cd1953e78f36c6d8) Signed-off-by: Willy Tarreau commit fb4921724172f89aa1d3309688aa793112839fe1 Author: Tim Duesterhus Date: Sun Jan 21 22:11:17 2018 +0100 BUG/MEDIUM: standard: Fix memory leak in str2ip2() An haproxy compiled with: > make -j4 all TARGET=linux2628 USE_GETADDRINFO=1 And running with a configuration like this: defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend fe bind :::8080 v4v6 default_backend be backend be server s example.com:80 check Will leak memory inside `str2ip2()`, because the list `result` is not properly freed in success cases: ==18875== 140 (76 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 87 of 111 ==18875== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==18875== by 0x537A565: gaih_inet (getaddrinfo.c:1223) ==18875== by 0x537DD5D: getaddrinfo (getaddrinfo.c:2425) ==18875== by 0x4868E5: str2ip2 (standard.c:733) ==18875== by 0x43F28B: srv_set_addr_via_libc (server.c:3767) ==18875== by 0x43F50A: srv_iterate_initaddr (server.c:3879) ==18875== by 0x43F50A: srv_init_addr (server.c:3944) ==18875== by 0x475B30: init (haproxy.c:1595) ==18875== by 0x40406D: main (haproxy.c:2479) The exists as long as the usage of getaddrinfo in that function exists, it was introduced in commit: d5f4328efd5f4eaa7c89cad9773124959195430a v1.5-dev8 is the first tag containing this comment, the fix should be backported to haproxy 1.5 and newer. (cherry picked from commit 7d58b4d156fe159775e240a73aaad1bb76075af5) Signed-off-by: Willy Tarreau commit 939124768d25723d6cf40d6182c1ddcb5f34fa19 Author: Willy Tarreau Date: Mon Feb 5 20:11:38 2018 +0100 BUG/MINOR: time/threads: ensure the adjusted time is always correct In the time offset calculation loop, we ensure we only commit the new date once it's futher in the future than the current one. However there is a small issue here on 32-bit platforms : if global_now is written in two cycles by another thread, starting with the tv_sec part, and the current thread reads it in the middle of a change, it may compute a wrong "adjusted" value on the first round, with the new (larger) tv_sec and the old (large) tv_usec. This will be detected as the CAS will fail, and another attempt will be made, but this time possibly with too large an adusted value, pushing the date further than needed (at worst almost one second). This patch addresses this by using a temporary adjusted time in the loop that always restarts from the last known one, and by assigning the result to the final value only once the CAS succeeds. The impact is very limited, it may cause the time to advance in small jumps on 32 bit platforms and in the worst case some timeouts might expire 1 second too early. This fix should be backported to 1.8. (cherry picked from commit a331544c33d287471ba0afaad5de5289f3cc35ea) Signed-off-by: Willy Tarreau commit bd9c62bc193678d44988eebb6ffa23b24d81652f Author: Christopher Faulet Date: Thu Feb 1 08:45:45 2018 +0100 BUG/MEDIUM: spoe: Allow producer to read and to forward shutdown on request side This is mandatory to correctly set right timeout on the stream. Else the client timeout is never set. So only SPOE processing timeout will be evaluated. If it is not defined (ie infinity), the stream can be blocked for a while, waiting the SPOA reply. Of course, this is not a good idea to let the SPOE processing timeout undefined, but it can happen. This patch must be backported in 1.8. (cherry picked from commit 9cdca976d325cfc69a085b7d9e53ee1789a18a5b) Signed-off-by: Willy Tarreau commit f13f3a4babdb1ce23a7e982c765704bca728111a Author: Christopher Faulet Date: Thu Feb 1 08:45:22 2018 +0100 BUG/MEDIUM: spoe: Always try to receive or send the frame to detect shutdowns Before, we checked if the buffer was allocated or not to avoid sending or receiving a frame. This was done to not call ci_putblk or co_getblk if there is nothing to do. But the checks on the buffers are also done in these functions. So this is not mandatory here. But in these functions, the channel state is also checked, so an error is returned if it is closed. By skipping the call, we also skip the checks on the channel state, delaying shutdowns detection. Now, we always try to send or receive a frame. So if the corresponding channel is closed, we can immediatly handle the error. This patch must be backported in 1.8 (cherry picked from commit d5216d474d69856a282e4443f180af2093a80d6c) Signed-off-by: Willy Tarreau commit dad5d4d4185b6392c0e866a1e6a4c297e2b28f18 Author: Willy Tarreau Date: Wed Jan 31 09:49:29 2018 +0100 BUG/MINOR: epoll/threads: only call epoll_ctl(DEL) on polled FDs Commit d9e7e36 ("BUG/MEDIUM: epoll/threads: use one epoll_fd per thread") addressed an issue with the polling and required that cloned FDs are removed from all polling threads on close. But in fact it does it for all bound threads, some of which may not necessarily poll the FD. This is harmless, but it may also make it harder later to deal with FD migration between threads. Better use polled_mask which only reports threads still aware of the FD instead of thread_mask. This fix should be backported to 1.8. (cherry picked from commit 497959290789002b814b9021a737a3c5f14e7407) Signed-off-by: Willy Tarreau commit 9e53b1986a3b6fc4a95254c828eb3aa551a263ce Author: Christopher Faulet Date: Tue Jan 30 11:04:29 2018 +0100 BUG/MINOR: threads: Update labels array because of changes in lock_label enum Recent changes to the enum were not synchronized with the lock debugging code. Now we use a switch/case instead of an array so that the compiler throws a warning if there is any inconsistency. To be backported to 1.8 (at least to add the START entry). (cherry picked from commit f51bac2ba83a27a75ef6bab58bfde6489b999c49) Signed-off-by: Willy Tarreau commit e0a80ca5b24fc3b9abf8e3a155d8dfdd80996f1c Author: Willy Tarreau Date: Mon Jan 29 15:17:05 2018 +0100 BUG/MINOR: cli: use global.maxsock and not maxfd to list all FDs The "show fd" command on the CLI doesn't list the last FD in use since it doesn't include maxfd. We don't need to use maxfd here anyway as global.maxsock will do the job pretty well and removes this dependency. This patch may be backported to 1.8. (cherry picked from commit ccea35c9800756f867aff8ec2752fa75480ac345) Signed-off-by: Willy Tarreau commit 283193a5dd4bbdfc6880c203f34ddbdfe3f5e8d9 Author: Tim Duesterhus Date: Thu Jan 25 16:24:47 2018 +0100 CLEANUP: Fix typo in ARGT_MSK6 comment The incorrect comment was introduced in commit: 2ac5718dbd4ec722ece228e9f613d2be74eee9da v1.5-dev9 is the first tag containing this comment, the fix should be backported to haproxy 1.5 and newer. (cherry picked from commit 92bb03420935c7fd844ddccd78acb7b1da2fffd4) Signed-off-by: Willy Tarreau commit 23528d0f23485c9914311f9f10dd0cd0fe4d734f Author: Tim Duesterhus Date: Thu Jan 25 16:24:46 2018 +0100 BUG/MINOR: sample: Fix output type of c_ipv62ip c_ipv62ip failed to set the output type of the cast to SMP_T_IPV4 even for a successful conversion. This bug exists as of commit cc4d1716a2e72516c2505a6459a9ddbbfb186da2 which is the first commit adding this function. v1.6-dev4 is the first tag containing this commit, the fix should be backported to haproxy 1.6 and newer. (cherry picked from commit bf5ce02effdc2d471fdb421010757320136cbb45) Signed-off-by: Willy Tarreau commit ffde75e94211d25b30b15687d10783d0b9000e9d Author: Tim Duesterhus Date: Thu Jan 25 16:24:45 2018 +0100 CLEANUP: sample: Fix outdated comment about sample casts functions The cast functions modify their output type as of commit: b805f71d1bb1487f01f78a6ffab26d44919e9944 v1.5-dev20 is the first tag containing this comment, the fix should be backported to haproxy 1.5 and newer. (cherry picked from commit ec6b0a2d189b84d743baccbcc14a4a4b97390549) Signed-off-by: Willy Tarreau commit 34e807e77d57b0a087b196b9d397c181285bb255 Author: Tim Duesterhus Date: Thu Jan 25 16:24:44 2018 +0100 CLEANUP: sample: Fix comment encoding of sample.c The file contained an 'e' with an gravis accent and thus was not US-ASCII, but ISO-8859-1. Also correct the spelling in the incorrect comment. The incorrect character was introduced in commit: 4d9a1d1a5c4720a169654ee47f9a4364261ffab4 v1.6-dev1 is the first tag containing this comment, the fix should be backported to haproxy 1.6 and newer. (cherry picked from commit c555ee0c45098278ddad695cbbd16a815bcb50da) Signed-off-by: Willy Tarreau commit 2dd90ea170614be1f1c6214c8236791ade96f737 Author: Christopher Faulet Date: Thu Jan 25 16:40:35 2018 +0100 BUILD: kqueue/threads: Add test on MAX_THREADS to avoid warnings when complied without threads This is the same patch than the previous one ("BUILD: epoll/threads: Add test on MAX_THREADS to avoid warnings when complied without threads "). It should be backported in 1.8 with the commit 7a2364d4 ("BUG/MEDIUM: kqueue/threads: use one kqueue_fd per thread"). (cherry picked from commit 727c89b3dfd158760a68656be0615f1a8f192933) Signed-off-by: Willy Tarreau commit f17eea8675d31ae31ef3154d95f5a1a7f98d326a Author: Christopher Faulet Date: Thu Jan 25 16:18:09 2018 +0100 BUILD: epoll/threads: Add test on MAX_THREADS to avoid warnings when complied without threads When HAProxy is complied without threads, gcc throws following warnings: src/ev_epoll.c:222:3: warning: array subscript is outside array bounds [-Warray-bounds] ... src/ev_epoll.c:199:11: warning: array subscript is outside array bounds [-Warray-bounds] ... Of course, this is not a bug. In such case, tid is always equal to 0. But to avoid the noise, a check on MAX_THREADS in "if (tid)" lines makes gcc happy. This patch should be backported in 1.8 with the commit d9e7e36c ("BUG/MEDIUM: epoll/threads: use one epoll_fd per thread"). (cherry picked from commit 3e805ed08ec1c16c29a59b6d08dc8d96f0b69e15) Signed-off-by: Willy Tarreau commit 0c44babeace11d34fb6c134df6b4eb80c1c2fd50 Author: Christopher Faulet Date: Thu Jan 25 16:10:16 2018 +0100 MINOR: threads: Use __decl_hathreads instead of #ifdef/#endif A #ifdef/#endif on USE_THREAD was added in the commit 0048dd04 ("MINOR: threads: Fix build when we're not compiling with threads.") to conditionally define the start_lock variable, because HA_SPINLOCK_T is only defined when HAProxy is compiled with threads. If fact, to do that, we should use the macro __decl_hathreads instead. If commit 0048dd04 is backported in 1.8, this one can also be backported. (cherry picked from commit da18b9db7b26080b899985191c6344d79f5497b5) Signed-off-by: Willy Tarreau commit b763e589b5f2a809b11e87a5598cc1008158773e Author: Christopher Faulet Date: Thu Jan 25 16:32:18 2018 +0100 BUG/MINOR: kqueue/threads: Don't forget to close kqueue_fd[tid] on each thread in deinit_kqueue_per_thread, kqueue_fd[tid] must be closed, except for the main thread (the first one, tid==0). This patch must be backported in 1.8 with commit 7a2364d4. (cherry picked from commit 13b007d583924d19d692268c2f51a2fc2b46fd0c) Signed-off-by: Willy Tarreau commit 616078521470b5c49325fdb9af8c345bedb770b2 Author: Christopher Faulet Date: Thu Jan 25 11:36:35 2018 +0100 BUG/MEDIUM: checks: Don't try to release undefined conn_stream when a check is freed When a healt-check is released, the attached conn_stream may be undefined. For instance, this happens when 'no-check' option is used on a server line. So we must check it is defined before trying to release it. This patch must be backported in 1.8. (cherry picked from commit 23d86d157edd6da8d3fe40d7883a3d2beeed3189) Signed-off-by: Willy Tarreau commit 757dd8c7407c94dd95272ba0bc92c18d0f106867 Author: Christopher Faulet Date: Wed Jan 24 21:49:41 2018 +0100 BUG/MEDIUM: threads/server: Fix deadlock in srv_set_stopping/srv_set_admin_flag Because of a typo (HA_SPIN_LOCK instead of HA_SPIN_UNLOCK), there is a deadlock in srv_set_stopping and srv_set_admin_flag when there is at least one trackers. This patch must be backported in 1.8. (cherry picked from commit 8d01fd6b3caf7fd97a21aa24bb946b12484ce1a3) Signed-off-by: Willy Tarreau commit 2c2cb94b059954208d3645768cf5ed723668934a Author: Willy Tarreau Date: Thu Jan 25 07:28:37 2018 +0100 BUG/MINOR: threads: always set an owner to the thread_sync pipe The owner of the fd used by the synchronization pipe was set to NULL, making it ignored by maxfd computation. The risk would be that some synchronization events get delayed between threads when using poll() or select(). However this is only theorical since the pipe is created before listeners are bound so normally its FD should be lower and this should normally not happen. The only possible situation would be if all listeners are bound to inherited FDs which are lower than the pipe's. This patch must be backported to 1.8. (cherry picked from commit c20d73733871ee820fe87880e076a00cbad1b7d6) Signed-off-by: Willy Tarreau commit bbedf00dbd0e4dfc002fb8007b1ca769006a91bb Author: Olivier Houchard Date: Wed Jan 24 15:41:04 2018 +0100 MINOR: threads: Fix build when we're not compiling with threads. Only declare the start_lock if threads are compiled in, otherwise HA_SPINLOCK_T won't be defined. This should be backported to 1.8 when/if 1605c7ae6154d8c2cfcf3b325872b1a7266c5bc2 is backported. (cherry picked from commit 0048dd04c9a6e9b9c7def8e7060793290e0c36ff) Signed-off-by: Willy Tarreau commit 945f4cf086b9029d892712c4bd382de6a02fb1e1 Author: Willy Tarreau Date: Tue Jan 23 19:20:19 2018 +0100 BUG/MINOR: mworker: only write to pidfile if it exists A missing test causes a write(-1, $PID) to appear in strace output when in master-worker mode. This is totally harmless though. This fix must be backported to 1.8. (cherry picked from commit 46ec48bc1ad81cdf10aacfd5d7b2911c0ef2b0df) Signed-off-by: Willy Tarreau commit a91f5578f2d8690f8ef0d956a65a248e845a4a86 Author: Willy Tarreau Date: Tue Jan 23 19:01:49 2018 +0100 BUG/MEDIUM: threads/mworker: fix a race on startup Marc Fournier reported an interesting case when using threads with the master-worker mode : sometimes, a listener would have its FD closed during startup. Sometimes it could even be health checks seeing this. What happens is that after the threads are created, and the pollers enabled on each threads, the master-worker pipe is registered, and at the same time a close() is performed on the write side of this pipe since the children must not use it. But since this is replicated in every thread, what happens is that the first thread closes the pipe, thus releases the FD, and the next thread starting a listener in parallel gets this FD reassigned. Then another thread closes the FD again, which this time corresponds to the listener. It can also happen with the health check sockets if they're started early enough. This patch splits the mworker_pipe_register() function in two, so that the close() of the write side of the FD is performed very early after the fork() and long before threads are created (we don't need to delay it anyway). Only the pipe registration is done in the threaded code since it is important that the pollers are properly allocated for this. The mworker_pipe_register() function now takes care of registering the pipe only once, and this is guaranteed by a new surrounding lock. The call to protocol_enable_all() looks fragile in theory since it scans the list of proxies and their listeners, though in practice all threads scan the same list and take the same locks for each listener so it's not possible that any of them escapes the process and finishes before all listeners are started. And the operation is idempotent. This fix must be backported to 1.8. Thanks to Marc for providing very detailed traces clearly showing the problem. (cherry picked from commit 1605c7ae6154d8c2cfcf3b325872b1a7266c5bc2) Signed-off-by: Willy Tarreau commit f839593dd26ec210ba66d74b2a4c2040dd1ed806 Author: Willy Tarreau Date: Fri Jan 19 08:56:14 2018 +0100 BUG/MEDIUM: kqueue/threads: use one kqueue_fd per thread This is the same principle as the previous patch (BUG/MEDIUM: epoll/threads: use one epoll_fd per thread) except that this time it's for kqueue. We don't want all threads to wake up because of activity on a single other thread that the other ones are not interested in. Just like with previous patch, this one shows that the polling state doesn't need to be changed here and that some simplifications are now possible. This patch only implements the minimum required for a stable backport. This should be backported to 1.8. (cherry picked from commit 7a2364d4741eed7d435019690b0f0d12878939b0) Signed-off-by: Willy Tarreau commit fadef1c842f7810df41fd6d56bfad50fb078697e Author: Willy Tarreau Date: Thu Jan 18 19:16:02 2018 +0100 BUG/MEDIUM: epoll/threads: use one epoll_fd per thread There currently is a problem regarding epoll(). While select() and poll() compute their polling state on the fly upon each call, epoll() keeps a shared state between all threads via the epoll_fd. The problem is that once an fd is registered on *any* thread, all other threads receive events for that FD as well. It is clearly visible when binding a listener to a single thread like in the configuration below where all 4 threads will work, 3 of them simply spinning to skip the event : global nbthread 4 frontend foo bind :1234 process 1/1 The worst case happens when some slow operations are in progress on a busy thread, preventing it from processing its task and causing the other ones to wake up not being able to do anything with this event. Typically computing a large TLS key will delay processing of next events on the same thread while others will still wake up. All this simply shows that the poller must remain thread-specific, with its own events and its own ability to sleep when it doesn't have anyhing to do. This patch does exactly this. For this, it proceeds like this : - have one epoll_fd per thread instead of one per process - initialize these epoll_fd when threads are created. - mark all known FDs as updated so that the next invocation of _do_poll() recomputes their polling status (including a possible removal of undesired polling from the original FD) ; - use each fd's polled_mask to maintain an accurate status of the current polling activity for this FD. - when scanning updates, only focus on events whose new polling status differs from the existing one - during updates, always verify the thread_mask to resist migration - on __fd_clo(), for cloned FDs (typically listeners inherited from the parent during a graceful shutdown), run epoll_ctl(DEL) on all epoll_fd. This is the reason why epoll_fd is stored in a shared array and not in a thread_local storage. Note: maybe this can be moved to an update instead. Interestingly, this shows that we don't need the FD's old state anymore and that we only use it to convert it to the new state based on stable information. It appears clearly that the FD code can be further improved by computing the final state directly when manipulating it. With this change, the config above goes from 22000 cps at 380% CPU to 43000 cps at 100% CPU : not only the 3 unused threads are not activated, but they do not disturb the activity anymore. The output of "show activity" before and after the patch on a 4-thread config where a first listener on thread 2 forwards over SSL to threads 3 & 4 shows this a much smaller amount of undesired events (thread 1 doesn't wake up anymore, poll_skip remains zero, fd_skip stays low) : // before: 400% CPU, 7700 cps, 13 seconds loops: 11380717 65879 5733468 5728129 wake_cache: 0 63986 317547 314174 wake_tasks: 0 0 0 0 wake_applets: 0 0 0 0 wake_signal: 0 0 0 0 poll_exp: 0 63986 317547 314174 poll_drop: 1 0 49981 48893 poll_dead: 65514 0 31334 31934 poll_skip: 46293690 34071 22867786 22858208 fd_skip: 66068135 174157 33732685 33825727 fd_lock: 0 2 2809 2905 fd_del: 0 494361 80890 79464 conn_dead: 0 0 0 0 stream: 0 407747 50526 49474 empty_rq: 11380718 1914 5683023 5678715 long_rq: 0 0 0 0 // after: 200% cpu, 9450 cps, 11 seconds loops: 17 66147 1001631 450968 wake_cache: 0 66119 865139 321227 wake_tasks: 0 0 0 0 wake_applets: 0 0 0 0 wake_signal: 0 0 0 0 poll_exp: 0 66119 865139 321227 poll_drop: 6 5 38279 60768 poll_dead: 0 0 0 0 poll_skip: 0 0 0 0 fd_skip: 54 172661 4411407 2008198 fd_lock: 0 0 10890 5394 fd_del: 0 492829 58965 105091 conn_dead: 0 0 0 0 stream: 0 406223 38663 61338 empty_rq: 18 40 962999 390549 long_rq: 0 0 0 0 This patch presents a few risks but fixes a real problem with threads, and as such it needs be backported to 1.8. It depends on previous patch ("MINOR: fd: add a bitmask to indicate that an FD is known by the poller"). Special thanks go to Samuel Reed for providing a large amount of useful debugging information and for testing fixes. (cherry picked from commit d9e7e36c6e5c0f9988a758d81fbe0a8fa8413922) Signed-off-by: Willy Tarreau commit 8560c73764b0ce3249c3098d6a4808fa0659e283 Author: Willy Tarreau Date: Wed Jan 17 18:44:46 2018 +0100 MINOR: fd: add a bitmask to indicate that an FD is known by the poller Some pollers like epoll() need to know if the fd is already known or not in order to compute the operation to perform (add, mod, del). For now this is performed based on the difference between the previous FD state and the new state but this will not be usable anymore once threads become responsible for their own polling. Here we come with a different approach : a bitmask is stored with the fd to indicate which pollers already know it, and the pollers will be able to simply perform the add/mod/del operations based on this bit combined with the new state. This patch only adds the bitmask declaration and initialization, it is it not yet used. It will be needed by the next two fixes and will need to be backported to 1.8. (cherry picked from commit c9c8378c2b7aed45a9b4733d81f69338dad7614a) Signed-off-by: Willy Tarreau commit 79ddc8190c1b1e9d6d29d763dca4250ea7257370 Author: Willy Tarreau Date: Sat Jan 20 23:53:50 2018 +0100 BUG/MEDIUM: fd: maintain a per-thread update mask Since the fd update tables are per-thread, we need to have a bit per thread to indicate whether an update exists, otherwise this can lead to lost update events every time multiple threads want to update the same FD. In practice *for now*, it only happens at start time when listeners are enabled and ask for polling after facing their first EAGAIN. But since the pollers are still shared, a lost event is still recovered by a neighbor thread. This will not reliably work anymore with per-thread pollers, where it has been observed a few times on startup that a single-threaded listener would not always accept incoming connections upon startup. It's worth noting that during this code review it appeared that the "new" flag in the fdtab isn't used anymore. This fix should be backported to 1.8. (cherry picked from commit ebc78d78a27ac3de7308eeb499c51d638e79ed6b) Signed-off-by: Willy Tarreau commit 3d4c732a698022b29186ec31cc36c653a1b11428 Author: Christopher Faulet Date: Mon Jan 15 12:16:34 2018 +0100 BUG/MEDIUM: threads/polling: Use fd_cache_mask instead of fd_cache_num fd_cache_num is the number of FDs in the FD cache. It is a global variable. So it is underoptimized because we may be lead to consider there are waiting FDs for the current thread in the FD cache while in fact all FDs are assigned to the other threads. So, in such cases, the polling loop will be evaluated many more times than necessary. Instead, we now check if the thread id is set in the bitfield fd_cache_mask. [wt: it's not exactly a bug, rather a design limitation of the thread which was not addressed in time for the 1.8 release. It can appear more often than we initially predicted, when more threads are running than the number of assigned CPU cores, or when certain threads spend milliseconds computing crypto keys while other threads spin on epoll_wait(0)=0] This patch should be backported to 1.8. (cherry picked from commit 32467fef98f9a4f14be8864bc44b3551f8e34759) Signed-off-by: Willy Tarreau commit 5e2865841a7f0e09ac3692f837253186e039d349 Author: Christopher Faulet Date: Mon Jan 15 11:57:03 2018 +0100 MINOR: threads/fd: Use a bitfield to know if there are FDs for a thread in the FD cache A bitfield has been added to know if there are some FDs processable by a specific thread in the FD cache. When a FD is inserted in the FD cache, the bits corresponding to its thread_mask are set. On each thread, the bitfield is updated when the FD cache is processed. If there is no FD processed, the thread is removed from the bitfield by unsetting its tid_bit. Note that this bitfield is updated but not checked in fd_process_cached_events. So, when this function is called, the FDs cache is always processed. [wt: should be backported to 1.8 as it will help fix a design limitation] (cherry picked from commit 69553fe62c5c69753d1862a3e74740a1ff6c4d8d) Signed-off-by: Willy Tarreau commit f3703dd53d5283ae2c0bc5d99c3569e24b7d996b Author: Willy Tarreau Date: Sat Jan 20 19:30:13 2018 +0100 MINOR: global: add some global activity counters to help debugging A number of counters have been added at special places helping better understanding certain bug reports. These counters are maintained per thread and are shown using "show activity" on the CLI. The "clear counters" commands also reset these counters. The output is sent as a single write(), which currently produces up to about 7 kB of data for 64 threads. If more counters are added, it may be necessary to write into multiple buffers, or to reset the counters. To backport to 1.8 to help collect more detailed bug reports. (cherry picked from commit d80cb4ee1386cb5853170371d11e41284739e9d4) Signed-off-by: Willy Tarreau commit 00065af94c9d726c954a32183036deabc7911517 Author: Willy Tarreau Date: Sat Jan 20 18:19:22 2018 +0100 MINOR: threads: add a MAX_THREADS define instead of LONGBITS This one allows not to inflate some structures when threads are disabled. Now struct global is 1.4 kB instead of 33 kB. Should be backported to 1.8 for ease of backporting of upcoming patches. (cherry picked from commit 421f02e738999dec9f52665023918e22580197fd) Signed-off-by: Willy Tarreau commit 7b7e3077e91cf82551c17a4912d0ed85daea5f13 Author: Willy Tarreau Date: Sat Jan 20 18:12:15 2018 +0100 MINOR: global/threads: move cpu_map at the end of the global struct The "thread" part is 32kB long, better move it at the end of the structure since it's only used during initialization, to keep the rest grouped together. Should be backported to 1.8 to ease backporting of upcoming patches, no functional impact. (cherry picked from commit f4571a027f20a3866018a9a6749fd456370943e1) Signed-off-by: Willy Tarreau commit eab977ce47868d17848bfde828c303c7e515abb6 Author: Olivier Houchard Date: Wed Jan 17 17:39:34 2018 +0100 MINOR: servers: Don't report duplicate dyncookies for disabled servers. Especially with server-templates, it can happen servers starts with a placeholder IP, in the disabled state. In this case, we don't want to report that the same cookie was generated for multiple servers. So defer the test until the server is enabled. This should be backported to 1.8. (cherry picked from commit e9bad0a9361a761f2f304dcca8ddd997c05627c2) Signed-off-by: Willy Tarreau commit 07325f084d469d8e8d38d1e5a80f1d82bb641871 Author: Emeric Brun Date: Mon Jan 22 15:10:08 2018 +0100 BUG/MEDIUM: peers: fix expire date wasn't updated if entry is modified remotely. The stktable_touch_remote considers the expire field stored in the stksess struct. The expire field was updated on the a newly created stksess to store. But if the stksess with a same key is still present the expire was not updated. This patch postpones the update of the expire field of the stksess just before processing the "touch". These bug was introduced in commit: MEDIUM: threads/stick-tables: handle multithreads on stick tables. And the fix should be backported on 1.8. (cherry picked from commit 55482913956581223dc9b54e480675610e2e177e) Signed-off-by: Willy Tarreau commit 13165eed6abb8569d56f6eec404f4462d0cd7115 Author: Willy Tarreau Date: Wed Jan 17 15:48:53 2018 +0100 BUG/MINOR: poll: too large size allocation for FD events Commit 80da05a ("MEDIUM: poll: do not use FD_* macros anymore") which appeared in 1.5-dev18 and which was backported to 1.4.23 made explicit use of arrays of FDs mapped to unsigned ints. The problem lies in the allocated size for poll(), as the resulting size is in bits and not bytes, resulting in poll() arrays being 8 times larger than necessary! In practice poll() is not used on highly loaded systems, explaining why nobody noticed. But it definetely has to be addressed. This fix needs to be backported to all stable versions. (cherry picked from commit cc35923c329bbadc78c1c026ce4e45e115852abe) Signed-off-by: Willy Tarreau commit e80014590f274df2edd3e0532c8a2b7570c7986d Author: Willy Tarreau Date: Mon Jan 15 18:59:16 2018 +0100 CONTRIB: debug: fix a few flags definitions Commit f4cfcf9 ("MINOR: debug/flags: Add missing flags") added a number of missing flags but a few of them were incorrect, hiding real values. This can be backported to 1.8. (cherry picked from commit 260bf5c1064e4d2da61b4c73f69b7716cc8eb7ed) Signed-off-by: Willy Tarreau commit a9430f93ab02a759c5dd490056247a9b872b9e94 Author: Jérôme Magnin Date: Mon Jan 15 14:01:17 2018 +0100 DOC: clarify the scope of ssl_fc_is_resumed Clarify that it's for incoming connections. (cherry picked from commit 4a326cba5b6720c95199e7f919f69a1ae682efb2) Signed-off-by: Willy Tarreau commit 72a0acaa4c5e8c4c09bd129ecac2352c8f6ed701 Author: Willy Tarreau Date: Fri Jan 12 10:42:12 2018 +0100 BUG/MEDIUM: stream: properly handle client aborts during redispatch James Mc Bride reported an interesting case affecting all versions since at least 1.5 : if a client aborts a connection on an empty buffer at the exact moment a server redispatch happens, the CF_SHUTW_NOW flag on the channel is immediately turned into CF_SHUTW, which is not caught by check_req_may_abort(), leading the redispatch to be performed anyway with the channel marked as shut in both directions while the stream interface correctly establishes. This situation makes no sense. Ultimately the transfer times out and the server-side stream interface remains in EST state while the client is in CLO state, and this case doesn't correspond to anything we can handle in process_stream, leading to poll() being woken up all the time without any progress being made. And the session cannot even be killed from the CLI. So we must ensure that check_req_may_abort() also considers the case where the channel is already closed, which is what this patch does. Thanks to James for providing detailed captures allowing to diagnose the problem. This fix must be backported to all maintained versions. (cherry picked from commit d651ba14d44a7350b4506e9de7c7b44cc18a6bff) Signed-off-by: Willy Tarreau commit a49366cebf074a247c1da642b472e6623a84c6c2 Author: David Carlier Date: Thu Jan 11 14:20:43 2018 +0000 BUILD/MINOR: ancient gcc versions atomic fix Commit 1a69af6d3892fe1946bb8babb3044d2d26afd46e introduced code for atomic prior to 4.7. Unfortunately clang uses as well those constants which is misleading. (cherry picked from commit ec5e84552a0a2e0767431731c4f40d5627dc7bdd) Signed-off-by: Willy Tarreau commit 74cf455e471e2d5cb98ca28ba98fe36a29e8f846 Author: Willy Tarreau Date: Thu Jan 4 18:49:31 2018 +0100 MINOR: hathreads: add support for gcc < 4.7 Till now the use of __atomic_* gcc builtins required gcc >= 4.7. Since some supported and quite common operating systems like CentOS 6 still come with older versions (4.4) and the mapping to the older builtins is reasonably simple, let's implement it. This code is only used for gcc < 4.7. It has been quickly tested on a machine using gcc 4.4.4 and provided expected results. This patch should be backported to 1.8. (cherry picked from commit 1a69af6d3892fe1946bb8babb3044d2d26afd46e) Signed-off-by: Willy Tarreau commit de445a05b47e5d01756e41db1d33677999a2adc3 Author: William Lallemand Date: Tue Jan 9 23:12:27 2018 +0100 BUG/MEDIUM: mworker: execvp failure depending on argv[0] The copy_argv() function lacks a check on '-' to remove the -x, -sf and -st parameters. When reloading a master process with a path starting by /st, /sf, or /x.. the copy_argv() function skipped argv[0] leading to an execvp() without the binary. (cherry picked from commit 29f690c94574666f0789af5254890c498011e2ed) Signed-off-by: Willy Tarreau commit 8b803aabd3816f8bee26810eee7bc6c914b4652b Author: Olivier Houchard Date: Mon Jan 8 16:28:57 2018 +0100 MINOR: dns: Handle SRV record weight correctly. A SRV record weight can range from 0 to 65535, while haproxy weight goes from 0 to 256, so we have to divide it by 256 before handing it to haproxy. Also, a SRV record with a weight of 0 doesn't mean the server shouldn't be used, so use a minimum weight of 1. This should probably be backported to 1.8. (cherry picked from commit 2ec2db9725fb54a76c726c5b8cc502071c575d28) Signed-off-by: Willy Tarreau commit c1a1297d0963f0a60e00d419fdc3c0b0bc836e60 Author: Tim Duesterhus Date: Sat Jan 6 19:16:25 2018 +0100 BUG/MINOR: lua: Fix return value of Socket.settimeout The `socket.tcp.settimeout` method of Lua returns `1` in all cases, while the `Socket.settimeout` method of haproxy returns `0` in all cases. This breaks the `socket.http` module, because it validates the return value of `settimeout`. This bug was introduced in commit 7e7ac32dad1e15c19152d37aaf9ea6b3f00a7226 (which is the very first commit adding the Socket class to Lua). This bugfix should be backported to every branch containing that commit: - 1.6 - 1.7 - 1.8 A test case for this bug is as follows: The 'Test' response header will contain an HTTP status code with the patch applied and will be zero (nil) without the patch applied. http.lua: http = require("socket.http") core.register_action("bug", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://93.184.216.34", headers = { Host = "example.com" }, create = core.tcp, redirect = false } txn:set_var("txn.foo", c) end) haproxy.cfg: global lua-load /scratch/haproxy/http.lua frontend fe bind 127.0.0.1:8080 http-request lua.bug http-response set-header Test %[var(txn.foo)] default_backend be backend be server s example.com:80 (cherry picked from commit 119a5f10e47f3507e58116002583e1226473485d) Signed-off-by: Willy Tarreau commit 9db449a701cd9e43a04f49e2e477193fa5636323 Author: Tim Duesterhus Date: Sat Jan 6 19:04:45 2018 +0100 BUG/MEDIUM: lua: Fix IPv6 with separate port support for Socket.connect The `socket.tcp.connect` method of Lua requires at least two parameters: The host and the port. The `Socket.connect` method of haproxy requires only one when a host with a combined port is provided. This stems from the fact that `str2sa_range` is used internally in `hlua_socket_connect`. This very fact unfortunately causes a diversion in the behaviour of Lua's socket class and haproxy's for IPv6 addresses: sock:connect("::1", "80") works fine with Lua, but fails with: connect: cannot parse destination address '::1' in haproxy, because `str2sa_range` parses the trailing `:1` as the port. This patch forcefully adds a `:` to the end of the address iff a port number greater than `0` is given as the second parameter. Technically this breaks backwards compatibility, because the docs state: > The syntax "127.0.0.1:1234" is valid. in this case, the > parameter *port* is ignored. But: The connect() call can only succeed if the second parameter is left out (which causes no breakage) or if the second parameter is an integer or a numeric string. It seems unlikely that someone would provide an address with a port number and would also provide a second parameter containing a number other than zero. Thus I feel this breakage is warranted to fix the mismatch between haproxy's socket class and Lua's one. This commit should be backported to haproxy 1.8 only, because of the possible breakage of existing Lua scripts. (cherry picked from commit 6edab865f661edf732d30232808118585cc2a1c7) Signed-off-by: Willy Tarreau commit c03d497583bbf2cf5892692b7e68f2d5302e878d Author: Tim Duesterhus Date: Thu Jan 4 19:32:14 2018 +0100 DOC: lua: Fix typos in comments of hlua_socket_receive (cherry picked from commit b33754ce86cdcc877344ccedfb3a336b63154c2e) Signed-off-by: Willy Tarreau commit 0c9d9a9621577582f18d3176257f26ae36d2acf4 Author: Tim Duesterhus Date: Thu Jan 4 19:32:13 2018 +0100 BUG/MINOR: lua: Fix default value for pattern in Socket.receive The default value of the pattern in `Socket.receive` is `*l` according to the documentation and in the `socket.tcp.receive` method of Lua. The default value of `wanted` in `int hlua_socket_receive(struct lua_State *)` reflects this requirement, but the function fails to ensure this nonetheless: If no parameter is given the top of the Lua stack will have the index 1. `lua_pushinteger(L, wanted);` then pushes the default value onto the stack (with index 2). The following `lua_replace(L, 2);` then pops the top index (2) and tries to replace the index 2 with it. I am not sure why exactly that happens (possibly, because one cannot replace non-existent stack indicies), but this causes the stack index to be lost. `hlua_socket_receive_yield` then tries to read the stack index 2, to determine what to read and get the value `0`, instead of the correct HLSR_READ_LINE, thus taking the wrong branch. Fix this by ensuring that the top of the stack is not replaced by itself. This bug was introduced in commit 7e7ac32dad1e15c19152d37aaf9ea6b3f00a7226 (which is the very first commit adding the Socket class to Lua). This bugfix should be backported to every branch containing that commit: - 1.6 - 1.7 - 1.8 A test case for this bug is as follows: The 'Test' response header will contain an HTTP status line with the patch applied and will be empty without the patch applied. Replacing the `sock:receive()` with `sock:receive("*l")` will cause the status line to appear with and without the patch http.lua: core.register_action("bug", { "http-req" }, function(txn) local sock = core.tcp() sock:settimeout(60) sock:connect("127.0.0.1:80") sock:send("GET / HTTP/1.0\r\n\r\n") response = sock:receive() sock:close() txn:set_var("txn.foo", response) end) haproxy.cfg (bits omitted for brevity): global lua-load /scratch/haproxy/http.lua frontend fe bind 127.0.0.1:8080 http-request lua.bug http-response set-header Test %[var(txn.foo)] default_backend be backend be server s 127.0.0.1:80 (cherry picked from commit c6e377e6bb7de1fdb25510b18e4b49768aef0909) Signed-off-by: Willy Tarreau commit 52a80823e8c2d04635cc95e5d0ca9440a53441cf Author: William Lallemand Date: Wed Jan 3 19:15:51 2018 +0100 BUG/MEDIUM: ssl: cache doesn't release shctx blocks Since the rework of the shctx with the hot list system, the ssl cache was putting session inside the hot list, without removing them. Once all block were used, they were all locked in the hot list, which was forbiding to reuse them for new sessions. Bug introduced by 4f45bb9 ("MEDIUM: shctx: separate ssl and shctx") Thanks to Jeffrey J. Persch for reporting this bug. Must be backported to 1.8. (cherry picked from commit 99b90af6213809a018e89988d7139f7048e97208) Signed-off-by: Willy Tarreau commit 914cf78f2e715db9e33bac0b40a92851af87ea17 Author: Willy Tarreau Date: Thu Jan 4 18:55:19 2018 +0100 BUILD: ssl: silence a warning when building without NPN nor ALPN support When building with a library not offering any of these, ssl_conf_cur is not used. Can be backported to 1.8. (cherry picked from commit 5d4cafb6105ac70b57de1df2358efd5ab3844f2f) Signed-off-by: Willy Tarreau commit 646d23d1b502bc07a4a846f2ca7d332506b3087e Author: Willy Tarreau Date: Thu Jan 4 14:41:00 2018 +0100 BUG/MEDIUM: h2: properly handle the END_STREAM flag on empty DATA frames Peter Lindegaard Hansen reported a problem affecting some POST requests sent by MSIE on 1.8.3. Lukas found that we incorrectly dealt with the END_STREAM flag on empty DATA frames. What happens in fact is that while we correctly report that we've read a zero-byte frame, since commit 8fc016d ("BUG/MEDIUM: h2: support uploading partial DATA frames") backported into 1.8.2, we've been able to return without updating the parser's state nor checking the frame flags in this case. The fix is trival, we just need not to return too early. This fix must be backported to 1.8. (cherry picked from commit 4a28da1e9deadf0c542b957a323c1ca015c90fe4) Signed-off-by: Willy Tarreau commit 205f6751891b60dac1958181f53cdcd2f1067651 Author: Willy Tarreau Date: Sat Dec 30 18:13:19 2017 +0100 [RELEASE] Released version 1.8.3 Released version 1.8.3 with the following main changes : - BUG/MEDIUM: h2: properly handle and report some stream errors - BUG/MEDIUM: h2: improve handling of frames received on closed streams - DOC/MINOR: configuration: typo, formatting fixes - BUG/MEDIUM: h2: ensure we always know the stream before sending a reset - BUG/MEDIUM: mworker: don't close stdio several time - MINOR: don't close stdio anymore - BUG/MEDIUM: http: don't automatically forward request close - BUG/MAJOR: hpack: don't return direct references to the dynamic headers table - MEDIUM: h2: prepare a graceful shutdown when the frontend is stopped commit 4dbce456a223de3d06873828185ba789d5043def Author: Willy Tarreau Date: Sat Dec 30 18:08:13 2017 +0100 MEDIUM: h2: prepare a graceful shutdown when the frontend is stopped During a reload operation, instead of keeping the H2 connections opened forever causing confusion during configuration changes, let's send a graceful shutdown so that the client knows that it would better open a new connection for future requests. We can't really catch the signal from H2, but we can advertise this graceful shutdown upon the next I/O event (eg: a WINDOW_UPDATE from the client or a new request). One of the visible effect is that the old process quits much faster. This patch should be backported to 1.8 since it is affected by this problem. (cherry picked from commit 8ec140604acfa15b976285f54755b2cf38552273) Signed-off-by: Willy Tarreau commit 1d45cd9e87d473400241187e736e8eabe836b617 Author: Willy Tarreau Date: Sat Dec 30 16:56:28 2017 +0100 BUG/MAJOR: hpack: don't return direct references to the dynamic headers table Maximilian Böhm and Lucas Rolff both reported some random failed requests with HTTP/2. Upon deep investigation on detailed traces provided by Lucas, it turned out that some header names were occasionally corrupted and used to point to random strings within the dynamic headers table. The HPACK decoder must always return copies of header names that point to the dynamic headers table. Otherwise, the insertion of a header after the current one leading to a reorganization of the table will change the data the pointer designates. Unfortunately, one such copy was missing for indexed names, leading to random request failures due to invalid header names. Many thanks to Lucas who ran a large number of tests with full traces helping to capture a reproduceable sequence exhibiting this issue. This patch must be backported to 1.8. (cherry picked from commit bb39b4945b5264f5e21414ceb52df2e16fa9a953) Signed-off-by: Willy Tarreau commit 57b496dbd609c4cd2dca722f9747a066916d0830 Author: Willy Tarreau Date: Fri Dec 29 16:30:31 2017 +0100 BUG/MEDIUM: http: don't automatically forward request close Maximilian Böhm, and Lucas Rolff reported some frequent HTTP/2 POST failures affecting version 1.8.2 that were not affecting 1.8.1. Lukas Tribus determined that these ones appeared consecutive to commit a48c141 ("BUG/MAJOR: connection: refine the situations where we don't send shutw()"). It turns out that the HTTP request forwarding engine lets a shutr from the client be automatically forwarded to the server unless chunked encoding is in use. It's a bit tricky to meet this condition as it only happens if the shutr is not reported in the initial request. So if a request is large enough or the body is delayed after the headers (eg: Expect: 100-continue), the the function quits with channel_auto_close() left enabled. The patch above was not really related in fact. It's just that a previous bug was causing this shutw to be skipped at the lower layers, and the two bugs used to cancel themselves. In the HTTP request we should only pass the close in tunnel mode, as other cases either need to keep the connection alive (eg: for reuse) or will force-close it. Also the forced close will properly take care of avoiding the painful time-wait, which is not possible with the early close. This patch must be backported to 1.8 as it directly impacts HTTP/2, and may be backported to older version to save them from being abused by clients causing TIME_WAITs between haproxy and the server. Thanks to Lukas and Lucas for running many tests with captures allowing the bug to be narrowed down. (cherry picked from commit ff47b3f41da2f27d0fc61dd8d1aa77beb217f313) Signed-off-by: Willy Tarreau commit 60520669ec120a04fcf7e8720f0d21567ab6c9b3 Author: William Lallemand Date: Thu Dec 28 16:09:36 2017 +0100 MINOR: don't close stdio anymore Closing the standard IO FDs (0,1,2) can be troublesome, especially in the case of the master-worker. Instead of closing those FDs, they are now pointing to /dev/null which prevents sending debugging messages to the wrong FDs. This patch could be backported in 1.8. (cherry picked from commit e134041910aa3ea908c3891c3f56e5468ba6ed4e) Signed-off-by: Willy Tarreau commit 9d6688691d8d58b77d1b719896d6ffaf8e40e89b Author: PiBa-NL Date: Mon Dec 25 21:03:31 2017 +0100 BUG/MEDIUM: mworker: don't close stdio several time This patch makes sure that a frontend socket that gets created after initialization won't be closed when the master gets re-executed. When used in daemon mode, the master-worker is closing the FDs 0, 1, 2 after the fork of the children. When the master was reloading, those FDs were assigned again during the parsing of the configuration (probably for some listeners), and the workers were closing them thinking it was the stdio. This patch must be backported to 1.8. (cherry picked from commit 149a81a443a30fd8b23184447cc8a571fcbe0451) Signed-off-by: Willy Tarreau commit 36feba6d4111a57e5ba4a1f63929d17f0aa97180 Author: Willy Tarreau Date: Fri Dec 29 11:34:40 2017 +0100 BUG/MEDIUM: h2: ensure we always know the stream before sending a reset The recent patch introducing the H2_CS_FRAME_E state to emit stream resets was not totally correct in that in the rare case where there is no room left to emit the reset, the next call to process it later could use an uninitialized stream. This only affects responses to frames that are sent on closed streams though. This fix must be backported to 1.8. (cherry picked from commit d790143d990859f4d5af2c5a9e9e0aac94baa69e) Signed-off-by: Willy Tarreau commit b3d3c84e80d78eb6dd799d7e1883e4289ac81217 Author: Davor Ocelic Date: Mon Dec 25 17:49:28 2017 +0100 DOC/MINOR: configuration: typo, formatting fixes - Add simple typo and formatting fixes - Eliminate a couple > 80 column lines Changes do not affect technical content and can be backported. (cherry picked from commit e9ed281e9fac52bbcc0edd84e16c13b2260f28f8) Signed-off-by: Willy Tarreau commit 6a6b57893b29f25504450c69bcbb84e639c32736 Author: Willy Tarreau Date: Wed Dec 27 15:07:30 2017 +0100 BUG/MEDIUM: h2: improve handling of frames received on closed streams The h2spec utility found certain situations where we're returning an RST_STREAM while a GOAWAY is expected. While we can't always reliably decide which one to use (eg: after a stream has been closed for a long time), in practice we often still have the stream available until it's destroyed at the application level. This provides the flags we need to verify the conditions that led to its closure, namely if RST was sent or received, or if it was regularly closed using a double ES. The first step consists in marking all closed streams as having already sent an RST_STREAM frame. This will ensure that we can send an RST_STREAM for a late transmission on a stream we have forgotten about instead of risking to break the connection. The next steps consist in re-arranging the H2_SS_CLOSED checks so that we can deliver a GOAWAY frame for the few cases where an unexpected frame was received after a double ES. By carefully taking care of these specificities, we can reduce by 4 the number of remaining compliance issues. Note: some tests start to become a bit long and to be repeated at various places. Probably that adding a bitmask of allowed/forbidden frame types per state and/or per situation could significantly help. It's likely that some deeper tests in the frame handlers could also be removed now as they can't be triggered anymore. This fix should be backported to 1.8. (cherry picked from commit ab83750a29191ea841c80ed7c170630d45bfea43) Signed-off-by: Willy Tarreau commit d8a609f13fb8564c822765126087b52562573723 Author: Willy Tarreau Date: Wed Dec 27 11:02:06 2017 +0100 BUG/MEDIUM: h2: properly handle and report some stream errors Some stream errors applied to half-closed and closed streams are not properly reported, especially after the stream transistions to the closed state. The reason is that the code checks for this "error" stream state in order to send an RST frame. But if the stream was just closed or was already closed, there's no way to validate this condition, and the error is never reported to the peer. In order to address this situation, we'll add a new FRAME_E demux state which indicates that the previously parsed frame triggered a stream error of type STREAM CLOSED that needs to be reported. Proceeding like this will ensure that we don't lose that information even if we can't immediately send the message. It also removes the confusion where FRAME_A could be used either for ACKs or for RST. The state transition has been added after every h2s_error() on the demux path. It seems that we might need to have two distinct h2s_error() functions, one for the mux and another one for the demux, though it would provide little benefit. It also becomes more apparent that the H2_SS_ERROR state is only used to detect the need to report an error on the mux direction. Maybe this will have to be revisited later. This simple change managed to eliminate 5 bugs reported by h2spec. This fix must be backported to 1.8. (cherry picked from commit a20a519b8f6d56b95f7f6b5e2b88087ac5889b2e) Signed-off-by: Willy Tarreau commit 08396fa0af8955072a2a479168a9945995ed7a43 Author: Willy Tarreau Date: Sat Dec 23 11:30:51 2017 +0100 [RELEASE] Released version 1.8.2 Released version 1.8.2 with the following main changes : - BUG/MINOR: action: Don't check http capture rules when no id is defined - BUG/MAJOR: hpack: don't pretend large headers fit in empty table - BUG/MINOR: ssl: support tune.ssl.cachesize 0 again - BUG/MEDIUM: mworker: also close peers sockets in the master - BUG/MEDIUM: ssl engines: Fix async engines fds were not considered to fix fd limit automatically. - BUG/MEDIUM: checks: a down server going to maint remains definitely stucked on down state. - BUG/MEDIUM: peers: set NOLINGER on the outgoing stream interface - BUG/MEDIUM: h2: fix handling of end of stream again - MINOR: mworker: Update messages referencing exit-on-failure - MINOR: mworker: Improve wording in `void mworker_wait()` - CONTRIB: halog: Add help text for -s switch in halog program - BUG/MEDIUM: email-alert: don't set server check status from a email-alert task - BUG/MEDIUM: threads/vars: Fix deadlock in register_name - MINOR: systemd: remove comment about HAPROXY_STATS_SOCKET - DOC: notifications: add precisions about thread usage - BUG/MEDIUM: lua/notification: memory leak - MINOR: conn_stream: add new flag CS_FL_RCV_MORE to indicate pending data - BUG/MEDIUM: stream-int: always set SI_FL_WAIT_ROOM on CS_FL_RCV_MORE - BUG/MEDIUM: h2: automatically set CS_FL_RCV_MORE when the output buffer is full - BUG/MEDIUM: h2: enable recv polling whenever demuxing is possible - BUG/MEDIUM: h2: work around a connection API limitation - BUG/MEDIUM: h2: debug incoming traffic in h2_wake() - MINOR: h2: store the demux padding length in the h2c struct - BUG/MEDIUM: h2: support uploading partial DATA frames - MINOR: h2: don't demand that a DATA frame is complete before processing it - BUG/MEDIUM: h2: don't switch the state to HREM before end of DATA frame - BUG/MEDIUM: h2: don't close after the first DATA frame on tunnelled responses - BUG/MEDIUM: http: don't disable lingering on requests with tunnelled responses - BUG/MEDIUM: h2: fix stream limit enforcement - BUG/MINOR: stream-int: don't try to receive again after receiving an EOS - BUG: MAJOR: lb_map: server map calculation broken - BUG: MINOR: http: don't check http-request capture id when len is provided - BUILD/MINOR: Makefile : enabling USE_CPU_AFFINITY - BUG/MEDIUM: mworker: Set FD_CLOEXEC flag on log fd - DOC/MINOR: intro: typo, wording, formatting fixes - MINOR: netscaler: respect syntax - MINOR: netscaler: remove the use of cip_magic only used once - MINOR: netscaler: rename cip_len to clarify its uage - BUG/MEDIUM: netscaler: use the appropriate IPv6 header size - BUG/MAJOR: netscaler: address truncated CIP header detection - CONTRIB: iprange: Fix compiler warning in iprange.c - CONTRIB: halog: Fix compiler warnings in halog.c - BUG/MINOR: h2: properly report a stream error on RST_STREAM - MINOR: mux: add flags to describe a mux's capabilities - MINOR: stream-int: set flag SI_FL_CLEAN_ABRT when mux supports clean aborts - BUG/MEDIUM: stream: don't consider abortonclose on muxes which close cleanly - MINOR: netscaler: check in one-shot if buffer is large enough for IP and TCP header - MEDIUM: netscaler: do not analyze original IP packet size - MEDIUM: netscaler: add support for standard NetScaler CIP protocol - BUG/MEDIUM: checks: a server passed in maint state was not forced down. - BUG/MEDIUM: lua: fix crash when using bogus mode in register_service() - MINOR: http: adjust the list of supposedly cacheable methods - MINOR: http: update the list of cacheable status codes as per RFC7231 - MINOR: http: start to compute the transaction's cacheability from the request - BUG/MINOR: http: do not ignore cache-control: public - BUG/MINOR: http: properly detect max-age=0 and s-maxage=0 in responses - BUG/MINOR: cache: do not force the TX_CACHEABLE flag before checking cacheability - MINOR: http: add a function to check request's cache-control header field - BUG/MEDIUM: cache: do not try to retrieve host-less requests from the cache - BUG/MEDIUM: cache: replace old object on store - BUG/MEDIUM: cache: respect the request cache-control header - BUG/MEDIUM: cache: don't cache the response on no-cache="set-cookie" - BUG/MAJOR: connection: refine the situations where we don't send shutw() - BUG/MEDIUM: checks: properly set servers to stopping state on 404 commit af09a19c3f02ae7192672a18d256ea2054e3f70c Author: Willy Tarreau Date: Sat Dec 23 11:16:49 2017 +0100 BUG/MEDIUM: checks: properly set servers to stopping state on 404 Paul Lockaby reported that since 1.8, disable-on-404 doesn't work anymore in that the server stay up despite returning 404. Cyril spotted that this was caused by a copy-paste error introduced by commit 5a13351 ("BUG/MEDIUM: log: check result details truncated.") causing set_server_running() to be called instead of set_server_stopping() in this case. It can be reproduced with the simple test config below : defaults mode http timeout connect 1s timeout client 10s timeout server 10s listen http bind :8888 option httpchk GET / http-check disable-on-404 server s1 127.0.0.1:9001 check server s2 127.0.0.1:9002 check http-response add-header x-served-by %s listen s1 bind :9001 server next 127.0.0.1:9002 http-response set-status 404 frontend s2 bind :9002 http-request redirect location / S1 is supposed to be stopping and s2 up, which is not the case. After calling the correct function, only S2 is used now. This needs to be backported to 1.8. (cherry picked from commit b26881a5d52c8e4abd61a8305c24baf3df990d59) Signed-off-by: Willy Tarreau commit 58d05368d690b0a5962bafa38595dca7bec9ee67 Author: Willy Tarreau Date: Fri Dec 22 18:46:33 2017 +0100 BUG/MAJOR: connection: refine the situations where we don't send shutw() Since commit f9ce57e ("MEDIUM: connection: make conn_sock_shutw() aware of lingering"), we refrain from performing the shutw() on the socket if there is no lingering risk. But there is a problem with this in tunnel and in TCP modes where a client is explicitly allowed to send a shutw to the server, eventhough it it risky. Not doing it creates this situation reported by Ricardo Fraile and diagnosed by Christopher : a typical HTTP client (eg: curl) connecting via the config below to an HTTP server would receive its response, immediately close while the server remains in keep-alive mode. The shutr() received by haproxy from the client is "propagated" to the server side but not acted upon because fdtab[fd].linger_risk is set, so we expect that the next close will immediately complete this operation. listen proxy-tcp bind 127.0.0.1:8888 mode tcp timeout connect 5s timeout server 10s timeout client 10s server server1 127.0.0.1:8000 But since the whole stream will not end until the server closes in turn, the server doesn't close and haproxy expires on server timeout. This problem has already struck by waking up an older bug and was partially fixed with commit 8059351 ("BUG/MEDIUM: http: don't disable lingering on requests with tunnelled responses") though it was not enough. The problem is that linger_risk is not suited here. In fact we need to know whether or not it is desired to close normally or silently, and whether or not a shutr() has already been received on this connection. This is the approach this patch takes, and it solves the problem for the various difficult modes (tcp, http-server-close, pretend-keepalive). This fix needs to be backported to 1.8. Many thanks to Ricardo for providing very detailed traces and configurations. (cherry picked from commit a48c141f448e04f6da8a5b40c677042fbc730f04) Signed-off-by: Willy Tarreau commit 013da51f08155dba75c4e21273ff79e6bed37e39 Author: Willy Tarreau Date: Fri Dec 22 18:03:04 2017 +0100 BUG/MEDIUM: cache: don't cache the response on no-cache="set-cookie" If the server mentions no-cache="set-cookie" in the response headers, we must guarantee that any set-cookie field will not be stored. We cannot edit the stored response on the fly to trim the set-cookie header so we can refrain from storing a response containing such a header. In theory we could use TX_SCK_PRESENT for this but this one is only set when the cookie is being watched by the configuration. Since these responses are not very frequent and often accompanied with a set-cookie header, let's simply refrain from caching whenever such directive is present. This needs to be backported to 1.8. (cherry picked from commit d4569d1937a160c17cb80c1848dadcad2a6f7d40) Signed-off-by: Willy Tarreau commit 6d37c43fb4327c58ee6ddac59ea006a779879583 Author: Willy Tarreau Date: Fri Dec 22 17:47:35 2017 +0100 BUG/MEDIUM: cache: respect the request cache-control header Till now if a client emitted a request featureing a cache-control header, this one was not respected and a stale object could still be delievered.r This patch ensures that : - cache-control: no-cache disables retrieval from the cache but does not prevent the newly fetched object from being stored ; - cache-control: no-store can safely retrieve from the cache but prevents from storing any fetched object - cache-control: max-age/max-stale/min-fresh act like no-cache - pragma: no-cache acts like cache-control: no-cache. This needs to be backported to 1.8. (cherry picked from commit 504455c5336c203ad1462e65a1579e04b02299e8) Signed-off-by: Willy Tarreau commit 5519d7f311a876c300525306271981ecdc39f73f Author: Willy Tarreau Date: Fri Dec 22 17:42:46 2017 +0100 BUG/MEDIUM: cache: replace old object on store Currently the cache aborts a store operation if the object to store already exists in the cache. This is used to avoid storing multiple copies at the same time on concurrent accesses. It causes an issue though, which is that existing unexpired objects cannot be updated. This happens when any request criterion disables the retrieval from the cache (eg: with max-age or any other cache-control condition). For now, let's simply replace the previous existing entry by unlinking it from the index. This could possibly be improved in the future if needed. This fix needs to be backported to 1.8. (cherry picked from commit c9bd34c7e0b0b4845d058c67c55fb0600a339919) Signed-off-by: Willy Tarreau commit 0d16433d4c13454a6e8a4f26953f00e79989959d Author: Willy Tarreau Date: Fri Dec 22 16:32:43 2017 +0100 BUG/MEDIUM: cache: do not try to retrieve host-less requests from the cache All HTTP/1.1 requests the Host header share the same hash key 0 and will be return the first cached object. Let's add the check on the call to sha1_hosturi() to prevent this from happening. This must be backported to 1.8. (cherry picked from commit 7704b1e89a202c5b09d89fbc50f042a18113a839) Signed-off-by: Willy Tarreau commit 2c3cd52e857cace750a2a248c5824e414a241ce5 Author: Willy Tarreau Date: Fri Dec 22 15:03:36 2017 +0100 MINOR: http: add a function to check request's cache-control header field The new function check_request_for_cacheability() is used to check if a request may be served from the cache, and/or allows the response to be stored into the cache. For this it checks the cache-control and pragma header fields, and adjusts the existing TX_CACHEABLE and a new TX_CACHE_IGNORE flags. For now, just like its response side counterpart, it only checks the first value of the header field. These functions should be reworked to improve their parsers and validate all elements. (cherry picked from commit 0ad8e0dfeaad3fe64890b6cbdcff23f1b123d5a1) Signed-off-by: Willy Tarreau commit 5a23fb5326f7e01c3000a2bb1717b0120be8c6f4 Author: Willy Tarreau Date: Thu Dec 21 15:59:17 2017 +0100 BUG/MINOR: cache: do not force the TX_CACHEABLE flag before checking cacheability The cache used to set this flag before calling check_response_for_cacheability() due to the way the flags were previously set (too late), but this is a bad idea as it loses the information of the implicit caching rules related to the method and the status code. Let's only rely on what was determined during the request and response parsing instead and not change it. This fix must be backported to 1.8, and it requires that the following patches are also merged : - MINOR: http: adjust the list of supposedly cacheable methods - MINOR: http: update the list of cacheable status codes as per RFC7231 - MINOR: http: start to compute the transaction's cacheability from the request - BUG/MINOR: http: do not ignore cache-control: public (cherry picked from commit faf2909f9f73a43b8db9a9ef884f64a0113ac1c9) Signed-off-by: Willy Tarreau commit b396103341dda1b73ff43f84ea8cdf270710db6f Author: Willy Tarreau Date: Fri Dec 22 15:35:11 2017 +0100 BUG/MINOR: http: properly detect max-age=0 and s-maxage=0 in responses In 1.3.8, commit a15645d ("[MAJOR] completed the HTTP response processing.") improved the response parser by taking care of the cache-control header field. The parser is wrong because it is split in two parts, one checking for elements containing an equal sign and the other one for those without. The "max-age=0" and "s-maxage=0" tests were located at the wrong place and thus have never matched. In practice the side effect was very minimal given that this code used to be enabled only when checking if a cookie had the risk of being cached or not. Recently in 1.8 it was also used to decide if the response could be cached but in practice the cache takes care of these values by itself so there is very limited impact. This fix can be backported to all stable versions. (cherry picked from commit d3900cc31d4ca14a9c45902b2543da2f39ce2c26) Signed-off-by: Willy Tarreau commit 2a0a3ac28ad0c105b7a9efc9578a5a627166f964 Author: Willy Tarreau Date: Thu Dec 21 16:08:09 2017 +0100 BUG/MINOR: http: do not ignore cache-control: public In check_response_for_cacheability(), we don't check the cache-control flags if the response is already supposed not to be cacheable. This was introduced very early when cache-control:public was not checked, and it basically results in this last one not being able to properly mark the response as cacheable if it uses a status code which is non-cacheable by default. Till now the impact is very limited as it doesn't check that cookies set on non-default status codes are not cacheable, and it prevents the cache from caching such responses. Let's fix this by doing two things : - remove the test for !TX_CACHEABLE in the aforementionned function - however take care of 1xx status codes here (which used to be implicitly dealt with by the test above) and remove the explicit check for 101 in the caller This fix must be backported to 1.8. (cherry picked from commit 12b32f212f4c7457eeae37edb06dd0a06995fce1) Signed-off-by: Willy Tarreau commit 1dd7765d3276124c366fe4c6a68c9e587dc34bfc Author: Willy Tarreau Date: Thu Dec 21 15:13:09 2017 +0100 MINOR: http: start to compute the transaction's cacheability from the request There has always been something odd with the way the cache-control flags are checked. Since it was made for checking for the risk of leaking cookies only, all the processing was done in the response. Because of this it is not possible to reuse the transaction flags correctly for use with the cache. This patch starts to change this by moving the method check in the request so that we know very early whether the transaction is expected to be cacheable and that this status evolves along with checked headers. For now it's not enough to use from the cache yet but at least it makes the flag more consistent along the transaction processing. (cherry picked from commit 83ece462b433b7b354036e33e810d1e8264d9c1f) Signed-off-by: Willy Tarreau commit f7391e84b0c66f812c919c261b329e36e0c21caf Author: Willy Tarreau Date: Thu Dec 21 11:41:38 2017 +0100 MINOR: http: update the list of cacheable status codes as per RFC7231 Since RFC2616, the following codes were added to the list of codes cacheable by default : 204, 404, 405, 414, 501. For now this it only checked by the checkcache option to detect cacheable cookies. (cherry picked from commit c55ddce65c6d727da3587074caea4733c6a91f96) Signed-off-by: Willy Tarreau commit 0e297f28869f906ced38aa4348b2e60bae50fb57 Author: Willy Tarreau Date: Thu Dec 21 11:32:55 2017 +0100 MINOR: http: adjust the list of supposedly cacheable methods We used to have a rule inherited from RFC2616 saying that the POST method was the only uncacheable one, but things have changed since and RFC7231+7234 made it clear that in fact only GET/HEAD/OPTIONS/TRACE are cacheable. Currently this rule is only used to detect cacheable cookies. (cherry picked from commit 24ea0bcb1d56ae0c99ed09273c556b0cee28e77e) Signed-off-by: Willy Tarreau commit 8a80594dc546b629b32b7197fbeb905c6ed06ee0 Author: Eric Salama Date: Thu Dec 21 14:30:07 2017 +0100 BUG/MEDIUM: lua: fix crash when using bogus mode in register_service() When using an incorrect 'mode' as 2nd argument of core.register_service(), HAProxy crashes while displaying the error message. To be backported to 1.8, 1.7 and 1.6. (cherry picked from commit fe7456f3b78ca1721d7f10d9df073ed8174f83c0) Signed-off-by: Willy Tarreau commit e196b56d15cb40bc24dc6193401f31465b8dcd3b Author: Emeric Brun Date: Thu Dec 21 14:42:26 2017 +0100 BUG/MEDIUM: checks: a server passed in maint state was not forced down. Setting a server in maint mode, the required next_state was not set before calling the 'lb_down' function and so the system state was never commited. This patch should be backported in 1.8 (cherry picked from commit e31148031f5c5327b342fe268d0cd4c882b0511d) Signed-off-by: Willy Tarreau commit 6447f18676f0f2eebdb87ee68e4017c19659b77d Author: Bertrand Jacquin Date: Tue Dec 12 01:17:23 2017 +0000 MEDIUM: netscaler: add support for standard NetScaler CIP protocol It looks like two version of the protocol exist as reported by Andreas Mahnke. This patch add support for both legacy and standard CIP protocol according to NetScaler specifications. (cherry picked from commit 72fa1ec24e96d37c66126918e3ce129da762803a) [wt: backported to 1.8 with Bertrand's approval to address the version handling issue Andreas was experiencing] Signed-off-by: Willy Tarreau commit 2bab25eabc364c05475fc99e8480b9d187a0da73 Author: Bertrand Jacquin Date: Wed Dec 13 01:29:56 2017 +0000 MEDIUM: netscaler: do not analyze original IP packet size Original informations about the client are stored in the CIP encapsulated IP header, hence there is no need to consider original IP packet length to determine if data are missing. Instead this change detect missing data if the remaining buffer is large enough to contain a minimal IP and TCP header and if the buffer has as much data as CIP is telling. (cherry picked from commit a341a2f479745adb486b153af61958f35f5612b0) [wt: needed by next patch] Signed-off-by: Willy Tarreau commit e09df553f9e15c55c1cfa99110f44a3d92d67b9b Author: Bertrand Jacquin Date: Wed Dec 13 01:15:05 2017 +0000 MINOR: netscaler: check in one-shot if buffer is large enough for IP and TCP header There is minimal gain in checking first the IP header length and then the TCP header length since we always want to capture information about both protocols. IPv4 length calculation was incorrect since IPv4 ip_len actually defines the total length of IPv4 header and following data. (cherry picked from commit 67de5a295c9f4693f0c65d69fed67a3618bee6dd) [wt: needed by next patch] Signed-off-by: Willy Tarreau commit 16c805f7304ad7eb7f4df4ba3de6959c37d6ee37 Author: Willy Tarreau Date: Wed Dec 20 16:56:50 2017 +0100 BUG/MEDIUM: stream: don't consider abortonclose on muxes which close cleanly The H2 mux can cleanly report an error when a client closes, which is not the case for the pass-through mux which only reports shutr. That was the reason why "option abortonclose" was created since there was no way to distinguish a clean shutdown after sending the request from an abort. The problem is that in case of H2, the streams are always shut read after the request is complete (when the END_STREAM flag is received), and that when this lands on a backend configured with "option abortonclose", this aborts the request. Disabling abortonclose is not always an option when H1 and H2 have to coexist. This patch makes use of the newly introduced mux capabilities reported via the stream interface's SI_FL_CLEAN_ABRT indicating that the mux is safe and that there is no need to turn a clean shutread into an abort. This way abortonclose has no effect on requests initiated from an H2 mux. This patch as well as these 3 previous ones need to be backported to 1.8 : - BUG/MINOR: h2: properly report a stream error on RST_STREAM - MINOR: mux: add flags to describe a mux's capabilities - MINOR: stream-int: set flag SI_FL_CLEAN_ABRT when mux supports clean aborts (cherry picked from commit 7aa15b072ea752399144c8193a85c55e26893591) Signed-off-by: Willy Tarreau commit c8f6440f7f142015338c776db0ea4776a38d398e Author: Willy Tarreau Date: Wed Dec 20 16:31:43 2017 +0100 MINOR: stream-int: set flag SI_FL_CLEAN_ABRT when mux supports clean aborts By copying the info in the stream interface that the mux cleanly reports aborts, we'll have the ability to check this flag wherever needed regardless of the presence of a mux or not. (cherry picked from commit 984fca936305d7552d92559436f3d5dbe4d917a0) [wt: backported since needed to fix abortonclose+h2 in 1.8] Signed-off-by: Willy Tarreau commit 1784dfdfa409c9bb6156541f85ab64f15b5145a9 Author: Willy Tarreau Date: Wed Dec 20 16:14:44 2017 +0100 MINOR: mux: add flags to describe a mux's capabilities This new field will be used to describe certain properties of some muxes. For now we only add MX_FL_CLEAN_ABRT to indicate that a mux is able to unambiguously report aborts using CS_FL_ERROR contrary to others who may only report it via a read0. This will be used to improve handling of the abortonclose option with H2. Other flags may come later to report multiplexing capabilities or not, support of client/server sides etc. (cherry picked from commit 28f1cb9da281ab340c034c3ccf0f8177d1ab9858) [wt: backported since needed to fix abortonclose+h2 in 1.8] Signed-off-by: Willy Tarreau commit 5012e89e9a69dc25ae954e34e3e9e0bff27ad561 Author: Willy Tarreau Date: Fri Dec 15 11:56:29 2017 +0100 BUG/MINOR: h2: properly report a stream error on RST_STREAM We want to report such an error since H2 allows to differenciate between an end of stream and an abort. To be backported to 1.8. (cherry picked from commit 2153d3ce7370711172967936344cdfc8ff25dd0a) Signed-off-by: Willy Tarreau commit 78981221ebf82713a12160dd269cea557e534024 Author: Ryan O'Hara Date: Fri Dec 15 10:21:39 2017 -0600 CONTRIB: halog: Fix compiler warnings in halog.c There were several unused variables in halog.c that each caused a compiler warning [-Wunused-but-set-variable]. This patch simply removes the declaration of said vairables and any instance where the unused variable was assigned a value. (cherry picked from commit 8cb99934698fd842b1ac8fc1316952670d4f477b) Signed-off-by: Willy Tarreau commit ceeeeb67b48846ad558a0d6b16f7859b41f28382 Author: Ryan O'Hara Date: Fri Dec 15 10:21:29 2017 -0600 CONTRIB: iprange: Fix compiler warning in iprange.c The declaration of main() in iprange.c did not specify a type, causing a compiler warning [-Wimplicit-int]. This patch simply declares main() to be type 'int' and calls exit(0) at the end of the function. (cherry picked from commit 957d12028e6a15cbbb42beab3dfe27445133998a) Signed-off-by: Willy Tarreau commit 5f100204baf68fcec9628c5d33a0aa39c1b51b8b Author: Bertrand Jacquin Date: Wed Dec 13 00:53:33 2017 +0000 BUG/MAJOR: netscaler: address truncated CIP header detection Buffer line is manually incremented in order to progress in the trash buffer but calculation are made omitting this manual offset. This leads to random packets being rejected with the following error: HTTP/1: Truncated NetScaler Client IP header received Instead, once original IP header is found, use the IP header length without considering the CIP encapsulation. (cherry picked from commit 43a66a96b3d39157687b6d53c57f431a7f83cec5) Signed-off-by: Willy Tarreau commit 87fbb80688939cec90c9610bc3e6b86fe938d6a2 Author: Bertrand Jacquin Date: Wed Dec 13 01:40:24 2017 +0000 BUG/MEDIUM: netscaler: use the appropriate IPv6 header size IPv6 header has a fixed size of 40 bytes, not 20. (cherry picked from commit c7cc69ac3676c0d3cc14e80c6f7ccde49b05de2c) Signed-off-by: Willy Tarreau commit 635e6197d95f37653412f18a5bda75c35a4077ab Author: Bertrand Jacquin Date: Wed Dec 13 01:23:39 2017 +0000 MINOR: netscaler: rename cip_len to clarify its uage cip_len was meant to be the length of the data encapsulated in the CIP protocol, the size the IP and TCP header (cherry picked from commit 7d668f9e76a9d381ad33784052fdd9072b2ef339) [wt: cleanup only backported to fix the protocol parser in next patches] Signed-off-by: Willy Tarreau commit 235dae9769152922420e405f0a601dd3c119fca8 Author: Bertrand Jacquin Date: Wed Dec 13 01:07:12 2017 +0000 MINOR: netscaler: remove the use of cip_magic only used once (cherry picked from commit 4b4c286bee603572323e72b56faebbe867498a01) [wt: cleanup only backported to fix the protocol parser in next patches] Signed-off-by: Willy Tarreau commit b4d6343e6e573a8c59e4686657fbac67d8a81d69 Author: Bertrand Jacquin Date: Wed Dec 13 00:58:51 2017 +0000 MINOR: netscaler: respect syntax As per doc/coding-style.txt (cherry picked from commit b387591f323a0f283e538b3fc3797b975a8febbc) [wt: actually it's only a harmless cleanup but alters the context of subsequent fixes] Signed-off-by: Willy Tarreau commit b844c4d9e030c54982b1c7a45f7e1a7ed2012969 Author: Davor Ocelic Date: Tue Dec 19 23:30:39 2017 +0100 DOC/MINOR: intro: typo, wording, formatting fixes - Fix a couple typos - Introduce a couple simple rewordings - Eliminate > 80 column lines Changes do not affect technical content and can be backported. (cherry picked from commit 4094ce1a239fbf2b604ba02917cfe9561060026e) Signed-off-by: Willy Tarreau commit 43845c32142a286b6dce384e86e8d2556e3b0d34 Author: Christopher Faulet Date: Tue Dec 19 10:35:53 2017 +0100 BUG/MEDIUM: mworker: Set FD_CLOEXEC flag on log fd A log socket (UDP or UNIX) is opened by the master during its startup, when the first log message is sent. So, to prevent FD leaks, we must ensure we correctly close it during a reload. By setting FD_CLOEXEC bit on it, we are sure it will be automatically closed it during a reload. This patch must be backported in 1.8. (cherry picked from commit 789691778fde0e792377b631e2c076f449ce6515) Signed-off-by: Willy Tarreau commit 36a160ef7ebed42d4b05aec941065dd341e66156 Author: David Carlier Date: Wed Nov 29 11:05:12 2017 +0000 BUILD/MINOR: Makefile : enabling USE_CPU_AFFINITY FreeBSD can handle cpuset matters just fine, we can hence enable it by default as linux2628 TARGET. (cherry picked from commit 903ddfd9c88e7c39fa001fe9f0b0b743667b9c69) Signed-off-by: Willy Tarreau commit cf7ff866a1147158b84208533d7d7bd80d04036e Author: Cyril Bonté Date: Thu Dec 14 22:44:41 2017 +0100 BUG: MINOR: http: don't check http-request capture id when len is provided Randomly, haproxy could fail to start when a "http-request capture" action is defined, without any change to the configuration. The issue depends on the memory content, which may raise a fatal error like : unable to find capture id 'xxxx' referenced by http-request capture rule Commit fd608dd2 already prevents the condition to happen, but this one should be included for completeness and to reclect the code on the response side. The issue was introduced recently by commit 29730ba5 and should only be backported to haproxy 1.8. (cherry picked from commit 9fc9e53763f0bda431879ceaef3968bf1f43509e) Signed-off-by: Willy Tarreau commit 22ca440f0ab444336deadc992059e2b5510b8c1d Author: Cyril Bonté Date: Thu Dec 14 16:39:26 2017 +0100 BUG: MAJOR: lb_map: server map calculation broken Adrian Williams reported that several balancing methods were broken and sent all requests to one backend. This is a regression in haproxy 1.8 where the server score was not correctly recalculated. This fix must be backported to the 1.8 branch. (cherry picked from commit 3906d5739c1c2f24bedeffcd0664190f85896523) Signed-off-by: Willy Tarreau commit 2c5fbcb4857052d0ba209d81b34c72d4632dc4a7 Author: Willy Tarreau Date: Tue Dec 12 09:58:40 2017 +0100 BUG/MINOR: stream-int: don't try to receive again after receiving an EOS When an end of stream has been reported, we should not try to receive again as the mux layer might not be prepared to this and could report unexpected errors. This is more of a strengthening measure that follows the introduction of conn_stream that came in 1.8. It's desired to backport this into 1.8 though it's uncertain at this time whether it may have caused real issues. (cherry picked from commit b78b80efe57fb7c6e195850626065d55ed5e4e3b) Signed-off-by: Willy Tarreau commit 66bde4277230771071e5f3327e77ef906a9e17b5 Author: Willy Tarreau Date: Thu Dec 14 12:00:14 2017 +0100 BUG/MEDIUM: h2: fix stream limit enforcement Commit 4974561 ("BUG/MEDIUM: h2: enforce the per-connection stream limit") implemented a stream limit enforcement on the connection but it was not correctly done as it would count streams still known by the connection, which includes the lingering ones that are already marked close. We need to count only the non-closed ones, which this patch does. The effect is that some streams are rejected a bit before the limit. This fix needs to be backported to 1.8. (cherry picked from commit 91bfdd7e04e105559c00f0d711842ad91ca96b26) Signed-off-by: Willy Tarreau commit f126a73be0e87eccf12eb515b8e1c5bab374bb78 Author: Willy Tarreau Date: Thu Dec 14 10:43:31 2017 +0100 BUG/MEDIUM: http: don't disable lingering on requests with tunnelled responses The HTTP forwarding engine needs to disable lingering on requests in case the connection to the server has to be suddenly closed due to http-server-close being used, so that we don't accumulate lethal TIME_WAIT sockets on the outgoing side. A problem happens when the server doesn't advertise a response size, because the response message quickly goes through the MSG_DONE and MSG_TUNNEL states, and once the client has transferred all of its data, it turns to MSG_DONE and immediately sets NOLINGER and closes before the server has a chance to respond. The problem is that this destroys some of the pending DATA being uploaded, the server doesn't receive all of them, detects an error and closes. This early NOLINGER is inappropriate in this situation because it happens before the response is transmitted. This state transition to MSG_TUNNEL doesn't happen when the response size is known since we stay in MSG_DATA (and related states) during all the transfer. Given that the issue is only related to connections not advertising a response length and that by definition these connections cannot be reused, there's no need for NOLINGER when the response's transfer length is not known, which can be verified when entering the CLOSED state. That's what this patch does. This fix needs to be backported to 1.8 and very likely to 1.7 and older as it affects the very rare case where a client immediately closes after the last uploaded byte (typically a script). However given that the risk of occurrence in HTTP/1 is extremely low, it is probably wise to wait before backporting it before 1.8. (cherry picked from commit 805935147a8d1520d8203b2bfe24726585e436a2) Signed-off-by: Willy Tarreau commit 3ff6e108dfcdf93c03327c70d3b65fef894c393d Author: Willy Tarreau Date: Thu Dec 14 10:55:21 2017 +0100 BUG/MEDIUM: h2: don't close after the first DATA frame on tunnelled responses Tunnelled responses are those without a content-length nor a chunked encoding. They are specially dealt with in the current code but the behaviour is not correct. The fact that the chunk size is left to zero with a state artificially set to CHUNK_SIZE validates the test on whether or not to set the end of stream flag. Thus the first DATA frame always carries the ES flag and subsequent ones remain blocked. This patch fixes it in two ways : - update h1m->curr_len to the size of the current buffer so that it is properly subtracted later to find the real end ; - don't set the state to CHUNK_SIZE when there's no content-length and instead set it to CHUNK_SIZE only when there's chunking. This fix needs to be backported to 1.8. (cherry picked from commit 13e4e94dae3e51b4076085fdac395a1d71e26ac1) Signed-off-by: Willy Tarreau commit c588002a3c99b8373326a81023b1c46f09bb669d Author: Willy Tarreau Date: Mon Dec 11 18:45:08 2017 +0100 BUG/MEDIUM: h2: don't switch the state to HREM before end of DATA frame We used to switch the stream's state to HREM when seeing and ES bit on the DATA frame before actually being able to process that frame, possibly resulting in the DATA frame being processed after the stream was seen as half-closed and possibly being rejected. The state must not change before the frame is really processed. Also fixes a harmless typo in the flag name which should have DATA and not HEADERS in its name (but all values are equal). Must be backported to 1.8. (cherry picked from commit c4134ba8b0dacfe008a7974a8d31630534030ca7) Signed-off-by: Willy Tarreau commit 6066e75230f8bc8e74aedf5166e6da0d17e1b924 Author: Willy Tarreau Date: Mon Dec 11 18:36:37 2017 +0100 MINOR: h2: don't demand that a DATA frame is complete before processing it Since last commit it's not required that the DATA frames are complete anymore so better start with what we have. Only the HEADERS frame requires this. This may be backported as part of the upload fixes. (cherry picked from commit 6847262211dd415140201ed312137055135f79cc) Signed-off-by: Willy Tarreau commit f54f27cb84d4e8bf477df3d400ccb5fbd55ac5b1 Author: Willy Tarreau Date: Mon Dec 11 18:27:15 2017 +0100 BUG/MEDIUM: h2: support uploading partial DATA frames We currently have a problem with DATA frames when they don't fit into the destination buffer. While it was imagined that in theory this never happens, in practice it does when "option http-buffer-request" is set, because the headers don't leave the target buffer before trying to read so if the frame is full, there's never enough room. This fix consists in reading what can be read from the frame and advancing the input buffer. Once the contents left are only the padding, the frame is completely processed. This also solves another problem we had which is that it was possible to fill a request buffer beyond its reserve because the argument was not respected in h2_rcv_buf(). Thus it's possible that some POST requests sent at once with a headers+body filling exactly a buffer could result in "400 bad req" when trying to add headers. This fix must be backported to 1.8. (cherry picked from commit 8fc016d0fee27e2ae46bbf29b05410b78e236899) Signed-off-by: Willy Tarreau commit 0e273d9ad52a9b5832b685e769c7f45a90128a5c Author: Willy Tarreau Date: Mon Dec 11 15:17:36 2017 +0100 MINOR: h2: store the demux padding length in the h2c struct We'll try to process partial frames and for this we need to know the padding length. The first step requires to extract it during the parsing and store it in the demux context in the connection. Till now it was only processed at once. (cherry picked from commit 05e5dafe9a3fb2c07fc9bcb1e5c684ce2d7c4f33) Signed-off-by: Willy Tarreau commit 0395f7b184c09a9c49f68f2b2252216b2c770f6e Author: Willy Tarreau Date: Thu Dec 14 10:34:52 2017 +0100 BUG/MEDIUM: h2: debug incoming traffic in h2_wake() Even after previous commit ("BUG/MEDIUM: h2: work around a connection API limitation") there is still a problem with some requests. Sometimes when polling for more request data while some pending data lies in the buffer, there's no way to enter h2_recv() because the FD is not marked ready for reading. We need to slightly change the approach and make h2_recv() only receive from the buffer and h2_wake() always attempt to demux if the demux is not blocked. However, if the connection is already being polled for reading, it will not wake up from polling. For this reason we need to cheat and also pretend a request for sending data, which ensures that as soon as any direction may move, we can continue to demux. This shows that in the long term we probably need a better way to resume an interrupted operation at the mux level. With this fix, no more hangups happen during uploads. Note that this time the setup required to provoke the hangups was a bit complex : - client is "curl" running on local host, uploading 1.7 MB of data via haproxy - haproxy running on local host, forwarding to a remote server through a 100 Mbps only switch - timeouts disabled on haproxy - remote server made of thttpd executing a cgi reading request data through "dd bs=10" to slow down everything. With such a setup, around 3-5% of the connections would hang up. This fix needs to be backported to 1.8. (cherry picked from commit d13bf27e7836a050ac2638ccb8f603c6ca743cd1) Signed-off-by: Willy Tarreau commit cf83eab931af79b6ffe920b356462d0623d7e928 Author: Willy Tarreau Date: Tue Dec 12 11:01:44 2017 +0100 BUG/MEDIUM: h2: work around a connection API limitation The connection API permits us to enable or disable receiving on a connection. The underlying FD layer arranges this with the polling and the fd cache. In practice, if receiving was allowed and an end of buffer was reached, the FD is subscribed to the polling. If later we want to process pending data from the buffer, we have to enable receiving again, but since it's already enabled (in polled mode), nothing happens and the pending data remain stuck until a new event happens on the connection to wake the FD up. This is a limitation of the internal connection API which is not very friendly to the new mux architecture. The visible effect is that certain uploads to slow servers experience truncation on timeout on their last blocks because nothing new comes from the connection to wake it up while it's being polled. In order to work around this, there are two solutions : - either cheat on the connection so that conn_update_xprt_polling() always performs a call to fd_may_recv() after fd_want_recv(), that we can trigger from the mux by always calling conn_xprt_stop_recv() before conn_xprt_want_recv(), but that's a bit tricky and may have side effects on other parts (eg: SSL) - or we refrain from receiving in the mux as soon as we're busy on anything else, regardless of whether or not some room is available in the receive buffer. This patch takes the second approach above. This way once we read some data, as soon as we detect that we're stuck, we immediately stop receiving. This ensures the event doesn't go into polled mode for this period and that as soon as we're unstuck we can continue. In fact this guarantees that we can only wait on one side of the mux for a given direction. A future improvement of the connection layer should make it possible to resume processing of an interrupted receive operation. This fix must be backported to 1.8. (cherry picked from commit 6042aeb1e8b19da25772a6f77fcce862ff87aaa2) Signed-off-by: Willy Tarreau commit dd8ea125889d0a55f883c2b86578f32173214c4c Author: Willy Tarreau Date: Sun Dec 10 22:17:57 2017 +0100 BUG/MEDIUM: h2: enable recv polling whenever demuxing is possible In order to allow demuxing when the dmux buffer is full, we need to enable data receipt in multiple conditions. Since the conditions are a bit complex, they have been delegated to a new function h2_recv_allowed() which follows these rules : - if an error or a shutdown was detected on the connection and the buffer is empty, we must not attempt to receive - if the demux buf failed to be allocated, we must not try to receive and we know there is nothing pending - if the buffer is not full, we may attempt to receive - if no flag indicates a blocking condition, we may attempt to receive - otherwise must may not attempt No more truncated payloads are detected in tests anymore, which seems to indicate that the issue was worked around. A better connection API will have to be created for new versions to make this stuff simpler and more intuitive. This fix needs to be backported to 1.8 along with the rest of the patches related to CS_FL_RCV_MORE. (cherry picked from commit 315d807cbc21ec329d547f6174d8e1136bf59187) Signed-off-by: Willy Tarreau commit 3ff736ae82c5a1a4c8be3b43b837f672ac3b9e03 Author: Willy Tarreau Date: Sun Dec 10 21:28:43 2017 +0100 BUG/MEDIUM: h2: automatically set CS_FL_RCV_MORE when the output buffer is full If we can't demux pending data due to a stream buffer full condition, we now set CS_FL_RCV_MORE on the conn_stream so that the stream layer knows it must call back as soon as possible to restart demuxing. Without this, some uploaded payloads are truncated if the server does not consume them fast enough and buffers fill up. Note that this is still not enough to solve the problem, some changes are required on the recv() and update_poll() paths to allow to restart reading even with a buffer full condition. This patch must be backported to 1.8. (cherry picked from commit c9ede6c43e40b255d7caa22c41a6790314699dc1) Signed-off-by: Willy Tarreau commit 7434c11fea7b9f9c5a02105e8e48ace66f8690c0 Author: Willy Tarreau Date: Sun Dec 10 21:19:33 2017 +0100 BUG/MEDIUM: stream-int: always set SI_FL_WAIT_ROOM on CS_FL_RCV_MORE When a stream interface tries to read data from a mux using rcv_buf(), sometimes it sees 0 as the return value and concludes that there's no more data while there are, resulting in the connection being polled for more data and no new attempt being made at reading these pending data. Now it will automatically check for flag CS_FL_RCV_MORE to know if the mux really did not have anything available or was not able to provide these data by lack of room in the destination buffer, and will set SI_FL_WAIT_ROOM accordingly. This will ensure that once current data lying in the buffer are forwarded to the other side, reading chk_rcv() will be called to re-enable reading. It's important to note that in practice it will rely on the mux's update_poll() function to re-enable reading and that where the calls are placed in the stream interface, it's not possible to perform a new synchronous rcv_buf() call. Thus a corner case remains where the mux cannot receive due to a full buffer or any similar condition, but needs to be able to wake itself up to deliver pending data. This is a limitation of the current connection/conn_stream API which will likely need a new event subscription to at least call ->wake() asynchronously (eg: mux->{kick,restart,touch,update} ?). For now the affected mux (h2 only) will have to take care of the extra logic to carefully enable polling to restart processing incoming data. This patch relies on previous one (MINOR: conn_stream: add new flag CS_FL_RCV_MORE to indicate pending data) and both must be backported to 1.8. (cherry picked from commit 6577b4861339e454f7f1fbed573ccb352f37cc30) Signed-off-by: Willy Tarreau commit 72b0779762e58e46473a1b3c26273bfbb3e4cd21 Author: Willy Tarreau Date: Sun Dec 10 21:13:25 2017 +0100 MINOR: conn_stream: add new flag CS_FL_RCV_MORE to indicate pending data Due to the nature of multiplexed protocols, it will often happen that some operations are only performed on full frames, preventing any partial operation from being performed. HTTP/2 is one such example. The current MUX API causes a problem here because the rcv_buf() function has no way to let the stream layer know that some data could not be read due to a lack of room in the buffer, but that data are definitely present. The problem with this is that the stream layer might not know it needs to call the function again after it has made some room. And if the frame in the buffer is not followed by any other, nothing will move anymore. This patch introduces a new conn_stream flag CS_FL_RCV_MORE whose purpose is to indicate on the stream that more data than what was received are already available for reading as soon as more room will be available in the buffer. This patch doesn't make use of this flag yet, it only declares it. It is expected that other similar flags may come in the future, such as reports of pending end of stream, errors or any such event that might save the caller from having to poll, or simply let it know that it can take some actions after having processed data. (cherry picked from commit 3c8294b6070137267feb1808b5271ca77b92dbdd) Signed-off-by: Willy Tarreau commit 44b5383987855b2ccdb717d4d764d3216fe8aa86 Author: Thierry FOURNIER Date: Sun Dec 10 17:10:57 2017 +0100 BUG/MEDIUM: lua/notification: memory leak The thread patches adds refcount for notifications. The notifications are used with the Lua cosocket. These refcount free the notifications when the session is cleared. In the Lua task case, it not have sessions, so the nofications are never cleraed. This patch adds a garbage collector for signals. The garbage collector just clean the notifications for which the end point is disconnected. This patch should be backported in 1.8 (cherry picked from commit cb14688496e3cca35a10b965fa644f14eb331595) Signed-off-by: Willy Tarreau commit d5a1553754de4aeda19cfd05db56397fdeedb027 Author: Thierry FOURNIER Date: Sun Dec 10 17:14:07 2017 +0100 DOC: notifications: add precisions about thread usage Precise the terms of use the notification functions. (cherry picked from commit d5b79835f8095cce3bb3f32374b571af46973112) Signed-off-by: Willy Tarreau commit 05bfbc44d9f06121114bef903aac4b2e7798df2e Author: Vincent Bernat Date: Sat Dec 9 08:32:13 2017 +0100 MINOR: systemd: remove comment about HAPROXY_STATS_SOCKET This variable was used by the wrapper which was removed in a6cfa9098e5a. The correct way to do seamless reload is now to enable "expose-fd listeners" on the stat socket. (cherry picked from commit 190bbb8f53e210d1cc997bc70ac1fd131c6abbe5) Signed-off-by: Willy Tarreau commit b5ee09f0dd95d47377200bf2beb31bd92f2b1d55 Author: Christopher Faulet Date: Fri Dec 8 09:17:39 2017 +0100 BUG/MEDIUM: threads/vars: Fix deadlock in register_name In register_name, before locking the var_names array, we check the variable name validity. So if we try to register an invalid or empty name, we need to return without unlocking it (because it was never locked). This patch must be backported in 1.8. (cherry picked from commit eb3e276d3911c4c95d1e2f270bab354440378701) Signed-off-by: Willy Tarreau commit 29e4360e6ef9a606f7358eafc0de165d0e591a01 Author: PiBa-NL Date: Wed Dec 6 01:35:43 2017 +0100 BUG/MEDIUM: email-alert: don't set server check status from a email-alert task This avoids possible 100% cpu usage deadlock on a EMAIL_ALERTS_LOCK and avoids sending lots of emails when 'option log-health-checks' is used. It is avoided to change the server state and possibly queue a new email while processing the email alert by setting check->status to HCHK_STATUS_UNKNOWN which will exit the set_server_check_status(..) early. This needs to be backported to 1.8. (cherry picked from commit 1714b9f28694d750d446917672dd59c46e16afd7) Signed-off-by: Willy Tarreau commit 85613070e1c0147f5175165ad4b49e2461a649fc Author: Aleksandar Lazic Date: Tue Dec 5 01:35:21 2017 +0100 CONTRIB: halog: Add help text for -s switch in halog program It was not documented. May be backported to older releases. (cherry picked from commit f2b5d75ae27eb4f19c0c528550ef0edc39996c88) Signed-off-by: Willy Tarreau commit 11536874d9661b39b79c4c9a4196a7cbc4fa46ff Author: Tim Duesterhus Date: Tue Dec 5 18:14:13 2017 +0100 MINOR: mworker: Improve wording in `void mworker_wait()` Replace "left" / "leaving" with "exit" / "exiting". This should be backported to haproxy 1.8. (cherry picked from commit d16f450c98dbc95470d9072fe924bacda1c7165d) Signed-off-by: Willy Tarreau commit e8b1a632811fd5b3b6bfa500f4dbd1e35d3cc99c Author: Tim Duesterhus Date: Tue Dec 5 18:14:12 2017 +0100 MINOR: mworker: Update messages referencing exit-on-failure Commit 4cfede87a313456fcbce7a185312460b4e1d05b7 removed `exit-on-failure` in favor of `no-exit-on-failure`, but failed to update references to the former in user facing messages. This should be backported to haproxy 1.8. (cherry picked from commit c578d9acfa50aac92c67a94f2b8977a8c56b7837) Signed-off-by: Willy Tarreau commit 57441d3daef1973f0b070cd2b641d7a331f478f4 Author: Willy Tarreau Date: Thu Dec 7 15:59:29 2017 +0100 BUG/MEDIUM: h2: fix handling of end of stream again Commit 9470d2c ("BUG/MINOR: h2: try to abort closed streams as soon as possible") tried to address the situations where a stream is closed by the client, but caused a side effect which is that in some cases, a regularly closed stream reports an error to the stream layer. The reason is that we purposely matched H2_SS_CLOSED in the test for H2_SS_ERROR to report this so that we can check for RST, but it accidently catches certain end of transfers as well. This results in valid requests to report flags "CD" in the logs. Instead, let's roll back to detecting H2_SS_ERROR and explicitly check for a received RST. This way we can correctly abort transfers without mistakenly reporting errors in normal situations. This fix needs to be backported to 1.8 as the fix above was merged into 1.8.1. (cherry picked from commit 0249219be8080df56f40cf147fecf3eafe050074) Signed-off-by: Willy Tarreau commit fe66fd82b13c9a2e32c52e7a016de6887f195349 Author: Willy Tarreau Date: Wed Dec 6 17:39:53 2017 +0100 BUG/MEDIUM: peers: set NOLINGER on the outgoing stream interface Since peers were ported to an applet in 1.5, an issue appeared which is that certain attempts to close an outgoing connection are a bit "too nice". Specifically, protocol errors and stream timeouts result in a clean shutdown to be sent, waiting for the other side to confirm. This is particularly problematic in the case of timeouts since by definition the other side will not confirm as it has disappeared. As found by Fred, this issue was further emphasized in 1.8 by commit f9ce57e ("MEDIUM: connection: make conn_sock_shutw() aware of lingering") which causes clean shutdowns not to be sent if the fd is marked as linger_risk, because now even a clean timeout will not be sent on an idle peers session, and the other one will have nothing to respond to. The solution here is to set NOLINGER on the outgoing stream interface to ensure we always close whenever we attempt a simple shutdown. However it is important to keep in mind that this also underlines some weaknesses of the shutr/shutw processing inside process_stream() and that all this part needs to be reworked to clearly consider the abort case, and to stop the confusion between linger_risk and NOLINGER. This fix needs to be backported as far as 1.5 (all versions are affected). However, during testing of the backport it was found that 1.5 never tries to close the peers connection on timeout, so it suffers for another issue. (cherry picked from commit dbd026792ac4d1cb9a2c14e7f10946fc86a4f6de) Signed-off-by: Willy Tarreau commit 80b92902146470f143a134c6057840fd962abb87 Author: Emeric Brun Date: Wed Dec 6 16:47:17 2017 +0100 BUG/MEDIUM: checks: a down server going to maint remains definitely stucked on down state. The new admin state was not correctly commited in this case. Checks were fully disabled but the server was not marked in MAINT state. It results with a server definitely stucked on the DOWN state. This patch should be backported on haproxy 1.8 (cherry picked from commit 8f29829e24667554c20993de5e1e7cedd3a047bb) commit b2454b102fe6ecf61867065f3e661e65910dc21e Author: Emeric Brun Date: Wed Dec 6 13:51:49 2017 +0100 BUG/MEDIUM: ssl engines: Fix async engines fds were not considered to fix fd limit automatically. The number of async fd is computed considering the maxconn, the number of sides using ssl and the number of engines using async mode. This patch should be backported on haproxy 1.8 (cherry picked from commit ece0c334bd16da66d56bece297b4495124a9d7c8) commit a6d10f3ac87a8d753c18fa2dc7384b8bd2def4f3 Author: Willy Tarreau Date: Tue Dec 5 11:14:12 2017 +0100 BUG/MEDIUM: mworker: also close peers sockets in the master There's a nasty case related to signaling all processes via SIGUSR1. Since the master process still holds the peers sockets, the old process trying to connect to the new one to teach it its tables has a risk to connect to the master instead, which will not do anything, causing the old process to hang instead of quitting. This patch ensures we correctly close the peers in the master process on startup, just like it is done for proxies. Ultimately we would rather have a complete list of listeners to avoid such issues. But that's a bit trickier as it would require using unbind_all() and avoiding side effects the master could cause to other processes (like unlinking unix sockets). To be backported to 1.8. (cherry picked from commit 473cf5d0cd2e950da7266bf88518442b583b66d7) Signed-off-by: Willy Tarreau commit 68848eb6c4159c67be51cb8bf1fbd897e5529bc8 Author: William Lallemand Date: Mon Dec 4 18:46:39 2017 +0100 BUG/MINOR: ssl: support tune.ssl.cachesize 0 again Since the split of the shctx and the ssl cache, we lost the ability to disable the cache with tune.ssl.cachesize 0. Worst than that, when using this configuration, haproxy segfaults during the configuration parsing. Must be backported to 1.8. (cherry picked from commit c61c0b371bebe0619cf4628b60b7c79d19e97269) Signed-off-by: Willy Tarreau commit 05bfeb72997a85cff1407fc46c7130424d6075fd Author: Willy Tarreau Date: Mon Dec 4 17:58:37 2017 +0100 BUG/MAJOR: hpack: don't pretend large headers fit in empty table In hpack_dht_make_room(), we try to fulfill this rule form RFC7541#4.4 : "It is not an error to attempt to add an entry that is larger than the maximum size; an attempt to add an entry larger than the maximum size causes the table to be emptied of all existing entries and results in an empty table." Unfortunately it is not consistent with the way it's used in hpack_dht_insert() as this last one will consider a success as a confirmation it can copy the header into the table, and a failure as an indexing error. This results in the two following issues : - if a client sends too large a header into an empty table, this header may overflow the table. Fortunately, most clients send small headers like :authority first, and never mark headers that don't fit into the table as indexable since it is counter-productive ; - if a client sends too large a header into a populated table, the operation fails after the table is totally flushed and the request is not processed. This patch fixes the two issues at once : - a header not fitting into an empty table is always a sign that it will never fit ; - not fitting into the table is not an error Thanks to Yves Lafon for reporting detailed traces demonstrating this issue. This fix must be backported to 1.8. (cherry picked from commit 6c71e4696b9bc439ded28c497eb8526c72999a3c) Signed-off-by: Willy Tarreau commit e5b90fae5df458de2c4b91dfd38f1f580a86c619 Author: Christopher Faulet Date: Mon Dec 4 09:45:15 2017 +0100 BUG/MINOR: action: Don't check http capture rules when no id is defined This is a regression in the commit 29730ba5 ("MINOR: action: Add a functions to check http capture rules"). We must check the capture id only when an id is defined. This patch must be backported in 1.8. (cherry picked from commit fd608dd2d212928e585da645d7d0c7089424a0de) Signed-off-by: Willy Tarreau commit bc1f797c2dcfe8a6b82697725e161f87b2d6c386 Author: Willy Tarreau Date: Sun Dec 3 22:19:05 2017 +0100 [RELEASE] Released version 1.8.1 Released version 1.8.1 with the following main changes : - BUG/MEDIUM: kqueue: Don't bother closing the kqueue after fork. - DOC: cache: update sections and fix some typos - BUILD/MINOR: deviceatlas: enable thread support - BUG/MEDIUM: tcp-check: Don't lock the server in tcpcheck_main - BUG/MEDIUM: ssl: don't allocate shctx several time - BUG/MEDIUM: cache: bad computation of the remaining size - BUILD: checks: don't include server.h - BUG/MEDIUM: stream: fix session leak on applet-initiated connections - BUILD/MINOR: haproxy : FreeBSD/cpu affinity needs pthread_np header - BUG/MINOR: ssl: CO_FL_EARLY_DATA removal is managed by stream - BUG/MEDIUM: threads/peers: decrement, not increment jobs on quitting - BUG/MEDIUM: h2: don't report an error after parsing a 100-continue response - BUG/MEDIUM: peers: fix some track counter rules dont register entries for sync. - BUG/MAJOR: thread/peers: fix deadlock on peers sync. - BUILD/MINOR: haproxy: compiling config cpu parsing handling when needed - BUG/MINOR: mworker: fix validity check for the pipe FDs - BUG/MINOR: mworker: detach from tty when in daemon mode - MINOR: threads: Fix pthread_setaffinity_np on FreeBSD. - BUG/MAJOR: thread: Be sure to request a sync between threads only once at a time - BUILD: Fix LDFLAGS vs. LIBS re linking order in various makefiles - BUG/MEDIUM: checks: Be sure we have a mux if we created a cs. - BUG/MINOR: hpack: fix debugging output of pseudo header names - BUG/MINOR: hpack: must reject huffman literals padded with more than 7 bits - BUG/MINOR: hpack: reject invalid header index - BUG/MINOR: hpack: dynamic table size updates are only allowed before headers - BUG/MAJOR: h2: correctly check the request length when building an H1 request - BUG/MINOR: h2: immediately close if receiving GOAWAY after the last stream - BUG/MINOR: h2: try to abort closed streams as soon as possible - BUG/MINOR: h2: ":path" must not be empty - BUG/MINOR: h2: fix a typo causing PING/ACK to be responded to - BUG/MINOR: h2: the TE header if present may only contain trailers - BUG/MEDIUM: h2: enforce the per-connection stream limit - BUG/MINOR: h2: do not accept SETTINGS_ENABLE_PUSH other than 0 or 1 - BUG/MINOR: h2: reject incorrect stream dependencies on HEADERS frame - BUG/MINOR: h2: properly check PRIORITY frames - BUG/MINOR: h2: reject response pseudo-headers from requests - BUG/MEDIUM: h2: remove connection-specific headers from request - BUG/MEDIUM: h2: do not accept upper case letters in request header names - BUG/MINOR: h2: use the H2_F_DATA_* macros for DATA frames commit 3fc8381cfbe7173b369e80485ef4c9d6d69a9fae Author: Willy Tarreau Date: Sun Dec 3 21:06:59 2017 +0100 BUG/MINOR: h2: use the H2_F_DATA_* macros for DATA frames A typo resulted in H2_F_HEADERS_* being used there, but it's harmless as they are equal. Better fix the confusion though. Should be backported to 1.8. (cherry picked from commit 7912781a306e3d04d896270d95bddbe9be144388) Signed-off-by: Willy Tarreau commit 3f4eb1ba441aede277d7c28b02fe5796b79f5e39 Author: Willy Tarreau Date: Sun Dec 3 20:28:13 2017 +0100 BUG/MEDIUM: h2: do not accept upper case letters in request header names This is explicitly forbidden by 7540#8.1.2, and may be used to bypass some of the other filters, so they must be blocked early. It removes another issue reported by h2spec. To backport to 1.8. (cherry picked from commit 637f64d56526ea87c935e75c3bd40a982a722f00) Signed-off-by: Willy Tarreau commit dbc29d67059f0737e58e9cd29466f8e5c033e3af Author: Willy Tarreau Date: Sun Dec 3 20:15:34 2017 +0100 BUG/MEDIUM: h2: remove connection-specific headers from request h2spec rightfully outlines that we used not to reject these ones, and they may cause trouble if presented, especially "upgrade". Must be backported to 1.8. (cherry picked from commit fe7c356be67335d92d41decd13468b862a562337) Signed-off-by: Willy Tarreau commit 2f2ecc18c4c5bc243f726cc101c6eae634aff133 Author: Willy Tarreau Date: Sun Dec 3 20:13:54 2017 +0100 BUG/MINOR: h2: reject response pseudo-headers from requests At the moment there's only ":status". Let's block it early when parsing the request. Otherwise it would be blocked by the HTTP/1 code anyway. This silences another h2spec issue. To backport to 1.8. (cherry picked from commit 520886990fbc4d40bd7dd8d788d4e79326fc64c5) Signed-off-by: Willy Tarreau commit 35faee261d4acbdded14c929bb1605fe1504c9f7 Author: Willy Tarreau Date: Sun Dec 3 19:46:19 2017 +0100 BUG/MINOR: h2: properly check PRIORITY frames We don't use them right now but it's better to ensure they're properly checked. This removes another 3 warnings in h2spec. To backport to 1.8. (cherry picked from commit 92153fccd349762ca59ed2a6d0646515476217d2) Signed-off-by: Willy Tarreau commit 114ef678bc267098737813b924e24d9bf24e624e Author: Willy Tarreau Date: Sun Dec 3 19:24:50 2017 +0100 BUG/MINOR: h2: reject incorrect stream dependencies on HEADERS frame We currently don't use stream dependencies, but as reported by h2spec, the spec requires that we reject streams that depend on themselves in HEADERS frames. To backport to 1.8. (cherry picked from commit 18b86cd074f8bfd67fa077b5e22b6b7528f72c8f) Signed-off-by: Willy Tarreau commit ef1414f74c1802b06a3f2274609c66d639decac5 Author: Willy Tarreau Date: Sun Dec 3 19:02:28 2017 +0100 BUG/MINOR: h2: do not accept SETTINGS_ENABLE_PUSH other than 0 or 1 We don't use yet it but for correctness, let's enforce the check. To backport to 1.8. (cherry picked from commit 1b38b46ab7c88d79e89e31cf8d22cd346c97d3ae) Signed-off-by: Willy Tarreau commit 8b2a58538a69d8ca634cc77f7f932ea70d44cc77 Author: Willy Tarreau Date: Sun Dec 3 18:56:02 2017 +0100 BUG/MEDIUM: h2: enforce the per-connection stream limit h2spec reports that we unfortunately didn't enforce the per-connection stream limit that we advertise. It's important to ensure it's never crossed otherwise it's cheap for a client to create many streams. This requires the addition of a stream count. The h2c struct could be cleaned up a bit, just like the h2_detach() function where an "if" block doesn't make sense anymore since it's always true. To backport to 1.8. (cherry picked from commit 497456154e3c9d1f719b939e1a943ea225abf8d1) Signed-off-by: Willy Tarreau commit 398b7519177c359eb1623ec87fc88d7ace760f7c Author: Willy Tarreau Date: Sun Dec 3 18:41:31 2017 +0100 BUG/MINOR: h2: the TE header if present may only contain trailers h2spec reports this issue which has no side effect for now, but is better cleared. To backport to 1.8. (cherry picked from commit d8d2ac75e89362637f3c21a2710f527355c86abe) Signed-off-by: Willy Tarreau commit 06c234d6ed5c10d6998d7d39a1b436bd35d89b93 Author: Willy Tarreau Date: Sun Dec 3 18:15:56 2017 +0100 BUG/MINOR: h2: fix a typo causing PING/ACK to be responded to The ACK flag was tested on the frame type instead of the frame flag. To backport to 1.8. (cherry picked from commit 68ed64148afcf253689241f0fade2e7c0ea4dcdb) Signed-off-by: Willy Tarreau commit 01d9430bdcf6b5626a240f35037275774cca1137 Author: Willy Tarreau Date: Sun Dec 3 11:51:31 2017 +0100 BUG/MINOR: h2: ":path" must not be empty As reported by h2spec, the h2->h1 gateway doesn't verify that ":path" is not empty. This is harmless since the H1 parser will reject such a request, but better fix it anyway. To backport to 1.8. (cherry picked from commit cd4fe17a2686c7af474340f920eed93f26cecab8) Signed-off-by: Willy Tarreau commit b997d88edf759c9879d60528c9420d59f1983f06 Author: Willy Tarreau Date: Sun Dec 3 10:42:59 2017 +0100 BUG/MINOR: h2: try to abort closed streams as soon as possible The purpose here is to be able to signal receipt of RST_STREAM to streams when they start to provide a response so that the response can be aborted ASAP. Given that RST_STREAM immediately switches the stream to the CLOSED state, we must check for CLOSED in addition to the existing ERROR check. To be backported to 1.8. (cherry picked from commit 9470d2cd35938db8e053f390c5de0e94eefe254b) Signed-off-by: Willy Tarreau commit c92287e44673e5c6d2fc28a0741785c4be5bedfa Author: Willy Tarreau Date: Sun Dec 3 10:27:47 2017 +0100 BUG/MINOR: h2: immediately close if receiving GOAWAY after the last stream The h2spec test suite reveals that a GOAWAY frame received after the last stream doesn't cause an immediate close, because we count on the last stream to quit to do so. By simply setting the last_sid to the received value in case it was not set, we can ensure to properly close an idle connection during h2_wake(). To be backported to 1.8. (cherry picked from commit 11cc2d6031a8fefe78f083c4d364c8489b7dbbaf) Signed-off-by: Willy Tarreau commit ac48c8a3d0d88739d5c38b83c90ec7464d895431 Author: Willy Tarreau Date: Sun Dec 3 09:44:50 2017 +0100 BUG/MAJOR: h2: correctly check the request length when building an H1 request Due to a typo in the request maximum length calculation, we count the request path twice instead of counting it added to the method's length. This has two effects, the first one being that a path cannot be larger than half a buffer, and the second being that the method's length isn't properly checked. Due to the way the temporary buffers are used internally, it is quite difficult to meet this condition. In practice, the only situation where this can cause a problem is when exactly one of either the method or the path are compressed and the other ones is sent as a literal. Thanks to Yves Lafon for providing useful traces exhibiting this issue. To be backported to 1.8. (cherry picked from commit 811ad12414e43608fd9d20865990496918ca2dd6) Signed-off-by: Willy Tarreau commit c42f6d25b8193f6494c2efacfa4893d381558c69 Author: Willy Tarreau Date: Sun Dec 3 18:09:21 2017 +0100 BUG/MINOR: hpack: dynamic table size updates are only allowed before headers h2spec reports that we used to support a dynamic table size update anywhere in the header block but it's only allowed before other headers (cf RFC7541#4.2.1). In practice we don't use these for now since we only use literals in responses. To backport to 1.8. (cherry picked from commit c611e6681b6ed4194b2e9ad0b096c275ab5c4012) Signed-off-by: Willy Tarreau commit f0f9797899b4314642fe198980a658b9cee689b7 Author: Willy Tarreau Date: Sun Dec 3 12:12:17 2017 +0100 BUG/MINOR: hpack: reject invalid header index If the hpack decoder sees an invalid header index, it emits value "### ERR ###" that was used during debugging instead of rejecting the block. This is harmless, and was detected by h2spec. To backport to 1.8. (cherry picked from commit d85ba4e092997d2a05cd811534c61946c57133a4) Signed-off-by: Willy Tarreau commit 1043f7a4991a9d9d2adfbe8600ecdd8888614e25 Author: Willy Tarreau Date: Sun Dec 3 12:00:36 2017 +0100 BUG/MINOR: hpack: must reject huffman literals padded with more than 7 bits h2spec reported that we didn't check that no more than 7 bits of padding were left after decoding an huffman-encoded literal. This is harmless but better fix it now. To backport to 1.8. (cherry picked from commit 4235d182143369d608c436f83004ce931ebb3635) Signed-off-by: Willy Tarreau commit 95d22e0a181bd75bfc56b02483241bc845f9d0b9 Author: Willy Tarreau Date: Sun Dec 3 09:43:38 2017 +0100 BUG/MINOR: hpack: fix debugging output of pseudo header names When a pseudo header is used, name.ptr is NULL and we must replace it with hpack_idx_to_name(). This only affects code built with DEBUG_HPACK. To be backported to 1.8. (cherry picked from commit 9e28f459b4d003072d84fce57811b26ff8245f54) Signed-off-by: Willy Tarreau commit 27b11f6cb05c152b35fc6c4d5e805a525c0d25cb Author: Olivier Houchard Date: Fri Dec 1 22:04:05 2017 +0100 BUG/MEDIUM: checks: Be sure we have a mux if we created a cs. In connect_conn_chk(), there were one case we could return with a new conn_stream created, but no mux attached. With no mux, cs_destroy() would segfault. Fix that by setting the mux before we can fail. This should be backported to 1.8. (cherry picked from commit 6377a0004fa25cc58a0e338a13012d3cf9416ef7) Signed-off-by: Willy Tarreau commit 16b04adbc516c3afafc7da7063b7b4e96bcbb665 Author: Christian Ruppert Date: Thu Nov 30 10:11:36 2017 +0100 BUILD: Fix LDFLAGS vs. LIBS re linking order in various makefiles Libraries should always be listed last. Should be backported to 1.8. Signed-off-by: Christian Ruppert (cherry picked from commit 57dc2830144bcd5b8cd16beb16679689d6f9f711) Signed-off-by: Willy Tarreau commit 21077740c306ee89db17a77c2b049af2cc890403 Author: Christopher Faulet Date: Sat Dec 2 09:53:24 2017 +0100 BUG/MAJOR: thread: Be sure to request a sync between threads only once at a time The first thread requesting a synchronization is responsible to write in the "sync" pipe to notify all others. But we must write only once in the pipe between two synchronizations to have exactly one character in the pipe. It is important because we only read 1 character in return when the last thread exits from the sync-point. Here there is a bug. If two threads request a synchronization, only the first writes in the pipe. But, if the same thread requests several times a synchronization before entering in the sync-point (because, for instance, it detects many servers down), it writes as many as characters in the pipe. And only one of them will be read. Repeating this bug many times will block HAProxy on the write because the pipe is full. To fix the bug, we just check if the current thread has already requested a synchronization before trying to notify all others. The patch must be backported in 1.8 (cherry picked from commit 81991d32853ed4c7ccf80871b0f73d3a2f3cc1d5) Signed-off-by: Willy Tarreau commit 3ce53f662a7bc3a66d32b639d7d102770c38dc03 Author: Olivier Houchard Date: Fri Dec 1 18:19:43 2017 +0100 MINOR: threads: Fix pthread_setaffinity_np on FreeBSD. As with the call to cpuset_setaffinity(), FreeBSD expects the argument to pthread_setaffinity_np() to be a cpuset_t, not an unsigned long, so the call was silently failing. This should probably be backported to 1.8. (cherry picked from commit 829aa24459c633896113e9e438c43b9dfcdfce27) Signed-off-by: Willy Tarreau commit f926969aa19058891ddaefc9fa2176837f2f8018 Author: PiBa-NL Date: Tue Nov 28 23:26:08 2017 +0100 BUG/MINOR: mworker: detach from tty when in daemon mode This allows a calling script to show the first startup output and know when to stop reading from stdout so haproxy can daemonize. To be backpored to 1.8. (cherry picked from commit baf6ea4bd5c6df7b9f91a612fb068df63b3d1eec) Signed-off-by: Willy Tarreau commit 4e612023671af84272fa6853acc90493e648c9d9 Author: PiBa-NL Date: Tue Nov 28 23:22:14 2017 +0100 BUG/MINOR: mworker: fix validity check for the pipe FDs Check if master-worker pipe getenv succeeded, also allow pipe fd 0 as valid. On FreeBSD in quiet mode the stdin/stdout/stderr are closed which lets the mworker_pipe to use fd 0 and fd 1. Additionally exit() upon failure to create or get the master-worker pipe. This needs to be backported to 1.8. (cherry picked from commit 4763ffdf0407e1c697da77a10599a2aecf7dcb04) Signed-off-by: Willy Tarreau commit 326179f046269a31976a0db559736290fcb83e89 Author: David Carlier Date: Fri Dec 1 09:14:02 2017 +0000 BUILD/MINOR: haproxy: compiling config cpu parsing handling when needed parse_cpu_set is only relevant where there is cpu affinity, avoiding in the process compilation warning as well. (cherry picked from commit 7e351eefe507a6617e05fd9ea499920ec1186562) commit 08aaa8a06fb1e1cf6467a14c39f633e85cff685b Author: Emeric Brun Date: Fri Dec 1 11:37:36 2017 +0100 BUG/MAJOR: thread/peers: fix deadlock on peers sync. Table lock was not released on an error path (if there is no enough room to write table switch message). [wt: needs to be backported to 1.8] (cherry picked from commit 088c9b73cab9a1958dceaa1ce6f8a76541f8d344) commit a80a2273fef154eefa2f757c787a01a9b34567a0 Author: Emeric Brun Date: Wed Nov 29 16:15:07 2017 +0100 BUG/MEDIUM: peers: fix some track counter rules dont register entries for sync. This BUG was introduced with: 'MEDIUM: threads/stick-tables: handle multithreads on stick tables' The API was reviewed to handle stick table entry updates asynchronously and the caller must now call a 'stkable_touch_*' function each time the content of an entry is modified to register the entry to be synced. There was missing call to stktable_touch_* resulting in not propagated entries to remote peers (or local one during reload) (cherry picked from commit 0fed0b0a381a90cbb72eb5e32e57e7ce68880d0d) commit 59fcb216085a7aa9744cffe39567c80de4ebd6bf Author: Willy Tarreau Date: Wed Nov 29 15:41:32 2017 +0100 BUG/MEDIUM: h2: don't report an error after parsing a 100-continue response Yves Lafon reported a breakage with 100-continue. In fact the problem is caused when an 1xx is the last response in the buffer (which commonly is the case). We loop back immediately into the parser with what remains of the input buffer (ie: nothing), while it is not expected to be called with an empty response, so it fails. Let's simply get back to the caller to decide whether or not more data are expected to be sent. This fix needs to be backported to 1.8. (cherry picked from commit 872855998bd03d5224e0e5cd6aef9b91e2a6de1d) Signed-off-by: Willy Tarreau commit d7c82793eca173deb7a0e0df9105d9892789a4ae Author: Willy Tarreau Date: Wed Nov 29 14:49:30 2017 +0100 BUG/MEDIUM: threads/peers: decrement, not increment jobs on quitting Commit 8d8aa0d ("MEDIUM: threads/listeners: Make listeners thread-safe") mistakenly placed HA_ATOMIC_ADD(job, 1) to replace a job--, so it maintains the job count too high preventing the process from cleanly exiting on reload. This needs to be backported to 1.8. (cherry picked from commit cea8537efdd8306742cda30faafb03732567a6ae) Signed-off-by: Willy Tarreau commit 8962857703c5a0c421ebbf3e8e01c03d4c016739 Author: Emmanuel Hocdet Date: Mon Nov 27 16:14:40 2017 +0100 BUG/MINOR: ssl: CO_FL_EARLY_DATA removal is managed by stream Manage BoringSSL early_data as it is with openssl 1.1.1. (cherry picked from commit cebd7962e2bcd31bfe67ef6426a5210d1ec0428b) Signed-off-by: Willy Tarreau commit c2b281443fb6fb5e4f4ca74f31ef16c57d668719 Author: David Carlier Date: Wed Nov 29 11:02:32 2017 +0000 BUILD/MINOR: haproxy : FreeBSD/cpu affinity needs pthread_np header for pthread_*_np calls, pthread_np.h is needed under FreeBSD. (cherry picked from commit 6d5c841d242a22573d1a66434df9e1e936e7bed6) Signed-off-by: Willy Tarreau commit ac6ea544cda7867391b80267d4b5769fe81048f6 Author: Willy Tarreau Date: Wed Nov 29 14:05:38 2017 +0100 BUG/MEDIUM: stream: fix session leak on applet-initiated connections Commit 3e13cba ("MEDIUM: session: make use of the connection's destroy callback") ensured that connections could be autonomous to destroy the session they initiated, but it didn't take care of doing the same for applets. Such applets are used for peers, Lua and SPOE outgoing connections. In this case, once the stream ends, it closes everything and nothing takes care of releasing the session. The problem is not immediately obvious since the only visible effect is that older processes will not quit on reload after having leaked one such session. For now we check in stream_free() if the session's origin is the applet we're releasing, and then free the session as well. Something more uniform should probably be done once we manage to unify applets and connections a bit more. This fix needs to be backported to 1.8. Thanks to Emmanuel Hocdet for reporting the problem. (cherry picked from commit 5bcfd56519b683386643c800558714893fd33910) Signed-off-by: Willy Tarreau commit 1205702cd287e5093698c709e18e1899071badc3 Author: Willy Tarreau Date: Wed Nov 29 10:52:29 2017 +0100 BUILD: checks: don't include server.h server.h needs checks.h since it references the struct check, but depending on the include order it will fail if check.h is included first due to this one including server.h in turn while it doesn't need it. (cherry picked from commit ec7464726f90a8c9536c21a693631a1193f81e19) Signed-off-by: Willy Tarreau commit 7732a3773924549002771c969bbf0f379559ad93 Author: William Lallemand Date: Tue Nov 28 11:33:02 2017 +0100 BUG/MEDIUM: cache: bad computation of the remaining size The cache was not setting the hdrs_len to zero when we are called in the http_forward_data with headers + body. The consequence is to always try to store a size - the size of headers, during the calls to http_forward_data even when it has already forwarded the headers. Thanks to Cyril Bonté for reporting this bug. Must be backported to 1.8. (cherry picked from commit bcd9101a6652856d2db30b254906a2c5eb1bb20a) Signed-off-by: Willy Tarreau commit 38df682ed119ace207566e7ce48c5f606c67c8fb Author: William Lallemand Date: Tue Nov 28 11:04:43 2017 +0100 BUG/MEDIUM: ssl: don't allocate shctx several time The shctx_init() function does not check anymore if the pointer is not NULL, this check must be done is the caller. The consequence was to allocate one shctx per ssl bind. Bug introduced by 4f45bb9 ("MEDIUM: shctx: separate ssl and shctx") Thanks to Maciej Zdeb for reporting this bug. Must be backported to 1.8. (cherry picked from commit c3cd35f96cbb6739dace9906448c5a68f554d96e) Signed-off-by: Willy Tarreau commit 8f42b1975068ae4deb0c5bd65915484ab796bc64 Author: Christopher Faulet Date: Tue Nov 28 10:06:29 2017 +0100 BUG/MEDIUM: tcp-check: Don't lock the server in tcpcheck_main There was a deadlock in tcpcheck_main function. The server's lock was already acquired by the caller (process_chk_conn or wake_srv_chk). This patch must be backported in 1.8. (cherry picked from commit b61028549e51b58abdafcf652ee2871767307a62) Signed-off-by: Willy Tarreau commit e0e9b8195a414b6b02aec183bf45fb0096633fb2 Author: David Carlier Date: Mon Nov 27 11:55:59 2017 +0000 BUILD/MINOR: deviceatlas: enable thread support DeviceAtlas detection being multi-thread safe, we enable the new thread feature support. Needs to be backported to 1.8 branch. (cherry picked from commit e78915a47a6427d1fb4a5347da4f7021c7350a1b) Signed-off-by: Willy Tarreau commit 00701cf1d4697e27dddb470d2ccf9fd5406bf4a9 Author: Cyril Bonté Date: Sun Nov 26 22:24:31 2017 +0100 DOC: cache: update sections and fix some typos Cache sections were not defined as the others, preventing them to be correctly parsed by the HTML converter. Also, the "Cache" subsections where not added to the summary. This patch should be backported to the 1.8 branch. (cherry picked from commit 7b888f123523c9a8ff4fc4ca5f04df6e9b2171da) Signed-off-by: Willy Tarreau commit 4ccc9e7c32b340620dcd1f3a49c4cfba9e45a7bf Author: Olivier Houchard Date: Sun Nov 26 19:53:46 2017 +0100 BUG/MEDIUM: kqueue: Don't bother closing the kqueue after fork. kqueue fd's are not shared with children after fork(), so the children don't have to close them, and it may in fact be dangerous, because we may end up closing a totally unrelated fd. [wt: to be backported to 1.8 where master-worker broke on this, and likely to older versions for completeness] (cherry picked from commit ba8e8c351866947c5fe4dac4db433c6a3d04ff7f) Signed-off-by: Willy Tarreau commit 0b78792bbe61fec420e4e7298d145ec7d498f8f2 Author: Willy Tarreau Date: Sun Nov 26 19:25:23 2017 +0100 [RELEASE] Released version 1.8.0 Released version 1.8.0 with the following main changes : - BUG/MEDIUM: stream: don't automatically forward connect nor close - BUG/MAJOR: stream: ensure analysers are always called upon close - BUG/MINOR: stream-int: don't try to read again when CF_READ_DONTWAIT is set - MEDIUM: mworker: Add systemd `Type=notify` support - BUG/MEDIUM: cache: free callback to remove from tree - CLEANUP: cache: remove unused struct - MEDIUM: cache: enable the HTTP analysers - CLEANUP: cache: remove wrong comment - MINOR: threads/atomic: rename local variables in macros to avoid conflicts - MINOR: threads/plock: rename local variables in macros to avoid conflicts - MINOR: threads/atomic: implement pl_mb() in asm on x86 - MINOR: threads/atomic: implement pl_bts() on non-x86 - MINOR: threads/build: atomic: replace the few inlines with macros - BUILD: threads/plock: fix a build issue on Clang without optimization - BUILD: ebtree: don't redefine types u32/s32 in scope-aware trees - BUILD: compiler: add a new type modifier __maybe_unused - BUILD: h2: mark some inlined functions "unused" - BUILD: server: check->desc always exists - BUG/MEDIUM: h2: properly report connection errors in headers and data handlers - MEDIUM: h2: add a function to emit an HTTP/1 request from a headers list - MEDIUM: h2: change hpack_decode_headers() to only provide a list of headers - BUG/MEDIUM: h2: always reassemble the Cookie request header field - BUG/MINOR: systemd: ignore daemon mode - CONTRIB: spoa_example: allow to compile outside HAProxy. - CONTRIB: spoa_example: remove bref, wordlist, cond_wordlist - CONTRIB: spoa_example: remove last dependencies on type "sample" - CONTRIB: spoa_example: remove SPOE enums that are useless for clients - CLEANUP: cache: reorder includes - MEDIUM: shctx: use unsigned int for len and block_count - MEDIUM: cache: "show cache" on the cli - BUG/MEDIUM: cache: use key=0 as a condition for freeing - BUG/MEDIUM: cache: refcount forbids to free the objects - BUG/MEDIUM: cache fix cli_kws structure - BUG/MEDIUM: deinit: correctly deinitialize the proxy and global listener tasks - BUG/MINOR: ssl: Always start the handshake if we can't send early data. - MINOR: ssl: Don't disable early data handling if we could not write. - MINOR: pools: prepare functions to override malloc/free in pools - MINOR: pools: implement DEBUG_UAF to detect use after free - BUG/MEDIUM: threads/time: fix time drift correction - BUG/MEDIUM: threads/time: maintain a common time reference between all threads - MINOR: sample: Add "thread" sample fetch - BUG/MINOR: Use crt_base instead of ca_base when crt is parsed on a server line - BUG/MINOR: stream: fix tv_request calculation for applets - BUG/MAJOR: h2: always remove a stream from the send list before freeing it - BUG/MAJOR: threads/task: dequeue expired tasks under the WQ lock - MINOR: ssl: Handle reading early data after writing better. - MINOR: mux: Make sure every string is woken up after the handshake. - MEDIUM: cache: store sha1 for hashing the cache key - MINOR: http: implement the "http-request reject" rule - MINOR: h2: send RST_STREAM before GOAWAY on reject - MEDIUM: h2: don't gracefully close the connection anymore on Connection: close - MINOR: h2: make use of client-fin timeout after GOAWAY - MEDIUM: config: ensure that tune.bufsize is at least 16384 when using HTTP/2 - MINOR: ssl: Handle early data with BoringSSL - BUG/MEDIUM: stream: always release the stream-interface on abort - BUG/MEDIUM: cache: free ressources in chn_end_analyze - MINOR: cache: move the refcount decrease in the applet release - BUG/MINOR: listener: Allow multiple "process" options on "bind" lines - MINOR: config: Support a range to specify processes in "cpu-map" parameter - MINOR: config: Slightly change how parse_process_number works - MINOR: config: Export parse_process_number and use it wherever it's applicable - MINOR: standard: Add my_ffsl function to get the position of the bit set to one - MINOR: config: Add auto-increment feature for cpu-map - MINOR: config: Support partial ranges in cpu-map directive - MINOR:: config: Remove thread-map directive - MINOR: config: Add the threads support in cpu-map directive - MINOR: config: Add threads support for "process" option on "bind" lines - MEDIUM: listener: Bind listeners on a thread subset if specified - CLEANUP: debug: Use DPRINTF instead of fprintf into #ifdef DEBUG_FULL/#endif - CLEANUP: log: Rename Alert/Warning in ha_alert/ha_warning - MINOR/CLEANUP: proxy: rename "proxy" to "proxies_list" - CLEANUP: pools: rename all pool functions and pointers to remove this "2" - DOC: update the roadmap file with the latest changes merged in 1.8 - DOC: fix mangled version in peers protocol documentation - DOC: add initial peers protovol v2.0 documentation. - DOC: mention William as maintainer of the cache and master-worker - DOC: add Christopher and Emeric as maintainers of the threads - MINOR: cache: replace a fprint() by an abort() - MEDIUM: cache: max-age configuration keyword - DOC: explain HTTP2 timeout behavior - DOC: cache: configuration and management - MAJOR: mworker: exits the master on failure - BUG/MINOR: threads: don't drop "extern" on the lock in include files - MINOR: task: keep a pointer to the currently running task - MINOR: task: align the rq and wq locks - MINOR: fd: cache-align fdtab and fdcache locks - MINOR: buffers: cache-align buffer_wq_lock - CLEANUP: server: reorder some fields in struct server to save 40 bytes - CLEANUP: proxy: slightly reorder the struct proxy to reduce holes - CLEANUP: checks: remove 16 bytes of holes in struct check - CLEANUP: cache: more efficiently pack the struct cache - CLEANUP: fd: place the lock at the beginning of struct fdtab - CLEANUP: pools: align pools on a cache line - DOC: config: add a few bits about how to configure HTTP/2 - BUG/MAJOR: threads/queue: avoid recursive locking in pendconn_get_next_strm() - BUILD: Makefile: reorder object files by size commit d73efb434954c5ede141b9ea62a5371f78f6dfd8 Author: Willy Tarreau Date: Sun Nov 26 17:58:17 2017 +0100 BUILD: Makefile: reorder object files by size We've added many files since last version, it was about time to reorder the makefile to improve parallel builds by having the slower files built first. This allows to consistently stay below 4 seconds when using a 20-core build farm. commit 103e5663c8419b1f821ba0d68c1704aeaaec7f91 Author: Willy Tarreau Date: Sun Nov 26 18:48:14 2017 +0100 BUG/MAJOR: threads/queue: avoid recursive locking in pendconn_get_next_strm() pendconn_get_next_strm() is called from process_srv_queue() under the server lock, and calls stream_add_srv_conn() with this lock held, while the latter tries to take it again. This results in a deadlock when a server's maxconn is reached and haproxy is built with thread support. commit 95c4e14d0c7d798b1f982541ee07e81f422ccf15 Author: Willy Tarreau Date: Sun Nov 26 12:18:55 2017 +0100 DOC: config: add a few bits about how to configure HTTP/2 There's hardly anything to say in this file beyond the ALPN configuration and some precisions about how the HTTP transaction model applies to HTTP/2. commit 1ca1b70cf937791011dd7cc607ceeffeabf7ac45 Author: Willy Tarreau Date: Sun Nov 26 10:50:36 2017 +0100 CLEANUP: pools: align pools on a cache line There are just a few pools, and they're stressed a lot, so it makes sense to dedicate them a cache line to avoid contention and to place the lock at the beginning. commit 5809052ae1066b0af9d36ae63608621f75d260f6 Author: Willy Tarreau Date: Sun Nov 26 10:41:47 2017 +0100 CLEANUP: fd: place the lock at the beginning of struct fdtab The struct is not cache line aligned but at least, every time the lock will appear in the same cache line as the fd it will benefit from being accessed first. This improves the performance by about 2% on fd-intensive workloads with 4 threads. commit fd5efb59364056e23b0dd9bded47b0a6c523b90d Author: Willy Tarreau Date: Sun Nov 26 08:54:31 2017 +0100 CLEANUP: cache: more efficiently pack the struct cache By having the cache id on 33 bytes as the first member, it was creating a hole and forcing the "hot" remaining part to be split across two cache lines. Let's move the id at the end as it's used only during config parsing. commit 08eaa787392f0b955d26041c08ef885ffc1196c5 Author: Willy Tarreau Date: Sun Nov 26 08:44:34 2017 +0100 CLEANUP: checks: remove 16 bytes of holes in struct check These ones were easily recovered by swapping two members. commit a51108443ee513eb51bda91453e7e6197483f3c0 Author: Willy Tarreau Date: Sun Nov 26 08:41:31 2017 +0100 CLEANUP: proxy: slightly reorder the struct proxy to reduce holes 16 bytes were recovered from the struct doing minimal reordering. commit d7e33bbe2f33f14929c36c1acfd296445dd9c762 Author: Willy Tarreau Date: Sun Nov 26 07:26:48 2017 +0100 CLEANUP: server: reorder some fields in struct server to save 40 bytes In 1.8 many holes were introduced in struct server, so let's slightly reorder a few fields to plug most of them. This saves 40 bytes in the struct. commit b6a2f5899361c0de45048b2d7b7104ad2a27f16b Author: Willy Tarreau Date: Sun Nov 26 11:08:14 2017 +0100 MINOR: buffers: cache-align buffer_wq_lock This lock is highly stressed, avoid cache-line sharing to limit stress. commit 8b94969054fcbecf5a8aad909852ca84086c3c65 Author: Willy Tarreau Date: Sun Nov 26 11:07:34 2017 +0100 MINOR: fd: cache-align fdtab and fdcache locks These locks are highly contended, let's not make them share cache lines. commit a24d1d0be40bdee6b1fc105080747e184b42b81f Author: Willy Tarreau Date: Sun Nov 26 10:19:16 2017 +0100 MINOR: task: align the rq and wq locks We really don't want them to share the same cache line as they are expected to be used in parallel. Adding a 64-byte alignment here shows a performance increase of about 4.5% on task-intensive workloads with 2 to 4 threads. commit 6d1222ce7392423c54be21d2c7ebf4adc87a6a5d Author: Willy Tarreau Date: Sun Nov 26 10:08:06 2017 +0100 MINOR: task: keep a pointer to the currently running task Very often when debugging, the current task's pointer isn't easy to recover (eg: from a core file). Let's keep a copy of it, it will likely help, especially with threads. commit 53bae85b8e9e8333e9a232fcd9a8aa085471f6cb Author: Willy Tarreau Date: Sun Nov 26 11:00:37 2017 +0100 BUG/MINOR: threads: don't drop "extern" on the lock in include files Commit 9dcf9b6 ("MINOR: threads: Use __decl_hathreads to declare locks") accidently lost a few "extern" in certain lock declarations, possibly causing certain entries to be declared at multiple places. Apparently it hasn't caused any harm though. The offending ones were : - fdtab_lock - fdcache_lock - poll_lock - buffer_wq_lock commit 4cfede87a313456fcbce7a185312460b4e1d05b7 Author: William Lallemand Date: Fri Nov 24 22:02:34 2017 +0100 MAJOR: mworker: exits the master on failure This patch changes the behavior of the master during the exit of a worker. When a worker exits with an error code, for example in the case of a segfault, all workers are now killed and the master leaves. If you don't want this behavior you can use the option "master-worker no-exit-on-failure". commit 86d0df0b665b53da42f64d97e50da9272153aac8 Author: William Lallemand Date: Fri Nov 24 21:36:45 2017 +0100 DOC: cache: configuration and management commit 75df9d7a7acb741e8413a6c2f3d6b6fe07b44bb8 Author: Lukas Tribus Date: Fri Nov 24 19:05:12 2017 +0100 DOC: explain HTTP2 timeout behavior Clarifies that in HTTP2 we don't consider "timeout http-keep-alive", but "timeout client" instead. commit 49b4453b586c30ab7659befb76b4244c31f9bae1 Author: William Lallemand Date: Fri Nov 24 18:53:43 2017 +0100 MEDIUM: cache: max-age configuration keyword Add a configuration keyword to change the max-age. The default one is still 60s. commit a71cd1d4072c3bb265cead9d11a4a24826d42be7 Author: William Lallemand Date: Fri Nov 24 18:53:42 2017 +0100 MINOR: cache: replace a fprint() by an abort() In the applet I/O handler we can never get an object bigger than a buffer, so we should never reach this case. commit 17536e40053bc602bc22352f53a7720bae0bb65f Author: Willy Tarreau Date: Fri Nov 24 18:50:13 2017 +0100 DOC: add Christopher and Emeric as maintainers of the threads We'll need to be extremely careful at the beginning regarding changes. commit 55612d4517d06d456171de657706f395bd6a00cb Author: Willy Tarreau Date: Fri Nov 24 18:23:03 2017 +0100 DOC: mention William as maintainer of the cache and master-worker The latter is very tricky, better not touch anything there without his approval. commit 9c05c48b1b844c2983240bcd0634b64227041e75 Author: Emeric Brun Date: Fri Nov 24 18:20:57 2017 +0100 DOC: add initial peers protovol v2.0 documentation. [wt: the new version is 2.1 but it's useful to document the different versions since they're found in field. There's some overlap with the new one and they complement on certain areas. Most likely they'll ultimately be merged.] commit 99795b195201f730aceca2761ffaba6c47b43e1d Author: Willy Tarreau Date: Fri Nov 24 18:10:24 2017 +0100 DOC: fix mangled version in peers protocol documentation Tim Düsterhus noticed that the create-release script had mangled the version in the peers protocol doc, forcing it to 1.8 due to its syntax matching the format of an haproxy version. Let's just slightly readjust the header not to match this by removing the word "version" and placing it on the same line as the title. commit d5961ad0385896e918032b62673d14566ce614fa Author: Willy Tarreau Date: Fri Nov 24 18:00:32 2017 +0100 DOC: update the roadmap file with the latest changes merged in 1.8 We're making progress :-) commit bafbe010282fc4fc37aa307736387da80321c46c Author: Willy Tarreau Date: Fri Nov 24 17:34:44 2017 +0100 CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something". commit fbc74e855622dab93dd6b90072160f977a44c4d5 Author: Olivier Houchard Date: Fri Nov 24 16:54:05 2017 +0100 MINOR/CLEANUP: proxy: rename "proxy" to "proxies_list" Rename the global variable "proxy" to "proxies_list". There's been multiple proxies in haproxy for quite some time, and "proxy" is a potential source of bugs, a number of functions have a "proxy" argument, and some code used "proxy" when it really meant "px" or "curproxy". It worked by pure luck, because it usually happened while parsing the config, and thus "proxy" pointed to the currently parsed proxy, but we should probably not rely on this. [wt: some of these are definitely fixes that are worth backporting] commit 767a84bcc07c9d42c851d09b26ce141c2a190a49 Author: Christopher Faulet Date: Fri Nov 24 16:50:31 2017 +0100 CLEANUP: log: Rename Alert/Warning in ha_alert/ha_warning commit 56803b1c989e886bccd708168bae34f883c6a8a1 Author: Christopher Faulet Date: Fri Nov 24 16:06:18 2017 +0100 CLEANUP: debug: Use DPRINTF instead of fprintf into #ifdef DEBUG_FULL/#endif commit 165f07e7b48da454da66d0dd8c45d590a31e80f8 Author: Christopher Faulet Date: Fri Nov 24 10:08:09 2017 +0100 MEDIUM: listener: Bind listeners on a thread subset if specified If a "process" option with a thread set is used on the bind line, we use the corresponding bitmask when the listener's FD is created. commit c644fa9bf5f9b94e4bcd1e806222b96f63f8d175 Author: Christopher Faulet Date: Thu Nov 23 22:44:11 2017 +0100 MINOR: config: Add threads support for "process" option on "bind" lines It is now possible on a "bind" line (or a "stats socket" line) to specify the thread set allowed to process listener's connections. For instance: # HTTPS connections will be processed by all threads but the first and HTTP # connection will be processed on the first thread. bind *:80 process 1/1 bind *:443 ssl crt mycert.pem process 1/2- commit cb6a94510dceab07cf66d27c14f64a3ffaef8bfc Author: Christopher Faulet Date: Wed Nov 22 16:50:41 2017 +0100 MINOR: config: Add the threads support in cpu-map directive Now, it is possible to bind CPU at the thread level instead of the process level by defining a thread set in "cpu-map" directives. Thus, its format is now: cpu-map [auto:][/] ... where and must follow the format: all | odd | even | number[-[number]] Having a process range and a thread range in same time with the "auto:" prefix is not supported. Only one range is supported, the other one must be a fixed number. But it is allowed when there is no "auto:" prefix. Because it is possible to define a mapping for a process and another for a thread on this process, threads will be bound on the intersection of their mapping and the one of the process on which they are attached. If the intersection is null, no specific binding will be set for the threads. commit 11da456e77eebaba9cc64ea02b7c281220088d22 Author: Christopher Faulet Date: Wed Nov 22 16:52:28 2017 +0100 MINOR:: config: Remove thread-map directive It was a temporary directive used for development purpose. Now, CPU mapping for at the thread level should be done using the cpu-map directive. This feature will be added in a next commit. commit ff4121f7419db90172562cab31f2ab7447f7ac05 Author: Christopher Faulet Date: Wed Nov 22 16:38:49 2017 +0100 MINOR: config: Support partial ranges in cpu-map directive Now, processa and CPU ranges can be partially defined. The higher bound can be omitted. In such case, it is replaced by the corresponding maximum value, 32 or 64 depending on the machine's word size. By extension, It is also true for the "bind-process" directive and "process" parameter on a "bind" or a "stats socket" line. commit 26028f620941a514417b3186841580739052ae56 Author: Christopher Faulet Date: Wed Nov 22 15:01:51 2017 +0100 MINOR: config: Add auto-increment feature for cpu-map The prefix "auto:" can be added before the process set to let HAProxy automatically bind a process to a CPU by incrementing process and CPU sets. To be valid, both sets must have the same size. No matter the declaration order of the CPU sets, it will be bound from the lower to the higher bound. Examples: # all these lines bind the process 1 to the cpu 0, the process 2 to cpu 1 # and so on. cpu-map auto:1-4 0-3 cpu-map auto:1-4 0-1 2-3 cpu-map auto:1-4 3 2 1 0 # bind each process to exaclty one CPU using all/odd/even keyword cpu-map auto:all 0-63 cpu-map auto:even 0-31 cpu-map auto:odd 32-63 # invalid cpu-map because process and CPU sets have different sizes. cpu-map auto:1-4 0 # invalid cpu-map auto:1 0-3 # invalid commit ff8131861f364a46030b854d34e67caf54d62569 Author: Christopher Faulet Date: Wed Nov 22 15:00:13 2017 +0100 MINOR: standard: Add my_ffsl function to get the position of the bit set to one commit f1f0c5f591334ae00d0eba7898b8b2d6a79594e7 Author: Christopher Faulet Date: Wed Nov 22 12:06:43 2017 +0100 MINOR: config: Export parse_process_number and use it wherever it's applicable This function is used when "bind-process" directive is parsed and when "process" parameter on a "bind" or a "stats socket" line is parsed. commit 5ab51775e736511b7e54f42e080dcef76a284da9 Author: Christopher Faulet Date: Wed Nov 22 11:21:58 2017 +0100 MINOR: config: Slightly change how parse_process_number works Now, this function returns a status code to indicate a success (0) or a failure (1) and the error message in set in parameter. And the result of the parsing is set in parameter. commit 1dcb9cb81c55366022e45de627bbdfc0154b2509 Author: Christopher Faulet Date: Wed Nov 22 10:24:40 2017 +0100 MINOR: config: Support a range to specify processes in "cpu-map" parameter Now, you can define processes concerned by a cpu-map line using a range. For instance, the following line binds the first 32 processes on CPUs 0 to 3: cpu-map 1-32 0-3 commit 15eb3a9a08639a53e477dd4cc73f63f559fbafda Author: Christopher Faulet Date: Thu Nov 23 22:23:08 2017 +0100 BUG/MINOR: listener: Allow multiple "process" options on "bind" lines The documentation specifies that you can have several "process" options to define several ranges on "bind" lines (or "stats socket" lines). It is uncommon, but it should be possible. So the bind_proc bitmask in bind_conf structure must not be overwritten at each new "process" option parsed. This bug also exists in 1.7, 1.6 and 1.5. So it may be backported. But no one seems to have noticed it, so it was probably never hitted. commit ecb73b12c1d8bead2bcf5bf8d06647708a20070e Author: William Lallemand Date: Fri Nov 24 14:33:55 2017 +0100 MINOR: cache: move the refcount decrease in the applet release Move the refcount decrease of the cache in the release callback of the applet. We don't need to decrease it in the applet code. commit 49dc048c25d04357bcdebdefd197b47666bf14bc Author: William Lallemand Date: Fri Nov 24 14:33:54 2017 +0100 BUG/MEDIUM: cache: free ressources in chn_end_analyze Upon an aborted HTTP connection, or an error, the filter cache does not decrement the refcount and does not free the allocated ressources. commit 0542c8b39a3346ee653bb16a81ae5b974e4befba Author: Willy Tarreau Date: Fri Nov 24 15:01:10 2017 +0100 BUG/MEDIUM: stream: always release the stream-interface on abort The cache exhibited a but in process_stream() where upon abort it is possible to switch the stream-int's state to SI_ST_CLO without calling si_release_endpoint(), resulting in a possibly missing ->release() for the applet. It should affect all other applets as well (eg: lua, spoe, peers) and should carefully be backported to stable branches after some observation period. commit ca6a957c5d9b55cfa026dd1813a1c9c46a5c2e1b Author: Emmanuel Hocdet Date: Thu Nov 23 12:40:07 2017 +0100 MINOR: ssl: Handle early data with BoringSSL BoringSSL early data differ from OpenSSL 1.1.1 implementation. When early handshake is done, SSL_in_early_data report if SSL_read will be done on early data. CO_FL_EARLY_SSL_HS and CO_FL_EARLY_DATA can be adjust accordingly. commit 45a66ccc5510beb635bf381a2982a2af62bab963 Author: Willy Tarreau Date: Fri Nov 24 11:28:00 2017 +0100 MEDIUM: config: ensure that tune.bufsize is at least 16384 when using HTTP/2 HTTP/2 mandates the support of 16384 bytes frames by default, so we need a large enough buffer to process them. Till now if tune.bufsize was too small, H2 connections were simply rejected during their establishment, making it quite hard to troubleshoot the issue. Now we detect when HTTP/2 is enabled on an HTTP frontend and emit an error if tune.bufsize is not large enough, with the appropriate recommendation. commit 599391a7c24732ba5e5b36083bb937fdbf182115 Author: Willy Tarreau Date: Fri Nov 24 10:16:00 2017 +0100 MINOR: h2: make use of client-fin timeout after GOAWAY At the moment, the "client" timeout is used on an HTTP/2 connection once it's idle with no active stream. With this patch, this timeout is replaced by client-fin once a GOAWAY frame is sent. This closely matches what is done on HTTP/1 since the principle is the same, as it indicates a willing ness to quickly close a connection on which we don't expect to see anything anymore. commit a76e4c21839cafd036fbe755416569206502c1d9 Author: Willy Tarreau Date: Fri Nov 24 08:17:28 2017 +0100 MEDIUM: h2: don't gracefully close the connection anymore on Connection: close As reported by Lukas, it causes more harm than good, for example on prompt for authentication. Now we have an "http-request reject" rule to use instead of "http-request deny" if we absolutely want to close the connection. commit 90c3232e54c7c0eeedb294a2ac80247a732678ac Author: Willy Tarreau Date: Fri Nov 24 08:00:30 2017 +0100 MINOR: h2: send RST_STREAM before GOAWAY on reject Apparently the h2c client has trouble reading the RST_STREAM frame after a GOAWAY was sent, so it's likely that other clients may face the same difficulty. Curl and Firefox don't care about this ordering, so let's send it first. commit 53275e8b02a0829ccdeb509fb3d9819da3d9afe9 Author: Willy Tarreau Date: Fri Nov 24 07:52:01 2017 +0100 MINOR: http: implement the "http-request reject" rule This one acts similarly to its tcp-request counterpart. It immediately closes the request without emitting any response. It can be suitable in certain DoS conditions, as well as to close an HTTP/2 connection. commit f528fff46bc00426f0329b46cbf77a88c92d3f8d Author: William Lallemand Date: Thu Nov 23 19:43:17 2017 +0100 MEDIUM: cache: store sha1 for hashing the cache key The cache was relying on the txn->uri for creating its key, which was a big problem when there was no log activated. This patch does a sha1 of the host + uri, and stores it in the txn. When a object is stored, the eb32node uses the first 32 bits of the hash as a key, and the whole hash is stored in the cache entry. During a lookup, the truncated hash is used, and when it matches an entry we check the real sha1. commit 7fc96d5a01b85cd324faa57da7bfedbe52f96dfe Author: Olivier Houchard Date: Thu Nov 23 18:25:47 2017 +0100 MINOR: mux: Make sure every string is woken up after the handshake. In case any stream was waiting for the handshake after receiving early data, we have to wake all of them. Do so by making the mux responsible for removing the CO_FL_EARLY_DATA flag after all of them are woken up, instead of doing it in si_cs_wake_cb(), which would then only work for the first one. This makes wait_for_handshake work with HTTP/2. commit 90084a133dc9fef84bd8bb30bafa947941701515 Author: Olivier Houchard Date: Thu Nov 23 18:21:29 2017 +0100 MINOR: ssl: Handle reading early data after writing better. It can happen that we want to read early data, write some, and then continue reading them. To do so, we can't reuse tmp_early_data to store the amount of data sent, so introduce a new member. If we read early data, then ssl_sock_to_buf() is now the only responsible for getting back to the handshake, to make sure we don't miss any early data. commit 51753458c419527eb706cc417692bda461df85a2 Author: Willy Tarreau Date: Thu Nov 23 18:36:50 2017 +0100 BUG/MAJOR: threads/task: dequeue expired tasks under the WQ lock There is a small unprotected window for a task between the wait queue and the run queue where a task could be woken up and destroyed at the same time. What typically happens is that a timeout is reached at the same time an I/O completes and wakes it up, and the I/O terminates the task, causing a use after free in wake_expired_tasks() possibly causing a crash and/or memory corruption : thread 1 thread 2 (wake_expired_tasks) (stream_int_notify) HA_SPIN_UNLOCK(TASK_WQ_LOCK, &wq_lock); task_wakeup(task, TASK_WOKEN_IO); ... process_stream() stream_free() task_free() pool_free(task) task_wakeup(task, TASK_WOKEN_TIMER); This case is reasonably easy to reproduce with a config using very short server timeouts (100ms) and client timeouts (10ms), while injecting on httpterm requesting medium sized objects (5kB) over SSL. All this is easier done with more threads than allocated CPUs so that pauses can happen anywhere and last long enough for process_stream() to kill the task. This patch inverts the lock and the wakeup(), but requires some changes in process_runnable_tasks() to ensure we never try to grab the WQ lock while having the RQ lock held. This means we have to release the RQ lock before calling task_queue(), so we can't hold the RQ lock during the loop and must take and drop it. It seems that a different approach with the scope-aware trees could be easier, but it would possibly not cover situations where a task is allowed to run on multiple threads. The current solution covers it and doesn't seem to have any measurable performance impact. commit 541dd82879b686dd208e8823154fb07bd945fb85 Author: Willy Tarreau Date: Thu Nov 23 18:12:50 2017 +0100 BUG/MAJOR: h2: always remove a stream from the send list before freeing it When a stream is aborted on timeout or any reason initiated by the stream, and this stream was subscribed to the send list, we forgot to detach it when freeing it, resulting in a dead node remaining present in the send list with all usual funny consequences (memory corruption, crashes, etc). Let's simply unconditionally delete the stream. commit ee8269e84d2c10328fe933646e74602c0d17f457 Author: Willy Tarreau Date: Thu Nov 23 17:34:29 2017 +0100 BUG/MINOR: stream: fix tv_request calculation for applets When the stats code was moved to an applet, it wasn't completely cleaned of its usage of the HTTP transaction and it used to store the HTTP status in txn->status and to set the HTTP request date to from within the applet. This is totally wrong because the applet is seen as a server from the HTTP engine, which parses its response, so the http_txn must not be touched there. This was made visible by the cache which would always exhibit a negative TR log, indicating that nowhere in the code we took care of setting s->logs.tv_request while the code above used to continue to hide this. Another side effect of this issue is that under load, if the stats applet call risks to be delayed, the reported t_queue can appear negative by being below tv_request-tv_accept. This patch removes the assignment of tv_request and txn->status from the applet code and instead sets the tv_request if still unset when connecting to the applet. This ensures that all applets report correct request timers now. commit ff3a41eb3f990ee1cc25a5fa961fe8e490254d0c Author: Christopher Faulet Date: Thu Nov 23 09:13:32 2017 +0100 BUG/MINOR: Use crt_base instead of ca_base when crt is parsed on a server line In srv_parse_crt, crt_base was checked but ca_base was used to build the certifacte path. This patch must be backported in 1.7, 1.6 and 1.5. commit 34adb2af967a12d7a9e445cfb4941b8ae9434f45 Author: Christopher Faulet Date: Tue Nov 21 21:45:38 2017 +0100 MINOR: sample: Add "thread" sample fetch It returns id of the thread calling the function. commit 9fefc51c569b692a09229182a76fae78883b1cd8 Author: Willy Tarreau Date: Thu Nov 23 14:52:28 2017 +0100 BUG/MEDIUM: threads/time: maintain a common time reference between all threads During high loads it becomes visible that the time drifts between threads, sometimes showing tens of seconds after several minutes. The root cause is the per-thread correction which is performed based on a local offset and local time. But we can't use a unique global time either as we need the thread-local time to be stable between two poll() calls. This commit takes a stab at this problem by proceeding this way : - a global "global_now" date is monotonous and common between all threads. - each thread has its own local which is resynced with on each invocation of tv_update_date() - each thread detects its own drift based on its poll() timeout and its local , and recalculates its adjusted local time - each thread then ensures its new local time is no older than the current global time, otherwise it readjusts its local time to match this one - finally threads do atomically update the global time to match its own local one This guarantees a monotonous global time and a monotonous+stable local time. It is still possible by definition for two threads to report a minor time variation on subsequent events but that variation will only be caused by the moment they watched the time and are very small. When a common global time is needed between all threads, global_now could be used as a reference (with care). The wallclock time used in logs is still anyway. commit 7649aacf7f17410b88f258b2b2eddea9e8445922 Author: Willy Tarreau Date: Thu Nov 23 11:52:55 2017 +0100 BUG/MEDIUM: threads/time: fix time drift correction With threads, it became mandatory to implement a thread-local time with its own correction. However, it was noticed that during high thread contention, the time correction could occasionally be wrong, reporting huge negative or positive timers in logs. This was caused by the conversion between struct timeval and a single 64-bit offset, due to an erroneous shift and due to a loss of sign during the conversion. Given that time_t is not always signed, and that timeval is not really needed here, better avoid playing dangerous games with these operations and use a single 64-bit offset representing a signed 32-bit offset, for the seconds part and an unsigned offset for the microsecond part. It still supports atomic updates and doesn't cause issues anymore. commit 158fa7581173234ab3bebf9dee91d18cf6a138d0 Author: Willy Tarreau Date: Wed Nov 22 15:47:29 2017 +0100 MINOR: pools: implement DEBUG_UAF to detect use after free This code has been used successfully a few times in the past to detect that a pool was used after being freed. Its main goal is to allocate a full page for each object so that they are always released individually and unmapped from memory. This way if any part of the code reference the object after is was freed and before it is reallocated, a segv occurs at the exact offending location. It does a few extra things such as writing to the memory area before freeing to detect double-frees and free of read-only areas, and placing the data at the end of the page instead of the beginning so that out of bounds accesses are easier to spot. The amount of memory used with this is huge (about 10 times the regular usage) but it can be useful sometimes. commit f13322ede1e4824e20fb871dabd22e91dcc8ab0a Author: Willy Tarreau Date: Wed Nov 22 10:50:54 2017 +0100 MINOR: pools: prepare functions to override malloc/free in pools This will be useful to add some debugging capabilities. For now it changes nothing. commit 424ecfb33ca412d8b8bc8edd531975a5440bc9ce Author: Olivier Houchard Date: Wed Nov 22 19:12:10 2017 +0100 MINOR: ssl: Don't disable early data handling if we could not write. If we can't write early data, for some reason, don't give up on reading them, they may still be early data to be read, and if we don't do so, openssl internal states might be inconsistent, and the handshake will fail. commit 777e4b98a3252f89615d528f686087a9ab22d169 Author: Olivier Houchard Date: Wed Nov 22 17:38:37 2017 +0100 BUG/MINOR: ssl: Always start the handshake if we can't send early data. The current code only tries to do the handshake in case we can't send early data if we're acting as a client, which is wrong, it has to be done on the server side too, or we end up in an infinite loop. commit 1f89b1805b308df51a1329aea9a81d2ba55dfedc Author: Willy Tarreau Date: Wed Nov 22 16:53:53 2017 +0100 BUG/MEDIUM: deinit: correctly deinitialize the proxy and global listener tasks While using mmap() to allocate pools for debugging purposes, kill -USR1 caused libc aborts in deinit() on two calls to free() on proxies' tasks and the global listener task. The issue comes from the fact that we're using free() to release a task instead of task_free(), so the task was allocated from a pool and released using a different method. This bug has been there since at least 1.5, so a backport is desirable to all maintained versions. commit e899af89b5f76bb79efae9fa3d7dda23d7a6f537 Author: William Lallemand Date: Wed Nov 22 16:41:26 2017 +0100 BUG/MEDIUM: cache fix cli_kws structure The cli_kws structure was not ended and was causing undefined behavior. commit 55e7674bc4a99fc20f10144053d0b5c3ab23bd0e Author: William Lallemand Date: Tue Nov 21 20:01:28 2017 +0100 BUG/MEDIUM: cache: refcount forbids to free the objects Some refcount decrementation were forgotten and they were forbidding to reuse the objects in some cases. commit 0872766e313651c73f5a3fef1d23ec1389f2203d Author: William Lallemand Date: Tue Nov 21 20:01:27 2017 +0100 BUG/MEDIUM: cache: use key=0 as a condition for freeing The cache was trying to remove objects from the tree while they were already removed from it. We set the key to 0 as a check for not trying to remove the object from the tree when we are still using the object. commit 1f49a366fd94f4c9f68c942c66e27ac4279d6370 Author: William Lallemand Date: Tue Nov 21 20:01:26 2017 +0100 MEDIUM: cache: "show cache" on the cli The cli command "show cache" displays the status of the cache, the first displayed line is the shctx informations with how much blocks available blocks it contains (blocks are 1k by default). The next lines are the objects stored in the cache tree, the pointer, the size of the object and how much blocks it uses, a refcount for the number of users of the object, and the remaining expiration time (which can be negative if expired) Example: $ echo "show cache" | socat - /run/haproxy.sock 0x7fa54e9ab03a: foobar (shctx:0x7fa54e9ab000, available blocks:3921) 0x7fa54ed65b8c (size: 43190 (43 blocks), refcount:2, expire: 2) 0x7fa54ecf1b4c (size: 45238 (45 blocks), refcount:0, expire: 2) 0x7fa54ed70cec (size: 61622 (61 blocks), refcount:0, expire: 2) 0x7fa54ecdbcac (size: 42166 (42 blocks), refcount:1, expire: 2) 0x7fa54ec9736c (size: 44214 (44 blocks), refcount:2, expire: 2) 0x7fa54eca28ec (size: 46262 (46 blocks), refcount:2, expire: -2) commit 111bfef33c97cf40dc9c136f51b9f670e92d6145 Author: William Lallemand Date: Tue Nov 21 20:01:25 2017 +0100 MEDIUM: shctx: use unsigned int for len and block_count Allows bigger objects to be cached in the shctx, the first implementation was only storing small ssl session, but we want to store bigger HTTP response. commit 75d93291c98c1ae39c93b5debeec4c182e62b596 Author: William Lallemand Date: Tue Nov 21 20:01:24 2017 +0100 CLEANUP: cache: reorder includes commit 543818327614646edb9dc432832f75057fda63f7 Author: Eric Salama Date: Wed Nov 15 14:14:30 2017 +0100 CONTRIB: spoa_example: remove SPOE enums that are useless for clients commit 75f42466c0eaa5104c9f6f643c4ce8c35417091f Author: Willy Tarreau Date: Tue Nov 14 15:01:22 2017 +0100 CONTRIB: spoa_example: remove last dependencies on type "sample" Being an external agent, it's confusing that it uses haproxy's internal types and it seems to have encouraged other implementations to do so. Let's completely remove any reference to struct sample and use the native DATA types instead of converting to and from haproxy's sample types. commit 9f95ff0647a51e935f1c759286b1e954b9e08ba8 Author: Willy Tarreau Date: Tue Nov 14 11:11:53 2017 +0100 CONTRIB: spoa_example: remove bref, wordlist, cond_wordlist These ones are not needed, let's further reduce the include file. commit 8a9c6c2154166579b93b7eabb4e0591ef37ca883 Author: Eric Salama Date: Fri Nov 10 11:02:23 2017 +0100 CONTRIB: spoa_example: allow to compile outside HAProxy. Don't include haproxy's includes anymore and use a local copy instead. commit f46bf95d2b4726d7f69a66824001bcdae0899b57 Author: Lukas Tribus Date: Tue Nov 21 12:39:34 2017 +0100 BUG/MINOR: systemd: ignore daemon mode Since we switched to notify mode in the systemd unit file in commit d6942c8, haproxy won't start if the daemon keyword is present in the configuration. This change makes sure that haproxy remains in foreground when using systemd mode and adds a note in the documentation. commit 2fb986ccb85cbce08ec5b270b2cf635b046c3b79 Author: Willy Tarreau Date: Tue Nov 21 21:01:29 2017 +0100 BUG/MEDIUM: h2: always reassemble the Cookie request header field The special case of the Cookie header field was overlooked in the implementation, considering that most servers do handle cookie lists, but as reported here on discourse it's not the case at all : https://discourse.haproxy.org/t/h2-cookie-header-splitted-header/1742 This patch fixes this by skipping all occurences of the Cookie header in the request while building the H1 request, and then building a single Cookie header with all values appended at once, according to what is requested in RFC7540#8.1.2.5. In order to build the list of values, the list struct is used as a linked list (as there can't be more cookies than headers). This makes the list walking quite efficient and ensures all values are quickly found without having to rescan the list. A test case provided by Lukas shows that it properly works : > GET /? HTTP/1.1 > user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0 > accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > accept-language: en-US,en;q=0.5 > accept-encoding: gzip, deflate > referer: https://127.0.0.1:4443/?expectValue=1511294406 > host: 127.0.0.1:4443 < HTTP/1.1 200 OK < Server: nginx < Date: Tue, 21 Nov 2017 20:00:13 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: PHP/5.3.10-1ubuntu3.26 < Set-Cookie: HAPTESTa=1511294413 < Set-Cookie: HAPTESTb=1511294413 < Set-Cookie: HAPTESTc=1511294413 < Set-Cookie: HAPTESTd=1511294413 < Content-Encoding: gzip > GET /?expectValue=1511294413 HTTP/1.1 > user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0 > accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > accept-language: en-US,en;q=0.5 > accept-encoding: gzip, deflate > host: 127.0.0.1:4443 > cookie: SERVERID=s1; HAPTESTa=1511294413; HAPTESTb=1511294413; HAPTESTc=1511294413; HAPTESTd=1511294413 Many thanks to @Nurza, @adrianw and @lukastribus for their helpful reports and investigations here. commit 59a10fb53d3f0841916d0d996db83c2bfe0b1378 Author: Willy Tarreau Date: Tue Nov 21 20:03:02 2017 +0100 MEDIUM: h2: change hpack_decode_headers() to only provide a list of headers The current H2 to H1 protocol conversion presents some issues which will require to perform some processing on certain headers before writing them so it's not possible to convert HPACK to H1 on the fly. This commit modifies the headers decoding so that it now works in two phases : hpack_decode_headers() only decodes the HPACK stream in the HEADERS frame and puts the result into a list. Headers which require storage (huffman-compressed or from the dynamic table) are stored in a chunk allocated by the H2 demuxer. Then once the headers are properly decoded into this list, h2_make_h1_request() is called with this list to produce the HTTP/1.1 request into the destination buffer. The list necessarily enforces a limit. Here we use 2*MAX_HTTP_HDR, which means that we can have as many individual cookies as we have regular headers if a client decides to break their cookies into multiple values. This seams reasonable and will allow the H1 parser to decide whether it's too much or not. Thus the output stream is not produced on the fly anymore and this will permit to deal with certain corner cases like reparing the Cookie header (which for now is not done). In order to limit header duplication and parsing, the known pseudo headers continue to be passed by their index : the name element in the list then has a NULL pointer and the value is the pseudo header's index. Given that these ones represent about half of the incoming requests and need to be found quickly, it maintains an acceptable level of performance. The code was significantly reduced by doing this because the orignal code had to deal with HPACK and H1 combinations (eg: index vs not indexed, etc) and now the HPACK decoding is totally focused on the decompression, and the H1 encoding doesn't have to deal with the issue of wrapping input for example. One bug was addressed here (though it couldn't happen at the moment). The H2 demuxer used to detect a failure to write the request into the H1 buffer and would then detect if the output buffer wraps, realign it and try again. The problem by doing so was that the HPACK context was already modified and not rewindable. Thus the size check is now performed first and a failure is reported if it doesn't fit. commit f24ea8e45e0242cc6a05d3c6bcbc825fd4c7d8b7 Author: Willy Tarreau Date: Tue Nov 21 19:55:27 2017 +0100 MEDIUM: h2: add a function to emit an HTTP/1 request from a headers list The current H2 to H1 protocol conversion presents some issues which will require to perform some processing on certain headers before writing them so it's not possible to convert HPACK to H1 on the fly. Here we introduce a function which performs half of what hpack_decode_header() used to do, which is to take a list of headers on input and emit the corresponding request in HTTP/1.1 format. The code is the same and functions were renamed to be prefixed with "h2" instead of "hpack", though it ends up being simpler as the various HPACK-specific cases could be fused into a single one (ie: add header). Moving this part here makes a lot of sense as now this code is specific to what is documented in HTTP/2 RFC 7540 and will be able to deal with special cases related to H2 to H1 conversion enumerated in section 8.1. Various error codes which were previously assigned to HPACK were never used (aside being negative) and were all replaced by -1 with a comment indicating what error was detected. The code could be further factored thanks to this but this commit focuses on compatibility first. This code is not yet used but builds fine. commit 8f650c369dbe1d6db3538c8fd16ace636c571fea Author: Willy Tarreau Date: Tue Nov 21 19:36:21 2017 +0100 BUG/MEDIUM: h2: properly report connection errors in headers and data handlers We used to return >0 indicating a success when an error was present on the connection, preventing the caller from detecting and handling it. This for example happens when sending too many headers in a frame, making the request impossible to decompress. commit 358847f026f7ca69e0c341326da3c34d45ec27db Author: Willy Tarreau Date: Mon Nov 20 21:33:21 2017 +0100 BUILD: server: check->desc always exists Clang reports this warning : src/server.c:872:14: warning: address of array 'check->desc' will always evaluate to 'true' [-Wpointer-bool-conversion] Indeed, check->desc used to be a pointer to a dynamically allocated area a long time ago and is now an array. Let's remove the useless test. commit 1f0946711414fa90e71d9525f373e9819f10f9dc Author: Willy Tarreau Date: Mon Nov 20 21:27:45 2017 +0100 BUILD: h2: mark some inlined functions "unused" Clang complains that h2_get_n64() is not used, and a few other protocol specific functions may fall in that category depending on how the code evolves. Better mark them unused to silence the warning since it's on purpose. commit dbd25fc75ad2e409c7347289d245e10521cc240a Author: Willy Tarreau Date: Mon Nov 20 21:22:17 2017 +0100 BUILD: compiler: add a new type modifier __maybe_unused While gcc only emits warnings about unused static functions, Clang also emits such a warning when the functions are inlined. This is a bit annoying at certain places where functions are provided to manipulate multiple data types and are not yet used. Let's have a type modifier "__maybe_unused" which sets the "unused" attribute like the Linux kernel does. It's elegant as it allows the code author to indicate that it knows that this element might be unused. It works on variables as well, which is convenient to remove ifdefs around local variables in certain functions, but doesn't work on labels. commit 0cb98b2451a6005c47ff1b49737ffbd9bb0cf179 Author: Willy Tarreau Date: Mon Nov 20 21:11:12 2017 +0100 BUILD: ebtree: don't redefine types u32/s32 in scope-aware trees Clang emits a warning about these types being redefined in eb32sctree while they are already defined in eb32tree. Let's simply not redefine them if eb32tree was already included. commit 2532bd2f8175d61a4303fef6f1aa7d3a24b746fc Author: Willy Tarreau Date: Mon Nov 20 19:25:18 2017 +0100 BUILD: threads/plock: fix a build issue on Clang without optimization [ plock commit 4c53fd3a0b2b1892817cebd0db012a52f4087850 ] Pieter Baauw reported a build issue affecting haproxy after plock was included. It happens that expressions of the form : if ((const) ? (expr1) : (expr2)) do_something() always produce code for both expr1 and expr2 on Clang when building without optimization. The resulting asm code is even funny, basically doing : mov reg, 1 cmp reg, 1 ... This causes our sizeof() tests to fail to build because we purposely dereference a fake function that reports the location and nature of the inconsistency, but this fake function appears in the object code despite all conditions being there to avoid it. However the compiler is still smart enough to optimize away code doing if (const) do_something() So we simply repeat the condition before do_something(), and the dummy function is not referenced anymore unless really required. commit b5f271555e9b86e60d60a6986a23c82adbde05fb Author: Willy Tarreau Date: Mon Nov 20 20:55:06 2017 +0100 MINOR: threads/build: atomic: replace the few inlines with macros [ plock commit 61e255286ae32e83e1a3174dd7c49eda99880a8b] There are a few inlines such as pl_barrier() and pl_cpu_relax() which are used a lot. Unfortunately, while building test code at -O0, inlining is disabled and these ones are called a lot and show up a lot in any profile, are traced into when single-stepping with a debugger, etc, thus they are polluting the landscape. Since they're single-asm statements, there is no reason for not turning them into macros. The result becomes fairly visible here at -O0 : $ size latency.inline latency.macro text data bss dec hex filename 11431 692 656 12779 31eb treelock.inline 10967 692 656 12315 301b treelock.macro And it was verified that regularly optimized code remains strictly identical. commit d0d8ba59d3334d51f9b9f3f8c1179747d2553564 Author: Willy Tarreau Date: Tue Jul 18 15:05:58 2017 +0200 MINOR: threads/atomic: implement pl_bts() on non-x86 [ plock commit da17ba320aad3a8faf08e36fca604de9cad21fdd ] This one was missing, it can be done using sync_fetch_and_or(). commit 01b8398b9ec8be853cd450e23dee50aba279c89f Author: Willy Tarreau Date: Tue Jul 18 15:01:39 2017 +0200 MINOR: threads/atomic: implement pl_mb() in asm on x86 [ plock commit 44081ea493dd78dab48076980e881748e9b33db5 ] Older compilers (eg: gcc 3.4) don't provide __sync_synchronize() so let's do it by hand on this platform. commit f7ba77eb807a5511dbb115634b2e2781dfe83384 Author: Willy Tarreau Date: Tue Jul 18 14:21:40 2017 +0200 MINOR: threads/plock: rename local variables in macros to avoid conflicts [ plock commit b155d5c762fb9a9793911881f80e61faa6b0e889 ] Local variables "l", "i" and "ret" were renamed "__pl_l", "__pl_i" and "__pl_r" respectively, to limit the risk of conflicts with existing variables in application code. commit 98409e34cab8df0285a7135348867ec7bc064c00 Author: Willy Tarreau Date: Tue Jul 18 14:20:41 2017 +0200 MINOR: threads/atomic: rename local variables in macros to avoid conflicts [ plock commit bfac5887ebabb8ef753b0351f162265767eb219b ] Local variable "t" was renamed "__pl_t" to limit the risk of conflicts with existing variables in application code. commit eee5c39715cb79566c6e33453a82421ef8e49e5d Author: William Lallemand Date: Mon Nov 20 19:13:15 2017 +0100 CLEANUP: cache: remove wrong comment commit 71bd11a1f3764d693adc38b9667482a0e31a09fe Author: William Lallemand Date: Mon Nov 20 19:13:14 2017 +0100 MEDIUM: cache: enable the HTTP analysers Enable the same analysers as the stats applet. Allows keepalive and termination flags to work. commit 44e259c0b762663f4f5193b5d240ee1cddceff47 Author: William Lallemand Date: Mon Nov 20 19:13:13 2017 +0100 CLEANUP: cache: remove unused struct Remove unused structure which remain from old dev. commit a400a3a6d07468161da77242fbb3e07aa07e3346 Author: William Lallemand Date: Mon Nov 20 19:13:12 2017 +0100 BUG/MEDIUM: cache: free callback to remove from tree Call the shctx free_blocks callback in order to remove the row from the cache tree. Put the row in the hot list during allocation, forbid the blocks to be stolen by a free or a row_reserve commit d6942c829746d9a28f1f92b531dd858db89a5c64 Author: Tim Duesterhus Date: Mon Nov 20 15:58:35 2017 +0100 MEDIUM: mworker: Add systemd `Type=notify` support This patch adds support for `Type=notify` to the systemd unit. Supporting `Type=notify` improves both starting as well as reloading of the unit, because systemd will be let known when the action completed. See this quote from `systemd.service(5)`: > Note however that reloading a daemon by sending a signal (as with the > example line above) is usually not a good choice, because this is an > asynchronous operation and hence not suitable to order reloads of > multiple services against each other. It is strongly recommended to > set ExecReload= to a command that not only triggers a configuration > reload of the daemon, but also synchronously waits for it to complete. By making systemd aware of a reload in progress it is able to wait until the reload actually succeeded. This patch introduces both a new `USE_SYSTEMD` build option which controls including the sd-daemon library as well as a `-Ws` runtime option which runs haproxy in master-worker mode with systemd support. When haproxy is running in master-worker mode with systemd support it will send status messages to systemd using `sd_notify(3)` in the following cases: - The master process forked off the worker processes (READY=1) - The master process entered the `mworker_reload()` function (RELOADING=1) - The master process received the SIGUSR1 or SIGTERM signal (STOPPING=1) Change the unit file to specify `Type=notify` and replace master-worker mode (`-W`) with master-worker mode with systemd support (`-Ws`). Future evolutions of this feature could include making use of the `STATUS` feature of `sd_notify()` to send information about the number of active connections to systemd. This would require bidirectional communication between the master and the workers and thus is left for future work. commit 62dd698070c80d1ee4e779b1f7622386a45508dc Author: Willy Tarreau Date: Sat Nov 18 11:26:20 2017 +0100 BUG/MINOR: stream-int: don't try to read again when CF_READ_DONTWAIT is set Commit 9aaf778 ("MAJOR: connection : Split struct connection into struct connection and struct conn_stream.") had to change the way the stream interface deals with incoming data to accomodate the mux. A break statement got lost during a change, leading to the receive call being performed twice even when CF_READ_DONTWAIT is set. The most noticeable effect is that it made the bug described in commit 33982cb ("BUG/MAJOR: stream: ensure analysers are always called upon close") much easier to reproduce as it would appear even with an HTTP frontend. Let's just restore the stream-interface flag and the break here, as in the previous code. No backport is needed as this was introduced during 1.8-dev. commit 33982cbdc062268d167a673274e3cc1cd7e3b232 Author: Willy Tarreau Date: Mon Nov 20 15:37:13 2017 +0100 BUG/MAJOR: stream: ensure analysers are always called upon close A recent issue affecting HTTP/2 + redirect + cache has uncovered an old problem affecting all existing versions regarding the way events are reported to analysers. It happens that when an event is reported, analysers see it and may decide to temporarily pause processing and prevent other analysers from processing the same event. Then the event may be cleared and upon the next call to the analysers, some of them will never see it. This is exactly what happens with CF_READ_NULL if it is received before the request is processed, like during redirects : the first time, some analysers see it, pause, then the event may be converted to a SHUTW and cleared, and on next call, there's nothing to process. In practice it's hard to get the CF_READ_NULL flag during the request because requests have CF_READ_DONTWAIT, preventing the read0 from happening. But on HTTP/2 it's presented along with any incoming request. Also on a TCP frontend the flag is not set and it's possible to read the NULL before the request is parsed. This causes a problem when filters are present because flt_end_analyse needs to be called to release allocated resources and remove the CF_FLT_ANALYZE flag. And the loss of this event prevents the analyser from being called and from removing itself, preventing the connection from ever ending. This problem just shows that the event processing needs a serious revamp after 1.8. In the mean time we can deal with the really problematic case which is that we *want* to call analysers if CF_SHUTW is set on any side ad it's the last opportunity to terminate a processing. It may occasionally result in some analysers being called for nothing in half- closed situations but it will take care of the issue. An example of problematic configuration triggering the bug in 1.7 is : frontend tcp bind :4445 default_backend http backend http redirect location / compression algo identity Then submitting requests which immediately close will have for effect to accumulate streams which will never be freed : $ printf "GET / HTTP/1.1\r\n\r\n" >/dev/tcp/0/4445 This fix must be backported to 1.7 as well as any version where commit c0c672a ("BUG/MINOR: http: Fix conditions to clean up a txn and to handle the next request") was backported. This commit didn't cause the bug but made it much more likely to happen. commit e223e3bc85393b36cbfe9057450adf2b9a3b7ca8 Author: Willy Tarreau Date: Sat Nov 18 15:39:10 2017 +0100 BUG/MEDIUM: stream: don't automatically forward connect nor close Upon stream instanciation, we used to enable channel auto connect and auto close to ease TCP processing. But commit 9aaf778 ("MAJOR: connection : Split struct connection into struct connection and struct conn_stream.") has revealed that it was a bad idea because this commit enables reading of the trailing shutdown that may follow a small requests, resulting in a read and a shutr turned into shutw before the stream even has a chance to apply the filters. This causes an issue with impossible situations where the backend stream interface is still in SI_ST_INI with a closed output, which blocks some streams for example when performing a redirect with filters enabled. Let's change this so that we only enable these two flags if there is no analyser on the stream. This way process_stream() has a chance to let the analysers decide whether or not to allow the shutdown event to be transferred to the other side. It doesn't seem possible to trigger this issue before 1.8, so for now it is preferable not to backport this fix. commit cfe14669f7f05c6b0d70eaa64944bcbcaa963218 Author: Willy Tarreau Date: Sun Nov 19 09:55:29 2017 +0100 [RELEASE] Released version 1.8-rc4 Released version 1.8-rc4 with the following main changes : - BUG/MEDIUM: cache: does not cache if no Content-Length - BUILD: thread/pipe: fix build without threads - BUG/MINOR: spoe: check buffer size before acquiring or releasing it - MINOR: debug/flags: Add missing flags - MINOR: threads: Use __decl_hathreads to declare locks - BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe - BUG/MAJOR: ebtree/scope: fix insertion and removal of duplicates in scope-aware trees - BUG/MAJOR: ebtree/scope: fix lookup of next node in scope-aware trees - MINOR: ebtree/scope: add a function to find next node from a parent - MINOR: ebtree/scope: simplify the lookup functions by using eb32sc_next_with_parent() - BUG/MEDIUM: mworker: Fix re-exec when haproxy is started from PATH - BUG/MEDIUM: cache: use msg->sov to forward header - MINOR: cache: forward data with headers - MINOR: cache: disable cache if shctx_row_data_append fail - BUG/MINOR: threads: tid_bit must be a unsigned long - CLEANUP: tasks: Remove useless double test on rq_next - BUG/MEDIUM: standard: itao_str/idx and quote_str/idx must be thread-local - MINOR: tools: add a function to dump a scope-aware tree to a file - MINOR: tools: improve the DOT dump of the ebtree - MINOR: tools: emphasize the node being worked on in the tree dump - BUG/MAJOR: ebtree/scope: properly tag upper nodes during insertion - DOC: peers: Add a first version of peers protocol v2.1. - CONTRIB: Wireshark dissector for HAProxy Peer Protocol. - MINOR: mworker: display an accurate error when the reexec fail - BUG/MEDIUM: mworker: wait again for signals when execvp fail - BUG/MEDIUM: mworker: does not deinit anymore - BUG/MEDIUM: mworker: does not close inherited FD - MINOR: tests: add a python wrapper to test inherited fd - BUG/MINOR: Allocate the log buffers before the proxies startup - MINOR: tasks: Use a bitfield to track tasks activity per-thread - MAJOR: polling: Use active_tasks_mask instead of tasks_run_queue - MINOR: applets: Use a bitfield to track applets activity per-thread - MAJOR: polling: Use active_appels_mask instead of applets_active_queue - MEDIUM: applets: Don't process more than 200 active applets at once - MINOR: stream: Add thread-mask of tasks/FDs/applets in "show sess all" command - MINOR: SSL: Store the ASN1 representation of client sessions. - MINOR: ssl: Make sure we don't shutw the connection before the handshake. - BUG/MEDIUM: deviceatlas: ignore not valuable HTTP request data commit 91a88b0c2568a28c4c830a7eb10c3d8fb5b1e8cb Author: David Carlier Date: Fri Nov 17 08:47:25 2017 +0000 BUG/MEDIUM: deviceatlas: ignore not valuable HTTP request data A customer reported a crash when within the HTTP request some headers were not set leading to the module to crash. So the module ignore them since empty data have no value for the detection. Needs to be backported to 1.7. commit e9bed53486888062f7f3c277c901aefab62959be Author: Olivier Houchard Date: Thu Nov 16 17:49:25 2017 +0100 MINOR: ssl: Make sure we don't shutw the connection before the handshake. Instead of trying to finish the handshake in ssl_sock_shutw, which may fail, try not to shutdown until the handshake is finished. commit e6060c5d877e78cfab04cca4f603340540206994 Author: Olivier Houchard Date: Thu Nov 16 17:42:52 2017 +0100 MINOR: SSL: Store the ASN1 representation of client sessions. Instead of storing the SSL_SESSION pointer directly in the struct server, store the ASN1 representation, otherwise, session resumption is broken with TLS 1.3, when multiple outgoing connections want to use the same session. commit f02050662b1699a3c107a3275bc624104df98a49 Author: Christopher Faulet Date: Wed Nov 15 20:56:43 2017 +0100 MINOR: stream: Add thread-mask of tasks/FDs/applets in "show sess all" command commit b4a4d9aed4342e387ead87e8c43ea88acaa70d64 Author: Christopher Faulet Date: Wed Nov 15 22:14:49 2017 +0100 MEDIUM: applets: Don't process more than 200 active applets at once Now, we process at most 200 active applets per call to applet_run_active. We use the same limit as the tasks. With the cache filter and the SPOE, the number of active applets can now be huge. So, it is important to limit the number of applets processed in applet_run_active. commit 7163056dc55e39e57768c0857040125e7b9db4f1 Author: Christopher Faulet Date: Tue Nov 14 11:30:47 2017 +0100 MAJOR: polling: Use active_appels_mask instead of applets_active_queue applets_active_queue is the active queue size. It is a global variable. So it is underoptimized because we may be lead to consider there are active applets for a thread while in fact all active applets are assigned to the otherthreads. So, in such cases, the polling loop will be evaluated many more times than necessary. Instead, we now check if the thread id is set in the bitfield active_applets_mask. This is specific to threads, no backport is needed. commit 595d7b72a6a58c175bfeb3c44dd51a05ffcdeab3 Author: Christopher Faulet Date: Tue Nov 14 11:28:52 2017 +0100 MINOR: applets: Use a bitfield to track applets activity per-thread a bitfield has been added to know if there are runnable applets for a thread. When an applet is woken up, the bits corresponding to its thread_mask are set. When all active applets for a thread is get to be processed, the thread is removed from active ones by unsetting its tid_bit from the bitfield. commit 8a48f675269e61721e2a5f8d08e2158503fceef0 Author: Christopher Faulet Date: Tue Nov 14 10:38:36 2017 +0100 MAJOR: polling: Use active_tasks_mask instead of tasks_run_queue tasks_run_queue is the run queue size. It is a global variable. So it is underoptimized because we may be lead to consider there are active tasks for a thread while in fact all active tasks are assigned to the other threads. So, in such cases, the polling loop will be evaluated many more times than necessary. Instead, we now check if the thread id is set in the bitfield active_tasks_mask. Another change has been made in process_runnable_tasks. Now, we always limit the number of tasks processed to 200. This is specific to threads, no backport is needed. commit 3911ee85df812e63d359f2d6e8ef0fb663254249 Author: Christopher Faulet Date: Tue Nov 14 10:26:53 2017 +0100 MINOR: tasks: Use a bitfield to track tasks activity per-thread a bitfield has been added to know if there are runnable tasks for a thread. When a task is woken up, the bits corresponding to its thread_mask are set. When all tasks for a thread have been evaluated without any wakeup, the thread is removed from active ones by unsetting its tid_bit from the bitfield. commit 96d4483df75458614339879e359b72169fa6b760 Author: Christopher Faulet Date: Tue Nov 14 22:02:30 2017 +0100 BUG/MINOR: Allocate the log buffers before the proxies startup Since the commit cd7879adc ("BUG/MEDIUM: threads: Run the poll loop on the main thread too"), the log buffers are allocated after the proxies startup. So log messages produced during this startup was ignored. To fix the bug, we restore the initialization of these buffers before proxies startup. This is specific to threads, no backport is needed. commit 51606feaf2a303a4d9f20c0f4fc33cbd3834255e Author: William Lallemand Date: Wed Nov 15 19:17:23 2017 +0100 MINOR: tests: add a python wrapper to test inherited fd commit 75ea0a06b070c982ef1de986532f535a3179b632 Author: William Lallemand Date: Wed Nov 15 19:02:58 2017 +0100 BUG/MEDIUM: mworker: does not close inherited FD At the end of the master initialisation, a call to protocol_unbind_all() was made, in order to close all the FDs. Unfortunately, this function closes the inherited FDs (fd@), upon reload the master wasn't able to reload a configuration with those FDs. The create_listeners() function now store a flag to specify if the fd was inherited or not. Replace the protocol_unbind_all() by mworker_cleanlisteners() + deinit_pollers() commit fade49d8fbe54ba9643d0fe30534489695bff2fb Author: William Lallemand Date: Wed Nov 15 19:02:57 2017 +0100 BUG/MEDIUM: mworker: does not deinit anymore Does not use the deinit() function during a reload, it's dangerous and might be subject to double free, segfault and hazardous behavior if it's called twice in the case of a execvp fail. commit 2f8b31c2c6e8cb4a15b57796e252226b3154c1bb Author: William Lallemand Date: Wed Nov 15 19:02:56 2017 +0100 BUG/MEDIUM: mworker: wait again for signals when execvp fail After execvp fails, the signals were ignored, preventing to try a reload again. It is now fixed by reaching the top of the mworker_wait() function once the execvp failed. commit 722d4ca0dd66aadf297b43296456289ef3b0f9da Author: William Lallemand Date: Wed Nov 15 19:02:55 2017 +0100 MINOR: mworker: display an accurate error when the reexec fail When the master worker fail the execvp, it returns the wrong error "Cannot allocate memory". We now display the accurate error corresponding to the errno value. commit 6d889500e2bc2a4f97d4f8b5ee63cc798ed53f96 Author: Frédéric Lécaille Date: Wed Nov 15 14:50:19 2017 +0100 CONTRIB: Wireshark dissector for HAProxy Peer Protocol. commit 4b6645d8a7b311c139f7bce84a175862932e0b03 Author: Frédéric Lécaille Date: Wed Nov 15 14:41:00 2017 +0100 DOC: peers: Add a first version of peers protocol v2.1. This documentation has to be completed. commit 318d0c205581aa1bbc493872b84e683f3bc3034b Author: Willy Tarreau Date: Wed Nov 15 19:38:29 2017 +0100 BUG/MAJOR: ebtree/scope: properly tag upper nodes during insertion Christopher found a case where some tasks would remain unseen in the run queue and would spontaneously appear after certain apparently unrelated operations performed by the other thread. It's in fact the insertion which is not correct, the node serving as the top of duplicate tree wasn't properly updated, just like the each top of subtree in a duplicate tree. This had the effect that after some removals, the incorrectly tagged node would hide the underlying ones, which would then suddenly re-appear once they were removed. This is 1.8-specific, no backport is needed. commit 9c1e15d8cd1ca748915cbc75094daae67f17b56c Author: Willy Tarreau Date: Wed Nov 15 18:51:29 2017 +0100 MINOR: tools: emphasize the node being worked on in the tree dump Now we can show in dotted red the node being removed or surrounded in red a node having been inserted, and add a description on the graph related to the operation in progress for example. commit 6c7f4deb21cdb1d359280af4fa0e7f598914760e Author: Willy Tarreau Date: Wed Nov 15 17:49:54 2017 +0100 MINOR: tools: improve the DOT dump of the ebtree Use a smaller and cleaner fixed font, use upper case to indicate sides on branches, remove the useless node/leaf markers on branches since the colors already indicate them, and show the node's key as it helps spot the matching leaf. commit ed3cda02ae8a94371e63d0c1def0318ce4a4d8af Author: Willy Tarreau Date: Wed Nov 15 15:04:05 2017 +0100 MINOR: tools: add a function to dump a scope-aware tree to a file It emits a dump in DOT format for graphing purposes during debugging sessions. It's convenient to dump the run queue. commit 99bca65f539844600ad7e89cd37da0472c6fe3b2 Author: Christopher Faulet Date: Tue Nov 14 16:47:26 2017 +0100 BUG/MEDIUM: standard: itao_str/idx and quote_str/idx must be thread-local This bug has an impact on the stats applet and easily leads to a crash of HAProxy. This is specific to threads, no backport is needed. commit 919b7398627ba9efc053cd8cc48861c772028e9a Author: Christopher Faulet Date: Tue Nov 14 10:17:48 2017 +0100 CLEANUP: tasks: Remove useless double test on rq_next No backport is needed, this is purely 1.8-specific. commit e9a896e09e8d56523f1af57cf454534b11e0fbe7 Author: Christopher Faulet Date: Tue Nov 14 10:16:04 2017 +0100 BUG/MINOR: threads: tid_bit must be a unsigned long This is specific to threads, no backport is needed. commit e1533f5790a3d6810250b278faa1ac7708b4fcae Author: William Lallemand Date: Tue Nov 14 14:39:24 2017 +0100 MINOR: cache: disable cache if shctx_row_data_append fail Disable the cache if the append of data failed, it should never happen because the allocated row size is at least equal to the size of the object to allocate. commit 10935bc547539001f2a35a0d9e0073787320b6f1 Author: William Lallemand Date: Tue Nov 14 14:39:23 2017 +0100 MINOR: cache: forward data with headers Forward the remaining headers with the data in the first call of cache_store_http_forward_data(). Previously the headers were forwarded first, and the function left, implying an additionnal call to cache_store_http_forward_data() for the data. Cc: Christopher Faulet commit 9d5f54daad88ddc67b75a6a34682e5bf9335f1aa Author: William Lallemand Date: Tue Nov 14 14:39:22 2017 +0100 BUG/MEDIUM: cache: use msg->sov to forward header Use msg->sov to forward headers instead of msg->eoh. It can causes some problem because eoh does not contains the last \r\n, and the filter does not support to send the headers partially. Cc: Christopher Faulet commit 0436ab78416e1066fb8f65d33496ac1ee4d685d1 Author: Tim Duesterhus Date: Sun Nov 12 17:39:18 2017 +0100 BUG/MEDIUM: mworker: Fix re-exec when haproxy is started from PATH If haproxy is started using the name of the binary only (i.e. not using a relative or absolute path) the `execv` in `mworker_reload` fails with `ENOENT`, because it does not examine the `PATH`: [WARNING] 315/161139 (7) : Reexecuting Master process [WARNING] 315/161139 (7) : Cannot allocate memory [WARNING] 315/161139 (7) : Failed to reexecute the master processs [7] The error messages are misleading, because the return value of `execv` is not checked. This should be fixed in a separate commit. Once this happened the master process ignores any further signals sent by the administrator. Replace `execv` with `execvp` to establish the expected behaviour. This bug was introduced in commit 73b85e75b3963086be889e1fb40a59e7ef2ad63b. commit cfaa6e7ef3b548039f551bc9b79973cb5c37d76e Author: Willy Tarreau Date: Mon Nov 13 19:17:54 2017 +0100 MINOR: ebtree/scope: simplify the lookup functions by using eb32sc_next_with_parent() This gets rid of the nasty loop we used to have at the end of the lookup function and instead falls back to the normal walk down code. commit f6ac365d8dac0aad8d0b8cf1cec918858ee17169 Author: Willy Tarreau Date: Mon Nov 13 19:13:06 2017 +0100 MINOR: ebtree/scope: add a function to find next node from a parent Several parts of the code need to access the next node but don't start from a node but a tagged parent link. Even eb32sc_next() does this. Let's provide this function to prepare a cleanup for the lookup function. commit 52743305861bf8da7220d402c1bb0d9bba4db06b Author: Willy Tarreau Date: Mon Nov 13 18:55:44 2017 +0100 BUG/MAJOR: ebtree/scope: fix lookup of next node in scope-aware trees The eb32sc_walk_down_left() function needs to be able to go up when it doesn't find a matching entry because this situation may always happen, especially when fixing two constraints (scope + value). It also happens after certain removal situations where some bits remain on some intermediary nodes in the tree. In addition, the algorithm for deciding to take the right branch is wrong as it would take it if the current node shows a scope that doesn't matchthe required one. The current code is flakey in that it returns NULL when the bottom has been reached and it's up to the caller to visit other nodes above. In addition to being complex it's not reliable, and it was noticed a few times that some tasks could remain lying in the tree after heavy insertion/removals under multi-threaded workloads. Now instead we make eb32sc_walk_down_left() visit the leftmost branch that matches the scope, and automatically go up to visit the closest matching right branch. This effectively does the same operations as a next() operation but in reverse order (down then up instead of up then down). The eb32sc_next() function now becomes very simple again and matches the original one, and the initial issues cannot be met anymore. No backport is needed, this is purely 1.8-specific. commit d19ec7d5021ec2a813ec9eed245f20bfade22e97 Author: Willy Tarreau Date: Mon Nov 13 16:16:09 2017 +0100 BUG/MAJOR: ebtree/scope: fix insertion and removal of duplicates in scope-aware trees Commit ca30839 and following ("MINOR: ebtree: implement the scope-aware functions for eb32") improperly dealt with the scope in duplicate trees. The insertion was too lenient in that it would always mark the whole rightmost chain below the insertion point, and the removal could leave marks of non-existing scopes causing next()/first() to visit the wrong branch and return NULL. For insertion, we must only tag the nodes between the head of the dup tree and the insertion point which is the top of the lowest subtree. For removal, the new scope must be be calculated by oring the scopes of the two new branches and is irrelevant to the previous values. No backport is needed, this is purely 1.8-specific. commit fa5c812a6bab78a0c0c53ccfeb9393dd3dcaec80 Author: Christopher Faulet Date: Fri Nov 10 10:39:16 2017 +0100 BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed. commit 9dcf9b6f031ed02a05cbbc76bf06b53d97d25fa3 Author: Christopher Faulet Date: Mon Nov 13 10:34:01 2017 +0100 MINOR: threads: Use __decl_hathreads to declare locks This macro should be used to declare variables or struct members depending on the USE_THREAD compile option. It avoids the encapsulation of such declarations between #ifdef/#endif. It is used to declare all lock variables. commit f4cfcf9a269ae3f106f3d5c2725fcf3abdd54e66 Author: Christopher Faulet Date: Fri Nov 10 14:10:35 2017 +0100 MINOR: debug/flags: Add missing flags commit 600d37edda6a4aebeffe85da97cd7277b5f2c217 Author: Christopher Faulet Date: Fri Nov 10 11:54:58 2017 +0100 BUG/MINOR: spoe: check buffer size before acquiring or releasing it In spoe_acquire_buffer and spoe_release_buffer, instead of checking the buffer against buf_empty, we now check its size. It is important because when an allocation fails, it will be set to buf_wanted. In both cases, the size is 0. It is a proactive bug fix, no real problem was observed till now. It cannot be backported as is in 1.7 because of all changes made on the SPOE in 1.8. commit 3e5e417060f04e2bf96ef98199014ad2c7b8e203 Author: Willy Tarreau Date: Sat Nov 11 17:58:31 2017 +0100 BUILD: thread/pipe: fix build without threads Marcus Rückert reported that commit d8b3b65 ("BUG/MEDIUM: splice/threads: pipe reuse list was not protected.") broke threadless support. Add the required #ifdef. commit 18f133adb3b48ce730ea1c53a934fb1a7123b8c9 Author: William Lallemand Date: Wed Nov 8 11:25:15 2017 +0100 BUG/MEDIUM: cache: does not cache if no Content-Length In the case of Transfer-Encoding: chunked, there is no Content-Length which causes the cache to allocate a too small shctx row for the data. It's not possible to allocate a shctx row for the chunks, we need to be able to allocate on-the-fly the shctx blocks during the data transfer. commit 34650d5a7b6e384ae02d095ff9e614b87203ace0 Author: Willy Tarreau Date: Sat Nov 11 09:06:48 2017 +0100 [RELEASE] Released version 1.8-rc3 Released version 1.8-rc3 with the following main changes : - BUILD: use MAXPATHLEN instead of NAME_MAX. - BUG/MAJOR: threads/checks: add 4 missing spin_unlock() in various functions - BUG/MAJOR: threads/server: missing unlock in CLI fqdn parser - BUG/MINOR: cli: do not perform an invalid action on "set server check-port" - BUG/MAJOR: threads/checks: wrong use of SPIN_LOCK instead of SPIN_UNLOCK - CLEANUP: checks: remove return statements in locked functions - BUG/MINOR: cli: add severity in "set server addr" parser - CLEANUP: server: get rid of return statements in the CLI parser - BUG/MAJOR: cli/streams: missing unlock on exit "show sess" - BUG/MAJOR: threads/dns: add missing unlock on allocation failure path - BUG/MAJOR: threads/lb: fix missing unlock on consistent hash LB - BUG/MAJOR: threads/lb: fix missing unlock on map-based hash LB - BUG/MEDIUM: threads/stick-tables: close a race condition on stktable_trash_expired() - BUG/MAJOR: h2: set the connection's task to NULL when no client timeout is set - BUG/MAJOR: thread/listeners: enable_listener must not call unbind_listener() - BUG/MEDIUM: threads: don't try to free build option message on exit - MINOR: applets: no need to check for runqueue's emptiness in appctx_res_wakeup() - MINOR: add master-worker in the warning about nbproc - MINOR: mworker: allow pidfile in mworker + foreground - MINOR: mworker: write parent pid in the pidfile - MINOR: mworker: do not store child pid anymore in the pidfile - MINOR: ebtree: implement the scope-aware functions for eb32 - MEDIUM: ebtree: specify the scope of every node inserted via eb32sc - MINOR: ebtree: update the eb32sc parent node's scope on delete - MEDIUM: ebtree: only consider the branches matching the scope in lookups - MINOR: ebtree: implement eb32sc_lookup_ge_or_first() - MAJOR: task: make use of the scope-aware ebtree functions - MINOR: task: simplify wake_expired_tasks() to avoid unlocking in the loop - MEDIUM: task: change the construction of the loop in process_runnable_tasks() - MINOR: threads: use faster locks for the spin locks - MINOR: tasks: only visit filled task slots after processing them - MEDIUM: tasks: implement a lockless scheduler for single-thread usage - BUG/MINOR: dns: Don't try to get the server lock if it's already held. - BUG/MINOR: dns: Don't lock the server lock in snr_check_ip_callback(). - DOC: Add note about encrypted password CPU usage - BUG/MINOR: h2: set the "HEADERS_SENT" flag on stream, not connection - BUG/MEDIUM: h2: properly send an RST_STREAM on mux stream error - BUG/MEDIUM: h2: properly send the GOAWAY frame in the mux - BUG/MEDIUM: h2: don't try (and fail) to send non-existing data in the mux - MEDIUM: h2: remove the H2_SS_RESET intermediate state - BUG/MEDIUM: h2: fix some wrong error codes on connections - BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix - BUILD: enable USE_THREAD for Solaris build. - BUG/MEDIUM: h2: don't close the connection is there are data left - MINOR: h2: don't re-enable the connection's task when we're closing - BUG/MEDIUM: h2: properly set H2_SF_ES_SENT when sending the final frame - BUG/MINOR: h2: correctly check for H2_SF_ES_SENT before closing - MINOR: h2: add new stream flag H2_SF_OUTGOING_DATA - BUG/MINOR: h2: don't send GOAWAY on failed response - BUG/MEDIUM: splice/threads: pipe reuse list was not protected. - BUG/MINOR: comp: fix compilation warning compiling without compression. - BUG/MINOR: stream-int: don't set MSG_MORE on closed request path - BUG/MAJOR: threads/tasks: fix the scheduler again - BUG/MINOR; ssl: Don't assume we have a ssl_bind_conf because a SNI is matched. - MINOR: ssl: Handle session resumption with TLS 1.3 - MINOR: ssl: Spell 0x10101000L correctly. - MINOR: ssl: Handle sending early data to server. - BUILD: ssl: fix build of backend without ssl - BUILD: shctx: do not depend on openssl anymore - BUG/MINOR: h1: the HTTP/1 make status code parser check for digits - BUG/MEDIUM: h2: reject non-3-digit status codes - BUG/MEDIUM: stream-int: Don't loss write's notifs when a stream is woken up - BUG/MINOR: pattern: Rely on the sample type to copy it in pattern_exec_match - BUG/MEDIUM: h2: split the function to send RST_STREAM - BUG/MEDIUM: h1: ensure the chunk size parser can deal with full buffers - MINOR: tools: don't use unlikely() in hex2i() - BUG/MEDIUM: h2: support orphaned streams - BUG/MEDIUM: threads/cli: fix "show sess" locking on release - CLEANUP: mux: remove the unused "release()" function - MINOR: cli: make "show fd" report the fd's thread mask - BUG/MEDIUM: stream: don't ignore res.analyse_exp anymore - CLEANUP: global: introduce variable pid_bit to avoid shifts with relative_pid - MEDIUM: http: always reject the "PRI" method commit 916597903cc0e1022db33e01dd832aee76218b99 Author: Willy Tarreau Date: Fri Nov 10 19:38:10 2017 +0100 MEDIUM: http: always reject the "PRI" method This method was reserved for the HTTP/2 connection preface, must never be used and must be rejected. In normal situations it doesn't happen, but it may be visible if a TCP frontend has alpn "h2" enabled, and forwards to an HTTP backend which tries to parse the request. Before this patch it would pass the wrong request to the backend server, now it properly returns 400 bad req. This patch should probably be backported to stable versions. commit 387bd4f69f675d7e35895ce35d04a8270eec45b6 Author: Willy Tarreau Date: Fri Nov 10 19:08:14 2017 +0100 CLEANUP: global: introduce variable pid_bit to avoid shifts with relative_pid At a number of places, bitmasks are used for process affinity and to map listeners to processes. Every time 1UL<<(relative_pid-1) is used. Let's create a "pid_bit" variable corresponding to this value to clean this up. commit 9a398beac321fdda9f6cf0cb7069960d1a29cfd6 Author: Willy Tarreau Date: Fri Nov 10 17:14:23 2017 +0100 BUG/MEDIUM: stream: don't ignore res.analyse_exp anymore It happens that no single analyser has ever needed to set res.analyse_exp, so that process_stream() didn't consider it when computing the next task expiration date. Since Lua actions were introduced in 1.6, this can be needed on http-response actions for example, so let's ensure it's properly handled. Thanks to Nick Dimov for reporting this bug. The fix needs to be backported to 1.7 and 1.6. commit 5d9846f4b3b4b92fca7fd49eb1007a0e9c18dd74 Author: Willy Tarreau Date: Fri Nov 10 16:53:09 2017 +0100 MINOR: cli: make "show fd" report the fd's thread mask This is useful to know what thread(s) an fd is scheduled to be handled on. It's worth noting that at the moment the "show fd"d doesn't seem totally thread-safe. commit 28b55c6fed212d0bf8f162360d1e5578c02d5ce2 Author: Willy Tarreau Date: Fri Nov 10 16:43:05 2017 +0100 CLEANUP: mux: remove the unused "release()" function In commit 53a4766 ("MEDIUM: connection: start to introduce a mux layer between xprt and data") we introduced a release() function which ends up never being used. Let's get rid of it now. commit 7ce3f09513f212a6cf74560d5d228479ccf35c79 Author: Willy Tarreau Date: Fri Nov 10 16:24:41 2017 +0100 BUG/MEDIUM: threads/cli: fix "show sess" locking on release The recent thread updates on the CLI broke "show sess" by unlocking the stream twice instead of lock+unlock. No backport is needed. commit 22cf59bbbac90573cc28bf1035a3dae046083236 Author: Willy Tarreau Date: Fri Nov 10 11:42:33 2017 +0100 BUG/MEDIUM: h2: support orphaned streams When a stream_interface performs a shutw() then a shutr(), the stream is marked closed. Then cs_destroy() calls h2_detach() and it cannot fail since we're on the leaving path of the caller. The problem is that in order to close streams we usually have to send either an emty DATA frame with the ES flag set or an RST_STREAM frame, and the mux buffer might already be full, forcing the stream to be queued. The forced removal of this stream causes this last message to silently disappear, and the client to wait forever for a response. This commit ensures we can detach the conn_stream from the h2 stream if the stream is blocked, effectively making the h2 stream an orphan, ensures that the mux can deal with orphaned streams after processing them, and that the demux can kill them upon receipt of GOAWAY. commit aa39860aeffbe4a6fe621bbdc5bfd1b12e16468b Author: Willy Tarreau Date: Fri Nov 10 11:19:54 2017 +0100 MINOR: tools: don't use unlikely() in hex2i() This small inline function causes some pain to the compiler when used inside other functions due to its use of the unlikely() hint for non-digits. It causes the letters to be processed far away in the calling function and makes the code less efficient. Removing these unlikely() hints has increased the chunk size parsing by around 5%. commit b15e3fefc96b772e5da54631a83a11c32b7e0111 Author: Willy Tarreau Date: Fri Nov 10 11:17:08 2017 +0100 BUG/MEDIUM: h1: ensure the chunk size parser can deal with full buffers The HTTP/1 code always has the reserve left available so the buffer is never full there. But with HTTP/2 we have to deal with full buffers, and it happens that the chunk size parser cannot tell the difference between a full buffer and an empty one since it compares the start and the stop pointer. Let's change this to instead deal with the number of bytes left to process. As a side effect, this code ends up being about 10% faster than the previous one, even on HTTP/1. commit 8c0ea7d21a2b197b2b9a101cb7e747a7cd788ae7 Author: Willy Tarreau Date: Fri Nov 10 10:05:24 2017 +0100 BUG/MEDIUM: h2: split the function to send RST_STREAM There is an issue with how the RST_STREAM frames are sent. Some of them are sent from the demux, either for valid or for closed streams, and some are sent from the mux always for valid streams. At the moment the demux stream ID is used, which is wrong for all streams being muxed, and sometimes results in certain bad HTTP responses causing the emission of an RST_STREAM referencing stream zero. In addition, the stream's blocked flags could be updated even if the stream was the closed or idle ones. We really need to split the function for the two distinct use cases where one is used to send an RST on a condition detected at the connection level (such as a closed stream) and the other one is used to send an RST for a condition detected at the stream level. The first one is used only in the demux, and the other one only by a valid stream. commit 09fdf4b1122670a0ecb86f100aa84353453e7474 Author: Christopher Faulet Date: Thu Nov 9 16:14:16 2017 +0100 BUG/MINOR: pattern: Rely on the sample type to copy it in pattern_exec_match To be thread safe, the function pattern_exec_match copy data (the pattern and the inner sample) in thread-local variables. But when the sample is duplicated, we must check its type and not the pattern one. This is specific to threads, no backport is needed. commit c5a9d5bf237bbc77cf90634a2a7e880478c696d6 Author: Christopher Faulet Date: Thu Nov 9 09:36:43 2017 +0100 BUG/MEDIUM: stream-int: Don't loss write's notifs when a stream is woken up When a write activity is reported on a channel, it is important to keep this information for the stream because it take part on the analyzers' triggering. When some data are written, the flag CF_WRITE_PARTIAL is set. It participates to the task's timeout updates and to the stream's waking. It is also used in CF_MASK_ANALYSER mask to trigger channels anaylzers. In the past, it was cleared by process_stream. Because of a bug (fixed in commit 95fad5ba4 ["BUG/MAJOR: stream-int: don't re-arm recv if send fails"]), It is now cleared before each send and in stream_int_notify. So it is possible to loss this information when process_stream is called, preventing analyzers to be called, and possibly leading to a stalled stream. Today, this happens in HTTP2 when you call the stat page or when you use the cache filter. In fact, this happens when the response is sent by an applet. In HTTP1, everything seems to work as expected. To fix the problem, we need to make the difference between the write activity reported to lower layers and the one reported to the stream. So the flag CF_WRITE_EVENT has been added to notify the stream of the write activity on a channel. It is set when a send succedded and reset by process_stream. It is also used in CF_MASK_ANALYSER. finally, it is checked in stream_int_notify to wake up a stream and in channel_check_timeouts. This bug is probably present in 1.7 but it seems to have no effect. So for now, no needs to backport it. commit a87f202b4973925e95029c2ce28108171c4b93c2 Author: Willy Tarreau Date: Thu Nov 9 11:23:00 2017 +0100 BUG/MEDIUM: h2: reject non-3-digit status codes If the H1 parser would report a status code length not consisting in exactly 3 digits, the error case was confused with a lack of buffer room and was causing the parser to loop infinitely. commit 1b4cf9b7545792aedbf9cf00b8948e6cf28824ed Author: Willy Tarreau Date: Thu Nov 9 11:15:45 2017 +0100 BUG/MINOR: h1: the HTTP/1 make status code parser check for digits The H1 parser used by the H2 gateway was a bit lax and could validate non-numbers in the status code. Since it computes the code on the fly it's problematic, as "30:" is read as status code 310. Let's properly check that it's a number now. No backport needed. commit ddfbd83780de0f140fc7195d1a16953ec18ef37a Author: Willy Tarreau Date: Wed Nov 8 14:33:36 2017 +0100 BUILD: shctx: do not depend on openssl anymore The build breaks on a machine without openssl/crypto.h because shctx still loads openssl-compat.h while it doesn't need it anymore since the code was moved : In file included from src/shctx.c:20:0: include/proto/openssl-compat.h:3:28: fatal error: openssl/crypto.h: No such file or directory #include Just remove include openssl-compat from shctx. commit 46c9d3e6cb47822b6372c6bb538a47ced8ad0141 Author: Willy Tarreau Date: Wed Nov 8 14:25:59 2017 +0100 BUILD: ssl: fix build of backend without ssl Commit 522eea7 ("MINOR: ssl: Handle sending early data to server.") added a dependency on SRV_SSL_O_EARLY_DATA which only exists when USE_OPENSSL is defined (which is probably not the best solution) and breaks the build when ssl is not enabled. Just add an ifdef USE_OPENSSL around the block for now. commit 522eea71106ad9486a3c01a0e5fd8b6d88a42693 Author: Olivier Houchard Date: Fri Nov 3 16:27:47 2017 +0100 MINOR: ssl: Handle sending early data to server. This adds a new keyword on the "server" line, "allow-0rtt", if set, we'll try to send early data to the server, as long as the client sent early data, as in case the server rejects the early data, we no longer have them, and can't resend them, so the only option we have is to send back a 425, and we need to be sure the client knows how to interpret it correctly. commit cfdef2e3123306a2d671ebc5fd05596295aa2b88 Author: Olivier Houchard Date: Fri Nov 3 13:50:53 2017 +0100 MINOR: ssl: Spell 0x10101000L correctly. Issue added in 1.8-dev by c2aae74 ("MEDIUM: ssl: Handle early data with OpenSSL 1.1.1"), no impact on older versions. commit bd84ac873784f61a94e80678c6e677ef4c4dc894 Author: Olivier Houchard Date: Fri Nov 3 13:43:35 2017 +0100 MINOR: ssl: Handle session resumption with TLS 1.3 With TLS 1.3, session aren't established until after the main handshake has completed. So we can't just rely on calling SSL_get1_session(). Instead, we now register a callback for the "new session" event. This should work for previous versions of TLS as well. commit 35a63cc1c78411c65d83b995d1fbd3d73d6bf020 Author: Olivier Houchard Date: Thu Nov 2 19:04:38 2017 +0100 BUG/MINOR; ssl: Don't assume we have a ssl_bind_conf because a SNI is matched. We only have a ssl_bind_conf if crt-list is used, however we can still match a certificate SNI, so don't assume we have a ssl_bind_conf. commit 9e45b33f7ee78953d3ec6ad32d4d9eed3bfc897a Author: Willy Tarreau Date: Wed Nov 8 14:05:19 2017 +0100 BUG/MAJOR: threads/tasks: fix the scheduler again My recent change in commit ce4e0aa ("MEDIUM: task: change the construction of the loop in process_runnable_tasks()") was bogus as it used to keep the rq_next across an unlock/lock sequence, occasionally leading to crashes for tasks that are eligible to any thread. We must use the lookup call for each new batch instead. The problem is easily triggered with such a configuration : global nbthread 4 listen check mode http bind 0.0.0.0:8080 redirect location / option httpchk GET / server s1 127.0.0.1:8080 check inter 1 server s2 127.0.0.1:8080 check inter 1 Thanks to Olivier for diagnosing this one. No backport is needed. commit ecd2e15919f31df2c0e42b3a1ac74f1344d9a2ae Author: Willy Tarreau Date: Tue Nov 7 15:07:25 2017 +0100 BUG/MINOR: stream-int: don't set MSG_MORE on closed request path Commit 4ac4928 ("BUG/MINOR: stream-int: don't set MSG_MORE on SHUTW_NOW without AUTO_CLOSE") was incomplete. H2 reveals another situation where the input stream is marked closed with the request and we set MSG_MORE, causing a delay before the request leaves. Better avoid setting the flag on the request path for close cases in general. commit 11f5886e5c177de9f77ff11c5848b19a4a8c6e37 Author: Emeric Brun Date: Tue Nov 7 11:57:54 2017 +0100 BUG/MINOR: comp: fix compilation warning compiling without compression. This is specific to threads, no backport is needed. commit d8b3b65faa62d8b9b4344014e7183e422769caea Author: Emeric Brun Date: Tue Nov 7 11:19:48 2017 +0100 BUG/MEDIUM: splice/threads: pipe reuse list was not protected. The list is now protected using a global spinlock. commit 926fa4c098fff29d54c9bf0a5a2c9d9bb245c143 Author: Willy Tarreau Date: Tue Nov 7 14:42:12 2017 +0100 BUG/MINOR: h2: don't send GOAWAY on failed response As part of the detection for intentional closes, we can kill the connection if a shutw() happens before the headers. But it can also happen that an invalid response is not properly parsed, preventing any headers frame from being sent and making the function believe it was an abort. Now instead we check if any response was received from the stream, regardless of the fact that it was properly converted. commit c4312d3dfd5129d8befa69208ff7132702fd5da7 Author: Willy Tarreau Date: Tue Nov 7 12:01:53 2017 +0100 MINOR: h2: add new stream flag H2_SF_OUTGOING_DATA This one indicates whether we've received data to mux out. It helps make the difference between a clean close and a an erroneous one. commit 58e3208714d8a1ee2fc186f99f9086f682e29447 Author: Willy Tarreau Date: Tue Nov 7 14:41:09 2017 +0100 BUG/MINOR: h2: correctly check for H2_SF_ES_SENT before closing In h2_shutw() we must not send another empty frame (nor RST) after one has been sent, as the stream is already in HLOC/CLOSED state. commit 6d8b682f9a60a26b4e73c63e8145757f98565616 Author: Willy Tarreau Date: Tue Nov 7 14:39:09 2017 +0100 BUG/MEDIUM: h2: properly set H2_SF_ES_SENT when sending the final frame When sending DATA+ES, it's important to set H2_SF_ES_SENT as we don't want to emit is several times nor to send an RST afterwards. commit e6ae77f64f6e5fb8fe5f9cc3ccd6f84545432699 Author: Willy Tarreau Date: Tue Nov 7 11:59:51 2017 +0100 MINOR: h2: don't re-enable the connection's task when we're closing It's pointless to requeue the task when we're closing, so swap the order of the task_queue() and h2_release(). It also matches what was written in the comment regarding re-arming the timer. commit 83906c2f912ea46373fda5795f398c5c657f01e8 Author: Willy Tarreau Date: Tue Nov 7 11:48:46 2017 +0100 BUG/MEDIUM: h2: don't close the connection is there are data left h2_detach() is called after a stream was closed, and it evaluates if it's worth closing the connection. The issue there is that the connection is closed too early in case there's demand for closing after the last stream, even if some data remain in the mux. Let's change the condition to check for this. commit f5d79ac5a84712aac1c4af53159e66ae45d32edf Author: Christopher Faulet Date: Tue Nov 7 10:47:44 2017 +0100 BUILD: enable USE_THREAD for Solaris build. commit 2a944ee16b4b6fa1d55b7eca8bb4d289baa75ba1 Author: Christopher Faulet Date: Tue Nov 7 10:42:54 2017 +0100 BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris. commit 7d8e4af46acbfad03cd813d7349ffbb9262d5539 Author: Willy Tarreau Date: Tue Nov 7 11:08:28 2017 +0100 BUG/MEDIUM: h2: fix some wrong error codes on connections When the assignment of the connection state was moved into h2c_error(), 3 of them were missed because they were wrong, using H2_SS_ERROR instead. This resulted in the connection's state being set to H2_CS_ERROR2 in fact, so the error was not properly sent. commit 721c974e5e0893d62902e3012463d35a332c3fcb Author: Willy Tarreau Date: Tue Nov 7 11:05:42 2017 +0100 MEDIUM: h2: remove the H2_SS_RESET intermediate state This one was created to maintain the knowledge that a stream was closed after having sent an RST_STREAM frame but that's not needed anymore and it confuses certain conditions on the error processing path. It's time to get rid of it. commit 319994a2e9e47985a8c1e9f589c579fc685b89cc Author: Willy Tarreau Date: Tue Nov 7 11:03:56 2017 +0100 BUG/MEDIUM: h2: don't try (and fail) to send non-existing data in the mux The call to xprt->snd_buf() was not conditionned on the presence of data in the buffer, resulting in snd_buf() returning 0 and never disabling the polling. It was revealed by the previous bug on error processing but must properly be handled. commit 3eabe9b174a245a8591f0fafc7f565914ae86f9a Author: Willy Tarreau Date: Tue Nov 7 11:03:01 2017 +0100 BUG/MEDIUM: h2: properly send the GOAWAY frame in the mux A typo on a condition prevented H2_CS_ERROR from being processed, leading to an infinite loop on connection error. commit c6795ca7c1845746838c27f4cc201aa560733af1 Author: Willy Tarreau Date: Tue Nov 7 09:43:06 2017 +0100 BUG/MEDIUM: h2: properly send an RST_STREAM on mux stream error Some stream errors are detected on the MUX path (eg: H1 response encoding). The ones forgot to emit an RST_STREAM frame, causing the client to wait and/or to see the connection being immediately closed. This is now fixed. commit 6743420778e65815184e36817e5e24fe0c5b4749 Author: Willy Tarreau Date: Mon Nov 6 20:20:51 2017 +0100 BUG/MINOR: h2: set the "HEADERS_SENT" flag on stream, not connection This flag was added after the GOAWAY flags were introduced and mistakenly placed in the connection, but that doesn't make sense as it's specific to the stream. The main impact is the risk of returning a DATA0+ES frame for an error instead of an RST_STREAM. commit d06f31c443bbaa231ef75221540dec605d20018b Author: Daniel Schneller Date: Mon Nov 6 16:51:04 2017 +0100 DOC: Add note about encrypted password CPU usage From first-hand experience I realized that using encrypted passwords in userlists can quickly become overwhelming for busy sites. In my case just about 100 rq/s were enough to drive (user) CPU usage from 2-3% up to >90%. While it is perfectly explicable why this is the case, having it mentioned in the relevant documentation section might spare someone some confusion in the future. commit 283810773afa55dd50e6b362e2b93f8f7a88edc4 Author: Olivier Houchard Date: Mon Nov 6 17:30:28 2017 +0100 BUG/MINOR: dns: Don't lock the server lock in snr_check_ip_callback(). snr_check_ip_callback() may be called with the server lock, so don't attempt to lock it again, instead, make sure the callers always have the lock before calling it. commit 55dcdf4c398b805770914ecfbe095c3b1b309ca2 Author: Olivier Houchard Date: Mon Nov 6 15:15:04 2017 +0100 BUG/MINOR: dns: Don't try to get the server lock if it's already held. dns_link_resolution() can be called with the server lock already held, so don't attempt to lock it again in that case. commit f0c531ab5516d179a83a44b1a34b26ffde5e2f51 Author: Willy Tarreau Date: Sun Nov 5 16:35:59 2017 +0100 MEDIUM: tasks: implement a lockless scheduler for single-thread usage The scheduler is complex and uses local queues to amortize the cost of locks. But all this comes with a cost that is quite observable with single-thread workloads. The purpose of this patch is to reimplement the much simpler scheduler for the case where threads are not used. The code is very small and simple. It doesn't impact the multi-threaded performance at all, and provides a nice 10% performance increase in single-thread by reaching 606kreq/s on the tests that showed 550kreq/s before. commit 9d4b56b88e01ac874a158f9bd4dd8c804ba61875 Author: Willy Tarreau Date: Mon Nov 6 08:36:53 2017 +0100 MINOR: tasks: only visit filled task slots after processing them process_runnable_tasks() needs to requeue or wake up tasks after processing them in batches. By only refilling the existing ones, we avoid revisiting all the queue. The performance gain is measurable starting with two threads, where the request rate climbs to 657k/s compared to 644k. commit 88ac59be4d7540bb7b6087397eb2ed9aa5f57066 Author: Willy Tarreau Date: Mon Nov 6 01:03:26 2017 +0100 MINOR: threads: use faster locks for the spin locks The spin locks used to rely on W locks, which involve a loop waiting for readers to leave, and this doesn't happen here. It's more efficient to use S locks instead, which are also mutually exclusive and do not have this loop. This saves one test per spinlock and a few tens of bytes allowing certain functions to be inlined. commit ce4e0aa7f387f589e293419269c5641c3966fbbc Author: Willy Tarreau Date: Sun Nov 5 23:57:00 2017 +0100 MEDIUM: task: change the construction of the loop in process_runnable_tasks() This patch slightly rearranges the loop to pack the locked code a little bit, and to try to concentrate accesses to the tree together to benefit more from the cache. It also fixes how the loop handles the right margin : now that is guaranteed that the retrieved nodes are filtered to only match the current thread, we don't need to rewind every 16 entries. Instead we can rewind each time we reach the right margin again. With this change, we now achieve the following performance for 10 H2 conns each containing 100 streams : 1 thread : 550kreq/s 2 thread : 644kreq/s 3 thread : 598kreq/s commit b992ba16ef4b7cb3e293a16229e21df599f412b0 Author: Willy Tarreau Date: Sun Nov 5 19:09:27 2017 +0100 MINOR: task: simplify wake_expired_tasks() to avoid unlocking in the loop This function is sensitive, let's make it shorter by factoring out the unlock and leave code. This reduced the function's size by a few tens of bytes and increased the overall performance by about 1%. commit 8d38805d3d9ee732d549918c29ea3446459fae63 Author: Willy Tarreau Date: Sun Nov 5 13:34:20 2017 +0100 MAJOR: task: make use of the scope-aware ebtree functions Currently the task scheduler suffers from an O(n) lookup when skipping tasks that are not for the current thread. The reason is that eb32_lookup_ge() has no information about the current thread so it always revisits many tasks for other threads before finding its own tasks. This is particularly visible with HTTP/2 since the number of concurrent streams created at once causes long series of tasks for the same stream in the scheduler. With only 10 connections and 100 streams each, by running on two threads, the performance drops from 640kreq/s to 11.2kreq/s! Lookup metrics show that for only 200000 task lookups, 430 million skips had to be performed, which means that on average, each lookup leads to 2150 nodes to be visited. This commit backports the principle of scope lookups for ebtrees from the ebtree_v7 development tree. The idea is that each node contains a mask indicating the union of the scopes for the nodes below it, which is fed during insertion, and used during lookups. Then during lookups, branches that do not contain any leaf matching the requested scope are simply ignored. This perfectly matches a thread mask, allowing a thread to only extract the tasks it cares about from the run queue, and to always find them in O(log(n)) instead of O(n). Thus the scheduler uses tid_bit and task->thread_mask as the ebtree scope here. Doing this has recovered most of the performance, as can be seen on the test below with two threads, 10 connections, 100 streams each, and 1 million requests total : Before After Gain test duration : 89.6s 4.73s x19 HTTP requests/s (DEBUG) : 11200 211300 x19 HTTP requests/s (PROD) : 15900 447000 x28 spin_lock time : 85.2s 0.46s /185 time per lookup : 13us 40ns /325 Even when going to 6 threads (on 3 hyperthreaded CPU cores), the performance stays around 284000 req/s, showing that the contention is much lower. A test showed that there's no benefit in using this for the wait queue though. commit 8878b6c4cb8bfcc84a1a7d007065c65bc9cc99a5 Author: Willy Tarreau Date: Sun Nov 5 21:23:21 2017 +0100 MINOR: ebtree: implement eb32sc_lookup_ge_or_first() In the scheduler we always have to loop back to the beginning after we don't find the last entry, so let's implement this in a new lookup function instead. The resulting code is slightly faster, mostly due to the fact that there's much less inlined code in the fast path. commit d1d55ac6196cd64f7408c0a9467139e45b59b732 Author: Willy Tarreau Date: Sun Nov 5 14:33:01 2017 +0100 MEDIUM: ebtree: only consider the branches matching the scope in lookups Now when looking up a node via eb32sc_first(), eb32sc_next(), and eb32sc_lookup_ge(), we only focus on the branches matching the requested scope. The code must be careful to miss no branch. It changes a little bit from the previous one because the scope stored on the intermediary nodes is not exact (since we don't propagate upwards during deletion), so in case a lookup fails, we have to walk up and pick the next matching entry. commit ef8d0dcefdf00b043697df971533daf01290811a Author: Willy Tarreau Date: Sun Nov 5 18:06:22 2017 +0100 MINOR: ebtree: update the eb32sc parent node's scope on delete During a delete operation, if the deleted node is above its leaf's parent, this parent will replace the node and then go up. In this case it is important to update the new parent's scope to reflect the presence of other branches. It's worth noting that in theory we should precisely recompute the exact node value, but it seems that it's not worth it for the rare cases there is a mismatch. commit 5d19fd479a825913a2168f0c30ff965ccca35630 Author: Willy Tarreau Date: Sun Nov 5 14:06:50 2017 +0100 MEDIUM: ebtree: specify the scope of every node inserted via eb32sc Here we mark each visited node with the scope bits of the node being inserted. This will allow the lookup to skip certain non-interesting nodes. commit ca30839a857049acc96e26a28e7515533d9ff153 Author: Willy Tarreau Date: Sun Nov 5 13:31:29 2017 +0100 MINOR: ebtree: implement the scope-aware functions for eb32 A new kind of tree nodes is currently being developed in ebtree v7, consisting in storing a scope in each node indicating a visibility mask so that certain nodes are not reported on certain lookups. The initial goal was to make this usable with a multi-thread scheduler. Since the ebtree v7 code is completely different from v6, this patch instead copies the minimally required functions from eb32 and ebtree and calls them "eb32sc_*". At the moment the scope is not implemented, it's only passed in arguments. commit 92159b29014281337ebf44d6267cba24f0f72162 Author: William Lallemand Date: Mon Nov 6 11:16:12 2017 +0100 MINOR: mworker: do not store child pid anymore in the pidfile The parent process supervises itself the children, we don't need to store the children pids anymore in the pidfile in master-worker mode. commit deed780a22d61e3c34f426f492f5735ac45109c1 Author: William Lallemand Date: Mon Nov 6 11:00:04 2017 +0100 MINOR: mworker: write parent pid in the pidfile The first pid in the pidfile is now the parent, it's more convenient for supervising the processus. You can now reload haproxy in master-worker mode with convenient command like: kill -USR2 $(head -1 /tmp/haproxy.pid) commit 8029300df64948cf2dbf87e4d71fb6c8d138d49d Author: William Lallemand Date: Mon Nov 6 11:00:03 2017 +0100 MINOR: mworker: allow pidfile in mworker + foreground This patch allows the use of the pidfile in master-worker mode without using the background option. commit cc113822a74ff0474d123b129dc2041d4f386060 Author: William Lallemand Date: Mon Nov 6 11:00:02 2017 +0100 MINOR: add master-worker in the warning about nbproc commit 62a124977bd900abb6cd7e07f0cccbd982d3225c Author: Willy Tarreau Date: Sun Nov 5 12:01:11 2017 +0100 MINOR: applets: no need to check for runqueue's emptiness in appctx_res_wakeup() The __appctx_wakeup() function already does it. It matters with threads enabled because it simplifies the code in appctx_res_wakeup() to get rid of this test. commit 6dbd3e963bc4f7c3fae545b623989afe5d68c198 Author: Willy Tarreau Date: Sun Nov 5 11:50:18 2017 +0100 BUG/MEDIUM: threads: don't try to free build option message on exit Commit 0493149 ("MINOR: thread: report multi-thread support in haproxy -vv") added information about thread support in haproxy -vv output but accidently marked the message as "must_free" while it's a constant. This causes a segv on the old process on clean exit if threads are enabled. It doesn't affect the stability during operations however. commit bbd09b930678af71ca6a09d75a9941282d2ed613 Author: Willy Tarreau Date: Sun Nov 5 11:38:44 2017 +0100 BUG/MAJOR: thread/listeners: enable_listener must not call unbind_listener() unbind_listener() takes the listener lock, which is already held by enable_listener(). This situation happens when starting with nbproc > 1 with some bind lines limited to a certain process, because in this case enable_listener() tries to stop unneeded listeners. This commit introduces __do_unbind_listeners() which must be called with the lock held, and makes enable_listener() use this one. Given that the only return code has never been used and that it starts to make the code more complicated to propagate it before throwing it to the trash, the function's return type was changed to void. commit 3340029b978126743f702ad3d967b42e026516af Author: Willy Tarreau Date: Sun Nov 5 11:23:40 2017 +0100 BUG/MAJOR: h2: set the connection's task to NULL when no client timeout is set If "timeout client" is missing from the frontend, the task is not initialized, causing a crash on connection teardown. commit 4d5f13cab3411cae504d579c4011a6707f133580 Author: Willy Tarreau Date: Sun Nov 5 11:04:47 2017 +0100 BUG/MEDIUM: threads/stick-tables: close a race condition on stktable_trash_expired() The spin_unlock() was called just before setting the expiry to TICK_ETERNITY, so if another thread has the time to perform its update and set a timeout, this would would clear it. commit 03071f69377f94fbe69f6ca00f10f4313432edf1 Author: Willy Tarreau Date: Sun Nov 5 10:59:12 2017 +0100 BUG/MAJOR: threads/lb: fix missing unlock on map-based hash LB We often left the function with the lock held on success. commit 1ed90ac3777d561771bc333b7f40549f83e86240 Author: Willy Tarreau Date: Sun Nov 5 10:54:50 2017 +0100 BUG/MAJOR: threads/lb: fix missing unlock on consistent hash LB If no matching node was found, the function was left without unlocking the tree. commit 5ec84574c761c883c0a7d768c417c3278c2c97d9 Author: Willy Tarreau Date: Sun Nov 5 10:35:57 2017 +0100 BUG/MAJOR: threads/dns: add missing unlock on allocation failure path An unlock was missing when a memory allocation failure is detected. commit 70124ce3e12403632511944c17c7fb8e8455fde2 Author: Willy Tarreau Date: Sun Nov 5 10:31:10 2017 +0100 BUG/MAJOR: cli/streams: missing unlock on exit "show sess" An unlock was missing on the situation where the session disappeared while watching it. commit 6ce38f3eab02634afcfb7be9ad836e1b6347ed7d Author: Willy Tarreau Date: Sun Nov 5 10:19:23 2017 +0100 CLEANUP: server: get rid of return statements in the CLI parser There were two many return, some of them missing a spin_unlock call, let's use a goto to a central place instead. commit a075258a2c21febd0b0940acea062ee67b387a50 Author: Willy Tarreau Date: Sun Nov 5 10:17:49 2017 +0100 BUG/MINOR: cli: add severity in "set server addr" parser Commit c3680ec ("MINOR: add severity information to cli feedback messages") introduced a severity level to CLI messages, but one of them was missed on "set server addr". No backport is needed. commit 62ac84f84300231f68a0ea08ce30e7ce7dc6236d Author: Willy Tarreau Date: Sun Nov 5 10:11:13 2017 +0100 CLEANUP: checks: remove return statements in locked functions Given that all spinning loops we've had since 1.8-rc1 were caused by unbalanced lock/unlock, let's get rid of all return statements in the locked check functions and only exit via a a single unlock place. commit 73247e0757eaf98cf1ae847f2675087b14efa088 Author: Willy Tarreau Date: Sun Nov 5 10:06:32 2017 +0100 BUG/MAJOR: threads/checks: wrong use of SPIN_LOCK instead of SPIN_UNLOCK Must unlock on exit, copy-paste error. commit 1c8980f9b54ceb2a35b0ecb9be71df6795773542 Author: Willy Tarreau Date: Sun Nov 5 09:58:50 2017 +0100 BUG/MINOR: cli: do not perform an invalid action on "set server check-port" The "set server check-port" CLI handler forgot to return after detecting an error on the port number, and still proceeds with the action. This needs to be backported to 1.7. commit 2a858a82ec874cca679a0938b103c6cf6329c17a Author: Willy Tarreau Date: Sun Nov 5 09:58:01 2017 +0100 BUG/MAJOR: threads/server: missing unlock in CLI fqdn parser This one didn't properly unlock before returning an error message. commit 1cd153aa895cd61808bffb0179f001e9ed0d5041 Author: Willy Tarreau Date: Sun Nov 5 09:56:14 2017 +0100 BUG/MAJOR: threads/checks: add 4 missing spin_unlock() in various functions Some unlocks were missing, resulting in deadlocks even with a single thread. We really need to make these functions safer by getting rid of all those remaining "return" calls and only leave using a goto! commit f143b8040bca7df02a2f501c6a68f2ab609d4269 Author: Olivier Houchard Date: Sat Nov 4 15:13:01 2017 +0100 BUILD: use MAXPATHLEN instead of NAME_MAX. This fixes building on at least Solaris, where NAME_MAX doesn't exist. commit a8d8d6e8f6a018ba7caed79c4847f4096ea17a16 Author: Willy Tarreau Date: Fri Nov 3 23:52:47 2017 +0100 [RELEASE] Released version 1.8-rc2 Released version 1.8-rc2 with the following main changes : - BUG/MINOR: send-proxy-v2: fix dest_len in make_tlv call - BUG/MINOR: send-proxy-v2: string size must include ('\0') - MINOR: mux: Only define pipe functions on linux. - MINOR: cache: Remove useless test for nonzero. - MINOR: cache: Don't confuse act_return and act_parse_ret. - BUG/MEDIUM: h2: don't try to parse incomplete H1 responses - BUG/MEDIUM: checks/mux: always enable send-polling after connecting - BUG/MAJOR: fix deadlock on healthchecks. - BUG/MINOR: thread: fix a typo in the debug code - BUILD: shctx: allow to be built without openssl - BUG/MEDIUM: cache: don't try to resolve wrong filters - BUG/MAJOR: buffers: fix get_buffer_nc() for data at end of buffer - BUG/MINOR: freq: fix infinite loop on freq_ctr_period. - BUG/MINOR: stdarg.h inclusion - BUG/MINOR: dns: fix missing lock protection on server. - BUG/MINOR: lua: fix missing lock protection on server. - BUILD: enable USE_THREAD for OpenBSD build. - BUG/MAJOR: mux_pt: don't dereference a connstream after ->wake() - MINOR: thread: report multi-thread support in haproxy -vv commit 0493149ac31d39f91a95d0e3516531d095948372 Author: Willy Tarreau Date: Fri Nov 3 23:39:25 2017 +0100 MINOR: thread: report multi-thread support in haproxy -vv Otherwise it's hard to know if it was enabled or not. commit ed339a375c1e4c1a30121dfb5469183be07e060e Author: Willy Tarreau Date: Fri Nov 3 15:55:24 2017 +0100 BUG/MAJOR: mux_pt: don't dereference a connstream after ->wake() The wake() callback may destroy a connstream, so it must not be dereferenced in case wake() returns negative. No backport needed, this is 1.8-only. commit 7567c4002fbaf7691bbba7153e9b5fc3e94d10cb Author: David Carlier Date: Fri Nov 3 14:35:44 2017 +0000 BUILD: enable USE_THREAD for OpenBSD build. commit 8c4954c5c267bffab7978cdbde9446f51f5775d8 Author: Emeric Brun Date: Thu Nov 2 17:21:40 2017 +0100 BUG/MINOR: lua: fix missing lock protection on server. To avoid inconsistencies server's attributes must be read or updated under lock. commit e9fd6b5916d30c020a8a12a9bfe105f786ef0b3e Author: Emeric Brun Date: Thu Nov 2 17:20:39 2017 +0100 BUG/MINOR: dns: fix missing lock protection on server. To avoid inconsistencies server's attributes must be read or updated under lock. commit 5222d8eb25c55ecd2fbb9d14f5e088780450d583 Author: David Carlier Date: Fri Nov 3 12:00:26 2017 +0000 BUG/MINOR: stdarg.h inclusion Needed for the memvprintf part, the va_list type. Spotted during OpenBSD build. commit f2fc1fda804962224664e3a23c8a9ed88e4076b6 Author: Emeric Brun Date: Thu Nov 2 17:32:43 2017 +0100 BUG/MINOR: freq: fix infinite loop on freq_ctr_period. Using peers or stick table we could update an freq_ctr using a tick value with the first bit set but this bit is reserved for lock since multithreading support. commit 4b75fffa2bb0c60f26affe3e784956a0b8087442 Author: Willy Tarreau Date: Thu Nov 2 17:16:07 2017 +0100 BUG/MAJOR: buffers: fix get_buffer_nc() for data at end of buffer This function incorrectly dealt with the case where data doesn't wrap but lies at the end of the buffer, resulting in Lukas' reported data corruption with HTTP/2. No backport is needed, it was introduced for HTTP/2 in 1.8-dev. commit 9c54c53f2fadea9aebe13ba61b0a45de0f63ec8b Author: William Lallemand Date: Thu Nov 2 16:38:42 2017 +0100 BUG/MEDIUM: cache: don't try to resolve wrong filters Don't try to resolve wrong filters which are not cache filters during the post configuration callback. commit b620e987d0ba5ad4c79f7c6acb0ee73f8c027a4b Author: William Lallemand Date: Thu Nov 2 16:00:51 2017 +0100 BUILD: shctx: allow to be built without openssl The shctx functions does not depend of openssl anymore, allows to build them without openssl. commit 7c2a2ad65c869a468802fb712de7edf993be9aba Author: Willy Tarreau Date: Thu Nov 2 16:26:02 2017 +0100 BUG/MINOR: thread: fix a typo in the debug code __spin_unlock() used to call RWLOCK_WRUNLOCK() to unlock in the debug code. It's harmless as they happen to be identical. commit f6ba17da201f3ea9fd9c5376ff7765e20a95fa72 Author: Emeric Brun Date: Thu Nov 2 14:35:27 2017 +0100 BUG/MAJOR: fix deadlock on healthchecks. Fix bugs due to missing unlock and recursive lock performing http health check. The server's lock scope was enlarged to protect all callers of 'set_server_check_status' and 'chk_report_conn_err'. This fix also protects tcpcheck against concurrency. commit 16257f648ffb49f639bb17956445acf934b992a7 Author: Willy Tarreau Date: Thu Nov 2 15:45:00 2017 +0100 BUG/MEDIUM: checks/mux: always enable send-polling after connecting Before introducing the mux layer, tcp_connect() would poll for sending to detect the connection establishment. It happens that the health checks have apparently never explicitly enabled this polling and have been relying on this implicit one. Now that there's the mux layer, the conn_stream needs to be enabled for polling as well and since it's not done in the checks, it's never done and the check's request doesn't leave the machine, as can be noticed with http checks. The solution simply consists in going back to the well-known case where we enable polling after connecting using cs_want_send() if we have anything but just a plain connect(). The regular data path is not affected because the stream interface code automatically computes the polling needs based on buffer contents. commit f13ef96e70fc6365fa7bbc078df2393a70bfb2d2 Author: Willy Tarreau Date: Thu Nov 2 15:14:19 2017 +0100 BUG/MEDIUM: h2: don't try to parse incomplete H1 responses This situation which must not happen does in fact happen when feeding artificial responses using errorfiles, Lua or an applet. For now it causes the H1 response parser to loop forever trying to get a more complete response. Since it cannot progress, let's return an error. commit fccf840cdf807197beaa93a16e80d9e70b5c0d9c Author: Olivier Houchard Date: Wed Nov 1 14:04:02 2017 +0100 MINOR: cache: Don't confuse act_return and act_parse_ret. commit cd2867a012483549bd11928c647c79d22a18c1b6 Author: Olivier Houchard Date: Wed Nov 1 13:58:21 2017 +0100 MINOR: cache: Remove useless test for nonzero. Don't bother testing if len is nonzero, we know it is, as we're in the "else" part of a if (!len), and testing it confuses clang into thinking ret may be left uninitialized. commit 7da120bb0e8b24e2d9878e2a719d4863ff5af596 Author: Olivier Houchard Date: Wed Nov 1 13:55:10 2017 +0100 MINOR: mux: Only define pipe functions on linux. Only define mux_pt_snd_pipe() and mux_pt_rcv_pipe() if splicing is available. commit 82913e4f79a1f1fb25aec84a2ce2f5f0e5ce1959 Author: Emmanuel Hocdet Date: Tue Oct 31 18:31:36 2017 +0100 BUG/MINOR: send-proxy-v2: string size must include ('\0') strlen() exclude the terminating null byte ('\0'), add it. commit 571c7ac0a56bc100ab2cbb8d790e281d1b5de9e8 Author: Emmanuel Hocdet Date: Tue Oct 31 18:24:05 2017 +0100 BUG/MINOR: send-proxy-v2: fix dest_len in make_tlv call Subtract already allocated size from buf_len. commit 901f75c4a67233e43b8992c5244f20dcf4946af6 Author: Willy Tarreau Date: Tue Oct 31 23:18:29 2017 +0100 [RELEASE] Released version 1.8-rc1 Released version 1.8-rc1 with the following main changes : - BUG/MEDIUM: server: Allocate tmptrash before using it. - CONTRIB: trace: add the possibility to place trace calls in the code - CONTRIB: trace: try to display the function's return value on exit - CONTRIB: trace: report the base name only for file names - BUILD: ssl: support OPENSSL_NO_ASYNC #define - MINOR: ssl: build with recent BoringSSL library - BUG/MINOR: ssl: OCSP_single_get0_status can return -1 - BUG/MINOR: cli: restore "set ssl tls-key" command - CLEANUP: cli: remove undocumented "set ssl tls-keys" command - IMPORT: sha1: import SHA1 functions - MINOR: sample: add the sha1 converter - MINOR: sample: add the hex2i converter - MINOR: stream-int: stop checking for useless connection flags in chk_snd_conn - MINOR: ssl: don't abort after sending 16kB - MINOR: connection: move the cleanup of flag CO_FL_WAIT_ROOM - MINOR: connection: add flag CO_FL_WILL_UPDATE to indicate when updates are granted - MEDIUM: connection: make use of CO_FL_WILL_UPDATE in conn_sock_shutw() - MINOR: raw_sock: make use of CO_FL_WILL_UPDATE - MINOR: ssl_sock: make use of CO_FL_WILL_UPDATE - BUG/MINOR: checks: Don't forget to release the connection on error case. - MINOR: buffer: add the buffer input manipulation functions - BUG/MEDIUM: prevent buffers being overwritten during build_logline() execution - MEDIUM: cfgparse: post section callback - MEDIUM: cfgparse: post parsing registration - MINOR: lua: add uuid to the Class Proxy - MINOR: hlua: Add regex class - MINOR: http: Mark the 425 code as "Too Early". - MEDIUM: ssl: convert CBS (BoringSSL api) usage to neutral code - MINOR: ssl: support Openssl 1.1.1 early callback for switchctx - MINOR: ssl: generated certificate is missing in switchctx early callback - MEDIUM: ssl: Handle early data with OpenSSL 1.1.1 - BUILD: Makefile: disable -Wunused-label - MINOR: ssl/proto_http: Add keywords to take care of early data. - BUG/MINOR: lua: const attribute of a string is overridden - MINOR: ssl: Don't abuse ssl_options. - MINOR: update proxy-protocol-v2 #define - MINOR: merge ssl_sock_get calls for log and ppv2 - MINOR: add ALPN information to send-proxy-v2 - MEDIUM: h1: ensure that 1xx, 204 and 304 don't have a payload body - CLEANUP: shctx: get ride of the shsess_packet{_hdr} structures - MEDIUM: lists: list_for_each_entry{_safe}_from functions - REORG: shctx: move lock functions and struct - MEDIUM: shctx: allow the use of multiple shctx - REORG: shctx: move ssl functions to ssl_sock.c - MEDIUM: shctx: separate ssl and shctx - MINOR: shctx: rename lock functions - MINOR: h1: store the status code in the H1 message - BUG/MINOR: spoe: Don't compare engine name and SPOE scope when both are NULL - BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded - MINOR: action: Add trk_idx inline function - MINOR: action: Use trk_idx instead of tcp/http_trk_idx - MINOR: action: Add a function pointer in act_rule struct to check its validity - MINOR: action: Add function to check rules using an action ACT_ACTION_TRK_* - MINOR: action: Add a functions to check http capture rules - MINOR: action: Factorize checks on rules calling check_ptr if defined - MINOR: acl: Pass the ACLs as an explicit parameter of build_acl_cond - MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages - MINOR: spoe: Check uniqness of SPOE engine names during config parsing - MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file - MEDIUM: spoe/rules: Add "send-spoe-group" action for tcp/http rules - MINOR: spoe: Move message encoding in its own function - MINOR: spoe: Add a type to qualify the message list during encoding - MINOR: spoe: Add a generic function to encode a list of SPOE message - MEDIUM: spoe/rules: Process "send-spoe-group" action - BUG/MINOR: dns: Fix CLI keyword declaration - MAJOR: dns: Refactor the DNS code - BUG/MINOR: mailers: Fix a memory leak when email alerts are released - MEDIUM: mailers: Init alerts during conf parsing and refactor their processing - MINOR: mailers: Use pools to allocate email alerts and its tcpcheck_rules - MINOR: standard: Add memvprintf function - MINOR: log: Save alerts and warnings emitted during HAProxy startup - MINOR: cli: Add "show startup-logs" command - MINOR: startup: Extend the scope the MODE_STARTING flag - MINOR: threads: Prepare makefile to link with pthread - MINOR: threads: Add THREAD_LOCAL macro - MINOR: threads: Add atomic-ops and plock includes in import dir - MEDIUM: threads: Add hathreads header file - MINOR: threads: Add mechanism to register per-thread init/deinit functions - MINOR: threads: Add nbthread parameter - MEDIUM: threads: Adds a set of functions to handle sync-point - MAJOR: threads: Start threads to experiment multithreading - MINOR: threads: Define the sync-point inside run_poll_loop - MEDIUM: threads/buffers: Define and register per-thread init/deinit functions - MEDIUM: threads/chunks: Transform trash chunks in thread-local variables - MEDIUM: threads/time: Many global variables from time.h are now thread-local - MEDIUM: threads/logs: Make logs thread-safe - MEDIUM: threads/pool: Make pool thread-safe by locking all access to a pool - MAJOR: threads/fd: Make fd stuffs thread-safe - MINOR: threads/fd: Add a mask of threads allowed to process on each fd in fdtab array - MEDIUM: threads/fd: Initialize the process mask during the call to fd_insert - MINOR: threads/fd: Process cached events of FDs depending on the process mask - MINOR: threads/polling: pollers now handle FDs depending on the process mask - WIP: SQUASH WITH SYNC POINT - MAJOR: threads/task: handle multithread on task scheduler - MEDIUM: threads/signal: Add a lock to make signals thread-safe - MEDIUM: threads/listeners: Make listeners thread-safe - MEDIUM: threads/proxy: Add a lock per proxy and atomically update proxy vars - MEDIUM: threads/server: Make connection list (priv/idle/safe) thread-safe - MEDIUM: threads/server: Add a lock per server and atomically update server vars - MINOR: threads/server: Add a lock to deal with insert in updates_servers list - MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe - MEDIUM: threads/stick-tables: handle multithreads on stick tables - MINOR: threads/sample: Change temp_smp into a thread local variable - MEDIUM: threads/http: Make http_capture_bad_message thread-safe - MINOR: threads/regex: Change Regex trash buffer into a thread local variable - MAJOR: threads/applet: Handle multithreading for applets - MAJOR: threads/peers: Make peers thread safe - MAJOR: threads/buffer: Make buffer wait queue thread safe - MEDIUM: threads/stream: Make streams list thread safe - MAJOR: threads/ssl: Make SSL part thread-safe - MEDIUM: threads/queue: Make queues thread-safe - MAJOR: threads/map: Make acls/maps thread safe - MEDIUM: threads/freq_ctr: Make the frequency counters thread-safe - MEDIUM: thread/vars: Make vars thread-safe - MEDIUM: threads/filters: Add init/deinit callback per thread - MINOR: threads/filters: Update trace filter to add _per_thread callbacks - MEDIUM: threads/compression: Make HTTP compression thread-safe - MEDIUM: threads/lua: Makes the jmpbuf and some other buffers local to the current thread. - MEDIUM: threads/lua: Add locks around the Lua execution parts. - MEDIUM: threads/lua: Ensure that the launched tasks runs on the same threads than me - MEDIUM: threads/lua: Cannot acces to the socket if we try to access from another thread. - MEDIUM: threads/xref: Convert xref function to a thread safe model - MEDIUM: threads/tasks: Add lock around notifications - MEDIUM: thread/spoe: Make the SPOE thread-safe - MEDIUM: thread/dns: Make DNS thread-safe - MINOR: threads: Add thread-map config parameter in the global section - MINOR: threads/checks: Add a lock to protect the pid list used by external checks - MINOR: threads/checks: Set the task process_mask when a check is executed - MINOR: threads/mailers: Add a lock to protect queues of email alerts - MEDIUM: threads/server: Use the server lock to protect health check and cli concurrency - MINOR: threads: Don't start when device a detection module is used - BUG/MEDIUM: threads: Run the poll loop on the main thread too - BUG/MINOR: threads: Add missing THREAD_LOCAL on static here and there - MAJOR: threads: Offically enable the threads support in HAProxy - BUG/MAJOR: threads/freq_ctr: fix lock on freq counters. - BUG/MAJOR: threads/time: Store the time deviation in an 64-bits integer - BUILD: stick-tables: silence an uninitialized variable warning - BUG/MINOR: dns: Fix SRV records with the new thread code. - MINOR: ssl: Remove the global allow-0rtt option. - CLEANUP: threads: replace the last few 1UL<detach() release the connection - MEDIUM: stream: do not forcefully close the client connection anymore - MEDIUM: checks: exclusively use cs_destroy() to release a connection - MEDIUM: connection: add a destroy callback - MINOR: session: release the listener with the session, not the stream - MEDIUM: session: make use of the connection's destroy callback - CONTRIB: hpack: implement a reverse huffman table generator for hpack - MINOR: hpack: implement the HPACK Huffman table decoder - MINOR: hpack: implement the header tables management - MINOR: hpack: implement the decoder - MEDIUM: hpack: implement basic hpack encoding - MINOR: h2: centralize all HTTP/2 protocol elements and constants - MINOR: h2: create a very minimalistic h2 mux - MINOR: h2: expose tune.h2.header-table-size to configure the table size - MINOR: h2: expose tune.h2.initial-window-size to configure the window size - MINOR: h2: expose tune.h2.max-concurrent-streams to limit the number of streams - MINOR: h2: create the h2c struct and allocate its pool - MINOR: h2: create the h2s struct and the associated pool - MINOR: h2: handle two extra stream states for errors - MINOR: h2: add a frame header descriptor for incoming frames - MEDIUM: h2: allocate and release the h2c context on connection init/end - MEDIUM: h2: implement basic recv/send/wake functions - MEDIUM: h2: dynamically allocate the demux buffer on Rx - MEDIUM: h2: implement the mux buffer allocator - MINOR: h2: add the connection and stream flags listing the causes for blocking - MINOR: h2: add function h2s_id() to report a stream's ID - MINOR: h2: small function to know when the mux is busy - MINOR: h2: new function h2c_error to mark an error on the connection - MINOR: h2: new function h2s_error() to mark an error on a stream - MINOR: h2: add h2_set_frame_size() to update the size in a binary frame - MINOR: h2: new function h2_peek_frame_hdr() to retrieve a new frame header - MINOR: h2: add a few functions to retrieve contents from a wrapping buffer - MINOR: h2: add stream lookup function based on the stream ID - MINOR: h2: create dummy idle and closed streams - MINOR: h2: add the function to create a new stream - MINOR: h2: update the {MUX,DEM}_{M,D}ALLOC flags on buffer availability - MEDIUM: h2: start to consider the H2_CF_{MUX,DEM}_* flags for polling - MINOR: h2: also terminate the connection on shutr - MEDIUM: h2: properly consider all conditions for end of connection - MEDIUM: h2: wake the connection up for send on pending streams - MEDIUM: h2: start to implement the frames processing loop - MINOR: h2: add a function to send a GOAWAY error frame - MINOR: h2: match the H2 connection preface on init - MEDIUM: h2: enable connection polling for send when a cs wants to emit - MEDIUM: h2: enable reading again on the connection if it was blocked on stream buffer full - MEDIUM: h2: process streams pending for sending - MINOR: h2: send a real SETTINGS frame based on the configuration - MEDIUM: h2: detect the presence of the first settings frame - MINOR: h2: create a stream parser for the demuxer - MINOR: h2: implement PING frames - MEDIUM: h2: decode SETTINGS frames and extract relevant settings - MINOR: h2: lookup the stream during demuxing - MEDIUM: h2: honor WINDOW_UPDATE frames - MINOR: h2: implement h2_send_rst_stream() to send RST_STREAM frames - MINOR: h2: handle CONTINUATION frames - MEDIUM: h2: partial implementation of h2_detach() - MEDIUM: h2: unblock a connection when its current stream detaches - MEDIUM: h2: basic processing of HEADERS frame - MEDIUM: h2: don't use trash to decode headers! - MEDIUM: h2: implement the response HEADERS frame to encode the H1 response - MEDIUM: h2: send the H1 response body as DATA frames - MEDIUM: h2: skip the response trailers if any - MEDIUM: h2: properly continue to parse header block when facing a 1xx response - MEDIUM: h2: send WINDOW_UPDATE frames for connection - MEDIUM: h2: handle request body in DATA frames - MINOR: h2: handle RST_STREAM frames - MEDIUM: h2: send DATA+ES or RST_STREAM on shutw/shutr - MINOR: h2: use a common function to signal some and all streams. - MEDIUM: h2: handle GOAWAY frames - MINOR: h2: centralize the check for the idle streams - MINOR: h2: centralize the check for the half-closed(remote) streams - MEDIUM: h2: silently ignore frames higher than last_id after GOAWAY - MINOR: h2: properly reject PUSH_PROMISE frames coming from the client - MEDIUM: h2: perform a graceful shutdown on "Connection: close" - MEDIUM: h2: send a GOAWAY frame when dealing with an empty response - MEDIUM: h2: apply a timeout to h2 connections - BUG/MEDIUM: h2: fix incorrect timeout handling on the connection - MEDIUM: shctx: forbid shctx to read more than expected - MEDIUM: cache: configuration parsing and initialization - MEDIUM: cache: store objects in cache - MEDIUM: cache: deliver objects from cache commit 77c1197bfb6b9ef92c655fa74d0907640b4a2be2 Author: William Lallemand Date: Tue Oct 31 20:43:01 2017 +0100 MEDIUM: cache: deliver objects from cache Lookup objects in the cache and deliver them using the http-request action "cache-use". commit 4da3f8a1f2c91f2d17378c5d12f43517e313701c Author: William Lallemand Date: Tue Oct 31 14:33:34 2017 +0100 MEDIUM: cache: store objects in cache Store object in the cache. The cache use an shctx for storage. It uses an http-response action to store the headers and a filter to store the body. The http-response action is used in order to allow modifications by other actions before caching. commit 41db46035ebf39a0c147fa96ecd5900af10123ea Author: William Lallemand Date: Mon Oct 30 11:15:51 2017 +0100 MEDIUM: cache: configuration parsing and initialization Parse a configuration section "cache" and a http-{response,request} actions. Example: listen frt mode http http-response cache-store foobar http-request cache-use foobar cache foobar total-max-size 4 # size in megabytes commit 7217c46dfefccf581583f6f38ba8032a0dfad643 Author: William Lallemand Date: Tue Oct 31 20:21:46 2017 +0100 MEDIUM: shctx: forbid shctx to read more than expected Forbid shctx to read more than expected, it allows you to use a greater value as a len with shctx_row_data_get(), the size of the destination buffer for example. commit 3f133570b862009c6100b6cc740519b8cfb2c420 Author: Willy Tarreau Date: Tue Oct 31 19:21:06 2017 +0100 BUG/MEDIUM: h2: fix incorrect timeout handling on the connection Previous commit ea3928 (MEDIUM: h2: apply a timeout to h2 connections) was wrong for two reasons. The first one is that if the client timeout is not set, it's used as zero, preventing connections from establishing. The second reason is that if the timeout triggers with active streams (normally it should not since the task is supposed to be disabled), the task is removed (h2c->task=NULL), and the last quitting stream might try to dereference it. Instead of doing this, we simply not register the task if there's no timeout (it's useless) and we always control its presence in the streams. commit ea39282e851c70ec9d192771deb7d949533fe6a2 Author: Willy Tarreau Date: Tue Oct 31 10:02:25 2017 +0100 MEDIUM: h2: apply a timeout to h2 connections Till now there was no way to deal with a dead H2 connection. Now each connection creates a task that wakes up to kill the connection. Its timeout is constantly refreshed when there's some activity. In case the timeout triggers, the best effort attempts are made at sending a clean GOAWAY message before closing and signaling the streams. The timeout is automatically disabled when there's an active stream on the connection, and restarted when the last stream finishes. This way it should not affect long sessions. commit a1349f020787ee7355c0d088f8054713fa25e069 Author: Willy Tarreau Date: Tue Oct 31 07:41:55 2017 +0100 MEDIUM: h2: send a GOAWAY frame when dealing with an empty response Given that we're processing data produced by haproxy, we know that the situations where haproxy doesn't return anything are : - request timeout with option http-ignore-probes : there's no reason to hit this since we're creating the stream with the request into it ; - tcp-request content reject : this definitely means we want to kill the connection and abort keep-alive and any further processing ; - using /dev/null as the error file to hide an error In practice it appears that using the abort on empty response as a hint to trigger a connection close is very appropriate to continue to give the control over the connection management. This patch thus tries to send a GOAWAY frame with the max_id presented as the last stream ID, then sends an RST_STREAM for the current stream. For the client, this means that the connection must be shut down immediately after processing the last pending streams and that the current stream is aborted. This way it's still possible to force connections to be closed using tcp-request rules. commit af1e4f5167062d5127a446e1bd63b4fd6185b7e6 Author: Willy Tarreau Date: Mon Oct 30 21:54:49 2017 +0100 MEDIUM: h2: perform a graceful shutdown on "Connection: close" After some long brainstorming sessions, it appears that "Connection: close" seems to be the best signal from the L7 layer to indicate the need to close the connection. Indeed, in H1 it is only present in very rare cases (eg: certain unrecoverable errors, some of which could remove it now by the way). It will also be added when the L7 layer wants to force the connection to terminate. By default when running in keep-alive mode it is not present. It's worth mentionning that in H1 with persistent connections, we have sort of a concurrency-1 mux and this header field is used the same way. Thus here this patch detects "Connection: close" in response headers and if seen, sends a GOAWAY frame with the highest possible ID so that the client knows that it can quit whenever it wants to. If more aggressive closures are needed in the future, we may decide to advertise the max_id to abort after the current requests and better honor "http-request deny". commit 1c661986a824251b4f6854c2e9facb4b723cff28 Author: Willy Tarreau Date: Mon Oct 30 13:52:01 2017 +0100 MINOR: h2: properly reject PUSH_PROMISE frames coming from the client These ones deserve a connection error as per 5.1. commit c0da1964ba086b0794043652f9429f7eb470f581 Author: Willy Tarreau Date: Mon Oct 30 18:38:00 2017 +0100 MEDIUM: h2: silently ignore frames higher than last_id after GOAWAY For a graceful shutdown, the specs requries to discard frames with a stream ID higher than the advertised last_id. (RFC7540#6.8). Well, finally for now the code is disabled (see last page of #6.8). Some frames need to be processed anyway to maintain the compression state and the flow control window state, but we don't have any trivial way to do this and ignore them at the same time. For the headers it's the worst case where we can't parse headers frames without coming from the streams, and we don't want to create such streams as we'd have to abort them, and aborting would cause errors to flow back. Possibly that a longterm solution might involve using some dummy streams and dummy buffers for this and calling the parsers directly. commit f182a9a8b422cbcf430c8890da180f10da421d2f Author: Willy Tarreau Date: Mon Oct 30 12:03:50 2017 +0100 MINOR: h2: centralize the check for the half-closed(remote) streams RFC7540#5.1 is pretty clear : "any frame other than WINDOW_UPDATE, PRIORITY, or RST_STREAM in this state MUST be treated as a connection error of type STREAM_CLOSED". Instead of dealing with this for each and every frame type, let's do it once for all in the main demux loop. commit f65b80dd47da42743b13327e358709eddb0edb73 Author: Willy Tarreau Date: Mon Oct 30 11:46:49 2017 +0100 MINOR: h2: centralize the check for the idle streams RFC7540#5.1 is pretty clear : "any frame other than HEADERS or PRIORITY in this state MUST be treated as a connection error". Instead of dealing with this for each and every frame type, let's do it once for all in the main demux loop. commit e96b0922e9b26260041fda7acc108a4ec901ef0a Author: Willy Tarreau Date: Mon Oct 30 00:28:29 2017 +0100 MEDIUM: h2: handle GOAWAY frames The ID is respected, and only IDs greater than the advertised last_id are woken up, with a CS_FL_ERROR flag to signal that the stream is aborted. This is necessary for a browser to abort a download or to reject a bad response that affects the connection's state. commit 23b92aa2bb5a4f26cc02859daa900e99feb3f6b2 Author: Willy Tarreau Date: Mon Oct 30 00:26:54 2017 +0100 MINOR: h2: use a common function to signal some and all streams. Let's replace h2_wake_all_streams() with h2_wake_some_streams(), to support signaling only streams by their ID (for GOAWAY frames) and to pass the flags to add on the conn_stream. commit c7576eac4603d80c787a276ba495f8471fc0a05b Author: Willy Tarreau Date: Sun Oct 29 22:00:09 2017 +0100 MEDIUM: h2: send DATA+ES or RST_STREAM on shutw/shutr When a stream sends a shutw, we send an empty DATA frame with the ES flag set, except if no HEADERS were sent, in which case we rather send RST_STREAM. On shutr(1) to abort a request, an RST_STREAM frame is sent if the stream is OPEN and the stream is closed. Care is taken to switch the stream's state accordingly and to avoid sending an ES bit again or another RST once already done. commit cd234e9fb0174da3d0f4a10fe0f27df4c8074704 Author: Willy Tarreau Date: Fri Aug 18 10:59:39 2017 +0200 MINOR: h2: handle RST_STREAM frames These ones are received when the browser aborts a page load, it's the only moment we can abort the stream. commit 454f90508452c1dce4f47cb855b95ba8ece79714 Author: Willy Tarreau Date: Thu Oct 26 19:40:35 2017 +0200 MEDIUM: h2: handle request body in DATA frames Data frames are received and transmitted. The per-connection and per-stream amount of data to ACK is automatically updated. Each DATA frame is ACKed because usually the downstream link is large and the upstream one is small, so it seems better to waste a few bytes every few kilobytes to maintain a low ACK latency and help the sender keep the link busy. The connection's ACK however is sent at the end of the demux loop and at the beginning of the mux loop so that a single aggregated one is emitted (connection windows tend to be much larger than stream windows). A future improvement would consist in sending a single ACK for multiple subsequent DATA frames of the same stream (possibly interleaved with window updates frames), but this is much trickier as it also requires to remember the ID of the stream for which DATA frames have to be sent. Ideally in the near future we should chunk-encode the body sent to HTTP/1 when there's no content length and when the request is not a CONNECT. It's just uncertain whether it's the best option or not for now. commit cc0b8c34a63bf55bfac455cd1c7a9342f2b57ea2 Author: Willy Tarreau Date: Thu Oct 26 16:55:59 2017 +0200 MEDIUM: h2: send WINDOW_UPDATE frames for connection When it is detected that the number of received bytes is > 0 on the connection at the end of the demux call or before starting to process pending output data, an attempt is made at sending a WINDOW UPDATE on the connection. In case of failure, it's attempted later. commit c199faf5bd54aa4b53053a0919b5f9afb1592d12 Author: Willy Tarreau Date: Tue Oct 31 08:35:27 2017 +0100 MEDIUM: h2: properly continue to parse header block when facing a 1xx response We still didn't handle the 1xx responses properly. commit 9d89ac8f42140e573b6d8afeb1c5722d7b2ae147 Author: Willy Tarreau Date: Tue Oct 31 17:15:59 2017 +0100 MEDIUM: h2: skip the response trailers if any For now we don't build a HEADERS frame with them, but at least we remove them from the response so that the L7 chunk parser inside isn't blocked on these (often two) remaining bytes that don't want to leave the buffer. It also ensures that trailers delivered progressively will correctly be skipped. commit c652dbde9dfb42a31281a2e9153401763c16c116 Author: Willy Tarreau Date: Thu Oct 19 11:16:37 2017 +0200 MEDIUM: h2: send the H1 response body as DATA frames The H1 response data are processed (either following content-length or chunks) and emitted as H2 DATA frames. In the case of content-length, the maximum size permitted by the mux buffer, the max frame size, the connection's window and the stream's window it used to determine the frame size. For chunked encoding, the same limitation applies, but in addition, each chunk leads to a distinct frame. This could be improved in the future to aggregate chunks into larger frames. Streams blocked on the connection's flow control subscribe to the connection's fctl_list to be woken up when the window opens again. Streams blocked on their own flow control don't subscribe to anything, they just sit waiting for window update frames to reopen the window. The connection-close mode (without content-length) partially works thanks to the fact that the SHUTW event leads to a close of the stream. In practice an empty DATA frame should be sent in this case though. commit 9e5ae1d7218ccf93a2d62e7b195396b1ab14857c Author: Willy Tarreau Date: Tue Oct 17 19:58:20 2017 +0200 MEDIUM: h2: implement the response HEADERS frame to encode the H1 response This calls the h1 response parser and feeds the output through the hpack encoder to produce stateless HPACK bytecode into an output chunk. For now it's a bit naive but reasonably efficient. The HPACK encoder relies on hpack_encode_header() so that the most common response header fields are encoded based on the static header table. The forbidden header field names (connection, proxy-connection, upgrade, transfer-encoding, keep-alive) are dropped before calling the hpack encoder. A new flag (H2_CF_HEADERS_SENT) is set once such a frame is emitted. It will be used to know if we can send an empty DATA+ES frame to use as a shutdown() signal or if we have to use RST_STREAM. commit 68dd9856cee6689a4be202a9ba6edadacc324f8b Author: Willy Tarreau Date: Mon Jul 3 14:44:26 2017 +0200 MEDIUM: h2: don't use trash to decode headers! The trash is already used by the hpack layer and for Huffman decoding, it's unsafe to use here as a buffer and results in corrupted data. Use a safely allocated trash instead. commit 13278b44b17cf63b0a9606a847eb7ea39b24587e Author: Willy Tarreau Date: Fri Oct 13 19:23:14 2017 +0200 MEDIUM: h2: basic processing of HEADERS frame This takes care of creating a new h2s and a new conn_stream when a HEADERS frame arrives. The recv() callback from the data layer is then called to extract the frame into the stream's buffer. It is verified that the stream ID is strictly greater than the known max stream ID. And the last_id is updated if the current request is properly converted. The streams are created in open or half-closed(remote) states. For now there are some limitations : - frames without END_HEADERS are rejected (CONTINUATION not supported yet, will require some more changes so that the stream processor checks the H2 frame header by itself and steals the frames from the connection) - padding/stream_dep/priority are currently ignored - limited error handling, could be improved But at least the request is properly decoded, transcoded and processed. commit 45f752e037b15fcd07d4620fbc39b8b192fa0bda Author: Willy Tarreau Date: Mon Oct 30 15:44:59 2017 +0100 MEDIUM: h2: unblock a connection when its current stream detaches If a stream is killed for whatever reason and it happens to be the one currently blocking the connection, we must unblock the connection and enable polling again so that it can attempt to make progress. This may happen for example on upload timeout, where the demux is blocked due to a full stream buffer, and the stream dies on server timeout and quits. commit 60935149334bf61d61cfbbbc30194733ba312008 Author: Willy Tarreau Date: Mon Oct 16 18:11:19 2017 +0200 MEDIUM: h2: partial implementation of h2_detach() This does the very minimum required to release a stream and/or a connection upon the stream's request. The only thing is that it doesn't kill the connection unless it's already closed or in error or the stream ID reached the one specified in GOAWAY frame. We're supposed to arm a timer to close after some idle timeout but it's not done. commit 61290ec774b311edad0285b90fab979a420d252c Author: Willy Tarreau Date: Tue Oct 17 08:19:21 2017 +0200 MINOR: h2: handle CONTINUATION frames For now we have nowhere to store partial header frames so we can't handle CONTINUATION frames and we must reject them. In this case we respond with a stream error of type INTERNAL_ERROR. commit 27a84c90ce57da78dff382d99d7829e90aee524a Author: Willy Tarreau Date: Tue Oct 17 08:10:17 2017 +0200 MINOR: h2: implement h2_send_rst_stream() to send RST_STREAM frames This one sends an RST_STREAM for a given stream, using the current demux stream ID. It's also used to send RST_STREAM for streams which have lost their CS part (ie were aborted). commit 26f95954fe4302a6ae558542ef9a73e9102b696c Author: Willy Tarreau Date: Thu Jul 27 17:18:30 2017 +0200 MEDIUM: h2: honor WINDOW_UPDATE frames Now they really increase the window size of connections and streams. If a stream was not queued but requested to send, it means it was flow-controlled so it's added again into the connection's send list. commit f3ee0697f36c2b6abc3fbbd17e5d8897a7e8cc2f Author: Willy Tarreau Date: Tue Oct 17 08:18:25 2017 +0200 MINOR: h2: lookup the stream during demuxing Several stream-oriented functions will need to perform this lookup, so better centralize it. commit 3421aba3dedc064c6697c0cf6151faa39ea66a83 Author: Willy Tarreau Date: Thu Jul 27 15:41:03 2017 +0200 MEDIUM: h2: decode SETTINGS frames and extract relevant settings The INITIAL_WINDOW_SIZE and MAX_FRAME_SIZE settings are now extracted from the settings frame, assigned to the connection, and attempted to be propagated to all existing streams as per the specification. In practice clients rarely update the settings after sending the first stream, so the propagation will rarely be used. The ACK is properly sent after the frame is completely parsed. commit cf68c787ae40fe870b96119ea7d8b516797dd676 Author: Willy Tarreau Date: Tue Oct 10 17:11:41 2017 +0200 MINOR: h2: implement PING frames Now we can detect and properly parse PING frames as well as emit a response containing the same payload. commit 7e98c057ff76f399be5de17d8bca98c954eb2bbb Author: Willy Tarreau Date: Tue Oct 10 15:56:59 2017 +0200 MINOR: h2: create a stream parser for the demuxer The function h2_process_demux() now tries to parse the incoming bytes to process as many streams as possible. For now it does nothing but dropping all incoming frames. commit 4c3690bf960e3e4274e1689b7d4faebbf6cf6e9c Author: Willy Tarreau Date: Tue Oct 10 15:16:55 2017 +0200 MEDIUM: h2: detect the presence of the first settings frame Instead of doing a special processing of the first SETTINGS frame, we simply parse its header, check that it matches the expected frame type and flags (ie no ACK), and switch to FRAME_P to parse it as any regular frame. The regular frame parser will take care of decoding it. commit be5b715fb2c5ff92448f9481d18cfee000cef6c9 Author: Willy Tarreau Date: Mon Sep 25 16:25:39 2017 +0200 MINOR: h2: send a real SETTINGS frame based on the configuration An initial settings frame is emitted upon receipt of the connection preface, which takes care of configured values. These settings are only emitted when they differ from the protocol's default value : - header_table_size (defaults to 4096) - initial_window_size (defaults to 65535) - max_concurrent_streams (defaults to unlimited) - max_frame_size (defaults to 16384) The max frame size is a copy of tune.bufsize. Clients will most often reject values lower than 16384 and currently there's no trivial way to check if H2 is going to be used at boot time. commit bacdf5a49b9c9a5d1215556c8dd8e68cca86d6e4 Author: Willy Tarreau Date: Tue Oct 17 10:57:04 2017 +0200 MEDIUM: h2: process streams pending for sending The send() callback calls h2_process_mux() which iterates over the list of flow controlled streams first, then streams waiting for room in the send_list. If a stream from the send_list ends up being flow controlled, it is then moved to the fctl_list. This way we can maintain the most accurate fairness by ensuring that flows are always processed in order of arrival except when they're blocked by flow control, in which case only the other ones may pass in front of them. It's a bit tricky as we want to remove a stream from the active lists if it doesn't block (ie it has no reason for staying there). commit d7739c8820be1d7a89498f6d01cd652f7548b583 Author: Willy Tarreau Date: Mon Oct 30 15:38:23 2017 +0100 MEDIUM: h2: enable reading again on the connection if it was blocked on stream buffer full If the polling update function is called with RD_ENA while H2_CF_DEM_SFULL indicates the demux had to block on a stream buffer full condition, we can remove the flag and re-enable polling for receiving because this is the indication that a consumer stream has made some room in the buffer. Probably that we should improve this to ensure that h2s->id == h2c->dsi and avoid trying to receive multiple times in a row for the wrong stream. commit 1d393228e09febd9e039d2fc6b247ba47d182fc6 Author: Willy Tarreau Date: Tue Oct 17 10:26:19 2017 +0200 MEDIUM: h2: enable connection polling for send when a cs wants to emit A conn_stream indicates its intent to send by setting the WR_ENA flag and calling mux->update_poll(). There's no synchronous write so the only way to emit a response from a stream is to proceed this way. The sender h2s is then queued into the h2c's send_list if it was not yet queued. Once the connection is ready, it will enter its send() callback to visit writers, calling their data->send_cb() callback to complete the operation using mux->snd_buf(). Also we enable polling if the mux contains data and wasn't enabled. This may happen just after a response has been transmitted using chk_snd(). It likely is incomplete for now and should probably be refined. commit 52eed75ced5d9a89807ca272ddbc618c74e62edd Author: Willy Tarreau Date: Fri Sep 22 15:05:09 2017 +0200 MINOR: h2: match the H2 connection preface on init The H2 preface is properly detected to switch to the settings state. It's important to note that for now we don't send out settings frame so the operation is not complete yet. commit 081d472f79398bd4fd2b4f7c80b3cf251b99b489 Author: Willy Tarreau Date: Tue May 16 21:51:05 2017 +0200 MINOR: h2: add a function to send a GOAWAY error frame For now it's only used to report immediate errors by announcing the highest known stream-id on the mux's error path. The function may be used both while processing a stream or directly in relation with the connection. The wake() callback will automatically ask for send access if an error is reported. The function should be usable for graceful shutdowns as well by simply setting h2c->last_sid to the highest acceptable stream-id (2^31-1) prior to calling the function. A connection flag (H2_CF_GOAWAY_SENT) is set once the frame was successfully sent. It will be usable to detect when it's safe to close the connection. Another flag (H2_CF_GOAWAY_FAILED) is set in case of unrecoverable error while trying to send. It will also be used to know when it's safe to close the connection. commit bc933930a709e9bf96450b98805df1f6b57bfa1f Author: Willy Tarreau Date: Mon Oct 9 16:21:43 2017 +0200 MEDIUM: h2: start to implement the frames processing loop The rcv_buf() callback now calls h2_process_demux() after an recv() call leaving some data in the buffer, and the snd_buf() callback calls h2_process_mux() to try to process pending data from streams. commit 5160683fc74e8684f8695404f928358fdbc17241 Author: Willy Tarreau Date: Tue Oct 17 15:30:07 2017 +0200 MEDIUM: h2: wake the connection up for send on pending streams If some streams were blocked on flow control and the connection's window was recently opened, or if some streams are waiting while no block flag remains, we immediately want to try to send again. This can happen if a recv() for a stream wants to send after the send() loop has already been processed. commit 29a98241441ba1f2081e0eda440cfd20834b9e7c Author: Willy Tarreau Date: Tue Oct 31 06:59:15 2017 +0100 MEDIUM: h2: properly consider all conditions for end of connection During h2_wake(), there are various situations that can lead to the connection being closed : - low-level connection error - read0 received - fatal error (ERROR2) - failed to emit a GOAWAY - empty stream list with max_id >= last_sid In such cases, all streams are notified and we have to wait for all streams to leave while doing nothing, or if the last stream is gone, we can simply terminate the connection. It's important to do this test there again because an error might arise while trying to send a pending GOAWAY after the last stream for example, thus there's possibly no way to get notified of a closing stream. commit 26bd761f0175737b33d76d8a3a4f17f070bfc9b8 Author: Willy Tarreau Date: Mon Oct 9 16:47:04 2017 +0200 MINOR: h2: also terminate the connection on shutr It happens that an H2 mux is totally unusable once the client has shut, so we must consider this situation equivalent to the connection error, and let the possible streams drain their data if needed then stop. commit fbe3b4fcbe7ecd8fc87264d73563871aca79d23e Author: Willy Tarreau Date: Mon Oct 9 15:14:19 2017 +0200 MEDIUM: h2: start to consider the H2_CF_{MUX,DEM}_* flags for polling Now we start to set the flags to indicate that the response buffer is being awaited or that it is full, it makes it possible to centralize a little bit the polling management into the wake() callback. In case of error, we wake all the streams up so that they are aware of the nature of the event and are able to detach if needed. commit 1b62c5caefbb3b29dea03f225404fe85cf4c98a2 Author: Willy Tarreau Date: Mon Sep 25 11:55:01 2017 +0200 MINOR: h2: update the {MUX,DEM}_{M,D}ALLOC flags on buffer availability Flag H2_CF_DEM_DALLOC is set when the demux buffer fails to be allocated in the recv() callback, and is cleared when it succeeds. Both flags H2_CF_MUX_MALLOC and H2_CF_DEM_MROOM are cleared when the mux buffer allocation succeeds. In both cases it will be up to the callers to report allocation failures. commit 3ccf4b2a2011245783b526340827075010a94daf Author: Willy Tarreau Date: Fri Oct 13 19:07:26 2017 +0200 MINOR: h2: add the function to create a new stream This one will be used by the HEADERS frame handler and maybe later by the PUSH frame handler. It creates a conn_stream in the mux's connection. The create streams are inserted in the h2c's tree sorted by IDs. The caller is expected to have verified that the stream doesn't exist yet. commit 2a8561895de6a70db82212f483c66ec94c1768db Author: Willy Tarreau Date: Tue May 16 15:20:39 2017 +0200 MINOR: h2: create dummy idle and closed streams It will be more convenient to always manipulate existing streams than null pointers. Here we create one idle stream and one closed stream. The idea is that we can easily point any stream to one of these states in order to merge maintenance operations. commit 2373acc3844edfbd3100f5d6f9f08bb48d7cd5c1 Author: Willy Tarreau Date: Thu Oct 12 17:35:14 2017 +0200 MINOR: h2: add stream lookup function based on the stream ID The function performs a simple lookup in the tree and returns either the matching h2s or NULL if not found. commit 54c150653dbaf4bee545051427223fbd2e7be46a Author: Willy Tarreau Date: Tue Oct 10 17:10:03 2017 +0200 MINOR: h2: add a few functions to retrieve contents from a wrapping buffer Functions h2_get_buf_n{16,32,64}() and h2_get_buf_bytes() respectively extract a network-ordered 16/32/64 bit value from a possibly wrapping buffer, or any arbitrary size. They're convenient to retrieve a PING payload or to parse SETTINGS frames. Since they copy one byte at a time, they will be less efficient than a memcpy-based implementation on large blocks. commit 715d5316e5f00a5ba7fdffcb5d7365e027a875d1 Author: Willy Tarreau Date: Tue Jul 11 15:20:24 2017 +0200 MINOR: h2: new function h2_peek_frame_hdr() to retrieve a new frame header This function extracts the next frame header but doesn't consume it. This will allow to detect a stream-id change and to perform a yielding window update without losing information. The result is stored into a temporary frame descriptor. We could also store the next frame header into the connection but parsing the header again is much cheaper than wasting bytes in the connection for a rare use case. A function (h2_skip_frame_hdr()) is also provided to skip the parsed header (always 9 bytes) and another one (h2_get_frame_hdr()) to do both at once. commit e482074c96b6d6c19bacec13488ff77977670751 Author: Willy Tarreau Date: Thu Jul 27 13:37:23 2017 +0200 MINOR: h2: add h2_set_frame_size() to update the size in a binary frame This function is called after preparing a frame, in order to update the frame's size in the frame header. It takes the frame payload length in argument. It simply writes a 24-bit frame size into a buffer, making use of the net_helper functions which try to optimize per platform (this is a frequently used operation). commit 2e43f08c60c738af1fbb4d2d1e32d5bc4036a69f Author: Willy Tarreau Date: Tue Oct 17 08:03:59 2017 +0200 MINOR: h2: new function h2s_error() to mark an error on a stream This one will store the error into the stream's errcode if it's neither idle nor closed (since these ones are read-only) and switch its state to H2_SS_ERROR. If a conn_stream is attached, it will be flagged with CS_FL_ERROR. commit 741d6df87095ffc510fd3de3a7fae7072be76a3d Author: Willy Tarreau Date: Tue Oct 17 08:00:59 2017 +0200 MINOR: h2: new function h2c_error to mark an error on the connection This one sets the error code in h2c->errcode and changes the connection's stat to H2_CS_ERROR. commit 5b5e68741ae9b29017b95fb2c0ae72589a108da7 Author: Willy Tarreau Date: Mon Sep 25 16:17:25 2017 +0200 MINOR: h2: small function to know when the mux is busy A mux is busy when any stream id >= 0 is currently being handled and the current stream's id doesn't match. When no stream is involved (ie: demuxer), stream 0 is considered. This will be necessary to know when it's possible to send frames. commit 71681174f37f3e32eacce1318e09c6d4f0056134 Author: Willy Tarreau Date: Mon Oct 23 14:39:06 2017 +0200 MINOR: h2: add function h2s_id() to report a stream's ID This one supports being called with NULL and returns 0 in this case, making it easier to check for stream IDs in various send functions. commit 2e5b60ee18321911450dc2b6405fa9185d8880b9 Author: Willy Tarreau Date: Mon Sep 25 11:49:03 2017 +0200 MINOR: h2: add the connection and stream flags listing the causes for blocking A demux may be prevented from receiving for the following reasons : - no receive buffer could be allocated - the receive buffer is full - a response is needed and the mux is currently being used by a stream - a response is needed and some room could not be found in the mux buffer (either full or waiting for allocation) - the stream buffer is waiting for allocation - the stream buffer is full A mux may stop accepting data for the following reasons : - the buffer could not be allocated - the buffer is full A stream may stop sending data to a mux for the following reaons : - the mux is busy processing another stream - the mux buffer lacks room (full or not allocated) - the mux's flow control prevents from sending - the stream's flow control prevents from sending All these conditions were turned into flags for use by the respective places. commit 1439812da87de5c8cd0e3cf5703ff2fb78f90702 Author: Willy Tarreau Date: Fri Sep 22 14:26:04 2017 +0200 MEDIUM: h2: implement the mux buffer allocator The idea is that we may need a mux buffer for anything, ranging from receiving to sending traffic. For now it's unclear where exactly the calls will be placed so let's block both send and recv when a buffer is missing, and re-enable both of them at the end. This will have to be changed later. commit 35dbd5d719167fd73a486cdafd571d9efabc64cd Author: Willy Tarreau Date: Fri Sep 22 09:13:49 2017 +0200 MEDIUM: h2: dynamically allocate the demux buffer on Rx This patch implements a very basic Rx buffer management. The mux needs an rx buffer to decode the connection's stream. If this buffer it available upon Rx events, we fill it with whatever input data are available. Otherwise we try to allocate it and subscribe to the buffer wait queue in case of failure. In such a situation, a function "h2_dbuf_available()" will be called once a buffer may be allocated. The buffer is released if it's still empty after recv(). commit a2af51291f5ee297bc4eb04f4c6efa440b2d4a36 Author: Willy Tarreau Date: Mon Oct 9 11:56:46 2017 +0200 MEDIUM: h2: implement basic recv/send/wake functions For now they don't do much since the buffers are not yet allocated, but the squeletton is here. commit 32218eb3448e6ef31f43ea56701d55d86f26bb49 Author: Willy Tarreau Date: Fri Sep 22 08:07:25 2017 +0200 MEDIUM: h2: allocate and release the h2c context on connection init/end The connection's h2c context is now allocated and initialized on mux initialization, and released on mux destruction. Note that for now the release() code is never called. commit c64051404def59c85462bd7b4f10bb862b20dea1 Author: Willy Tarreau Date: Thu Sep 21 20:23:50 2017 +0200 MINOR: h2: add a frame header descriptor for incoming frames This descriptor will be used by the frame parser, it's designed to ease manipulation of frame length, type, flags and sid. commit 96060bad266b9b402d3ea79ec84419f3e0c4fed4 Author: Willy Tarreau Date: Mon Oct 16 18:34:34 2017 +0200 MINOR: h2: handle two extra stream states for errors We need to deal with stream error notifications (RST_STREAM) as well as internal reporting. The problem is that we don't know in which order this will be done so we can't unilaterally decide to deallocate the stream. In order to help, we add two extra stream states, H2_SS_ERROR and H2_SS_RESET. The former mentions that the stream has an error pending and the latter indicates that the error was already sent and that the stream is now closed. It's equivalent to H2_SS_CLOSED except that in this state we'll avoid sending new RST_STREAM as per RFC7540#5.4.2. With this it will be possible to only detach or deallocate the h2s once the stream is closed. commit 183126488b7b749ab60f5e4963fc854f5f680981 Author: Willy Tarreau Date: Wed Oct 11 07:57:07 2017 +0200 MINOR: h2: create the h2s struct and the associated pool This describes an HTTP/2 stream with its relation to the connection and to the conn_stream on the other side. For now we also allocate request and response state for HTTP/1 because the internal HTTP representation is HTTP/1 at the moment. Later this should evolve towards a version-agnostic representation and this H1 message state will disappear. It's important to consider that the streams are necessarily polarized depending on h2c : if the connection is incoming, streams initiated by the connection receive requests and send responses. Otherwise it's the other way around. Such information is known during the connection instanciation by h2c_frt_init() and will normally be reflected in the stream ID (odd=demux from client, even=demux from server). The initial H2_CS_PREFACE state will also depend on the direction. The current h2c state machine doesn't allow for outgoing connections as it uses a single state for both (rx state only). It should be the demux state only. commit 5ab6b57c6f35f33be594b8137847e25e07222ff2 Author: Willy Tarreau Date: Fri Sep 22 08:05:00 2017 +0200 MINOR: h2: create the h2c struct and allocate its pool The h2c struct describes an H2 connection context and is assigned as the mux's context. It has its own pool, allocated at boot time and released after deinit(). commit 5242ef8095d60cdf2d48b97595aeebf624c253e9 Author: Willy Tarreau Date: Thu Jul 27 11:47:28 2017 +0200 MINOR: h2: expose tune.h2.max-concurrent-streams to limit the number of streams This will be advertised in the settings frame. commit e6baec0e231db8fe915e67375fa715f40d107bc4 Author: Willy Tarreau Date: Thu Jul 27 11:45:11 2017 +0200 MINOR: h2: expose tune.h2.initial-window-size to configure the window size This will be advertised in the settings frame. commit fe20e5b8c706cf6393a7cd37e89d54ec5e1b23d5 Author: Willy Tarreau Date: Thu Jul 27 11:42:14 2017 +0200 MINOR: h2: expose tune.h2.header-table-size to configure the table size It's the HPACK header table size which is to be advertised in the settings frames. It defaults to 4096. commit 62f5269d05ebd53c21f81eade32515bc76825b68 Author: Willy Tarreau Date: Sun Oct 8 23:01:42 2017 +0200 MINOR: h2: create a very minimalistic h2 mux This one currently does nothing and rejects every connection. It registers ALPN token "h2". commit ffca736401f8c1156c899faf0cb40058c140f4a4 Author: Willy Tarreau Date: Tue Dec 13 18:25:15 2016 +0100 MINOR: h2: centralize all HTTP/2 protocol elements and constants These constants from RFC7540 will be centralized into common/h2.h for use by the future h2 mux and other places. commit 1be4f3d8af8a23484bd8e81bbd40b69cd1f5ded6 Author: Willy Tarreau Date: Thu Sep 21 14:35:57 2017 +0200 MEDIUM: hpack: implement basic hpack encoding For now it only supports literals and a bit of static header table references for the 9 most common header field names (date, server, content-type, content-length, last-modified, accept-ranges, etag, cache-control, location). A previous incarnation of this commit used to strip the forbidden H2 header names (connection, proxy-connection, upgrade, transfer-encoding, keep-alive) but this is no longer the case as this filtering is irrelevant to HPACK encoding and is specific to H2, so this will have to be done by the caller. It's quite not optimal but works fine enough to prepare some valid and partially compressed responses during development. commit 679790baae3367afb3e07dfb3d2392ecf026cb30 Author: Willy Tarreau Date: Tue May 30 19:09:44 2017 +0200 MINOR: hpack: implement the decoder The decoder is now fully functional. It makes use of the dynamic header table. Dynamic header table size updates are currently ignored, as our initially advertised value is the highest we support. Strictly speaking, the impact is that a client referencing a header field after such an update wouldn't observe an error instead of the connection being dropped if it was implemented. Decoded header fields are copied into a target buffer in HTTP/1 format using HTTP/1.1 as the version. The Host header field is automatically appended if a ":authority" header field is present. All decoded header fields can be displayed if the file is compiled with DEBUG_HPACK. commit ce04094c4a5a7d3db4115dfb4cd20ab654a90aff Author: Willy Tarreau Date: Tue May 30 18:46:58 2017 +0200 MINOR: hpack: implement the header tables management This code deals with header insertion, retrieval and eviction, as well as with dynamic header table defragmentation. It is functional for use as a decoder and was heavily tested in this context. There's still some room for optimization (eg: the defragmentation code currently does it in place using a memcpy). Also for now the dynamic header table is allocated using malloc() while a pool needs to be created instead. This code was mostly imported from https://github.com/wtarreau/http2-exp with "hpack_" prepended in front of most names to avoid risks of conflicts. Some small cleanups and renamings were applied during the import. This version must be considered more recent. Some HPACK error codes were placed here (HPACK_ERR_*), not exactly because they're needed by the decoder but they'll be needed by all callers. Maybe a different location should be found. commit a004ade512cddb5f9ed4a077c6f3cc85aeaa72b0 Author: Willy Tarreau Date: Tue May 30 17:22:18 2017 +0200 MINOR: hpack: implement the HPACK Huffman table decoder The code was borrowed from the HPACK experimental implementations available here : https://github.com/wtarreau/http2-exp It contains the Huffman table as specified in RFC7541 Appendix B, and a set of reverse tables used to decode a Huffman byte stream, and produced by contrib/h2/gen-rht. The encoder is not finalized, it doesn't emit the byte stream but this is not needed for now. commit 8071eae6b92f00f001af7eef89ee6ac129e45af6 Author: Willy Tarreau Date: Fri May 19 18:14:51 2017 +0200 CONTRIB: hpack: implement a reverse huffman table generator for hpack This one was built by studying the HPACK Huffman table (RFC7541 appendix B). It creates 5 small tables (4*512 bytes, 1*64 bytes) to map one byte at a time from the input stream based on the following observations : * rht_bit31_24[256] is indexed on bits 31..24 when < 0xfe * rht_bit24_17[256] is indexed on bits 24..17 when 31..24 >= 0xfe * rht_bit15_11_fe[32] is indexed on bits 15..11 when 24..17 == 0xfe * rht_bit15_8[256] is indexed on bits 15..8 when 24..17 == 0xff * rht_bit11_4[256] is indexed on bits 11..4 when 15..8 == 0xff * when 11..4 == 0xff, 3..2 provide the following mapping : * 00 => 0x0a, 01 => 0x0d, 10 => 0x16, 11 => EOS commit 3e13cbafe2612dc026494d90ce8604f08cdaf58d Author: Willy Tarreau Date: Sun Oct 8 11:26:30 2017 +0200 MEDIUM: session: make use of the connection's destroy callback Now we don't remove the session when a stream dies, instead we detach the stream and let the mux decide to release the connection and call session_free() instead. commit 4f0c64cad7a9d3dc0bde92bea671c493c7b284a5 Author: Willy Tarreau Date: Wed Oct 18 15:01:14 2017 +0200 MINOR: session: release the listener with the session, not the stream Since multiple streams can share one session attached to one listener, the listener_release() call must be done in session_free() and not in stream_free(), otherwise we end up with a negative count in H2. commit 436d333124604db40b20344fa58bc41e1148cb2b Author: Willy Tarreau Date: Sun Oct 8 11:16:46 2017 +0200 MEDIUM: connection: add a destroy callback This callback will be used to release upper layers when a mux is in use. Given that the mux can be asynchronously deleted, we need a way to release the extra information such as the session. This callback will be called directly by the mux upon releasing everything and before the connection itself is released, so that the callee can find its information inside the connection if needed. The way it currently works is not perfect, and most likely this should instead become a mux release callback, but for now we have no easy way to add mux-specific stuff, and since there's one mux per connection, it works fine this way. commit ac59f361badb83abf960d1a71c9648a36a6c164e Author: Willy Tarreau Date: Sun Oct 8 11:10:19 2017 +0200 MEDIUM: checks: exclusively use cs_destroy() to release a connection This way we're using the more consistent API everywhere. commit 3256073976d4f43e12e7ff97d243fdb8eb56165a Author: Willy Tarreau Date: Sun Oct 8 11:33:44 2017 +0200 MEDIUM: stream: do not forcefully close the client connection anymore Now that the mux will take care of closing the client connection at the right moment, we don't need to close the client connection anymore, and we just need to close the conn_stream. commit 2c52a2b9ee472574d59c97427e5149df1e5eb496 Author: Willy Tarreau Date: Sun Oct 8 11:00:17 2017 +0200 MEDIUM: connection: make mux->detach() release the connection For H2, only the mux's timeout or other conditions might cause a release of the mux and the connection, no stream should be allowed to kill such a shared connection. So a stream will only detach using cs_destroy() which will call mux->detach() then free the cs. For now it's only handled by mux_pt. The goal is that the data layer never has to care about the connection, which will have to be released depending on the mux's mood. commit a553ae96f56070c9dada03d2598177ccca5b2689 Author: Willy Tarreau Date: Thu Oct 5 18:52:17 2017 +0200 MEDIUM: connection: replace conn_full_close() with cs_close() At all call places where a conn_stream is in use, we can now use cs_close() to get rid of a conn_stream and of its underlying connection if the mux estimates it makes sense. This is what is currently being done for the pass-through mux. commit 4b795245914eac02c7d2b776f5b5086afd6f3a2f Author: Willy Tarreau Date: Thu Oct 5 18:47:38 2017 +0200 MEDIUM: mux_pt: make cs_shutr() / cs_shutw() properly close the connection Now these functions are able to automatically close both the transport and the socket layer, causing the whole connection to be torn down if needed. The two shutdown modes are implemented for both directions, and when a direction is closed, if it sees the other one is closed as well, it completes by closing the connection. This is similar to what is performed in the stream interface. It's not deployed yet but the purpose is to get rid of conn_full_close() where only conn_stream should be known. commit 6978db35e92ccf4777d040b92a4c1b2ef144f202 Author: Willy Tarreau Date: Thu Oct 5 18:19:43 2017 +0200 MINOR: connection: add cs_close() to close a conn_stream This basically calls cs_shutw() followed by cs_shutr(). Both of them are called in the most conservative mode so that any previous call is still respected. The CS flags are cleared so that it can be reused (this is important for connection retries when conn and CS are reused without being reallocated). commit 9fbbff6de4222c79236043a2f9c2f5e6d7fb6c81 Author: Willy Tarreau Date: Thu Oct 5 17:24:42 2017 +0200 MEDIUM: connection: make conn_sock_shutw() aware of lingering Instead of having to manually handle lingering outside, let's make conn_sock_shutw() check for it before calling shutdown(). We simply don't want to emit the FIN if we're going to reset the connection due to lingering. It's particularly important for silent-drop where it's absolutely mandatory that no packet leaves the machine. commit ecdb3fe9f4b27ffe053d34d92a4c37692af49737 Author: Willy Tarreau Date: Thu Oct 5 15:25:48 2017 +0200 MINOR: conn_stream: modify cs_shut{r,w} API to pass the desired mode Now we can specify how we want to shutdown (drain vs reset, and normal vs silent), and this propagates to the mux then the transport layer. commit 79dadb5335e933d085baf1a8b518ceb951f87a9d Author: Willy Tarreau Date: Thu Oct 5 15:06:07 2017 +0200 MINOR: conn_stream: new shutr/w status flags In order to support all shutdown modes on the CS, we introduce the following flags : CS_FL_SHRD : shut read, drain extra data CS_FL_SHRR : shut read, reset extra data CS_FL_SHWN : shut write, normal notification CS_FL_SHWS : shut write, silent mode (no notification) And the following modes for shutr/shutw : CS_SHR_DRAIN, CS_SHR_RESET, CS_SHW_NORMAL, CS_SHW_SILENT. Note: it's possible that we won't need to distinguish the two shutw above as they're only an action. For now they are not used. commit 4ff3b89643d710750d5e7eec17479685e905dd80 Author: Willy Tarreau Date: Mon Oct 16 15:17:17 2017 +0200 MINOR: connection: make conn_stream users also check for per-stream error flag In a 1:1 connection:stream there's no problem relying on the connection flags alone to check for errors. But in a mux, it will be possible to mark certain streams in error without having to mark all of them. An example is an H2 client sending RST_STREAM frames to abort a long download, or a parse error requiring to abort only this specific stream. This commit ensures that stream-interface and checks properly check for CS_FL_ERROR in cs->flags wherever CO_FL_ERROR was in use. Most likely over the long term, any check for CO_FL_ERROR will have to disappear. commit 9aaf778129f47819bab4b3dec7f1579cf0e0f84b Author: Olivier Houchard Date: Wed Sep 13 18:30:23 2017 +0200 MAJOR: connection : Split struct connection into struct connection and struct conn_stream. All the references to connections in the data path from streams and stream_interfaces were changed to use conn_streams. Most functions named "something_conn" were renamed to "something_cs" for this. Sometimes the connection still is what matters (eg during a connection establishment) and were not always renamed. The change is significant and minimal at the same time, and was quite thoroughly tested now. As of this patch, all accesses to the connection from upper layers go through the pass-through mux. commit 7a3f0dfb7b72d4497da55efb3097c0abc4a78356 Author: Olivier Houchard Date: Wed Sep 13 18:30:23 2017 +0200 MINOR: mux_pt: implement remaining mux_ops methods This is a basic pass-through implementation which is now basic but complete and operational, just not used yet. commit 63dd75d9344ebd33dd516556d223c63f0ef3f34a Author: Willy Tarreau Date: Sun Oct 8 15:16:00 2017 +0200 MINOR: connection: introduce the conn_stream manipulation functions Most of the functions dealing with conn_streams are here. They act at the data layer and interact with the mux. For now they are not used yet but everything builds. commit 8e6147292ea38b865cd0341e9c7c9827c9b611b5 Author: Olivier Houchard Date: Wed Sep 13 18:30:23 2017 +0200 MINOR: mux: add more methods to mux_ops We'll need to support reading/writing from both sides, with buffers and pipes, as well as retrieving/updating flags. commit e2b40b9eabdc36fabaeacfc1a075427be8062b7f Author: Olivier Houchard Date: Wed Sep 13 18:30:23 2017 +0200 MINOR: connection: introduce conn_stream This patch introduces a new struct conn_stream. It's the stream-side of a multiplexed connection. A pool is created and destroyed on exit. For now the conn_streams are not used at all. commit 60ca10a372186e7cda30f387e947c83e00acdc37 Author: Willy Tarreau Date: Fri Aug 18 15:26:54 2017 +0200 MINOR: connection: report the major HTTP version from the MUX for logging (fc_http_major) A new sample fetch function reports either 1 or 2 for the on-wire encoding, to indicate if the request was received using the HTTP/1.x format or HTTP/2 format. Note that it reports the on-wire encoding, not the version presented in the request header. This will possibly have to evolve if it becomes necessary to report the encoding on the server side as well. commit 2e0b2b5f8363da05abab2e2012d298f4941a4f75 Author: Willy Tarreau Date: Fri Sep 15 06:59:55 2017 +0200 MEDIUM: session: use the ALPN token and proxy mode to select the mux When an incoming connection is made on an HTTP mode frontend, the session now looks up the mux to use based on the ALPN token and the proxy mode. This will allow easier mux registration, and we don't need to hard-code the mux_pt_ops anymore. commit f64908294c807cf3efdb16f23df6629c24752e7d Author: Willy Tarreau Date: Thu Sep 21 19:43:21 2017 +0200 MINOR: mux: register the pass-through mux for any ALPN string The pass-through mux is the fallback used on any incoming connection unless another mux claims the ALPN name and the proxy mode. Thus mux_pt registers ALPN token "" (empty name) which catches everything. commit 2386be64ba3f13f767a5ef0af4912d7361198664 Author: Willy Tarreau Date: Thu Sep 21 19:40:52 2017 +0200 MINOR: connection: implement alpn registration of muxes Selecting a mux based on ALPN and the proxy mode will quickly become a pain. This commit provides new functions to register/lookup a mux based on the ALPN string and the proxy mode to make this easier. Given that we're not supposed to support a wide range of muxes, the lookup should not have any measurable performance impact. commit 53a4766e400aa60835b6994c67100aaa0b936017 Author: Willy Tarreau Date: Mon Aug 28 10:53:00 2017 +0200 MEDIUM: connection: start to introduce a mux layer between xprt and data For HTTP/2 and QUIC, we'll need to deal with multiplexed streams inside a connection. After quite a long brainstorming, it appears that the connection interface to the existing streams is appropriate just like the connection interface to the lower layers. In fact we need to have the mux layer in the middle of the connection, between the transport and the data layer. A mux can exist on two directions/sides. On the inbound direction, it instanciates new streams from incoming connections, while on the outbound direction it muxes streams into outgoing connections. The difference is visible on the mux->init() call : in one case, an upper context is already known (outgoing connection), and in the other case, the upper context is not yet known (incoming connection) and will have to be allocated by the mux. The session doesn't have to create the new streams anymore, as this is performed by the mux itself. This patch introduces this and creates a pass-through mux called "mux_pt" which is used for all new connections and which only calls the data layer's recv,send,wake() calls. One incoming stream is immediately created when init() is called on the inbound direction. There should not be any visible impact. Note that the connection's mux is purposely not set until the session is completed so that we don't accidently run with the wrong mux. This must not cause any issue as the xprt_done_cb function is always called prior to using mux's recv/send functions. commit d7bddda1517ee870381528f8c49f97a41b201d89 Author: Christopher Faulet Date: Tue Oct 31 17:30:12 2017 +0100 BUG/MEDIUM: threads: Initialize the sync-point The sync point must be initialized before starting threads. This line was lost in one of merges preparing the threads support integration. commit a06a580941002f77fba182fd9d922670c7e2c5fc Author: Willy Tarreau Date: Tue Oct 31 17:54:15 2017 +0100 BUG/MAJOR: threads/freq_ctr: use a memory barrier to detect changes commit 6e01286 (BUG/MAJOR: threads/freq_ctr: fix lock on freq counters) attempted to fix the loop using volatile but that doesn't work depending on the level of optimization, resulting in situations where the threads could remain looping forever. Here we use memory barriers between reads to enforce a strict ordering and the asm code produced does exactly what the C code does and works perfectly, with a 3-digit measurement accuracy observed during a test. commit b29dc95a9748732898996540768bbed7a01f9a03 Author: Willy Tarreau Date: Tue Oct 31 18:00:20 2017 +0100 MINOR: threads: add a portable barrier for threads and non-threads HA_BARRIER() is just a simple memory barrier to prevent the compiler from reordering our code. commit 2510f702f923ae83f8324bc94703fc77a37601bc Author: Willy Tarreau Date: Tue Oct 31 17:14:16 2017 +0100 MINOR: h1: add a function to measure the trailers length This is needed in the H2->H1 gateway so that we know how long the trailers block is in chunked encoding. It returns the number of bytes, or 0 if some are missing, or -1 in case of parse error. commit f65610a83dbddb7a7a42fa95c6ae7dd95f718d7d Author: Willy Tarreau Date: Tue Oct 31 16:06:06 2017 +0100 CLEANUP: threads: rename process_mask to thread_mask It was a leftover from the last cleaning session; this mask applies to threads and calling it process_mask is a bit confusing. It's the same in fd, task and applets. commit 5f4a47b70125c1f5ca0e6e6a72774a94ddbc0b37 Author: Willy Tarreau Date: Tue Oct 31 15:59:32 2017 +0100 CLEANUP: threads: replace the last few 1UL< Date: Tue Oct 31 13:32:10 2017 +0100 MINOR: ssl: Remove the global allow-0rtt option. commit d16bfe6c01d50d4dee8e788a82dd241f68419ae6 Author: Olivier Houchard Date: Tue Oct 31 15:21:19 2017 +0100 BUG/MINOR: dns: Fix SRV records with the new thread code. srv_set_fqdn() may be called with the DNS lock already held, but tries to lock it anyway. So, add a new parameter to let it know if it was already locked or not; commit a5e0590b80502b29d38ce2b704231bdcd103cda5 Author: Willy Tarreau Date: Tue Oct 31 15:45:42 2017 +0100 BUILD: stick-tables: silence an uninitialized variable warning Commit 819fc6f ("MEDIUM: threads/stick-tables: handle multithreads on stick tables") introduced a valid warning about an uninitialized return value in stksess_kill_if_expired(). It just happens that this result is never used, so let's turn the function back to void as previously. commit 99aad9295b76e5de0f4c6faf321ce69d51beb02f Author: Christopher Faulet Date: Tue Oct 31 09:03:51 2017 +0100 BUG/MAJOR: threads/time: Store the time deviation in an 64-bits integer In function tv_update_date, we keep an offset reprenting the time deviation to adjust the system time. At every call, we check if this offset must be updated or not. Of course, It must be shared by all threads. It was store in a timeval. But it cannot be atomically updated. So now, instead, we store it in a 64-bits integer. And in tv_update_date, we convert this integer in a timeval. Once updated, it is converted back in an integer to be atomically stored. To store a tv_offset into an integer, we use 32 bits from tv_sec and 32 bits tv_usec to avoid shift operations. commit 6e0128630b80d691c915cba5cec43db59d191898 Author: Emeric Brun Date: Mon Oct 30 18:04:28 2017 +0100 BUG/MAJOR: threads/freq_ctr: fix lock on freq counters. The wrong bit was set to keep the lock on freq counter update. And the read functions were re-worked to use volatile. Moreover, when a freq counter is updated, it is now rotated only if the current counter is in the past (now.tv_sec > ctr->curr_sec). It is important with threads because the current time (now) is thread-local. So, rounded to the second, the time may vary by more or less 1 second. So a freq counter rotated by one thread may be see 1 second in the future. In this case, it is updated but not rotated. commit a1ae7e81cd2553472a19dd3bce06590ea6ac06d0 Author: Christopher Faulet Date: Wed Oct 25 17:55:27 2017 +0200 MAJOR: threads: Offically enable the threads support in HAProxy Now, USE_THREAD option is implicitly enabled when HAProxy is compiled, for targets linux2628 and freebsd. To enable it for other targets, you can set "USE_THREAD=1" explicitly on the command line. And to disable it explicitly, you must set "USE_THREAD=" on the command line. Now, to be clear. This does not means it is bug free, far from that. But it seems stable enough to be tested. You can try to experiment it and to report bugs of course by setting nbthread parameter. By leaving it to 1 (or not using it at all), it should be as safe as an HAProxy compiled without threads. Between the commit "MINOR: threads: Prepare makefile to link with pthread" and this one, the feature was in development and really unstable. It could be hard to track a bug using a bisect for all these commits. commit 1bc04c766432a9180cb011f7ca5390fc77789223 Author: Christopher Faulet Date: Sun Oct 29 20:14:08 2017 +0100 BUG/MINOR: threads: Add missing THREAD_LOCAL on static here and there commit cd7879adc2c408cd607bb287318d8152eb059872 Author: Christopher Faulet Date: Fri Oct 27 13:53:47 2017 +0200 BUG/MEDIUM: threads: Run the poll loop on the main thread too There was a flaw in the way the threads was created. the main one was just used to create all the others and just wait to exit. Now, it is used to run a poll loop. So we only create nbthread-1 threads. This also fixes a bug about the compression filter when there is only 1 thread (nbthread == 1 or no threads support). The bug was in the way thread-local resources was initialized. per-thread init/deinit callbacks were never called for the main process. So, with nthread set to 1, some buffers remained uninitialized. commit e8ca434074a0a5e2319b2ced6c780f9228d07e01 Author: Christopher Faulet Date: Wed Oct 25 17:23:02 2017 +0200 MINOR: threads: Don't start when device a detection module is used For now, we don't know if device detection modules (51degrees, deviceatlas and wurfl) are thread-safe or not. So HAproxy exits with an error when you try to use one of them with nbthread greater than 1. We will ask to maintainers of these modules to make them thread-safe or to give us hints to do so. commit 9f0b45852550088b9dc04224c33ddde387469d4b Author: Emeric Brun Date: Mon Oct 23 14:39:51 2017 +0200 MEDIUM: threads/server: Use the server lock to protect health check and cli concurrency commit c2a89a6aeda57207514bc9c719ef294ba1e33816 Author: Christopher Faulet Date: Mon Oct 23 15:54:24 2017 +0200 MINOR: threads/mailers: Add a lock to protect queues of email alerts commit 88ce5d18a58950ff525a1e6267303f42bb1e6f4d Author: Christopher Faulet Date: Fri Oct 20 15:41:18 2017 +0200 MINOR: threads/checks: Set the task process_mask when a check is executed Tasks used to process checks are created to be processed by any threads. But, once a check is started, we must be sure to be sticky on the running thread because I/O will be also sticky on it. This is a requirement for now: Tasks and I/O handlers linked to the same session must be executed on the same thread. commit cfda8476436ca90a8ebee67c3d84bf8f69ac56b6 Author: Christopher Faulet Date: Fri Oct 20 15:40:23 2017 +0200 MINOR: threads/checks: Add a lock to protect the pid list used by external checks commit 6251902e6731182c5bec30e4d1fa94c9fe0def22 Author: Christopher Faulet Date: Mon Oct 16 15:49:32 2017 +0200 MINOR: threads: Add thread-map config parameter in the global section By default, no affinity is set for threads. To bind threads on CPU, you must define a "thread-map" in the global section. The format is the same than the "cpu-map" parameter, with a small difference. The process number must be defined, with the same format than cpu-map ("all", "even", "odd" or a number between 1 and 31/63). A thread will be bound on the intersection of its mapping and the one of the process on which it is attached. If the intersection is null, no specific bind will be set for the thread. commit b2812a624096280489a0c7a1aabfbe1c94884d12 Author: Christopher Faulet Date: Wed Oct 4 16:17:58 2017 +0200 MEDIUM: thread/dns: Make DNS thread-safe commit 24289f2e07d2a92b1376b851b0c8878d957b38b5 Author: Christopher Faulet Date: Mon Sep 25 14:48:02 2017 +0200 MEDIUM: thread/spoe: Make the SPOE thread-safe Because there is not migration mechanism yet, all runtime information about an SPOE agent are thread-local and async exchanges with agents are disabled when we have serveral threads. Howerver, pipelining is still available. So for now, the thread part of the SPOE is pretty simple. commit 738a6d76f6afc92e53776f6aa9f8d8493598dc8a Author: Thierry FOURNIER Date: Mon Jul 17 00:14:07 2017 +0200 MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls commit 952939d294f5b38c7e2c55171f8844e1ffc97f69 Author: Thierry FOURNIER Date: Fri Sep 1 14:17:32 2017 +0200 MEDIUM: threads/xref: Convert xref function to a thread safe model Ensure that the unlink is done safely between thread and that the peer struct will not destroy between the usage of the peer. commit 94a6bfce9b714f13dfd94c2f5df3666e2601c053 Author: Thierry FOURNIER Date: Wed Jul 12 12:10:44 2017 +0200 MEDIUM: threads/lua: Cannot acces to the socket if we try to access from another thread. We have two y for nsuring that the data is not concurently manipulated: - locks - running task on the same thread. locks are expensives, it is better to avoid it. This patch cecks that the Lua task run on the same thread that the stream associated to the coprocess. TODO: in a next version, the error should be replaced by a yield and thread migration request. commit 4325ab727ca4ba07a68d37bd35c464d46b7f8cdb Author: Thierry FOURNIER Date: Wed Jul 12 11:53:38 2017 +0200 MEDIUM: threads/lua: Ensure that the launched tasks runs on the same threads than me The applet manipulates the session and its buffers. We have two methods for ensuring that the memory of the session will not change during its manipulation by the task: 1 - adding mutex 2 - running on the same threads than the task. The second point is smart because it cannot lock the execution of another thread. commit 61ba0e2b6d073126a857f71e01068ef319f8dfb4 Author: Thierry FOURNIER Date: Wed Jul 12 11:41:21 2017 +0200 MEDIUM: threads/lua: Add locks around the Lua execution parts. Note that the Lua processing is not really thread safe. It provides heavy system which consists to add our own lock function in the Lua code and recompile the library. This system will probably not accepted by maintainers of various distribs. Our main excution point of the Lua is the function lua_resume(). A quick looking on the Lua sources displays a lua_lock() a the start of function and a lua_unlock() at the end of the function. So I conclude that the Lua thread safe mode just perform a mutex around all execution. So I prefer to do this in the HAProxy code, it will be easier for distro maintainers. Note that the HAProxy lua functions rounded by the macro SET_SAFE_LJMP and RESET_SAFE_LJMP manipulates the Lua stack, so it will be careful to set mutex around these functions. commit ffbad790276486fe16326571d9d406e4188aebdf Author: Thierry FOURNIER Date: Wed Jul 12 11:39:04 2017 +0200 MEDIUM: threads/lua: Makes the jmpbuf and some other buffers local to the current thread. The jmpbuf contains pointer on the stack memory address currently use when the jmpbuf is set. So the information is local to each thread. The struct field is too big to put it in the stack, but it is used as buffer for retriving stats values. So, this buffer si local to each threads. Each function using this buffer, use it whithout break (yield) so, the consistency of local buffer is ensured. commit 8ca3b4bc46748dbf16865e17c43551bb62073ba6 Author: Christopher Faulet Date: Tue Jul 25 11:07:15 2017 +0200 MEDIUM: threads/compression: Make HTTP compression thread-safe commit f2273728976083e9661107765a9b942af941cc9e Author: Christopher Faulet Date: Thu Jul 27 16:58:42 2017 +0200 MINOR: threads/filters: Update trace filter to add _per_thread callbacks commit 71a6a8efaa6770a505788f872174c1b82c31c1f5 Author: Christopher Faulet Date: Thu Jul 27 16:33:28 2017 +0200 MEDIUM: threads/filters: Add init/deinit callback per thread Now, it is possible to define init_per_thread and deinit_per_thread callbacks to deal with ressources allocation for each thread. This is the filter responsibility to deal with concurrency. This is also the filter responsibility to know if HAProxy is started with some threads. A good way to do so is to check "global.nbthread" value. If it is greater than 1, then _per_thread callbacks will be called. commit e95f2c3ef5489143286e89b288d4950c1d95eb21 Author: Christopher Faulet Date: Mon Jul 24 16:30:34 2017 +0200 MEDIUM: thread/vars: Make vars thread-safe A RW lock has been added to the vars structure to protect each list of variables. And a global RW lock is used to protect registered names. When a varibable is fetched, we duplicate sample data because the variable could be modified by another thread. commit 94b712337d40c6b5bdf54c18025af1caf06ca7ac Author: Christopher Faulet Date: Thu Oct 12 09:49:09 2017 +0200 MEDIUM: threads/freq_ctr: Make the frequency counters thread-safe When a frequency counter must be updated, we use the curr_sec/curr_tick fields as a lock, by setting the MSB to 1 in a compare-and-swap to lock and by reseting it to unlock. And when we need to read it, we loop until the counter is unlocked. This way, the frequency counters are thread-safe without any external lock. It is important to avoid increasing the size of many structures (global, proxy, server, stick_table). commit b5997f740b21ebb197e10a0f2fe9dc13163e1772 Author: Emeric Brun Date: Mon Jul 3 11:34:05 2017 +0200 MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe. commit 8ba59148aec9992943907a137688e8397da2edfa Author: Christopher Faulet Date: Tue Jun 27 15:43:53 2017 +0200 MEDIUM: threads/queue: Make queues thread-safe The list of pending connections are now protected using the proxy or server lock, depending on the context. commit 821bb9beaa40b4793ad4617fdf4fdab5ad4a4c7b Author: Emeric Brun Date: Thu Jun 15 16:37:39 2017 +0200 MAJOR: threads/ssl: Make SSL part thread-safe First, OpenSSL is now initialized to be thread-safe. This is done by setting 2 callbacks. The first one is ssl_locking_function. It handles the locks and unlocks. The second one is ssl_id_function. It returns the current thread id. During the init step, we create as much as R/W locks as needed, ie the number returned by CRYPTO_num_locks function. Next, The reusable SSL session in the server context is now thread-local. Shctx is now also initialized if HAProxy is started with several threads. And finally, a global lock has been added to protect the LRU cache used to store generated certificates. The function ssl_sock_get_generated_cert is now deprecated because the retrieved certificate can be removed by another threads in same time. Instead, a new function has been added, ssl_sock_assign_generated_cert. It must be used to search a certificate in the cache and set it immediatly if found. commit 6b35e9bfbf0f37c8a584c7aabb475afb81b7008c Author: Emeric Brun Date: Fri Jun 30 16:23:45 2017 +0200 MEDIUM: threads/stream: Make streams list thread safe Adds a global lock to protect the full streams list used to dump sessions on stats socket. commit a1dd243adb0b9ebf78263a5f2ec5729e903abe32 Author: Emeric Brun Date: Wed Jun 21 15:42:52 2017 +0200 MAJOR: threads/buffer: Make buffer wait queue thread safe Adds a global lock to protect the buffer wait queue. commit 80527f5bb605ab7a50533175c611686e62fe23b6 Author: Emeric Brun Date: Mon Jun 19 17:46:37 2017 +0200 MAJOR: threads/peers: Make peers thread safe A lock is used to protect accesses to a peer structure. A the lock is taken in the applet handler when the peer is identified and released living the applet handler. In the scheduling task for peers section, the lock is taken for every listed peer and released at the end of the process task function. The peer 'force shutdown' function was also re-worked. commit 1138fd0c576d8ac9ba28a57da3463de72022bea5 Author: Emeric Brun Date: Mon Jun 19 12:38:55 2017 +0200 MAJOR: threads/applet: Handle multithreading for applets A global lock has been added to protect accesses to the list of active applets. A process mask has also been added on each applet. Like for FDs and tasks, it is used to know which threads are allowed to process an applet. Because applets are, most of time, linked to a session, it should be sticky on the same thread. But in all cases, it is the responsibility of the applet handler to lock what have to be protected in the applet context. commit 272e252e61cbb34959d111ed067caf2cf0d11f4a Author: Emeric Brun Date: Thu Jun 15 11:53:49 2017 +0200 MINOR: threads/regex: Change Regex trash buffer into a thread local variable commit 8c1aaa201aa2b57c98285bdda7fbff8005f16f96 Author: Emeric Brun Date: Thu Jun 15 11:30:06 2017 +0200 MEDIUM: threads/http: Make http_capture_bad_message thread-safe This is done by passing the right stream's proxy (the frontend or the backend, depending on the context) to lock the error snapshot used to store the error info. commit e5c918bcefbaa1bd203bc58ec7c3c6b2e52d70a5 Author: Emeric Brun Date: Wed Jun 14 14:15:36 2017 +0200 MINOR: threads/sample: Change temp_smp into a thread local variable commit 819fc6f5636b08e252b46c5de4fec406c13b7505 Author: Emeric Brun Date: Tue Jun 13 19:37:32 2017 +0200 MEDIUM: threads/stick-tables: handle multithreads on stick tables The stick table API was slightly reworked: A global spin lock on stick table was added to perform lookup and insert in a thread safe way. The handling of refcount on entries is now handled directly by stick tables functions under protection of this lock and was removed from the code of callers. The "stktable_store" function is no more externalized and users should now use "stktable_set_entry" in any case of insertion. This last one performs a lookup followed by a store if not found. So the code using "stktable_store" was re-worked. Lookup, and set_entry functions automatically increase the refcount of the returned/stored entry. The function "sticktable_touch" was renamed "sticktable_touch_local" and is now able to decrease the refcount if last arg is set to true. It is allowing to release the entry without taking the lock twice. A new function "sticktable_touch_remote" is now used to insert entries coming from remote peers at the right place in the update tree. The code of peer update was re-worked to use this new function. This function is also able to decrease the refcount if wanted. The function "stksess_kill" also handle a parameter to decrease the refcount on the entry. A read/write lock is added on each entry to protect the data content updates of the entry. commit 5b51755aefd1dfed507772fdd13fae1ce572c2bd Author: Christopher Faulet Date: Fri Jun 9 14:17:53 2017 +0200 MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe A lock for LB parameters has been added inside the proxy structure and atomic operations have been used to update server variables releated to lb. The only significant change is about lb_map. Because the servers status are updated in the sync-point, we can call recalc_server_map function synchronously in map_set_server_status_up/down function. commit 5d42e099c565c8002cf0f6fd0f29e3e6fbfdc723 Author: Christopher Faulet Date: Mon Oct 16 12:00:40 2017 +0200 MINOR: threads/server: Add a lock to deal with insert in updates_servers list This list is used to save changes on the servers state. So when serveral threads are used, it must be locked. The changes are then applied in the sync-point. To do so, servers_update_status has be moved in the sync-point. So this is useless to lock it at this step because the sync-point is a protected area by iteself. commit 29f77e846bd9cad8b9984dc5304e6edcaa4f435d Author: Christopher Faulet Date: Thu Jun 8 14:04:45 2017 +0200 MEDIUM: threads/server: Add a lock per server and atomically update server vars The server's lock is use, among other things, to lock acces to the active connection list of a server. commit 40a007cf2a387bc2c4257840bafc54290297cbf9 Author: Christopher Faulet Date: Mon Jul 3 15:41:01 2017 +0200 MEDIUM: threads/server: Make connection list (priv/idle/safe) thread-safe For now, we have a list of each type per thread. So there is no need to lock them. This is the easiest solution for now, but not the best one because there is no sharing between threads. An idle connection on a thread will not be able be used by a stream on another thread. So it could be a good idea to rework this patch later. commit ff8abcd31d390c5bc3b72c8acf991144c91c28ef Author: Christopher Faulet Date: Fri Jun 2 15:33:24 2017 +0200 MEDIUM: threads/proxy: Add a lock per proxy and atomically update proxy vars Now, each proxy contains a lock that must be used when necessary to protect it. Moreover, all proxy's counters are now updated using atomic operations. commit 8d8aa0d681c001891839588c0d51fa3cc9f652c7 Author: Christopher Faulet Date: Tue May 30 15:36:50 2017 +0200 MEDIUM: threads/listeners: Make listeners thread-safe First, we use atomic operations to update jobs/totalconn/actconn variables, listener's nbconn variable and listener's counters. Then we add a lock on listeners to protect access to their information. And finally, listener queues (global and per proxy) are also protected by a lock. Here, because access to these queues are unusal, we use the same lock for all queues instead of a global one for the global queue and a lock per proxy for others. commit b79a94c9f3c6b741a219decfeb004896cd978795 Author: Christopher Faulet Date: Tue May 30 15:34:30 2017 +0200 MEDIUM: threads/signal: Add a lock to make signals thread-safe A global lock has been added to protect the signal processing. So when a signal it triggered, only one thread will catch it. commit c60def8368f73e578685c4f02df7d6dbe63c85d3 Author: Emeric Brun Date: Wed Sep 27 14:59:38 2017 +0200 MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts. commit 209d02a257d795c3b682c858277e7cbc738aefdf Author: Christopher Faulet Date: Fri Oct 27 23:01:38 2017 +0200 WIP: SQUASH WITH SYNC POINT commit 63e2ce61a84670ea46637fb7026fde0b6af9372b Author: Christopher Faulet Date: Fri Jun 2 14:36:39 2017 +0200 MINOR: threads/polling: pollers now handle FDs depending on the process mask commit 8aae8b1d610479b22860b75121abbcb3b1d04c76 Author: Christopher Faulet Date: Wed Aug 30 10:56:25 2017 +0200 MINOR: threads/fd: Process cached events of FDs depending on the process mask commit 36716a7fec1d84ab4c590dfb038aa30a4ad92b57 Author: Christopher Faulet Date: Tue May 30 11:07:16 2017 +0200 MEDIUM: threads/fd: Initialize the process mask during the call to fd_insert Listeners will allow any threads to process the corresponding fd. But for other FDs, we limit the processing to the current thread. commit a7c5d43085d93d7d02d5181d696eaeaf9905633f Author: Christopher Faulet Date: Tue May 30 11:05:09 2017 +0200 MINOR: threads/fd: Add a mask of threads allowed to process on each fd in fdtab array commit d4604adeaa8c86c378c8042487637db64e0b9910 Author: Christopher Faulet Date: Mon May 29 10:40:41 2017 +0200 MAJOR: threads/fd: Make fd stuffs thread-safe Many changes have been made to do so. First, the fd_updt array, where all pending FDs for polling are stored, is now a thread-local array. Then 3 locks have been added to protect, respectively, the fdtab array, the fd_cache array and poll information. In addition, a lock for each entry in the fdtab array has been added to protect all accesses to a specific FD or its information. For pollers, according to the poller, the way to manage the concurrency is different. There is a poller loop on each thread. So the set of monitored FDs may need to be protected. epoll and kqueue are thread-safe per-se, so there few things to do to protect these pollers. This is not possible with select and poll, so there is no sharing between the threads. The poller on each thread is independant from others. Finally, per-thread init/deinit functions are used for each pollers and for FD part for manage thread-local ressources. Now, you must be carefull when a FD is created during the HAProxy startup. All update on the FD state must be made in the threads context and never before their creation. This is mandatory because fd_updt array is thread-local and initialized only for threads. Because there is no pollers for the main one, this array remains uninitialized in this context. For this reason, listeners are now enabled in run_thread_poll_loop function, just like the worker pipe. commit b349e48ede90afe1c7b37741b87694429ddfc4da Author: Christopher Faulet Date: Tue Aug 29 09:52:38 2017 +0200 MEDIUM: threads/pool: Make pool thread-safe by locking all access to a pool A lock has been added for each memory pool. It is used to protect the pool during allocations and releases. It is also used when pool info are dumped. commit f8188c69fa5d19e3e12ef92a30cd154c7fa514e1 Author: Christopher Faulet Date: Fri Jun 2 16:20:16 2017 +0200 MEDIUM: threads/logs: Make logs thread-safe log buffers and static variables used in log functions are now thread-local. So there is no need to lock anything to log messages. Moreover, per-thread init/deinit functions are now used to initialize these buffers. commit 9a6557178164996f15454301b9f94b0e4720d249 Author: Christopher Faulet Date: Thu May 11 11:00:15 2017 +0200 MEDIUM: threads/time: Many global variables from time.h are now thread-local commit 6adad11283458f7dafc0b52008840b788072eac6 Author: Christopher Faulet Date: Fri Apr 21 16:47:03 2017 +0200 MEDIUM: threads/chunks: Transform trash chunks in thread-local variables So, per-thread init/deinit functions are registered to allocate/release them. commit ba39f23a9da91662922bef96b52fc250e0018ba8 Author: Christopher Faulet Date: Tue Aug 29 14:43:04 2017 +0200 MEDIUM: threads/buffers: Define and register per-thread init/deinit functions For now, only the swap_buffer is handled in these functions. Moreover, swap_buffer has been changed to be a thread-local variable. commit dc628a3a76907caecd653596527086d7054ce04b Author: Christopher Faulet Date: Thu Oct 19 11:59:44 2017 +0200 MINOR: threads: Define the sync-point inside run_poll_loop The function sync_poll_loop is called at the end of each loop inside run_poll_loop function. It is a protected area where all threads have a chance to execute tricky tasks with the warranty that no concurrent access is possible. Of course, it comes with a cost because all threads must be syncrhonized. So changes must be uncommon. commit 1d17c10d8b82662be48bf9f7a4fc7df910a08d28 Author: Christopher Faulet Date: Tue Aug 29 15:38:48 2017 +0200 MAJOR: threads: Start threads to experiment multithreading [WARNING] For now, HAProxy is not thread-safe, so from this commit, it will be broken for a while, when compiled with threads. When nbthread parameter is greater than 1, HAProxy will create the corresponding number of threads. If nbthread is set to 1, nothing should be done. So if there are concurrency issues (and be sure there will be, unfortunatly), an obvious workaround is to disable the multithreading... Each created threads will run a polling loop. So, in a certain way, it is pretty similar to the nbproc mode ("outside" the bugs and the lock contention). Nevertheless, there are an init and a deinit steps for each thread to deal with per-thread allocation. Each thread has a tid (thread-id), numbered from 0 to (nbtread-1). It is used in many place to do bitwise operations or to improve debugging information. commit 339fff8a18046aada616ac41437451b1f180534f Author: Christopher Faulet Date: Thu Oct 19 11:59:15 2017 +0200 MEDIUM: threads: Adds a set of functions to handle sync-point A sync-point is a protected area where you have the warranty that no concurrency access is possible. It is implementated as a thread barrier to enter in the sync-point and another one to exit from it. Inside the sync-point, all threads that must do some syncrhonous processing will be called one after the other while all other threads will wait. All threads will then exit from the sync-point at the same time. A sync-point will be evaluated only when necessary because it is a costly operation. To limit the waiting time of each threads, we must have a mechanism to wakeup all threads. This is done with a pipe shared by all threads. By writting in this pipe, we will interrupt all threads blocked on a poller. The pipe is then flushed before exiting from the sync-point. commit be0faa2e47939e2672266a1a23b8a6de542b520e Author: Christopher Faulet Date: Tue Aug 29 15:37:10 2017 +0200 MINOR: threads: Add nbthread parameter It is only parsed and initialized for now. It will be used later. This parameter is only available when support for threads was built in. commit 415f611ff48916cb46a8f29c7510667d3fd01a1a Author: Christopher Faulet Date: Tue Jul 25 16:52:58 2017 +0200 MINOR: threads: Add mechanism to register per-thread init/deinit functions hap_register_per_thread_init and hap_register_per_thread_deinit functions has been added to register functions to do, for each thread, respectively, some initialization and deinitialization. These functions are added in the global lists per_thread_init_list and per_thread_deinit_list. These functions are called only when HAProxy is started with more than 1 thread (global.nbthread > 1). commit 1a2b56ea8e33805996994e64aa74c348cf543116 Author: Christopher Faulet Date: Thu Oct 12 16:09:09 2017 +0200 MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks. commit 7122ab31b195edb511fecf9c20904701970b195f Author: Emeric Brun Date: Fri Jul 7 10:26:46 2017 +0200 MINOR: threads: Add atomic-ops and plock includes in import dir atomic-ops header contains some low-level functions to do atomic operations. These operations are used by the progressive locks (plock). commit e9bd686b68279406b954239ce38f5bf1b24462be Author: Christopher Faulet Date: Tue Aug 29 09:51:52 2017 +0200 MINOR: threads: Add THREAD_LOCAL macro When compiled with threads support, this marco is set to __thread. Else it is empty. commit 5f271850bdc6c3c5c05be0b2ba875f5ea7012f83 Author: Emeric Brun Date: Mon Jun 26 18:41:42 2017 +0200 MINOR: threads: Prepare makefile to link with pthread USE_THREAD option has been added to enable the compilation with the experimental support of threads . Of course for now, there is nothing. And for a while, HAProxy will be unstable. When we will be confident enough, this option will be removed. For this implementation and probably for a while, only the pthread library will be supported. commit e3a5e35708d98b2bf906bf0fe719b903c266d215 Author: Christopher Faulet Date: Tue Oct 24 13:53:54 2017 +0200 MINOR: startup: Extend the scope the MODE_STARTING flag Now, MODE_STARTING is set at the begining to init function and it is removed just before the polling loop. So more alerts or warnings are saved. commit c1b730a41a56f34308a8d5c8796760b04f4f007b Author: Christopher Faulet Date: Tue Oct 24 12:00:51 2017 +0200 MINOR: cli: Add "show startup-logs" command This command will dump all startup_logs buffer containing all alerts and warnings emitted during HAProxy startup. commit d46963865e9dda276b63bd841912e1a7078b1845 Author: Christopher Faulet Date: Tue Oct 24 11:44:05 2017 +0200 MINOR: log: Save alerts and warnings emitted during HAProxy startup Because we can't always display the standard error messages when HAProxy is started, all alerts and warnings emitted during the startup will now be saved in a buffer. It can also be handy to store these messages just in case you missed something during the startup To implement this feature, Alert and Warning functions now relies on display_message. The difference is just on conditions to call this function and it remains unchanged. In display_message, if MODE_STARTING flag is set, we save the message. commit 93a518f02a8048fabd34c2f364e3ca9e42e4db06 Author: Christopher Faulet Date: Tue Oct 24 11:25:33 2017 +0200 MINOR: standard: Add memvprintf function Now memprintf relies on memvprintf. This new function does exactly what memprintf did before, but it must be called with a va_list instead of a variable number of arguments. So there is no change for every functions using memprintf. But it is now also possible to have same functionnality from any function with variadic arguments. commit 31dff9b1bde9263c331d197a24e485e89934cb9b Author: Christopher Faulet Date: Mon Oct 23 15:45:20 2017 +0200 MINOR: mailers: Use pools to allocate email alerts and its tcpcheck_rules commit 0108bb3e401e515d4c304600e822af0de2e45043 Author: Christopher Faulet Date: Fri Oct 20 21:34:32 2017 +0200 MEDIUM: mailers: Init alerts during conf parsing and refactor their processing Email alerts relies on checks to send emails. The link between a mailers section and a proxy was resolved during the configuration parsing, But initialization was done when the first alert is triggered. This implied memory allocations and tasks creations. With this patch, everything is now initialized during the configuration parsing. So when an alert is triggered, only the memory required by this alert is dynamically allocated. Moreover, alerts processing had a flaw. The task handler used to process alerts to be sent to the same mailer, process_email_alert, was designed to give back the control to the scheduler when an alert was sent. So there was a delay between the sending of 2 consecutives alerts (the min of "proxy->timeout.connect" and "mailer->timeout.mail"). To fix this problem, now, we try to process as much queued alerts as possible when the task is woken up. commit de1a75b86954ede54cdc82cc913ef6b17e2db6f5 Author: Christopher Faulet Date: Mon Oct 23 15:38:19 2017 +0200 BUG/MINOR: mailers: Fix a memory leak when email alerts are released An email alert contains a list of tcpcheck_rule. Each one is dynamically allocated, just like its internal members. So, when an email alerts is freed, we must be sure to properly free each tcpcheck_rule too. This patch must be backported in 1.7 and 1.6. commit 67957bd59e3ccd7be1174b50a5bf402bd676ecb0 Author: Christopher Faulet Date: Wed Sep 27 11:00:59 2017 +0200 MAJOR: dns: Refactor the DNS code This is a huge patch with many changes, all about the DNS. Initially, the idea was to update the DNS part to ease the threads support integration. But quickly, I started to refactor some parts. And after several iterations, it was impossible for me to commit the different parts atomically. So, instead of adding tens of patches, often reworking the same parts, it was easier to merge all my changes in a uniq patch. Here are all changes made on the DNS. First, the DNS initialization has been refactored. The DNS configuration parsing remains untouched, in cfgparse.c. But all checks have been moved in a post-check callback. In the function dns_finalize_config, for each resolvers, the nameservers configuration is tested and the task used to manage DNS resolutions is created. The links between the backend's servers and the resolvers are also created at this step. Here no connection are kept alive. So there is no needs anymore to reopen them after HAProxy fork. Connections used to send DNS queries will be opened on demand. Then, the way DNS requesters are linked to a DNS resolution has been reworked. The resolution used by a requester is now referenced into the dns_requester structure and the resolution pointers in server and dns_srvrq structures have been removed. wait and curr list of requesters, for a DNS resolution, have been replaced by a uniq list. And Finally, the way a requester is removed from a DNS resolution has been simplified. Now everything is done in dns_unlink_resolution. srv_set_fqdn function has been simplified. Now, there is only 1 way to set the server's FQDN, independently it is done by the CLI or when a SRV record is resolved. The static DNS resolutions pool has been replaced by a dynamoc pool. The part has been modified by Baptiste Assmann. The way the DNS resolutions are triggered by the task or by a health-check has been totally refactored. Now, all timeouts are respected. Especially hold.valid. The default frequency to wake up a resolvers is now configurable using "timeout resolve" parameter. Now, as documented, as long as invalid repsonses are received, we really wait all name servers responses before retrying. As far as possible, resources allocated during DNS configuration parsing are releases when HAProxy is shutdown. Beside all these changes, the code has been cleaned to ease code review and the doc has been updated. commit ff88efbd7a187bee28f26eeea432e807f3798851 Author: Christopher Faulet Date: Tue Oct 3 16:00:57 2017 +0200 BUG/MINOR: dns: Fix CLI keyword declaration The cli command to show resolvers stats is in conflict with the command to show proxies and servers stats. When you use the command "show stat resolvers [id]", instead of printing stats about resolvers, you get the stats about all proxies and servers. Now, to avoid conflict, to print resolvers stats, you must use the following command: show resolvers [id] This patch must be backported in 1.7. commit 344c4ab6a9b3b39aa3593ac46bd746d13ee99b9b Author: Christopher Faulet Date: Fri Sep 22 10:20:13 2017 +0200 MEDIUM: spoe/rules: Process "send-spoe-group" action The messages processing is done using existing functions. So here, the main task is to find the SPOE engine to use. To do so, we loop on all filter instances attached to the stream. For each, we check if it is a SPOE filter and, if yes, if its name is the one used to declare the "send-spoe-group" action. We also take care to return an error if the action processing is interrupted by HAProxy (because of a timeout or an error at the HAProxy level). This is done by checking if the flag ACT_FLAG_FINAL is set. The function spoe_send_group is the action_ptr callback ot commit 58d036858887a290f88bb3400e7cbf80f1ca66e2 Author: Christopher Faulet Date: Thu Sep 21 16:57:24 2017 +0200 MINOR: spoe: Add a generic function to encode a list of SPOE message So it will be possible to encode messages chained by event or by group. For now, it is only possible to do it by event. commit c718b82dfe4698be791c3382a6e9a6f889d46142 Author: Christopher Faulet Date: Thu Sep 21 16:50:56 2017 +0200 MINOR: spoe: Add a type to qualify the message list during encoding Because we can have messages chained by event or by group, we need to have a way to know which kind of list we manipulate during the encoding. So 2 types of list has been added, SPOE_MSGS_BY_EVENT and SPOE_MSGS_BY_GROUP. And the right type is passed when spoe_encode_messages is called. commit 10e376727a51229a56e7e644170aadf7462607f6 Author: Christopher Faulet Date: Thu Sep 21 16:38:22 2017 +0200 MINOR: spoe: Move message encoding in its own function Instead of having a big function to encode a list of messages, now we have a function to unroll the list and a function to encode the message itself. commit 76c09ef8de396e7a269578f430b1c9b29b5c8d04 Author: Christopher Faulet Date: Thu Sep 21 11:03:52 2017 +0200 MEDIUM: spoe/rules: Add "send-spoe-group" action for tcp/http rules This action is used to trigger sending of a group of SPOE messages. To do so, the SPOE engine used to send messages must be defined, as well as the SPOE group to send. Of course, the SPOE engine must refer to an existing SPOE filter. If not engine name is provided on the SPOE filter line, the SPOE agent name must be used. For example: http-request send-spoe-group my-engine some-group This action is available for "tcp-request content", "tcp-response content", "http-request" and "http-response" rulesets. It cannot be used for tcp connection/session rulesets because actions for these rulesets cannot yield. For now, the action keyword is parsed and checked. But it does nothing. Its processing will be added in another patch. commit 11610f3b5afe52589792383fe8349ff4ea40228a Author: Christopher Faulet Date: Thu Sep 21 10:23:10 2017 +0200 MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group messages ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt ... groups ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group. commit 7ee8667c999e2c2a3c0341ecbff1c98cb0b913a2 Author: Christopher Faulet Date: Tue Sep 19 11:08:28 2017 +0200 MINOR: spoe: Check uniqness of SPOE engine names during config parsing The engine name is now kept in "spoe_config" struture. Because a SPOE filter can be declared without engine name, we use the SPOE agent name by default. Then, its uniqness is checked against all others SPOE engines configured for the same proxy. * TODO: Add documentation commit 57583e474e3ee73d960403f2515437b6349ab7f1 Author: Christopher Faulet Date: Mon Sep 4 15:41:09 2017 +0200 MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections. commit 1b421eab872384a56976059a8642f8a94aba367f Author: Christopher Faulet Date: Fri Sep 22 14:38:56 2017 +0200 MINOR: acl: Pass the ACLs as an explicit parameter of build_acl_cond So it is possible to use anothers ACLs to build ACL conditions than those of proxies. commit e4e830d909cb224e39c672df5fffbd02de448704 Author: Christopher Faulet Date: Mon Sep 18 14:51:41 2017 +0200 MINOR: action: Factorize checks on rules calling check_ptr if defined commit 29730ba570f17790193ede6dbf9edb259b434f3b Author: Christopher Faulet Date: Mon Sep 18 15:26:32 2017 +0200 MINOR: action: Add a functions to check http capture rules "check_http_req_capture" and "check_http_res_capture" functions have been added to check validity of "http-request capture" and "http-response capture" rules. Code for these functions come from cfgparse.c. commit 78880fb1968d82dadb2b3b08305ae71f780e702a Author: Christopher Faulet Date: Mon Sep 18 14:43:55 2017 +0200 MINOR: action: Add function to check rules using an action ACT_ACTION_TRK_* The function "check_trk_action" has been added to find and check the target table for rules using an action ACT_ACTION_TRK_*. commit 6d950b92cdf59b5d897b545fab347ec472c39db4 Author: Christopher Faulet Date: Mon Sep 18 15:12:39 2017 +0200 MINOR: action: Add a function pointer in act_rule struct to check its validity It is possible to define the field "act_rule.check_ptr" if you want to check the validity of a tcp/http rule. commit 4fce0d844752fbdd5e4613f99b23727f4829fdd6 Author: Christopher Faulet Date: Mon Sep 18 11:57:31 2017 +0200 MINOR: action: Use trk_idx instead of tcp/http_trk_idx So tcp_trk_idx and http_trk_idx have been removed. commit 7421b14c220e6639ec67b3ff372962c9537602b5 Author: Christopher Faulet Date: Mon Sep 18 11:54:51 2017 +0200 MINOR: action: Add trk_idx inline function It returns tracking index corresponding to an action ACT_ACTION_TRK_SC*. It will replace http_trk_idx and tcp_trk_idx. commit 94bb4c6a488f5f872d7ceacc1ce685aa02fd6598 Author: Christopher Faulet Date: Tue Sep 26 11:49:23 2017 +0200 BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded The same buffer is used for a request and its response. So we need to be sure to correctly reset info when the response is encoded. And here there was a bug. The pointer on the end of the frame was not updated. So it was not possible to encode a response bigger than the corresponding request. commit e1405e5f802123eec8270ab87d6e9637c40dfd6a Author: Christopher Faulet Date: Tue Sep 19 10:35:35 2017 +0200 BUG/MINOR: spoe: Don't compare engine name and SPOE scope when both are NULL SPOE filter can be declared without engine name. This is an optional parameter. But in this case, no scope must be used in the SPOE configuration file. So engine name and scope are both undefined, and, obviously, we must not try to compare them. This patch must be backported in 1.7. commit d22e83abd90a6b52101c0a9efb59186f56154572 Author: Willy Tarreau Date: Tue Oct 31 08:02:24 2017 +0100 MINOR: h1: store the status code in the H1 message It was painful not to have the status code available, especially when it was computed. Let's store it and ensure we don't claim content-length anymore on 1xx, only 0 body bytes. commit a3c77cfdd7d80ecd5f341913902410c5827f8840 Author: William Lallemand Date: Mon Oct 30 23:44:40 2017 +0100 MINOR: shctx: rename lock functions Rename lock functions to shctx_lock() and shctx_unlock() to be coherent with the new API. commit 4f45bb9c461f462290b77bf2511badb7a4453c0a Author: William Lallemand Date: Mon Oct 30 20:08:51 2017 +0100 MEDIUM: shctx: separate ssl and shctx This patch reorganize the shctx API in a generic storage API, separating the shared SSL session handling from its core. The shctx API only handles the generic data part, it does not know what kind of data you use with it. A shared_context is a storage structure allocated in a shared memory, allowing its usage in a multithread or a multiprocess context. The structure use 2 linked list, one containing the available blocks, and another for the hot locked blocks. At initialization the available list is filled with blocks of size . An space is initialized outside the list in case you need some specific storage. +-----------------------+--------+--------+--------+--------+---- | struct shared_context | extra | block1 | block2 | block3 | ... +-----------------------+--------+--------+--------+--------+---- <-------- maxblocks ---------> * blocksize The API allows to store content on several linked blocks. For example, if you allocated blocks of 16 bytes, and you want to store an object of 60 bytes, the object will be allocated in a row of 4 blocks. The API was made for LRU usage, each time you get an object, it pushes the object at the end of the list. When it needs more space, it discards The functions name have been renamed in a more logical way, the part regarding shctx have been prefixed by shctx_ and the functions for the shared ssl session cache have been prefixed by sh_ssl_sess_. commit ed0b5ad1aaca4312d26d1e54e9acb8e4f474ab42 Author: William Lallemand Date: Mon Oct 30 19:36:36 2017 +0100 REORG: shctx: move ssl functions to ssl_sock.c Move the ssl callback functions of the ssl shared session cache to ssl_sock.c. The shctx functions still needs to be separated of the ssl tree and data. commit 3f85c9aec8ea5bce1bd1119e142af872a99b0939 Author: William Lallemand Date: Mon Oct 9 16:30:50 2017 +0200 MEDIUM: shctx: allow the use of multiple shctx Add an shctx argument which permits to create new independent shctx area. commit 24a7a75be662bd111ee07e0265112e969fedc8aa Author: William Lallemand Date: Mon Oct 9 14:17:39 2017 +0200 REORG: shctx: move lock functions and struct Move locks functions to proto/shctx.h, and structures to types/shctx.h in order to simplify the split ssl/shctx. commit 83215a44b8ce6b6a8d4c132c70bc2a888ff46f3d Author: William Lallemand Date: Sun Sep 24 11:26:02 2017 +0200 MEDIUM: lists: list_for_each_entry{_safe}_from functions Add list_for_each_entry_from and list_for_each_entry_safe_from which allows to iterate in a list starting from a specific item. commit 2a97966b08f9d577236aaf3482489fc08422ecd2 Author: William Lallemand Date: Mon Sep 18 17:37:07 2017 +0200 CLEANUP: shctx: get ride of the shsess_packet{_hdr} structures This patch removes remaining structures and fields which were never used in the shctx code. commit 8ea0f38c75016a5ca4d5410355133f456b6db5c5 Author: Willy Tarreau Date: Mon Oct 30 19:31:59 2017 +0100 MEDIUM: h1: ensure that 1xx, 204 and 304 don't have a payload body It's important for the H2 to H1 gateway that the response parser properly clears the H1 message's body_len when seeing these status codes so that we don't hang waiting to transfer data that will not come. commit 404d978d402d6bb3a69fe1c9579d4dd79592f89d Author: Emmanuel Hocdet Date: Tue Oct 24 10:55:14 2017 +0200 MINOR: add ALPN information to send-proxy-v2 Send ALPN information in proxy-protocol-v2 if an alpn have been negotiated. commit 01da571e21b47ad6cd26d034477bf144f9b99f9f Author: Emmanuel Hocdet Date: Fri Oct 13 16:59:49 2017 +0200 MINOR: merge ssl_sock_get calls for log and ppv2 Merge ssl_sock_get_version and ssl_sock_get_proto_version. Change ssl_sock_get_cipher to be used in ppv2. commit 58118b43b12d89f5db6cacc5c62cb88d8bd767ed Author: Emmanuel Hocdet Date: Fri Oct 13 12:15:28 2017 +0200 MINOR: update proxy-protocol-v2 #define Report #define from doc/proxy-protocol.txt. commit 9679ac997a8d93e22f629b5a838e7f3aadc5fade Author: Olivier Houchard Date: Fri Oct 27 14:58:08 2017 +0200 MINOR: ssl: Don't abuse ssl_options. A bind_conf does contain a ssl_bind_conf, which already has a flag to know if early data are activated, so use that, instead of adding a new flag in the ssl_options field. commit 7c210e6aa87365b9b87b9d7e904b41f860f278b9 Author: Thierry FOURNIER Date: Fri Oct 27 14:13:51 2017 +0200 BUG/MINOR: lua: const attribute of a string is overridden If HAProxy is compiled without PCRE regexes, this can cause a write in const memory. The probability of a consequence is very low. commit ccaa7de72e847a920d9cbe29e6959af46450e4d9 Author: Olivier Houchard Date: Mon Oct 2 11:51:03 2017 +0200 MINOR: ssl/proto_http: Add keywords to take care of early data. Add a new sample fetch, "ssl_fc_has_early", a boolean that will be true if early data were sent, and a new action, "wait-for-handshake", if used, the request won't be forwarded until the SSL handshake is done. commit eff9a9ef950caa7327515a9f3063228dd9559b29 Author: Willy Tarreau Date: Fri Oct 27 11:06:11 2017 +0200 BUILD: Makefile: disable -Wunused-label It's becoming extremely tricky not to make gcc warn about unused labels with support for openssl 1.1 and 1.1.1, because some error paths only exist for certain versions. Latest patch causes a warning for me on 1.0.2. There is no real point it warning about an unused error label so let's disable this warning. commit c2aae74f010f97a3415542fe649198a5d3be1ea8 Author: Olivier Houchard Date: Fri Sep 22 18:26:28 2017 +0200 MEDIUM: ssl: Handle early data with OpenSSL 1.1.1 When compiled with Openssl >= 1.1.1, before attempting to do the handshake, try to read any early data. If any early data is present, then we'll create the session, read the data, and handle the request before we're doing the handshake. For this, we add a new connection flag, CO_FL_EARLY_SSL_HS, which is not part of the CO_FL_HANDSHAKE set, allowing to proceed with a session even before an SSL handshake is completed. As early data do have security implication, we let the origin server know the request comes from early data by adding the "Early-Data" header, as specified in this draft from the HTTP working group : https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-replay commit 253c62b257c137e7da5c273f42bc5d6eacd31d2c Author: Emmanuel Hocdet Date: Mon Aug 14 11:01:25 2017 +0200 MINOR: ssl: generated certificate is missing in switchctx early callback Openssl 1.1.1 supports switchctx early callback and generated certificate. Generated certificate calls must be available in switchctx early callback. commit 84e417d8593438b56654125d6e86d948038cfa45 Author: Emmanuel Hocdet Date: Wed Aug 16 11:33:17 2017 +0200 MINOR: ssl: support Openssl 1.1.1 early callback for switchctx Use Openssl-1.1.1 SSL_CTX_set_client_hello_cb to mimic BoringSSL early callback. Native multi certificate and SSL/TLS method per certificate is now supported by Openssl >= 1.1.1. commit 48e875563908c0ed0363aef27e2ca20ee00f13bf Author: Emmanuel Hocdet Date: Wed Aug 16 11:28:44 2017 +0200 MEDIUM: ssl: convert CBS (BoringSSL api) usage to neutral code switchctx early callback is only supported for BoringSSL. To prepare the support of openssl 1.1.1 early callback, convert CBS api to neutral code to work with any ssl libs. commit 51a76d84e4c44b2a9ba1cbf91cba1707b41d917d Author: Olivier Houchard Date: Mon Oct 2 16:12:07 2017 +0200 MINOR: http: Mark the 425 code as "Too Early". This adds a new status code for use with the "http-request deny" ruleset. The use case for this code is currently handled by this draft dedicated to 0-RTT processing : https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-replay commit 31904278dc8ee68c015538cb1a91e246b1e5d1ea Author: Thierry FOURNIER Date: Wed Oct 25 12:59:51 2017 +0200 MINOR: hlua: Add regex class This patch simply brings HAProxy internal regex system to the Lua API. Lua doesn't embed regexes, now it inherits from the regexes compiled with haproxy. commit 46c72551f36b982d117cf89537430699305a3c15 Author: Baptiste Assmann Date: Thu Oct 26 21:51:58 2017 +0200 MINOR: lua: add uuid to the Class Proxy the proxy UUID parameter is not set in the Lua Proxy Class. This patches adds it. commit 48b4bb4b09a51a8ef8c0d3692550831bd0c67d95 Author: William Lallemand Date: Mon Oct 23 14:36:34 2017 +0200 MEDIUM: cfgparse: post parsing registration Allow to register a function which will be called after the configuration file parsing, at the end of the check_config_validity(). It's useful fo checking dependencies between sections or for resolving keywords, pointers or values. commit d2ff56d2a35ac4cbe07ad2c0e0575ecca565d331 Author: William Lallemand Date: Mon Oct 16 11:06:50 2017 +0200 MEDIUM: cfgparse: post section callback This commit implements a post section callback. This callback will be used at the end of a section parsing. Every call to cfg_register_section must be modified to use the new prototype: int cfg_register_section(char *section_name, int (*section_parser)(const char *, int, char **, int), int (*post_section_parser)()); commit 2ae327edaf6c96d33dd5acebdb5a9291c3d55ec4 Author: Dragan Dosen Date: Thu Oct 26 11:25:10 2017 +0200 BUG/MEDIUM: prevent buffers being overwritten during build_logline() execution Calls to build_logline() are audited in order to use dynamic trash buffers allocated by alloc_trash_chunk() instead of global trash buffers. This is similar to commits 07a0fec ("BUG/MEDIUM: http: Prevent replace-header from overwriting a buffer") and 0d94576 ("BUG/MEDIUM: http: prevent redirect from overwriting a buffer"). This patch should be backported in 1.7, 1.6 and 1.5. It relies on commit b686afd ("MINOR: chunks: implement a simple dynamic allocator for trash buffers") for the trash allocator, which has to be backported as well. commit 145746c2d55f6524fdb47191479d84bcd6e3d5d1 Author: Willy Tarreau Date: Thu Oct 26 15:26:17 2017 +0200 MINOR: buffer: add the buffer input manipulation functions We used to have bo_{get,put}_{chr,blk,str} to retrieve/send data to the output area of a buffer, but not the equivalent ones for the input area. This will be needed to copy uploaded data frames in HTTP/2. commit 390485a68d8a7cc2243351fc153df19a2d2cb743 Author: Olivier Houchard Date: Tue Oct 24 19:03:30 2017 +0200 BUG/MINOR: checks: Don't forget to release the connection on error case. When switching the check code to a non-permanent connection, the new code forgot to free the connection if an error happened and was returned by connect_conn_chk(), leading to the check never be ran again. commit 31d4dbe82508e9b525ea0700a587341ba275e770 Author: Willy Tarreau Date: Wed Oct 25 09:32:15 2017 +0200 MINOR: ssl_sock: make use of CO_FL_WILL_UPDATE Now when ssl_sock_{to,from}_buf are called, if the connection doesn't feature CO_FL_WILL_UPDATE, they will first retrieve the updated flags using conn_refresh_polling_flags() before changing any flag, then call conn_cond_update_sock_polling() before leaving, to commit such changes. commit 256b9c59932b21063e21dd9aab62d6052b8a42b9 Author: Willy Tarreau Date: Wed Oct 25 09:30:13 2017 +0200 MINOR: raw_sock: make use of CO_FL_WILL_UPDATE Now when raw_sock_{to,from}_{pipe,buf} are called, if the connection doesn't feature CO_FL_WILL_UPDATE, they will first retrieve the updated flags using conn_refresh_polling_flags() before changing any flag, then call conn_cond_update_sock_polling() before leaving, to commit such changes. Note that the only real call to one of the __conn_* functions is in fact in conn_sock_read0() which is called from here. commit 7b271b214fa5a750d6c93d429b72e7004b6168eb Author: Willy Tarreau Date: Wed Oct 25 09:59:22 2017 +0200 MEDIUM: connection: make use of CO_FL_WILL_UPDATE in conn_sock_shutw() This one may be called by upper layers (eg: si_shutw()) or lower layers (si_shutw() as well during stream_int_notify()) so we want it to take care of updating the connection's flags if it's not going to be done by the caller. commit 916e12dcfbee0d66f2ac04f7c033b6946d71bfe2 Author: Willy Tarreau Date: Wed Oct 25 09:22:43 2017 +0200 MINOR: connection: add flag CO_FL_WILL_UPDATE to indicate when updates are granted In transport-layer functions (snd_buf/rcv_buf), it's very problematic never to know if polling changes made to the connection will be propagated or not. This has led to some conn_cond_update_polling() calls being placed at a few places to cover both the cases where the function is called from the upper layer and when it's called from the lower layer. With the arrival of the MUX, this becomes even more complicated, as the upper layer will not have to manipulate anything from the connection layer directly and will not have to push such updates directly either. But the snd_buf functions will need to see their updates committed when called from upper layers. The solution here is to introduce a connection flag set by the connection handler (and possibly any other similar place) indicating that the caller is committed to applying such changes on return. This way, the called functions will be able to apply such changes by themselves before leaving when the flag is not set, and the upper layer will not have to care about that anymore. commit bc97cc4fd14972d68e20f6b6031c30b872aeb8e6 Author: Willy Tarreau Date: Wed Oct 25 10:28:45 2017 +0200 MINOR: connection: move the cleanup of flag CO_FL_WAIT_ROOM This flag is only used when reading using splicing for now, and is only set when a pipe full condition is met, so we can simplify its reset condition in conn_refresh_polling_flags so that it's cleared at the same time as the other ones, only when the control layer is ready. This flag could be used more, to mark that a buffer full condition was met with any receive method in order to simplify polling management. This should probably be revisited after 1.8. commit 8de70bcb54200ba217447941eeeb4f14ed2d803e Author: Willy Tarreau Date: Wed Oct 25 15:34:39 2017 +0200 MINOR: ssl: don't abort after sending 16kB SSL records are 16kB max. When trying to send larger data chunks at once, SSL_read() only processes 16kB and ssl_sock_from_buf() believes it means the system buffers are full, which is not the case, contrary to raw_sock. This is particularly noticeable with HTTP/2 when using a 64kB buffer with multiple streams, as the mux buffer can start to fill up pretty quickly in this situation, slowing down the data delivery. commit 3b9c850271589751540b7eb0d9ae8816c38875c0 Author: Willy Tarreau Date: Wed Oct 25 14:22:28 2017 +0200 MINOR: stream-int: stop checking for useless connection flags in chk_snd_conn We've been keep this test for a connection being established since 1.5-dev14 when the stream-interface was still accessing the FD directly. The test on CO_FL_HANDSHAKE and L{4,6}_CONN is totally useless here, and can even be counter-productive on pure TCP where it could prevent a request from being sent on a connection still attempting to complete its establishment. And it creates an abnormal dependency between the layers that will complicate the implementation of the mux, so let's get rid of it now. commit 3f957b2f83b876f5f5282c2affc3bebc5aaa54c2 Author: Dragan Dosen Date: Tue Oct 24 09:27:34 2017 +0200 MINOR: sample: add the hex2i converter Converts a hex string containing two hex digits per input byte to an integer. If the input value can not be converted, then zero is returned. commit 6e5a9ca94830ba40ac97a43e6232114601e1ae6f Author: Dragan Dosen Date: Tue Oct 24 09:18:23 2017 +0200 MINOR: sample: add the sha1 converter This converter can be used to generate a SHA1 digest from binary type sample. The result is a binary sample with length of 20 bytes. commit 7389dd086c6dcc6d045adfb38e5a390868c92766 Author: Dragan Dosen Date: Tue Oct 24 08:48:25 2017 +0200 IMPORT: sha1: import SHA1 functions This is based on the git SHA1 implementation and optimized to do word accesses rather than byte accesses, and to avoid unnecessary copies into the context array. commit 8c6a3647ddb8fd5d38228d2a97c75b89b625b804 Author: Lukas Tribus Date: Tue Oct 24 12:26:32 2017 +0200 CLEANUP: cli: remove undocumented "set ssl tls-keys" command The documented "set ssl tls-key" command must be used instead. This is for 1.8 only. commit f4bbc438fccc2bb2fdd45c81feafc7d7d9f3fb88 Author: Lukas Tribus Date: Tue Oct 24 12:26:31 2017 +0200 BUG/MINOR: cli: restore "set ssl tls-key" command in 32af203b75 ("REORG: cli: move ssl CLI functions to ssl_sock.c") "set ssl tls-key" was accidentally replaced with "set ssl tls-keys" (keys instead of key). This is undocumented and breaks upgrades from 1.6 to 1.7. This patch restores "set ssl tls-key" and also registers a helptext. This should be backported to 1.7. commit ef607050754981f4ed9280e96e94f77ce471fd68 Author: Emmanuel Hocdet Date: Tue Oct 24 14:57:16 2017 +0200 BUG/MINOR: ssl: OCSP_single_get0_status can return -1 Commit 872085ce "BUG/MINOR: ssl: ocsp response with 'revoked' status is correct" introduce a regression. OCSP_single_get0_status can return -1 and haproxy must generate an error in this case. Thanks to Sander Hoentjen who have spotted the regression. This patch should be backported in 1.7, 1.6 and 1.5 if the patch above is backported. commit 019f9b10ef4e99d2ddd017f75d5e2050cf593678 Author: Emmanuel Hocdet Date: Mon Oct 2 17:12:06 2017 +0200 MINOR: ssl: build with recent BoringSSL library BoringSSL switch OPENSSL_VERSION_NUMBER to 1.1.0 for compatibility. Fix BoringSSL call and openssl-compat.h/#define occordingly. This will not break openssl/libressl compat. commit e966e4e45118f1f83b060001a5ffb9d8315a1bdf Author: Emmanuel Hocdet Date: Tue Oct 24 18:11:48 2017 +0200 BUILD: ssl: support OPENSSL_NO_ASYNC #define Support build without ASYNC support. This #define is set per default in BoringSSL. commit f5e0d1f9fdae4ca7e184e94c71446b4ccbc39984 Author: Willy Tarreau Date: Tue Oct 24 11:55:37 2017 +0200 CONTRIB: trace: report the base name only for file names The output is hard to read when the full path to the sources files is emitted, let's strip it. commit e8f0f1265f1b68578e82ae180c4be55647ad0675 Author: Willy Tarreau Date: Tue Oct 24 10:58:20 2017 +0200 CONTRIB: trace: try to display the function's return value on exit On x86_64, when gcc instruments functions and compiles at -O0, it saves the function's return value in register rbx before calling the trace callback. It provides a nice opportunity to display certain useful values (flags, booleans etc) during trace sessions. It's absolutely not guaranteed that it will always work but it provides a considerable help when it does so it's worth activating it. When building on a different architecture, the value 0 is always reported as the return value. On x86_64 with optimizations (-O), the RBX register will not necessarily match and random values will be reported, but since it's not the primary target it's not a problem. commit 1296382d0bf7e600bbb60e927e546053d381e473 Author: Willy Tarreau Date: Tue Oct 24 10:54:08 2017 +0200 CONTRIB: trace: add the possibility to place trace calls in the code Now any call to trace() in the code will automatically appear interleaved with the call sequence and timestamped in the trace file. They appear with a '#' on the 3rd argument (caller's pointer) in order to make them easy to spot. If the trace functionality is not used, a dmumy weak function is used instead so that it doesn't require to recompile every time traces are enabled/disabled. The trace decoder knows how to deal with these messages, detects them and indents them similarly to the currently traced function. This can be used to print function arguments for example. Note that we systematically flush the log when calling trace() to ensure we never miss important events, so this may impact performance. The trace() function uses the same format as printf() so it should be easy to setup during debugging sessions. commit 796a2b3324907e0349c8dd071208dd621a528725 Author: Olivier Houchard Date: Tue Oct 24 17:42:47 2017 +0200 BUG/MEDIUM: server: Allocate tmptrash before using it. Don't forget to allocate tmptrash before using it, and free it once we're done. [wt: introduced by commit 64cc49cf ("MAJOR: servers: propagate server status changes asynchronously"), no backport needed] commit f08137c43486314f82c39101bb17610da3b00106 Author: Willy Tarreau Date: Sun Oct 22 10:13:45 2017 +0200 [RELEASE] Released version 1.8-dev3 Released version 1.8-dev3 with the following main changes : - REORG: ssl: move defines and methodVersions table upper - MEDIUM: ssl: ctx_set_version/ssl_set_version func for methodVersions table - MINOR: ssl: support ssl-min-ver and ssl-max-ver with crt-list - MEDIUM: ssl: disable SSLv3 per default for bind - BUG/MAJOR: ssl: fix segfault on connection close using async engines. - BUG/MAJOR: ssl: buffer overflow using offloaded ciphering on async engine - BUG/MINOR: ssl: do not call directly the conn_fd_handler from async_fd_handler - BUG/MINOR: haproxy/cli : fix for solaris/illumos distros for CMSG* macros - BUG/MEDIUM: build without openssl broken - BUG/MINOR: warning: need_resend may be used uninitialized - BUG/MEDIUM: misplaced exit and wrong exit code - BUG/MINOR: Makefile: fix compile error with USE_LUA=1 in ubuntu16.04 - BUILD: scripts: make publish-release support bare repositories - BUILD: scripts: add an automatic mode for publish-release - BUILD: scripts: add a "quiet" mode to publish-release - BUG/MAJOR: http: call manage_client_side_cookies() before erasing the buffer - BUG/MINOR: buffers: Fix bi/bo_contig_space to handle full buffers - CONTRIB: plug qdiscs: Plug queuing disciplines mini HOWTO. - BUG/MINOR: acls: Set the right refflag when patterns are loaded from a map - BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0 - BUG/MINOR: http/filters: Be sure to wait if a filter loops in HTTP_MSG_ENDING - BUG/MEDIUM: peers: Peers CLOSE_WAIT issue. - BUG/MAJOR: server: Segfault after parsing server state file. - BUG/MEDIUM: unix: never unlink a unix socket from the file system - scripts: create-release pass -n to tail - SCRIPTS: create-release: enforce GIT_COMMITTER_{NAME|EMAIL} validity - BUG/MEDIUM: fix segfault when no argument to -x option - MINOR: warning on multiple -x - MINOR: mworker: don't copy -x argument anymore in copy_argv() - BUG/MEDIUM: mworker: don't reuse PIDs passed to the master - BUG/MINOR: Wrong peer task expiration handling during synchronization processing. - BUG/MINOR: cfgparse: Check if tune.http.maxhdr is in the range 1..32767 - BUG/MINOR: log: pin the front connection when front ip/ports are logged - DOC: fix references to the section about the unix socket - BUG/MINOR: stream: flag TASK_WOKEN_RES not set if task in runqueue - MAJOR: task: task scheduler rework. - MINOR: task/stream: tasks related to a stream must be init by the caller. - MINOR: queue: Change pendconn_get_next_strm into private function - MINOR: backends: Change get_server_sh/get_server_uh into private function - MINOR: queue: Change pendconn_from_srv/pendconn_from_px into private functions - MEDIUM: stream: make stream_new() always set the target and analysers - MINOR: frontend: initialize HTTP layer after the debugging code - MINOR: connection: add a .get_alpn() method to xprt_ops - MINOR: ssl: add a get_alpn() method to ssl_sock - MINOR: frontend: retrieve the ALPN name when available - MINOR: frontend: report the connection's ALPN in the debug output - MINOR: stream: don't set backend's nor response analysers on SF_TUNNEL - MINOR: connection: send data before receiving - MAJOR: applet: applet scheduler rework. - BUG/MAJOR: frontend: don't dereference a null conn on outgoing connections - BUG/MAJOR: cli: fix custom io_release was crushed by NULL. - BUG/MAJOR: map: fix segfault during 'show map/acl' on cli. - BUG/MAJOR: compression: Be sure to release the compression state in all cases - MINOR: compression: Use a memory pool to allocate compression states - BUG/MAJOR: applet: fix a freeze if data is immedately forwarded. - DOC: fix references to the section about time format. - BUG/MEDIUM: map/acl: fix unwanted flags inheritance. - BUG/MAJOR: http: fix buffer overflow on loguri buffer. - MINOR: ssl: compare server certificate names to the SNI on outgoing connections - BUG/MINOR: stream: Don't forget to remove CF_WAKE_ONCE flag on response channel - BUG/MINOR: http: Don't reset the transaction if there are still data to send - BUG/MEDIUM: filters: Be sure to call flt_end_analyze for both channels - MINOR: peers: Add additional information to stick-table definition messages. - BUG/MINOR: http: properly handle all 1xx informational responses - OPTIM: ssl: don't consider a small ssl_read() as an indication of end of buffer - BUG/MINOR: peers: peer synchronization issue (with several peers sections). - CLEANUP: hdr_idx: make some function arguments const where possible - BUG/MINOR: Prevent a use-after-free on error scenario on option "-x". - BUG/MINOR: lua: In error case, the safe mode is not removed - BUG/MINOR: lua: executes the function destroying the Lua session in safe mode - BUG/MAJOR: lua/socket: resources not detroyed when the socket is aborted - BUG/MEDIUM: lua: bad memory access - BUG/MINOR: Lua: variable already initialized - DOC: update CONTRIBUTING regarding optional parts and message format - DOC: update the list of OpenSSL versions in the README - BUG/MINOR: http: Set the response error state in http_sync_res_state - MINOR: http: Reorder/rewrite checks in http_resync_states - MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flags - BUG/MEDIUM: http: Switch HTTP responses in TUNNEL mode when body length is undefined - MINOR: http: Rely on analyzers mask to end processing in forward_body functions - BUG/MINOR: http: Fix bug introduced in previous patch in http_resync_states - BUG/MINOR: contrib/modsecurity: BSD build fix - BUG/MINOR: contrib/mod_defender: build fix - BUG/MINOR: ssl: remove haproxy SSLv3 support when ssl lib have no SSLv3 - MINOR: ssl: remove an unecessary SSL_OP_NO_* dependancy - BUILD: ssl: fix compatibility with openssl without TLSEXT_signature_* - MINOR: tools: add a portable timegm() alternative - BUILD: lua: replace timegm() with my_timegm() to fix build on Solaris 10 - DOC: Updated 51Degrees git URL to point to a stable version. - BUG/MAJOR: http: Fix possible infinity loop in http_sync_(req|res)_state - MINOR: memory: remove macros - BUG/MINOR: lua: Fix Server.get_addr() port values - BUG/MINOR: lua: Correctly use INET6_ADDRSTRLEN in Server.get_addr() - MINOR: samples: Handle the type SMP_T_METH when we duplicate a sample in smp_dup - MINOR: samples: Handle the type SMP_T_METH in smp_is_safe and smp_is_rw - MINOR: samples: Don't allocate memory for SMP_T_METH sample when method is known - BUG/MINOR: lua: always detach the tcp/http tasks before freeing them - MINOR: task: always preinitialize the task's timeout in task_init() - CLEANUP: task: remove all initializations to TICK_ETERNITY after task_new() - BUG/MAJOR: lua: properly dequeue hlua_applet_wakeup() for new scheduler - MINOR: lua: Add proxy as member of proxy object. - DOC: lua: Proxy class doc update - MINOR: lua: Add lists of frontends and backends - BUG/MINOR: ssl: Fix check against SNI during server certificate verification - BUG/MINOR: ssl: make use of the name in SNI before verifyhost - MINOR: ssl: add a new error codes for wrong server certificates - BUG/MEDIUM: stream: don't retry SSL connections which fail the SNI name check - MINOR: ssl: add "no-ca-names" parameter for bind - BUG/MINOR: lua: Fix bitwise logic for hlua_server_check_* functions. - DOC: fix alphabetical order of "show commands" in management.txt - MINOR: listener: add a function to return a listener's state as a string - MINOR: cli: add a new "show fd" command - BUG/MEDIUM: ssl: Fix regression about certificates generation - MINOR: Add server port field to server state file. - MINOR: ssl: allow to start without certificate if strict-sni is set - MINOR: dns: Cache previous DNS answers. - MINOR: obj: Add a new type of object, OBJ_TYPE_SRVRQ. - Add a few functions to do unaligned access. - MINOR: dns: Handle SRV records. - MINOR: check: Fix checks when using SRV records. - MINOR: doc: Document SRV label usage. - BUILD/MINOR: cli: shut a minor gcc warning in "show fd" - BUILD: ssl: replace SSL_CTX_get0_privatekey for openssl < 1.0.2 - BUILD/MINOR: build without openssl still broken - BUG/MAJOR: stream: in stream_free(), close the front endpoint and not the origin - CLEANUP: raw_sock: Use a better name for the constructor than __ssl_sock_deinit() - MINOR: init: Fix CPU affinity setting on FreeBSD. - MINOR: dns: Update analysis of TRUNCATED response for SRV records - MINOR: dns: update record dname matching for SRV query types - MINOR: dns: update dns response buffer reading pointer due to SRV record - MINOR: dns: duplicate entries in resolution wait queue for SRV records - MINOR: dns: make debugging function dump_dns_config() compatible with SRV records - MINOR: dns: ability to use a SRV resolution for multiple backends - MINOR: dns: enable caching of responses for server set by a SRV record - MINOR: dns: new dns record type (RTYPE) for OPT - MINOR: dns: enabled edns0 extension and make accpeted payload size tunable - MINOR: dns: default "hold obsolete" timeout set to 0 - MINOR: chunks: add chunk_memcpy() and chunk_memcat() - MINOR: session: add a streams field to the session struct - MINOR: stream: link the stream to its session - MEDIUM: session: do not free a session until no stream references it - MINOR: ist: implement very simple indirect strings - TESTS: ist: add a test file for the functions - MINOR: http: export some of the HTTP parser macros - BUG/MINOR: Wrong type used as argument for spoe_decode_buffer(). - BUG/MINOR: dns: server set by SRV records stay in "no resolution" status - MINOR: dns: Maximum DNS udp payload set to 8192 - MINOR: dns: automatic reduction of DNS accpeted payload size - MINOR: dns: make SRV record processing more verbose - CLEANUP: dns: remove duplicated code in dns_resolve_recv() - CLEANUP: dns: remove duplicated code in dns_validate_dns_response() - BUG/MINOR: dns: wrong resolution interval lead to 100% CPU - BUG/MEDIUM: dns: fix accepted_payload_size parser to avoid integer overflow - BUG/MAJOR: lua: fix the impact of the scheduler changes again - BUG/MEDIUM: lua: HTTP services must take care of body-less status codes - MINOR: lua: properly process the contents of the content-length field - BUG/MEDIUM: stream: properly set the required HTTP analysers on use-service - OPTIM: lua: don't use expensive functions to parse headers in the HTTP applet - OPTIM: lua: don't add "Connection: close" on the response - REORG/MEDIUM: connection: introduce the notion of connection handle - BUG/MINOR: stream-int: don't check the CO_FL_CURR_WR_ENA flag - MEDIUM: connection: get rid of data->init() which was not for data - MEDIUM: stream: make stream_new() allocate its own task - CLEANUP: listener: remove the unused handler field - MEDIUM: session: add a pointer to a struct task in the session - MINOR: stream: provide a new stream creation function for connections - MEDIUM: connection: remove useless flag CO_FL_DATA_RD_SH - CLEANUP: connection: remove the unused conn_sock_shutw_pending() - MEDIUM: connection: remove useless flag CO_FL_DATA_WR_SH - DOC: add CLI info on privilege levels - DOC: Refer to Mozilla TLS info / config generator - MINOR: ssl: remove duplicate ssl_methods in struct bind_conf - BUG/MEDIUM: http: Fix a regression bug when a HTTP response is in TUNNEL mode - DOC: Add note about "* " prefix in CSV stats - CLEANUP: memory: Remove unused function pool_destroy - MINOR: listeners: Change listener_full and limit_listener into private functions - MINOR: listeners: Change enable_listener and disable_listener into private functions - MINOR: fd: Don't forget to reset fdtab[fd].update when a fd is added/removed - MINOR: fd: Set owner and iocb field before inserting a new fd in the fdtab - MINOR: backends: Make get_server_* functions explicitly static - MINOR: applet: Check applets_active_queue before processing applets queue - MINOR: chunks: Use dedicated function to init/deinit trash buffers - MEDIUM: chunks: Realloc trash buffers only after the config is parsed and checked - MINOR: logs: Use dedicated function to init/deinit log buffers - MINOR: logs: Realloc log buffers only after the config is parsed and checked - MINOR: buffers: Move swap_buffer into buffer.c and add deinit_buffer function - MINOR: stick-tables: Make static_table_key a struct variable instead of a pointer - MINOR: http: Use a trash chunk to store decoded string of the HTTP auth header - MINOR: fd: Add fd_active function - MINOR: fd: Use inlined functions to check fd state in fd_*_send/recv functions - MINOR: fd: Move (de)allocation of fdtab and fdinfo in (de)init_pollers - MINOR: freq_ctr: Return the new value after an update - MEDIUM: check: server states and weight propagation re-work - BUG/MEDIUM: epoll: ensure we always consider HUP and ERR - MINOR: fd: Add fd_update_events function - MINOR: polling: Use fd_update_events to update events seen for a fd - BUG/MINOR: server: Remove FQDN requirement for using init-addr and state file - Revert "BUG/MINOR: server: Remove FQDN requirement for using init-addr and state file" - MINOR: ssl: rework smp_fetch_ssl_fc_cl_str without internal ssl use - BUG/MEDIUM: http: Close streams for connections closed before a redirect - BUG/MINOR: Lua: The socket may be destroyed when we try to access. - MINOR: xref: Add a new xref system - MEDIUM: xref/lua: Use xref for referencing cosocket relation between stream and lua - MINOR: tasks: Move Lua notification from Lua to tasks - MINOR: net_helper: Inline functions meant to be inlined. - MINOR: cli: add socket commands and config to prepend informational messages with severity - MINOR: add severity information to cli feedback messages - BUILD: Makefile: add a function to detect support by the compiler of certain options - BUILD: Makefile: shut certain gcc/clang stupid warnings - BUILD: Makefile: improve detection of support for compiler warnings - MINOR: peers: don't reference the incoming listener on outgoing connections - MINOR: frontend: don't retrieve ALPN on the critical path - MINOR: protocols: always pass a "port" argument to the listener creation - MINOR: protocols: register the ->add function and stop calling them directly - MINOR: unix: remove the now unused proto_uxst.h file - MINOR: listeners: new function create_listeners - MINOR: listeners: make listeners count consistent with reality - MEDIUM: session: take care of incrementing/decrementing jobs - MINOR: listener: new function listener_release - MINOR: session: small cleanup of conn_complete_session() - MEDIUM: session: factor out duplicated code for conn_complete_session - MEDIUM: session: count the frontend's connections at a single place - BUG/MEDIUM: compression: Fix check on txn in smp_fetch_res_comp_algo - BUG/MINOR: compression: Check response headers before http-response rules eval - BUG/MINOR: spoe: Don't rely on SPOE ctx in debug message when its creation failed - BUG/MINOR: dns: Fix check on nameserver in snr_resolution_cb - MINOR: ssl: Remove useless checks on bind_conf or bind_conf->is_ssl - BUG/MINOR: contrib/mod_defender: close the va_list argp before return - BUG/MINOR: contrib/modsecurity: close the va_list ap before return - MINOR: tools: make my_htonll() more efficient on x86_64 - MINOR: buffer: add b_del() to delete a number of characters - MINOR: buffer: add b_end() and b_to_end() - MINOR: net_helper: add functions to read from vectors - MINOR: net_helper: add write functions - MINOR: net_helper: add 64-bit read/write functions - MINOR: connection: adjust CO_FL_NOTIFY_DATA after removal of flags - MINOR: ist: add a macro to ease const array initialization - BUG/MEDIUM: server: unwanted behavior leaving maintenance mode on tracked stopping server - BUG/MEDIUM: server: unwanted behavior leaving maintenance mode on tracked stopping server (take2) - BUG/MINOR: log: fixing small memory leak in error code path. - BUG/MINOR: contrib/halog: fixing small memory leak - BUG/MEDIUM: tcp/http: set-dst-port action broken - CLEANUUP: checks: don't set conn->handle.fd to -1 - BUG/MEDIUM: tcp-check: properly indicate polling state before performing I/O - BUG/MINOR: tcp-check: don't quit with pending data in the send buffer - BUG/MEDIUM: tcp-check: don't call tcpcheck_main() from the I/O handlers! - BUG/MINOR: unix: properly check for octal digits in the "mode" argument - MINOR: checks: make chk_report_conn_err() take a check, not a connection - CLEANUP: checks: remove misleading comments and statuses for external process - CLEANUP: checks: don't report report the fork() error twice - CLEANUP: checks: do not allocate a connection for process checks - TESTS: checks: add a simple test config for external checks - BUG/MINOR: tcp-check: don't initialize then break a connection starting with a comment - TESTS: checks: add a simple test config for tcp-checks - MINOR: tcp-check: make tcpcheck_main() take a check, not a connection - MINOR: checks: don't create then kill a dummy connection before tcp-checks - MEDIUM: checks: make tcpcheck_main() indicate if it recycled a connection - MEDIUM: checks: do not allocate a permanent connection anymore - BUG/MEDIUM: cli: fix "show fd" crash when dumping closed FDs - BUG/MEDIUM: http: Return an error when url_dec sample converter failed - BUG/MAJOR: stream-int: don't re-arm recv if send fails - BUILD/MINOR: 51d: fix warning when building with 51Degrees release version 3.2.12.12 - DOC: 51d: add 51Degrees git URL that points to release version 3.2.12.12 - DOC: 51d: Updated git URL and instructions for getting Hash Trie data files. - MINOR: compiler: restore the likely() wrapper for gcc 5.x - MINOR: session: remove the list of streams from struct session - DOC: fix some typos - MINOR: server: add the srv_queue() sample fetch method - MINOR: payload: add new sample fetch functions to process distcc protocol - MAJOR: servers: propagate server status changes asynchronously. - BUG/MEDIUM: ssl: fix OCSP expiry calculation - BUG/MINOR: stream-int: don't set MSG_MORE on SHUTW_NOW without AUTO_CLOSE - MINOR: server: Handle weight increase in consistent hash. - MINOR: checks: Add a new keyword to specify a SNI when doing SSL checks. - BUG/MINOR: tools: fix my_htonll() on x86_64 - BUG/MINOR: stats: Clear a bit more counters with in cli_parse_clear_counters(). - BUG/MAJOR: lua: scheduled task is freezing. - MINOR: buffer: add bo_del() to delete a number of characters from output - MINOR: buffer: add a function to match against string patterns - MINOR: buffer: add two functions to inject data into buffers - MINOR: buffer: add buffer_space_wraps() - REORG: channel: finally rename the last bi_* / bo_* functions - MINOR: buffer: add bo_getblk() and bo_getblk_nc() - MINOR: channel: make use of bo_getblk{,_nc} for their channel equivalents - MINOR: channel: make the channel be a const in all {ci,co}_get* functions - MINOR: ist: add ist0() to add a trailing zero to a string. - BUG/MEDIUM: log: check result details truncated. - MINOR: buffer: make bo_getblk_nc() not return 2 for a full buffer - REORG: http: move some very http1-specific parts to h1.{c,h} - REORG: http: move the HTTP/1 chunk parser to h1.{c,h} - REORG: http: move the HTTP/1 header block parser to h1.c - MEDIUM: http: make the chunk size parser only depend on the buffer - MEDIUM: http: make the chunk crlf parser only depend on the buffer - MINOR: h1: add struct h1m for basic HTTP/1 messages - MINOR: http: add very simple header management based on double strings - MEDIUM: h1: reimplement the http/1 response parser for the gateway - REORG: connection: rename CO_FL_DATA_* -> CO_FL_XPRT_* - MEDIUM: connection: make conn_sock_shutw() aware of lingering - MINOR: connection: ensure conn_ctrl_close() also resets the fd - MINOR: connection: add conn_stop_tracking() to disable tracking - MINOR: tcp: use conn_full_close() instead of conn_force_close() - MINOR: unix: use conn_full_close() instead of conn_force_close() - MINOR: checks: use conn_full_close() instead of conn_force_close() - MINOR: session: use conn_full_close() instead of conn_force_close() - MINOR: stream: use conn_full_close() instead of conn_force_close() - MINOR: stream: use conn_full_close() instead of conn_force_close() - MINOR: backend: use conn_full_close() instead of conn_force_close() - MINOR: stream-int: use conn_full_close() instead of conn_force_close() - MINOR: connection: remove conn_force_close() - BUG/MINOR: ssl: ocsp response with 'revoked' status is correct commit 872085ce375c76723ff023d2a79e9d9a0852aa98 Author: Emmanuel Hocdet Date: Tue Oct 10 15:18:52 2017 +0200 BUG/MINOR: ssl: ocsp response with 'revoked' status is correct ocsp_status can be 'good', 'revoked', or 'unknown'. 'revoked' status is a correct status and should not be dropped. In case of certificate with OCSP must-stapling extension, response with 'revoked' status must be provided as well as 'good' status. This patch can be backported in 1.7, 1.6 and 1.5. commit cbc6524a198400b96c28d226a0a042687b622dc0 Author: Willy Tarreau Date: Thu Oct 5 17:49:56 2017 +0200 MINOR: connection: remove conn_force_close() Now only conn_full_close() will be used. It will become more obvious when the tracking is in place or not and will make it easier to convert remaining call places to conn_streams. commit 3b737c9894e15bb349e335fe8adebe88d0fc2b9b Author: Willy Tarreau Date: Sun Oct 22 09:39:11 2017 +0200 MINOR: stream-int: use conn_full_close() instead of conn_force_close() We simply disable tracking before calling it. commit f098fd061f0af46fe51e6588023fb7f57359791a Author: Willy Tarreau Date: Sun Oct 22 09:35:01 2017 +0200 MINOR: backend: use conn_full_close() instead of conn_force_close() There's no point in using conn_force_close() in outgoing connect() since XPRT_TRACKED is not set so both functions are equivalent. commit 630f99a7e9da6ae233ce2124c69eb163648c0d19 Author: Willy Tarreau Date: Thu Oct 5 18:13:15 2017 +0200 MINOR: stream: use conn_full_close() instead of conn_force_close() We simply disable tracking before calling it. commit 151a2387ece5489313cd4955a2ba17e132ee8b2d Author: Willy Tarreau Date: Thu Oct 5 18:02:55 2017 +0200 MINOR: stream: use conn_full_close() instead of conn_force_close() There's no point in using conn_force_close() in outgoing connect() since XPRT_TRACKED is not set so both functions are equivalent. commit 5b78a9dd0467d6c158e444b9e3989396609ee328 Author: Willy Tarreau Date: Thu Oct 5 18:12:51 2017 +0200 MINOR: session: use conn_full_close() instead of conn_force_close() We simply disable tracking before calling it. commit 402dbc11d57a584a7b69aaababd076a8b462b24b Author: Willy Tarreau Date: Thu Oct 5 17:53:13 2017 +0200 MINOR: checks: use conn_full_close() instead of conn_force_close() There's no point in using conn_force_close() in the checks since they do not use XPRT_TRACKED so both functions are equivalent. commit 8c82901958bc21970f90b334c5e0f36bf7918d69 Author: Willy Tarreau Date: Thu Oct 5 18:02:11 2017 +0200 MINOR: unix: use conn_full_close() instead of conn_force_close() There's no point in using conn_force_close() in outgoing connect() since XPRT_TRACKED is not set so both functions are equivalent. commit 3f2770ba2759fce9c6876781fd34ad2b7ec7ad86 Author: Willy Tarreau Date: Thu Oct 5 18:01:29 2017 +0200 MINOR: tcp: use conn_full_close() instead of conn_force_close() There's no point in using conn_force_close() in outgoing connect() since XPRT_TRACKED is not set so both functions are equivalent. commit dc42acddb662014a4bce640b098951ad5dec332c Author: Willy Tarreau Date: Thu Oct 5 18:09:20 2017 +0200 MINOR: connection: add conn_stop_tracking() to disable tracking This will be used before conn_full_close() instead of using conn_force_close(), resulting in a clearer exit path in various situations. commit 6a0a80adaf8649bda5caa6315cd1fd03a646beae Author: Willy Tarreau Date: Thu Oct 5 17:43:39 2017 +0200 MINOR: connection: ensure conn_ctrl_close() also resets the fd The connection's fd was reset to DEAD_FD_MAGIC on conn_force_close() but not on conn_full_close(), which is a bit strange. Let's do it on both. commit f9ce57e86c9126785a25e5b6625f0f43b17ce210 Author: Willy Tarreau Date: Thu Oct 5 17:24:42 2017 +0200 MEDIUM: connection: make conn_sock_shutw() aware of lingering Instead of having to manually handle lingering outside, let's make conn_sock_shutw() check for it before calling shutdown(). We simply don't want to emit the FIN if we're going to reset the connection due to lingering. It's particularly important for silent-drop where it's absolutely mandatory that no packet leaves the machine. commit 1a0545f3d74630ee124540c2fc77f8c88c81df04 Author: Olivier Houchard Date: Wed Sep 13 18:30:23 2017 +0200 REORG: connection: rename CO_FL_DATA_* -> CO_FL_XPRT_* These flags are not exactly for the data layer, they instead indicate what is expected from the transport layer. Since we're going to split the connection between the transport and the data layers to insert a mux layer, it's important to have a clear idea of what each layer does. All function conn_data_* used to manipulate these flags were renamed to conn_xprt_*. commit 794f9af8949d0d9edfdfe2453167325e75cac2e4 Author: Willy Tarreau Date: Wed Jul 26 09:07:47 2017 +0200 MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len). commit 306924ecb828a01edf1e8012deb02299284d3cb8 Author: Willy Tarreau Date: Thu Sep 21 14:25:39 2017 +0200 MINOR: http: add very simple header management based on double strings This will be used initially by the hpack table and hopefully later by a new native http processor. These headers are made of name and value, both an immediate string (ie: pointer and length). commit 4093a4dc0138ef59e56bca3b9b954a66740b2045 Author: Willy Tarreau Date: Thu Sep 21 11:46:43 2017 +0200 MINOR: h1: add struct h1m for basic HTTP/1 messages This one is much simpler than http_msg and will be used in the HTTP parsers involved in the H2 to H1 gateway. commit b28925675d257db1ac8ec9a3b3d819575554345c Author: Willy Tarreau Date: Thu Sep 21 11:33:54 2017 +0200 MEDIUM: http: make the chunk crlf parser only depend on the buffer The chunk crlf parser used to depend on the channel and on the HTTP message, eventhough it's not really needed. Let's remove this dependency so that it can be used within the H2 to H1 gateway. As part of this small API change, it was renamed to h1_skip_chunk_crlf() to mention that it doesn't depend on http_msg anymore. commit e56cdd362983009863a417d599145593249b5acf Author: Willy Tarreau Date: Thu Sep 21 08:36:33 2017 +0200 MEDIUM: http: make the chunk size parser only depend on the buffer The chunk parser used to depend on the channel and on the HTTP message but it's not really needed as they're only used to retrieve the buffer as well as to return the number of bytes parsed and the chunk size. Here instead we pass the (few) relevant information in arguments so that the function may be reused without a channel nor an HTTP message (ie from the H2 to H1 gateway). As part of this API change, it was renamed to h1_parse_chunk_size() to mention that it doesn't depend on http_msg anymore. commit 8740c8b1b2fffd33ab3b54c4b304f05d803f0063 Author: Willy Tarreau Date: Thu Sep 21 10:22:25 2017 +0200 REORG: http: move the HTTP/1 header block parser to h1.c Since it still depends on http_msg, it was not renamed yet. commit db4893d6a4168601cc855114d58e9cdbe587df42 Author: Willy Tarreau Date: Thu Sep 21 08:40:02 2017 +0200 REORG: http: move the HTTP/1 chunk parser to h1.{c,h} Functions http_parse_chunk_size(), http_skip_chunk_crlf() and http_forward_trailers() were moved to h1.h and h1.c respectively so that they can be called from outside. The parts that were inline remained inline as it's critical for performance (+41% perf difference reported in an earlier test). For now the "http_" prefix remains in their name since they still depend on the http_msg type. commit 0da5b3bddce07dc4a57f5ce9dd97af9a8ab626f0 Author: Willy Tarreau Date: Thu Sep 21 09:30:46 2017 +0200 REORG: http: move some very http1-specific parts to h1.{c,h} Certain types and enums are very specific to the HTTP/1 parser, and we'll need to share them with the HTTP/2 to HTTP/1 translation code. Let's move them to h1.c/h1.h. Those with very few occurrences or only used locally were renamed to explicitly mention the relevant HTTP version : enum ht_state -> h1_state. http_msg_state_str -> h1_msg_state_str HTTP_FLG_* -> H1_FLG_* http_char_classes -> h1_char_classes Others like HTTP_IS_*, HTTP_MSG_* are left to be done later. commit 0621da5f5bbd089e3562a412d7e73cf8971030b0 Author: Willy Tarreau Date: Fri Oct 20 18:21:49 2017 +0200 MINOR: buffer: make bo_getblk_nc() not return 2 for a full buffer Thus function returns the number of blocks. When a buffer is full and properly aligned, buf->p loops back the beginning, and the test in the code doesn't cover that specific case, so it returns two chunks, a full one and an empty one. It's harmless but can sometimes have a small impact on performance and definitely makes the code hard to debug. commit 5a1335110ceeadc9aca6783588a5ee39d431b9fa Author: Emeric Brun Date: Thu Oct 19 14:42:30 2017 +0200 BUG/MEDIUM: log: check result details truncated. Fix regression introduced by commit: 'MAJOR: servers: propagate server status changes asynchronously.' The building of the log line was re-worked to be done at the postponed point without lack of data. [wt: this only affects 1.8-dev, no backport needed] commit e67c4e5744edfe865137f9699a3bd4be368b5b6f Author: Willy Tarreau Date: Thu Oct 19 06:28:23 2017 +0200 MINOR: ist: add ist0() to add a trailing zero to a string. This function modifies the string to add a zero after the end, and returns the start pointer. The purpose is to use it on strings extracted by parsers from larger strings cut with delimiters that are not important and can be destroyed. It allows any such string to be used with regular string functions. It's also convenient to use with printf() to show data extracted from writable areas. commit 41ab86898e4e14b61fd2e1a35effdff9ab4fb4a2 Author: Willy Tarreau Date: Thu Oct 19 14:58:40 2017 +0200 MINOR: channel: make the channel be a const in all {ci,co}_get* functions There's no point having the channel marked writable as these functions only extract data from the channel. The code was retrieved from their ci/co ancestors. commit 6b3f353bcfdc136eda68a03b5b63a3798911129b Author: Willy Tarreau Date: Thu Oct 19 14:57:54 2017 +0200 MINOR: channel: make use of bo_getblk{,_nc} for their channel equivalents Let's reuse the buffer-level functions to perform the operations. commit e0e734ccc5322e844eb9d3e2bc82dd6479c803f3 Author: Willy Tarreau Date: Thu Oct 19 14:56:49 2017 +0200 MINOR: buffer: add bo_getblk() and bo_getblk_nc() These functions respectively extract a block from an output buffer by copying it or by just passing pointers and lengths for zero copy operation. commit 06d80a9a9c49b3b17f343a7131b4cf12ef7c6636 Author: Willy Tarreau Date: Thu Oct 19 14:32:15 2017 +0200 REORG: channel: finally rename the last bi_* / bo_* functions For HTTP/2 we'll need some buffer-only equivalent functions to some of the ones applying to channels and still squatting the bi_* / bo_* namespace. Since these names have kept being misleading for quite some time now and are really getting annoying, it's time to rename them. This commit will use "ci/co" as the prefix (for "channel in", "channel out") instead of "bi/bo". The following ones were renamed : bi_getblk_nc, bi_getline_nc, bi_putblk, bi_putchr, bo_getblk, bo_getblk_nc, bo_getline, bo_getline_nc, bo_inject, bi_putchk, bi_putstr, bo_getchr, bo_skip, bi_swpbuf commit 5b9834f12a72f6c2ee7be8cc3da13e83ae3d92b9 Author: Willy Tarreau Date: Mon Oct 16 14:01:18 2017 +0200 MINOR: buffer: add buffer_space_wraps() This function returns true if the available buffer space wraps. This will be used to detect if it's worth realigning a buffer when it lacks contigous space. commit e5676e71034a26e338552702e7100b99806b1538 Author: Willy Tarreau Date: Fri Sep 22 15:47:51 2017 +0200 MINOR: buffer: add two functions to inject data into buffers bi_istput() injects the ist string into the input region of the buffer, it will be used to feed small data chunks into the conn_stream. bo_istput() does the same into the output region of the buffer, it will be used to send data via the transport layer and assumes there's no input data. commit 6634b63c7807dcda56e2abd86f6360f4043b9818 Author: Willy Tarreau Date: Fri Sep 22 15:02:54 2017 +0200 MINOR: buffer: add a function to match against string patterns In order to match known patterns in wrapping buffer, we'll introduce new string manipulation functions for buffers. The new function b_isteq() relies on an ist string for the pattern and compares it against any location in the buffer relative to

. The second function bi_eat() is specially designed to match input contents. commit 7f564d2b60e169f7dd0b1d233ec3c41849296026 Author: Willy Tarreau Date: Wed Oct 18 08:32:12 2017 +0200 MINOR: buffer: add bo_del() to delete a number of characters from output This simply reduces the amount of output data from the buffer after they have been transferred, in a way that is more natural than by fiddling with buf->o. b_del() was renamed to bi_del() to avoid any ambiguity (it's not yet used). commit 253e53e661c49fb9723535319cf511152bf09bc7 Author: Emeric Brun Date: Tue Oct 17 18:58:40 2017 +0200 BUG/MAJOR: lua: scheduled task is freezing. Since commit 'MAJOR: task: task scheduler rework' 0194897e540cec67d7d1e9281648b70efe403f08. LUA's scheduling tasks are freezing. A running task should not handle the scheduling itself but let the task scheduler to handle it based on the 'expire' field. [wt: no backport needed] commit 00bc3cb59f61638ee48cbca126016bc20abee47f Author: Olivier Houchard Date: Tue Oct 17 19:23:25 2017 +0200 BUG/MINOR: stats: Clear a bit more counters with in cli_parse_clear_counters(). Clear MaxSslRate, SslFrontendMaxKeyRate and SslBackendMaxKeyRate when clear counters is used, it was probably forgotten when those counters were added. [wt: this can probably be backported as far as 1.5 in dumpstats.c] commit dea7c5c03de727f492e6ce2418a7651b4f0f6144 Author: Willy Tarreau Date: Wed Oct 18 11:39:33 2017 +0200 BUG/MINOR: tools: fix my_htonll() on x86_64 Commit 36eb3a3 ("MINOR: tools: make my_htonll() more efficient on x86_64") brought an incorrect asm statement missing the input constraints, causing the input value not necessarily to be placed into the same register as the output one, resulting in random output. It happens to work when building at -O0 but not above. This was only detected in the HTTP/2 parser, but in mainline it could only affect the integer to binary sample cast. No backport is needed since this bug was only introduced in the development branch. commit 9130a9605d10a0c155bbc7e808770318606ede35 Author: Olivier Houchard Date: Tue Oct 17 17:33:43 2017 +0200 MINOR: checks: Add a new keyword to specify a SNI when doing SSL checks. Add a new keyword, "check-sni", to be able to specify the SNI to be used when doing health checks over SSL. commit f8eb8d56a79359a99d4076d3af9fdaaa070b680a Author: Olivier Houchard Date: Tue Oct 17 15:52:59 2017 +0200 MINOR: server: Handle weight increase in consistent hash. When the server weight is rised using the CLI, extra nodes have to be allocated, or the weight will be effectively the same as the original one. [wt: given that the doc made no explicit mention about this limitation, this patch could even be backported as it fixes an unexpected behaviour] commit 4ac49287182dfefc69e7088332348ea0948022fa Author: Willy Tarreau Date: Tue Oct 17 16:33:46 2017 +0200 BUG/MINOR: stream-int: don't set MSG_MORE on SHUTW_NOW without AUTO_CLOSE Since around 1.5-dev12, we've been setting MSG_MORE on send() on various conditions, including the fact that SHUTW_NOW is present, but we don't check that it's accompanied with AUTO_CLOSE. The result is that on requests immediately followed by a close (where AUTO_CLOSE is not set), the request gets delayed in the TCP stack before being sent to the server. This is visible with the H2 code where the end-of-stream flag is set on requests, but probably happens when a POLL_HUP is detected along with the request. The (lack of) presence of option abortonclose has no effect here since we never send the SHUTW along with the request. This fix can be backported to 1.7, 1.6 and 1.5. commit 953917abc9fb72c49fa49cb41bc2cdbecd7c9e93 Author: Frederik Deweerdt Date: Mon Oct 16 07:37:31 2017 -0700 BUG/MEDIUM: ssl: fix OCSP expiry calculation The hour part of the timezone offset was multiplied by 60 instead of 3600, resulting in an inaccurate expiry. This bug was introduced in 1.6-dev1 by commit 4f3c87a ("BUG/MEDIUM: ssl: Fix to not serve expired OCSP responses."), so this fix must be backported into 1.7 and 1.6. commit 64cc49cf7e0db452e1467b2ac4cbf71c674dbc5a Author: Emeric Brun Date: Tue Oct 3 14:46:45 2017 +0200 MAJOR: servers: propagate server status changes asynchronously. In order to prepare multi-thread development, code was re-worked to propagate changes asynchronoulsy. Servers with pending status changes are registered in a list and this one is processed and emptied only once 'run poll' loop. Operational status changes are performed before administrative status changes. In a case of multiple operational status change or admin status change in the same 'run poll' loop iteration, those changes are merged to reach only the targeted status. commit d716f9bacfa6c363c5eb52e98b536c140a671392 Author: Willy Tarreau Date: Fri Oct 13 11:03:15 2017 +0200 MINOR: payload: add new sample fetch functions to process distcc protocol When using haproxy in front of distccd, it's possible to provide significant improvements by only connecting when the preprocessing is completed, and by selecting different farms depending on the payload size. This patch provides two new sample fetch functions : distcc_param([,]) : integer distcc_body([,]) : binary commit ff2b7afe0b72874e158400aaa58f413dda030489 Author: Willy Tarreau Date: Fri Oct 13 11:46:26 2017 +0200 MINOR: server: add the srv_queue() sample fetch method srv_queue([/]) : integer Returns an integer value corresponding to the number of connections currently pending in the designated server's queue. If is omitted, then the server is looked up in the current backend. It can sometimes be used together with the "use-server" directive to force to use a known faster server when it is not much loaded. See also the "srv_conn", "avg_queue" and "queue" sample fetch methods. commit dce734e10f5c2218e94b48dd28e1d4df86a7c3d0 Author: Patrick Starr Date: Mon Oct 9 13:17:12 2017 +0700 DOC: fix some typos [wt: ~25 typos, most of which should be eligible for backporting] commit bf08beb2a349403c3b491831fc7e5d3d8d518566 Author: Willy Tarreau Date: Sun Oct 8 21:32:53 2017 +0200 MINOR: session: remove the list of streams from struct session Commit bcb86ab ("MINOR: session: add a streams field to the session struct") added this list of streams that is not needed anymore. Let's get rid of it now. commit c939835f77f9785e96380949c13b290abac7e4c2 Author: Willy Tarreau Date: Sun Oct 8 22:26:03 2017 +0200 MINOR: compiler: restore the likely() wrapper for gcc 5.x After some tests, gcc 5.x produces better code with likely() than without, contrary to gcc 4.x where it was better to disable it. Let's re-enable it for 5 and above. commit 636e6afcfab3d6c8cd3df852bdc920ff468681c2 Author: Ben51Degrees Date: Thu Oct 5 19:54:18 2017 +0100 DOC: 51d: Updated git URL and instructions for getting Hash Trie data files. Use branch, not tag for download URL, and recommend switching to Hash Trie. commit 16586e635b0761ea154e457f7743bca390431a3a Author: Dragan Dosen Date: Wed Sep 27 12:55:07 2017 +0200 DOC: 51d: add 51Degrees git URL that points to release version 3.2.12.12 The 51Degrees C library version 3.2.12.12 has support for a new Hash Trie algorithm. This patch can be backported in 1.7. commit 483b93cc9a7c63fefe7366f749edddf7c8cd20bc Author: Dragan Dosen Date: Wed Sep 27 12:46:44 2017 +0200 BUILD/MINOR: 51d: fix warning when building with 51Degrees release version 3.2.12.12 The warning appears when building with 51Degrees release that uses a new Hash Trie algorithm (release version 3.2.12.12): src/51d.c: In function init_51degrees: src/51d.c:566:2: warning: enumeration value DATA_SET_INIT_STATUS_TOO_MANY_OPEN_FILES not handled in switch [-Wswitch] switch (_51d_dataset_status) { ^ This patch can be backported in 1.7. commit 95fad5ba4b2c07ab1ac610d119380ffd7709f0e6 Author: Bin Wang Date: Fri Sep 15 14:56:40 2017 +0800 BUG/MAJOR: stream-int: don't re-arm recv if send fails When 1) HAProxy configured to enable splice on both directions 2) After some high load, there are 2 input channels with their socket buffer being non-empty and pipe being full at the same time, sitting in `fd_cache` without any other fds. The 2 channels will repeatedly be stopped for receiving (pipe full) and waken for receiving (data in socket), thus getting out and in of `fd_cache`, making their fd swapping location in `fd_cache`. There is a `if (entry < fd_cache_num && fd_cache[entry] != fd) continue;` statement in `fd_process_cached_events` to prevent frequent polling, but since the only 2 fds are constantly swapping location, `fd_cache[entry] != fd` will always hold true, thus HAProxy can't make any progress. The root cause of the issue is dual : - there is a single fd_cache, for next events and for the ones being processed, while using two distinct arrays would avoid the problem. - the write side of the stream interface wakes the read side up even when it couldn't write, and this one really is a bug. Due to CF_WRITE_PARTIAL not being cleared during fast forwarding, a failed send() attempt will still cause ->chk_rcv() to be called on the other side, re-creating an entry for its connection fd in the cache, causing the same sequence to be repeated indefinitely without any opportunity to make progress. CF_WRITE_PARTIAL used to be used for what is present in these tests : check if a recent write operation was performed. It's part of the CF_WRITE_ACTIVITY set and is tested to check if timeouts need to be updated. It's also used to detect if a failed connect() may be retried. What this patch does is use CF_WROTE_DATA() to check for a successful write for connection retransmits, and to clear CF_WRITE_PARTIAL before preparing to send in stream_int_notify(). This way, timeouts are still updated each time a write succeeds, but chk_rcv() won't be called anymore after a failed write. It seems the fix is required all the way down to 1.5. Without this patch, the only workaround at this point is to disable splicing in at least one direction. Strictly speaking, splicing is not absolutely required, as regular forwarding could theorically cause the issue to happen if the timing is appropriate, but in practice it appears impossible to reproduce it without splicing, and even with splicing it may vary. The following config manages to reproduce it after a few attempts (haproxy going 100% CPU and having to be killed) : global maxpipes 50000 maxconn 10000 listen srv1 option splice-request option splice-response bind :8001 server s1 127.0.0.1:8002 server$ tcploop 8002 L N20 A R10 S1000000 R10 S1000000 R10 S1000000 R10 S1000000 R10 S1000000 client$ tcploop 8001 N20 C T S1000000 R10 J commit a258479e3fe17fc525d3c82b23e26e311453fd56 Author: Christopher Faulet Date: Thu Oct 5 10:03:12 2017 +0200 BUG/MEDIUM: http: Return an error when url_dec sample converter failed url_dec sample converter uses url_decode function to decode an URL. This function fails by returning -1 when an invalid character is found. But the sample converter never checked the return value and it used it as length for the decoded string. Because it always succeeded, the invalid sample (with a string length set to -1) could be used by other sample fetches or sample converters, leading to undefined behavior like segfault. The fix is pretty simple, url_dec sample converter just needs to return an error when url_decode fails. This patch must be backported in 1.7 and 1.6. commit 017af2477eb478e95b2e2e09e3daffaf34979f2d Author: Willy Tarreau Date: Wed Oct 4 20:24:54 2017 +0200 BUG/MEDIUM: cli: fix "show fd" crash when dumping closed FDs I misplaced the "if (!fdt.owner)" test so it can occasionally crash when dumping an fd that's already been closed but still appears in the table. It's not critical since this was not pushed into any release nor backported though. commit 00149121b78c42963a5924fda0fa94d7ddb27957 Author: Willy Tarreau Date: Wed Oct 4 18:05:01 2017 +0200 MEDIUM: checks: do not allocate a permanent connection anymore Health check currently cheat, they allocate a connection upon startup and never release it, it's only recycled. The problem with doing this is that this code is preventing the connection code from evolving towards multiplexing. This code ensures that it's safe for the checks to run without a connection all the time. Given that the code heavily relies on CO_FL_ERROR to signal check errors, it is not trivial but in practice this is the principle adopted here : - the connection is not allocated anymore on startup - new checks are not supposed to have a connection, so an attempt is made to allocate this connection in the check task's context. If it fails, the check is aborted on a resource error, and the rare code on this path verifying the connection was adjusted to check for its existence (in practice, avoid to close it) - returning checks necessarily have a valid connection (which may possibly be closed). - a "tcp-check connect" rule tries to allocate a new connection before releasing the previous one (but after closing it), so that if it fails, it still keeps the previous connection in a closed state. This ensures a connection is always valid here Now it works well on all tested cases (regular and TCP checks, even with multiple reconnections), including when the connection is forced to NULL or randomly allocated. commit 6bdcab014980a5f1793b3e27834f1bfcdab9c9f5 Author: Willy Tarreau Date: Wed Oct 4 18:41:00 2017 +0200 MEDIUM: checks: make tcpcheck_main() indicate if it recycled a connection The tcp-checks are very fragile. They can modify a connection's FD by closing and reopening a socket without informing the connection layer, which may then possibly touch the wrong fd. Given that the events are only cleared and that the fd is just created, there should be no visible side effect because the old fd is deleted so even if its flags get cleared they were already, and the new fd already has them cleared as well so it's a NOP. Regardless, this is too fragile and will not resist to threads. In order to address this situation, this patch makes tcpcheck_main() indicate if it closed a connection and report it to wake_srv_chk(), which will then report it to the connection's fd handler so that it refrains from updating the connection polling and the fd. Instead the connection polling status is updated in the wake() function. commit f411cce456dabbe3cf6637f2bb11435af0499ae7 Author: Willy Tarreau Date: Wed Oct 4 16:21:19 2017 +0200 MINOR: checks: don't create then kill a dummy connection before tcp-checks When tcp-checks are in use, a connection starts to be created, then it's destroyed so that tcp-check can recreate its own. Now we directly move to tcpcheck_main() when it's detected that tcp-check is in use. commit be74b88be89bf849802e922a3bc04c91ef3f8963 Author: Willy Tarreau Date: Wed Oct 4 16:22:49 2017 +0200 MINOR: tcp-check: make tcpcheck_main() take a check, not a connection We want this one to allocate its own connection so it must not take a connection but a check. commit 668730fd009d9148d5ba260260b61d0f0b827e12 Author: Willy Tarreau Date: Wed Oct 4 16:28:34 2017 +0200 TESTS: checks: add a simple test config for tcp-checks tcp-check.cfg tests various arrangements of initial tcp-check rules. commit 894c642fbf33e635fdd9422ca0e640dc61202342 Author: Willy Tarreau Date: Wed Oct 4 15:58:52 2017 +0200 BUG/MINOR: tcp-check: don't initialize then break a connection starting with a comment The following config : backend tcp9000 option tcp-check tcp-check comment "this is a comment" tcp-check connect port 10000 server srv 127.0.0.1:9000 check inter 1s will result in a connection being first made to port 9000 then immediately destroyed and re-created on port 10000, because the first rule is a comment and doesn't match the test for the first rule being a connect(). It's mostly harmless (unless the server really must not receive empty connections) and the workaround simply consists in removing the comment. Let's proceed like in other places where we simply skip leading comments. A new function was made to make this lookup les boring. The fix should be backported to 1.7 and 1.6. commit 59070784fcac855b1a7e6da7c239800aeed622bb Author: Willy Tarreau Date: Wed Oct 4 15:42:00 2017 +0200 TESTS: checks: add a simple test config for external checks ext-check.cfg tests both for success and failure in two different backends. commit b398e643d44565d1a10f772186aa1f92d9c375d1 Author: Willy Tarreau Date: Wed Oct 4 15:25:38 2017 +0200 CLEANUP: checks: do not allocate a connection for process checks Since this connection is not used at all anymore, do not allocate it. It was verified that check successes and failures (both synchronous and asynchronous) continue to be properly reported. commit d7c3fbd5c304b695a73fc530b0472a276b1b4217 Author: Willy Tarreau Date: Wed Oct 4 15:19:26 2017 +0200 CLEANUP: checks: don't report report the fork() error twice Upon fork() error, a first report is immediately made by connect_proc_chk() via set_server_check_status(), then process_chk_proc() detects the error code and makes up a dummy connection error to call chk_report_conn_err(), which tries to retrieve the errno code from the connection, fails, then saves the status message from the check, fails all "if" tests on its path related to the connection then resets the check's state to the current one with the current status message. All this useless chain is the only reason why process checks require a connection! Let's simply get rid of this second useless call. commit 1e62e2a780a9118e478ba05c4bdb546f133d7e4d Author: Willy Tarreau Date: Wed Oct 4 15:07:02 2017 +0200 CLEANUP: checks: remove misleading comments and statuses for external process The external process check code abused a little bit from copy-pasting to the point of making think it requires a connection... The initialization code only returns SF_ERR_NONE and SF_ERR_RESOURCE, so the other one can be folded there. The code now only uses the connection to report the error status. commit b5259bf44f7b95174c1daa38e9dfd49f66e6ac01 Author: Willy Tarreau Date: Wed Oct 4 14:47:29 2017 +0200 MINOR: checks: make chk_report_conn_err() take a check, not a connection Amazingly, this function takes a connection to report an error and is used by process checks, placing a hard dependency between the connection and the check preventing the mux from being completely implemented. Let's first get rid of this. commit a1a247bd90248f92005b3a456b0ccabc11efe602 Author: Willy Tarreau Date: Wed Oct 4 14:43:44 2017 +0200 BUG/MINOR: unix: properly check for octal digits in the "mode" argument A config containing "stats socket /path/to/socket mode admin" used to silently start and be unusable (mode 0, level user) because the "mode" parser doesn't take care of non-digits. Now it properly reports : [ALERT] 276/144303 (7019) : parsing [ext-check.cfg:4] : 'stats socket' : ''mode' : missing or invalid mode 'admin' (octal integer expected)' This can probably be backported to 1.7, 1.6 and 1.5, though reporting parsing errors in very old versions probably isn't a good idea if the feature was left unused for years. commit c09572fd8be6e7d7a1207bca31e14856ee7ac9d2 Author: Willy Tarreau Date: Wed Oct 4 11:58:22 2017 +0200 BUG/MEDIUM: tcp-check: don't call tcpcheck_main() from the I/O handlers! This function can destroy a socket and create a new one, resulting in a change of FD on the connection between recv() and send() for example, which is absolutely not permitted, and can result in various funny games like polling not being properly updated (or with the flags from a previous fd) etc. Let's only call this from the wake() callback which is more tolerant. Ideally the operations should be made even more reliable by returning a specific value to indicate that the connection was released and that another one was created. But this is hasardous for stable releases as it may reveal other issues. This fix should be backported to 1.7 and 1.6. commit 82feaaf0423919c5af61b59f6f3234bc0423a791 Author: Willy Tarreau Date: Wed Oct 4 11:38:08 2017 +0200 BUG/MINOR: tcp-check: don't quit with pending data in the send buffer In the rare case where the "tcp-check send" directive is the last one in the list, it leaves the loop without sending the data. Fortunately, the polling is still enabled on output, resulting in the connection handler calling back to send what remains, but this is ugly and not very reliable. This may be backported to 1.7 and 1.6. commit a3782e7594b67cb82afb6a02c417b4c955b11c16 Author: Willy Tarreau Date: Wed Oct 4 08:45:19 2017 +0200 BUG/MEDIUM: tcp-check: properly indicate polling state before performing I/O While porting the connection to use the mux layer, it appeared that tcp-checks wouldn't receive anymore because the polling is not enabled before attempting to call xprt->rcv_buf() nor xprt->snd_buf(), and it is illegal to call these functions with polling disabled as they directly manipulate the FD state, resulting in an inconsistency where the FD is enabled and the connection's polling flags disabled. Till now it happened to work only because when recv() fails on EAGAIN it calls fd_cant_recv() which enables polling while signaling the failure, so that next time the message is received. But the connection's polling is never enabled, and any tiny change resulting in a call to conn_data_update_polling() immediately disables reading again. It's likely that this problem already happens on some corner cases such as multi-packet responses. It definitely breaks as soon as the response buffer is full but we don't support consuming more than one response buffer. This fix should be backported to 1.7 and 1.6. In order to check for the proper behaviour, this tcp-check must work and clearly show an SSH banner in recvfrom() as observed under strace, otherwise it's broken : tcp-check connect port 22 tcp-check expect rstring SSH tcp-check send blah commit 3cad39452004d74c2364ee19074105c2c02a9bfb Author: Willy Tarreau Date: Wed Oct 4 07:48:56 2017 +0200 CLEANUUP: checks: don't set conn->handle.fd to -1 This used to be needed to know whether there was a check in progress a long time ago (before tcp_checks) but this is not true anymore and even becomes wrong after the check is reused as conn_init() initializes it to DEAD_FD_MAGIC. commit 46392fdd083ee36f5ee5a82e138080a25e3340f1 Author: Baptiste Assmann Date: Tue Oct 3 23:16:36 2017 +0200 BUG/MEDIUM: tcp/http: set-dst-port action broken A regression has been introduced in commit 00005ce5a14310d248c9f20af9ef258d245d43b1: the port being changed is the one from 'cli_conn->addr.from' instead of 'cli_conn->addr.to'. This patch fixes the regression. Backport status: should be backported to HAProxy 1.7 and above. commit 4473a2e9aa7bc786275d1f111cf84570e4a93633 Author: Ilya Shipitsin Date: Fri Sep 22 22:33:16 2017 +0500 BUG/MINOR: contrib/halog: fixing small memory leak Issue was identified by cppcheck commit 93e8b88f06122523c1dfdc3b81f33f78e1fa8d90 Author: David Carlier Date: Thu Sep 21 14:36:43 2017 +0000 BUG/MINOR: log: fixing small memory leak in error code path. since we do not log the sample fetch when it is invalid, we can free the log data. commit 6fb4ba38e0bdd7e57ea9af5a3544f972ca11f7b9 Author: Willy Tarreau Date: Thu Sep 21 17:37:38 2017 +0200 BUG/MEDIUM: server: unwanted behavior leaving maintenance mode on tracked stopping server (take2) Previous patch got accidently broken. This one fixes it. commit e1e3947e7e6c2346038725b6ff09b38cfe7af8bf Author: Emeric Brun Date: Thu Sep 21 15:45:44 2017 +0200 BUG/MEDIUM: server: unwanted behavior leaving maintenance mode on tracked stopping server Leaving the maintenance state and if the server remains in stopping mode due to a tracked one: - We mistakenly try to grab some pending conns and shutdown backup sessions. - The proxy down time and last change were also mistakenly updated commit 2ba672726c05f55dc70d3e887444c64a8decca64 Author: Willy Tarreau Date: Thu Sep 21 15:24:10 2017 +0200 MINOR: ist: add a macro to ease const array initialization It's not possible to use strlen() in const arrays even with const strings, but we can use sizeof-1 via a macro. Let's provide this in the IST() macro, as it saves the developer from having to count the characters. commit 82967bf9b3e551d93fc5afc24958cc631f0c7fbf Author: Willy Tarreau Date: Wed Sep 20 17:46:46 2017 +0200 MINOR: connection: adjust CO_FL_NOTIFY_DATA after removal of flags After the removal of CO_FL_DATA_RD_SH and CO_FL_DATA_WR_SH, the aggregate mask CO_FL_NOTIFY_DATA was not updated. It happens that now CO_FL_NOTIFY_DATA and CO_FL_NOTIFY_DONE are similar, which may reveal some overlap between the ->wake and ->xprt_done callbacks. We'll see after the mux changes if both are still required. commit 5531d5732d9ec36f0e1d23e06c73d82a656f2210 Author: Willy Tarreau Date: Wed Sep 20 08:14:52 2017 +0200 MINOR: net_helper: add 64-bit read/write functions These ones are the same as the previous ones but for 64 bit values. We're using my_ntohll() and my_htonll() from standard.h for the byte order conversion. commit 2888c08346448db9eb521d9b65a406016049842f Author: Willy Tarreau Date: Tue Sep 19 17:27:05 2017 +0200 MINOR: net_helper: add write functions These ones are the equivalent of the read_* functions. They support writing unaligned words, possibly wrapping, in host and network order. The write_i*() functions were not implemented since the caller can already use the unsigned version. commit d5370e1d6c2007f07d1bf743aa6b8034a8196fda Author: Willy Tarreau Date: Tue Sep 19 14:59:52 2017 +0200 MINOR: net_helper: add functions to read from vectors This patch adds the ability to read from a wrapping memory area (ie: buffers). The new functions are called "readv_". The original ones were renamed to start with "read_" to make the difference more obvious between the read method and the returned type. It's worth noting that the memory barrier in readv_bytes() is critical, as otherwise gcc decides that it doesn't need the resulting data, but even worse, removes the length checks in readv_u64() and happily performs an out-of-bounds unaligned read using read_u64()! Such "optimizations" are a bit borderline, especially when they impact security like this... commit 26488ad358ab7c6d8cd801b3c63fa4cf87cedf8e Author: Willy Tarreau Date: Tue Sep 19 21:14:08 2017 +0200 MINOR: buffer: add b_end() and b_to_end() These ones return respectively the pointer to the end of the buffer and the distance between b->p and the end. These will simplify a bit some new code needed to parse directly from a wrapping buffer. commit 4a6425d37374f5f514a67fdb67454b24da64d7ef Author: Willy Tarreau Date: Tue Sep 19 14:18:46 2017 +0200 MINOR: buffer: add b_del() to delete a number of characters This will be used by code which directly parses buffers with no channel in the middle (eg: h2, might be used by checks as well). commit 36eb3a3ac83793748e9e5b644a848e6fa23c84cb Author: Willy Tarreau Date: Wed Sep 20 08:18:49 2017 +0200 MINOR: tools: make my_htonll() more efficient on x86_64 The current construct was made when developing on a 32-bit machine. Having a simple bswap operation replaced with 2 bswap, 2 shift and 2 or is quite of a waste of precious cycles... Let's provide a trivial asm-based implementation for x86_64. commit 2f1cacb1aaf0369cb39e14147f022fa6c321cb15 Author: Dragan Dosen Date: Mon Sep 18 09:20:43 2017 +0200 BUG/MINOR: contrib/modsecurity: close the va_list ap before return Make sure the va_list ap is closed before return in case the function vsnprintf() returned an error. commit ccf6100e11769b1650ba56fa727cb51993d0f3d2 Author: Dragan Dosen Date: Mon Sep 18 08:20:33 2017 +0200 BUG/MINOR: contrib/mod_defender: close the va_list argp before return Fix the case when the va_list argp could be left open if the function vsnprintf() returned a negative value. commit f8bb0ce4509460d07fcf554fec1db9576ab0bc82 Author: Christopher Faulet Date: Fri Sep 15 09:52:49 2017 +0200 MINOR: ssl: Remove useless checks on bind_conf or bind_conf->is_ssl bind_conf always exists at these steps and it is always for SSL listeners. commit 3bbd65b23eba214abf07d5728fe74909fc5d286d Author: Christopher Faulet Date: Fri Sep 15 11:55:45 2017 +0200 BUG/MINOR: dns: Fix check on nameserver in snr_resolution_cb snr_resolution_cb can be called with parameter set to NULL. So we must check it before using it. This is done most of time, except when we deal with invalid DNS response. commit ccbc3fd9f932958902c7ac78f95eba0badb01d57 Author: Christopher Faulet Date: Fri Sep 15 11:51:18 2017 +0200 BUG/MINOR: spoe: Don't rely on SPOE ctx in debug message when its creation failed If the SPOE context creation failed, we must not try to use it in the debug message used to notice the error. This patch must be backported in 1.7. commit 3dc860d19dfa573f2327ee7255b97a10444ff89b Author: Christopher Faulet Date: Fri Sep 15 11:39:36 2017 +0200 BUG/MINOR: compression: Check response headers before http-response rules eval This is required if we want to use res.comp or res.comp_algo sample fetches in http-response rules. This patch must be backported in 1.7. commit 03d85538b35ac63d7126cbd5ad06c64513aba3a7 Author: Christopher Faulet Date: Fri Sep 15 10:14:43 2017 +0200 BUG/MEDIUM: compression: Fix check on txn in smp_fetch_res_comp_algo The check was totally messed up. In the worse case, it led to a crash, when res.comp_algo sample fetch was retrieved on uncompressed response (with the compression enabled). This patch must be backported in 1.7. commit 0bf6fa5e40f75456880ec5e235c6d2c8264b2df5 Author: Willy Tarreau Date: Fri Sep 15 10:25:14 2017 +0200 MEDIUM: session: count the frontend's connections at a single place There are several places where we see feconn++, feconn--, totalconn++ and an increment on the frontend's number of connections and connection rate. This is done exactly once per session in each direction, so better take care of this counter in the session and simplify the callers. At least it ensures a better symmetry. It also ensures consistency as till now the lua/spoe/peers frontend didn't have these counters properly set, which can be useful at least for troubleshooting. commit 0c4ed35225080118466e574ff310c8b37a308496 Author: Willy Tarreau Date: Fri Sep 15 10:06:28 2017 +0200 MEDIUM: session: factor out duplicated code for conn_complete_session session_accept_fd() may either successfully complete a session creation, or defer it to conn_complete_session() depending of whether a handshake remains to be performed or not. The problem is that all the code after the handshake was duplicated between the two functions. This patch make session_accept_fd() synchronously call conn_complete_session() to finish the session creation. It is only needed to check if the session's task has to be released or not at the end, which is fairly minimal. This way there is now a single place where the sessions are created. commit eaa7e44ad749f0ff689f4cf07f762abcf20d1d6c Author: Willy Tarreau Date: Fri Sep 15 09:49:59 2017 +0200 MINOR: session: small cleanup of conn_complete_session() Commit 8e3c6ce ("MEDIUM: connection: get rid of data->init() which was not for data") simplified conn_complete_session() but introduced a confusing check which cannot happen on CO_FL_HANDSHAKE. Make it clear that this call is final and will either succeed and complete the session or fail. commit 05f5047d40a9623f96501aca3af66e4f82a4dd20 Author: Willy Tarreau Date: Fri Sep 15 09:19:58 2017 +0200 MINOR: listener: new function listener_release Instead of duplicating some sensitive listener-specific code in the session and in the stream code, let's call listener_release() when releasing a connection attached to a listener. commit 6f5e4b98dfecb8852bda07420680c2028d95d0d2 Author: Willy Tarreau Date: Fri Sep 15 09:07:56 2017 +0200 MEDIUM: session: take care of incrementing/decrementing jobs Each user of a session increments/decrements the jobs variable at its own place, resulting in a real mess and inconsistencies between them. Let's have session_new() increment jobs and session_free() decrement it. commit 2cc5bae0b897a377d9f6e2740a091008fbbe42e3 Author: Willy Tarreau Date: Fri Sep 15 08:18:11 2017 +0200 MINOR: listeners: make listeners count consistent with reality Some places call delete_listener() then decrement the number of listeners and jobs. At least one other place calls delete_listener() without doing so, but since it's in deinit(), it's harmless and cannot risk to cause zombie processes to survive. Given that the number of listeners and jobs is incremented when creating the listeners, it's much more logical to symmetrically decrement them when deleting such listeners. commit 0de59fd53a117a2878141522acd03e3338320218 Author: Willy Tarreau Date: Fri Sep 15 08:10:44 2017 +0200 MINOR: listeners: new function create_listeners This function is used to create a series of listeners for a specific address and a port range. It automatically calls the matching protocol handlers to add them to the relevant lists. This way cfgparse doesn't need to manipulate listeners anymore. As an added bonus, the memory allocation is checked. commit 31794892af34e2c52877928b14f9416a3017d27b Author: Willy Tarreau Date: Fri Sep 15 07:59:31 2017 +0200 MINOR: unix: remove the now unused proto_uxst.h file Since everything is self contained in proto_uxst.c there's no need to export anything. The same should be done for proto_tcp.c but the file contains other stuff that's not related to the TCP protocol itself and which should first be moved somewhere else. commit 9d5be5c82378ba3b02b2ff94178e5c8aa835f5c6 Author: Willy Tarreau Date: Fri Sep 15 07:55:51 2017 +0200 MINOR: protocols: register the ->add function and stop calling them directly cfgparse has no business directly calling each individual protocol's 'add' function to create a listener. Now that they're all registered, better perform a protocol lookup on the family and have a standard ->add method for all of them. commit 3228238c73e944d7e1efef5093d8bdb0de706dd0 Author: Willy Tarreau Date: Fri Sep 15 07:44:44 2017 +0200 MINOR: protocols: always pass a "port" argument to the listener creation It's a shame that cfgparse() has to make special cases of each protocol just to cast the port to the target address family. Let's pass the port in argument to the function. The unix listener simply ignores it. commit 20814ff1fcb7067d0d434cdbf00a34daeb22e193 Author: Willy Tarreau Date: Fri Sep 15 11:43:32 2017 +0200 MINOR: frontend: don't retrieve ALPN on the critical path It's pointless to read it on each and every accept(), as we only need it for reporting in debugging mode a few lines later. Let's move this part to the relevant block. commit 04b928693322932339840bcd3c5772263488d5e5 Author: Willy Tarreau Date: Fri Sep 15 11:01:04 2017 +0200 MINOR: peers: don't reference the incoming listener on outgoing connections Since v1.7 it's pointless to reference a listener when greating a session for an outgoing connection, it only complicates the code. SPOE and Lua were cleaned up in 1.8-dev1 but the peers code was forgotten. This patch fixes this by not assigning such a listener for outgoing connections. It also has the extra benefit of not discounting the outgoing connections from the number of allowed incoming connections (the code currently adds a safety marging of 3 extra connections to take care of this). commit b780166499ba95fe0a2356a2c5193d405817d77f Author: Willy Tarreau Date: Thu Sep 14 19:05:45 2017 +0200 BUILD: Makefile: improve detection of support for compiler warnings Some compiler versions don't emit an error when facing an unknown no-warning unless another error is reported, resulting in all -Wno-* options being enabled by default and being reported as wrong with build errors. Let's create a new "cc-nowarn" function to disable warnings only after checking that the positive one is supported. commit 065843c8767cd7851637b517990221709116b848 Author: Willy Tarreau Date: Wed Sep 13 16:58:27 2017 +0200 BUILD: Makefile: shut certain gcc/clang stupid warnings The recent gcc and clang are utterly broken and apparently written by people who don't use them anymore, because they emit warnings that are impossible to disable in the code, which is the opposite of what a warning should do. It is however possible to disable these warnings on the command line. This patch adds when supported : -Wno-format-truncation: bogus warning which is triggered on each snprintf() call based on the input type instead of the variables ranges, resulting in the impossibility to use "%02d" and similar. -Wno-address-of-packed-member: emitted for each and every line in ebtree.h by recent clang. Probably that the warning's author has never understood the use cases of packed structs and should be taught the use cases of the language he writes the compiler for. -Wno-null-dereference: emitted by clang on *(int *)0 = 0. The code will be updated to use a volatile instead but this recent change of behaviour will certainly cause quite some bugs in decades of existing code. Feel free to report new such stupid warnings and to propose patches to complete this list. commit d3a7f40359e59a0c44e929efa91761ad21d38167 Author: Willy Tarreau Date: Wed Sep 13 16:54:28 2017 +0200 BUILD: Makefile: add a function to detect support by the compiler of certain options The recent gcc and clang are utterly broken and apparently written by people who don't use them anymore, because they emit warnings that are impossible to disable in the code, which is the opposite of what a warning should do. It is however possible to disable these warnings on the command line, but not in a backwards-compatible way. Thus here we create a new function which detect if the compiler supports certain options, and which adds them if supported. commit c3680ecdf83e813fda6b9dd9231dfdb8611b3ea7 Author: Andjelko Iharos Date: Thu Jul 20 16:49:14 2017 +0200 MINOR: add severity information to cli feedback messages commit c4df59e91487ca78851a742adb70322264bd6992 Author: Andjelko Iharos Date: Thu Jul 20 11:59:48 2017 +0200 MINOR: cli: add socket commands and config to prepend informational messages with severity Adds cli commands to change at runtime whether informational messages are prepended with severity level or not, with support for numeric and worded severity in line with syslog severity level. Adds stats socket config keyword severity-output to set default behavior per socket on startup. commit ed0d96cac4c2235be3ab44b94269bbb8ab55dc6f Author: Olivier Houchard Date: Wed Sep 13 11:49:22 2017 +0200 MINOR: net_helper: Inline functions meant to be inlined. commit d697596c6c7633130253d39064cc10e1e9bfb69b Author: Thierry FOURNIER Date: Wed Jul 12 14:31:10 2017 +0200 MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references. commit 2da788e7558f28d44b96f5d43b4e3d625d694f14 Author: Thierry FOURNIER Date: Mon Sep 11 18:37:23 2017 +0200 MEDIUM: xref/lua: Use xref for referencing cosocket relation between stream and lua This relation will ensure that each was informed about death of another one. commit 3c65b7a91673895d69dfad8d022865af8a42a305 Author: Thierry FOURNIER Date: Thu Aug 31 20:35:18 2017 +0200 MINOR: xref: Add a new xref system xref is used to create a relation between two elements. Once an element is released, it breaks the relation. If the relation is already broken, it frees the xref struct. The pointer between two elements is a sort of refcount with max value 1. The relation is only between two elements. The pointer and the type of element a and b are conventional. Note that xref is initialised from Lua files because Lua is the only one user. commit b01d28f9766c96ebe4d29b9e8cc0c24c689fc03f Author: Thierry FOURNIER Date: Tue Jul 11 18:38:39 2017 +0200 BUG/MINOR: Lua: The socket may be destroyed when we try to access. When we try to access to other proxy context, we must check its existence because haproxy can kill it between the creation and the usage. This patch should be backported in 1.6 and 1.7 commit 5d468ca97b9437eb12403ce25d71622e508eac5c Author: Christopher Faulet Date: Mon Sep 11 09:27:29 2017 +0200 BUG/MEDIUM: http: Close streams for connections closed before a redirect A previous fix was made to prevent the connection to a server if a redirect was performed during the request processing when we wait to keep the client connection alive. This fix introduced a pernicious bug. If a client closes its connection immediately after sending a request, it is possible to keep stream alive infinitely. This happens when the connection closure is caught when the request is received, before the request parsing. To be more specific, this happens because the close event is not "forwarded", first because of the call to "channel_dont_connect" in the function "http_apply_redirect_rule", then because we want to keep the client connection alive, we explicitly call "channel_dont_close" in the function "http_request_forward_body". So, to fix the bug, instead of blocking the server connection, we force its shutdown. This will force the stream to re-evaluate all connexions states. So it will detect the client has closed its connection. This patch must be backported in 1.7. commit ddcde195ebb47a9707e97905501959e39808eb9b Author: Emmanuel Hocdet Date: Fri Sep 1 17:32:08 2017 +0200 MINOR: ssl: rework smp_fetch_ssl_fc_cl_str without internal ssl use smp_fetch_ssl_fc_cl_str as very limited usage (only work with openssl == 1.0.2 compiled with the option enable-ssl-trace). It use internal cipher.algorithm_ssl attribut and SSL_CIPHER_standard_name (available with ssl-trace). This patch implement this (debug) function in a standard way. It used common SSL_CIPHER_get_name to display cipher name. It work with openssl >= 1.0.2 and boringssl. commit 3d609a755e228cb6e601311e9268ad1eeac964e3 Author: Willy Tarreau Date: Wed Sep 6 14:22:45 2017 +0200 Revert "BUG/MINOR: server: Remove FQDN requirement for using init-addr and state file" This reverts commit 19e8aa58f7c42e602a95b4ceb4b254c424aed11c. It causes some trouble reported by Manu : listen tls [...] server bla 127.0.0.1:8080 [ALERT] 248/130258 (21960) : parsing [/etc/haproxy/test.cfg:53] : 'server bla' : no method found to resolve address '(null)' [ALERT] 248/130258 (21960) : Failed to initialize server(s) addr. According to Nenad : "It's not a good way to fix the issue we were experiencing before. It will need a bigger rewrite, because the logic in srv_iterate_initaddr needs to be changed." commit 19e8aa58f7c42e602a95b4ceb4b254c424aed11c Author: Nenad Merdanovic Date: Tue Sep 5 15:32:47 2017 +0200 BUG/MINOR: server: Remove FQDN requirement for using init-addr and state file Historically the DNS was the only way of updating the server IP dynamically and the init-addr processing and state file load required the server to have an FQDN defined. Given that we can now update the IP through the socket as well and also can have different init-addr values (like IP and 'none') - this requirement needs to be removed. This patch should be backported to 1.7. commit ab62f51959ff79a3b193be6e5b44f74ed6885648 Author: Christopher Faulet Date: Wed Aug 30 10:34:36 2017 +0200 MINOR: polling: Use fd_update_events to update events seen for a fd Now, the same function is used by all pollers to update events seen for a fd. This will ease the threads support integration. commit 21e9267ac3bd8642307b0f9791d3e46998141620 Author: Christopher Faulet Date: Wed Aug 30 10:30:04 2017 +0200 MINOR: fd: Add fd_update_events function This function should be called by the poller to set FD_POLL_* flags on an FD and update its state if needed. This function has been added to ease threads support integration. commit 9fab7bedfb1cdbd9bf390bdcc0a1d043c0bcc0b0 Author: Willy Tarreau Date: Tue Sep 5 15:32:56 2017 +0200 BUG/MEDIUM: epoll: ensure we always consider HUP and ERR Since commit 5be2f35 ("MAJOR: polling: centralize calls to I/O callbacks") that came into 1.6-dev1, each poller deals with its own events and decides to signal ability to receive or send on a file descriptor based on the active events on the file descriptor. The commit above was incorrectly done for the epoll code. Instead of checking the active events on the fd, it checks for the new events. In general these ones are the same for POLL_IN and POLL_OUT since they are always cleared prior to being computed, but it is possible that POLL_HUP and POLL_ERR were initially reported and are not reported again (especially for HUP). This could happen for example if POLL_HUP and POLL_IN were received together, the pending data exactly correspond to a full buffer which is read at once, preventing the POLL_HUP from being dealt with in the same call, and on the next call only POLL_OUT is reported (eg: to emit some response or peers protocol ACKs). In this case fd_may_recv() will not be enabled anymore and the close event will be missed. It seems quite hard to trigger this case, though it might explain some of the rare missed close events that were detected in the past on the peers. This fix needs to be backported to 1.6 and 1.7. commit 52a91d3d487350ca7229b849153b193d75efab07 Author: Emeric Brun Date: Thu Aug 31 14:41:55 2017 +0200 MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_' commit de2075fd2186ccf0563163e1b4295ff152860c3f Author: Christopher Faulet Date: Fri Sep 1 12:18:36 2017 +0200 MINOR: freq_ctr: Return the new value after an update This will ease threads support integration. commit 63fe65277a74101431a898703ebe73d71a1b573d Author: Christopher Faulet Date: Thu Aug 31 17:52:09 2017 +0200 MINOR: fd: Move (de)allocation of fdtab and fdinfo in (de)init_pollers This will be useful for the threads support integration. commit d82b180d6b57033a2d093329c6109bb7f14bd529 Author: Christopher Faulet Date: Wed Aug 30 10:07:47 2017 +0200 MINOR: fd: Use inlined functions to check fd state in fd_*_send/recv functions It these functions, the test is inverted and we rely on fd_recv/send_* function to check the fd state. This will ease threads support integration. commit 8db2fdfabaada02eabc92193b951d1821e49c1ba Author: Christopher Faulet Date: Wed Aug 30 09:59:38 2017 +0200 MINOR: fd: Add fd_active function This inlined function is used to check if a fd is active for receive or send. It will ease threads support integration. commit 6988f678cd6c7c91c5ded5b13a4958b0d3f8ae9d Author: Christopher Faulet Date: Thu Jul 27 15:18:52 2017 +0200 MINOR: http: Use a trash chunk to store decoded string of the HTTP auth header This string is used in sample fetches so it is safe to use a preallocated trash chunk instead of a buffer dynamically allocated during HAProxy startup. commit ca20d02ea8d5508fdbccf4319c0d7c83e67817bd Author: Christopher Faulet Date: Tue Aug 29 15:30:31 2017 +0200 MINOR: stick-tables: Make static_table_key a struct variable instead of a pointer First, this variable does not need to be publicly exposed because it is only used by stick_table functions. So we declare it as a global static in stick_table.c file. Then, it is useless to use a pointer. Using a plain struct variable avoids any dynamic allocation. commit ad405f171493fae6ff63f0400f575277a8cd1ecf Author: Christopher Faulet Date: Tue Aug 29 15:30:11 2017 +0200 MINOR: buffers: Move swap_buffer into buffer.c and add deinit_buffer function swap_buffer is a global variable only used by buffer_slow_realign. So it has been moved from global.h to buffer.c and it is allocated by init_buffer function. deinit_buffer function has been added to release it. It is also used to destroy the buffers' pool. commit 084aa9615b827d67b82df3d61fa3112fa588c786 Author: Christopher Faulet Date: Tue Aug 29 16:54:41 2017 +0200 MINOR: logs: Realloc log buffers only after the config is parsed and checked During the configuration parsing, log buffers are reallocated when global.max_syslog_len is updated. This can be done serveral time. So, instead of doing it serveral time, we do it only once after the configuration parsing. commit 0132d06f6880ca016f62ce418786d7d2bcb44d2b Author: Christopher Faulet Date: Wed Jul 26 15:33:35 2017 +0200 MINOR: logs: Use dedicated function to init/deinit log buffers Now, we use init_log_buffers and deinit_log_buffers to, respectively, initialize and deinitialize log buffers used for syslog messages. These functions have been introduced to be used by threads, to deal with thread-local log buffers. commit 3ef26398703a8f83a710278df8ec585ada161184 Author: Christopher Faulet Date: Tue Aug 29 16:46:57 2017 +0200 MEDIUM: chunks: Realloc trash buffers only after the config is parsed and checked Trash buffers are reallocated when "tune.bufsize" parameter is changed. Here, we just move the realloc after the configuration parsing. Given that the config parser doesn't rely on the trash size, it should be harmless. commit 748919a4c75495bc6464e82b52dd9b71cbbc23ca Author: Christopher Faulet Date: Wed Jul 26 14:59:46 2017 +0200 MINOR: chunks: Use dedicated function to init/deinit trash buffers Now, we use init_trash_buffers and deinit_trash_buffers to, respectively, initialize and deinitialize trash buffers (trash, trash_buf1 and trash_buf2). These functions have been introduced to be used by threads, to deal with thread-local trash buffers. commit 6c57dc9145f05ea3be76bba588e2a4d8e2ed40b7 Author: Christopher Faulet Date: Tue Jun 27 16:07:01 2017 +0200 MINOR: applet: Check applets_active_queue before processing applets queue This is useless for now, but it will allow a huge improvement when the multithreading will be merged. commit 8fe4891b116c6f48567e1eed333d73b564ed9ac0 Author: Christopher Faulet Date: Fri Jun 9 14:23:09 2017 +0200 MINOR: backends: Make get_server_* functions explicitly static Not used outside. commit 576c5aa25c871acadc1e2dbdc0121aacd0035a87 Author: Christopher Faulet Date: Tue Sep 5 09:51:57 2017 +0200 MINOR: fd: Set owner and iocb field before inserting a new fd in the fdtab This will be needed for concurrent accesses. commit d531f88622490af3c740b4f2935ea5e46ff30c47 Author: Christopher Faulet Date: Thu Jun 1 16:55:03 2017 +0200 MINOR: fd: Don't forget to reset fdtab[fd].update when a fd is added/removed It used to be guaranteed by the polling functions on a later call but with concurrent accesses it cannot be granted anymore. commit f5b8adc5c0e00c0b83439ded9822cb90fc0c51a2 Author: Christopher Faulet Date: Fri Jun 2 10:00:35 2017 +0200 MINOR: listeners: Change enable_listener and disable_listener into private functions These functions are only used in listener.c. commit 5580ba2e1182605e8cea3c043c1bb023a04c28d0 Author: Christopher Faulet Date: Mon Aug 28 15:29:20 2017 +0200 MINOR: listeners: Change listener_full and limit_listener into private functions These functions are only used in listener_accept. So there is no need to export them. commit ae459fd2068899055d8f0d93e3cbe2148d7b8ffc Author: Christopher Faulet Date: Mon Aug 28 14:28:44 2017 +0200 CLEANUP: memory: Remove unused function pool_destroy This one was never used. commit b6c8b0db043fb4a67ee826ee5ac2db113ef8ec50 Author: Daniel Schneller Date: Fri Sep 1 19:13:55 2017 +0200 DOC: Add note about "* " prefix in CSV stats The check_status field in the CSV stats output is conditionally prefixed with "* " if a check is currently underway. This can trip tools that parse the CSV output and compare against a well known list of values. This commit just adds this bit to the documentation. commit 35fe699ec73f769562526eb8279d0cc123f681f4 Author: Christopher Faulet Date: Tue Aug 29 16:06:38 2017 +0200 BUG/MEDIUM: http: Fix a regression bug when a HTTP response is in TUNNEL mode Unfortunatly, a regression bug was introduced in the commit 1486b0ab ("BUG/MEDIUM: http: Switch HTTP responses in TUNNEL mode when body length is undefined"). HTTP responses with undefined body length are blocked until timeout when the compression is enabled. This bug was fixed in commit 69744d92 ("BUG/MEDIUM: http: Fix blocked HTTP/1.0 responses when compression is enabled"). The bug is still the same. We do not forward response data because we are waiting for the synchronization between the HTTP request and the response. To fix the bug, conditions to infinitly forward channel data has been slightly relaxed. Now, it is done if there is no more analyzer registered on the channel or if _FLT_END analyzer is still there but without the flag CF_FLT_ANALYZE. This last condition is only possible when a channel is waiting the end of the other side. So, fundamentally, it means that no one is analyzing the channel anymore. This is a transitional state during a sync phase. This patch must be backported in 1.7. commit 43664768520e11bf96413732c3d41adb1e55cf8e Author: Emmanuel Hocdet Date: Wed Aug 9 18:26:20 2017 +0200 MINOR: ssl: remove duplicate ssl_methods in struct bind_conf Patch "MINOR: ssl: support ssl-min-ver and ssl-max-ver with crt-list" introduce ssl_methods in struct ssl_bind_conf. struct bind_conf have now ssl_methods and ssl_conf.ssl_methods (unused). It's error-prone. This patch remove the duplicate structure to avoid any confusion. commit 87e4302707adc2c9c36d640a8f7571d5e6d0e987 Author: Daniel Schneller Date: Fri Sep 1 19:29:57 2017 +0200 DOC: Refer to Mozilla TLS info / config generator As per a recent mailing list discussion, suggesting specific cipher settings is not too helpful, because they depend on a lot of factors, ranging from client capabilities, available TLS libraries, new security research, and others. To avoid the documentation from become stale -- and potentially wrong/dangerous, this commit adds links to Mozilla's well-reknowned TLS blog, as well as to their configuration generator. commit d8703e8cd72ba2579fb41b9529243117a8c7bf55 Author: Olivier Doucet Date: Thu Aug 31 11:05:10 2017 +0200 DOC: add CLI info on privilege levels commit bbae3f017060e2890104588000d91bcdc6c4ef2d Author: Willy Tarreau Date: Wed Aug 30 09:59:52 2017 +0200 MEDIUM: connection: remove useless flag CO_FL_DATA_WR_SH After careful inspection, this flag is set at exactly two places : - once in the health-check receive callback after receipt of a response - once in the stream interface's shutw() code where CF_SHUTW is always set on chn->flags The flag was checked in the checks before deciding to send data, but when it is set, the wake() callback immediately closes the connection so the CO_FL_SOCK_WR_SH flag is also set. The flag was also checked in si_conn_send(), but checking the channel's flag instead is enough and even reveals that one check involving it could never match. So it's time to remove this flag and replace its check with a check of CF_SHUTW in the stream interface. This way each layer is responsible for its shutdown, this will ease insertion of the mux layer. commit cde5651c4d03c718f3036a69ef51b8384f2b93ba Author: Willy Tarreau Date: Wed Aug 30 07:44:03 2017 +0200 CLEANUP: connection: remove the unused conn_sock_shutw_pending() This has never been used anywhere. commit 54e917cfa1e7b0539550ae32c48c76da2f169041 Author: Willy Tarreau Date: Wed Aug 30 07:35:35 2017 +0200 MEDIUM: connection: remove useless flag CO_FL_DATA_RD_SH This flag is both confusing and wrong. It is supposed to report the fact that the data layer has received a shutdown, but in fact this is reported by CO_FL_SOCK_RD_SH which is set by the transport layer after this condition is detected. The only case where the flag above is set is in the stream interface where CF_SHUTR is also set on the receiving channel. In addition, it was checked in the health checks code (while never set) and was always test jointly with CO_FL_SOCK_RD_SH everywhere, except in conn_data_read0_pending() which incorrectly doesn't match the second time it's called and is fortunately protected by an extra check on (ic->flags & CF_SHUTR). This patch gets rid of the flag completely. Now conn_data_read0_pending() accurately reports the fact that the transport layer has detected the end of the stream, regardless of the fact that this state was already consumed, and the stream interface watches ic->flags&CF_SHUTR to know if the channel was already closed by the upper layer (which it already used to do). The now unused conn_data_read0() function was removed. commit 5790eb0a762b79176a0469fbd5d0c690e3c381ae Author: Willy Tarreau Date: Mon Aug 28 17:18:36 2017 +0200 MINOR: stream: provide a new stream creation function for connections The purpose will be to create new streams for a given connection so that we can later abstract this from a mux. commit 0b74eae1f1bd54c00c4b799044ef67febdbd79a5 Author: Willy Tarreau Date: Mon Aug 28 19:02:51 2017 +0200 MEDIUM: session: add a pointer to a struct task in the session The session may need to enforce a timeout when waiting for a handshake. Till now we used a trick to avoid allocating a pointer, we used to set the connection's owner to the task and set the task's context to the session, so that it was possible to circle between all of them. The problem is that we'll really need to pass the pointer to the session to the upper layers during initialization and that the only place to store it is conn->owner, which is squatted for this trick. So this patch moves the struct task* into the session where it should always have been and ensures conn->owner points to the session until the data layer is properly initialized. commit ca3610251b3903c8db4f6d79a5d064dbe0d04962 Author: Willy Tarreau Date: Mon Aug 28 16:28:47 2017 +0200 CLEANUP: listener: remove the unused handler field Historically listeners used to have a handler depending on the upper layer. But now it's exclusively process_stream() and nothing uses it anymore so it can safely be removed. commit 87787acf724eeaf413393b5fce0047ad74356815 Author: Willy Tarreau Date: Mon Aug 28 16:22:54 2017 +0200 MEDIUM: stream: make stream_new() allocate its own task Currently a task is allocated in session_new() and serves two purposes : - either the handshake is complete and it is offered to the stream via the second arg of stream_new() - or the handshake is not complete and it's diverted to be used as a timeout handler for the embryonic session and repurposed once we land into conn_complete_session() Furthermore, the task's process() function was taken from the listener's handler in conn_complete_session() prior to being replaced by a call to stream_new(). This will become a serious mess with the mux. Since it's impossible to have a stream without a task, this patch removes the second arg from stream_new() and make this function allocate its own task. In session_accept_fd(), we now only allocate the task if needed for the embryonic session and delete it later. commit 8e3c6ce75a4f4b61e01ddb72d278ad447dbc1ae7 Author: Willy Tarreau Date: Mon Aug 28 15:46:01 2017 +0200 MEDIUM: connection: get rid of data->init() which was not for data The ->init() callback of the connection's data layer was only used to complete the session's initialisation since sessions and streams were split apart in 1.6. The problem is that it creates a big confusion in the layers' roles as the session has to register a dummy data layer when waiting for a handshake to complete, then hand it off to the stream which will replace it. The real need is to notify that the transport has finished initializing. This should enable a better splitting between these layers. This patch thus introduces a connection-specific callback called xprt_done_cb() which informs about handshake successes or failures. With this, data->init() can disappear, CO_FL_INIT_DATA as well, and we don't need to register a dummy data->wake() callback to be notified of errors. commit 8ff5a8d87fa99f3e9bbc2b7d68f922fcc0fedd31 Author: Willy Tarreau Date: Tue Aug 29 16:40:59 2017 +0200 BUG/MINOR: stream-int: don't check the CO_FL_CURR_WR_ENA flag The stream interface chk_snd() code checks if the connection has already subscribed to write events in order to avoid attempting a useless write() which will fail. But it used to check both the CO_FL_CURR_WR_ENA and the CO_FL_DATA_WR_ENA flags, while the former may only be present without the latterif either the other side just disabled writing did not synchronize yet (which is harmless) or if it's currently performing a handshake, which is being checked by the next condition and will be better dealt with by properly subscribing to the data events. This code was added back in 1.5-dev20 to limit the number of useless calls to splice() but both flags were checked at once while only CO_FL_DATA_WR_ENA was needed. This bug seems to have no impact other than making code changes more painful. This fix may be backported down to 1.5 though is unlikely to be needed there. commit 585744bf2ecf05a0761c2be10ebf866648491cd3 Author: Willy Tarreau Date: Thu Aug 24 14:31:19 2017 +0200 REORG/MEDIUM: connection: introduce the notion of connection handle Till now connections used to rely exclusively on file descriptors. It was planned in the past that alternative solutions would be implemented, leading to member "union t" presenting sock.fd only for now. With QUIC, the connection will need to continue to exist but will not rely on a file descriptor but a connection ID. So this patch introduces a "connection handle" which is either a file descriptor or a connection ID, to replace the existing "union t". We've now removed the intermediate "struct sock" which was never used. There is no functional change at all, though the struct connection was inflated by 32 bits on 64-bit platforms due to alignment. commit ee1bdd5a036117e9d038ee6338f206fc9294d523 Author: Willy Tarreau Date: Wed Aug 23 11:37:48 2017 +0200 OPTIM: lua: don't add "Connection: close" on the response Haproxy doesn't need this anymore, we're wasting cycles checking for a Connection header in order to add "Connection: close" only in the 1.1 case so that haproxy sees it and removes it. All tests were run in 1.0 and 1.1, with/without the request header, and in the various keep-alive/close modes, with/without compression, and everything works fine. It's worth noting that this header was inherited from the stats applet and that the same cleanup probably ought to be done there as well. commit a329463655af8bc3e286f50c437727f21865b550 Author: Willy Tarreau Date: Wed Aug 23 11:24:47 2017 +0200 OPTIM: lua: don't use expensive functions to parse headers in the HTTP applet In the HTTP applet, we have to parse the response headers provided by the application and to produce a response. strcasecmp() is expensive, and chunk_append() even more as it uses a format string. Here we check the string length before calling strcasecmp(), which results in strcasecmp() being called only on the relevant header in practise due to very few collisions on the name lengths, effectively dividing the number of calls by 3, and we replace chunk_appendf() with memcpy() as we already know the string lengths. Doing just this makes the "hello-world" applet 5% faster, reaching 41400 requests/s on a core i5-3320M. commit 85cb0aecf5b29d09d151086858bdcbe0e6f183fc Author: Willy Tarreau Date: Wed Aug 23 10:52:20 2017 +0200 BUG/MEDIUM: stream: properly set the required HTTP analysers on use-service Commit 4850e51 ("BUG/MAJOR: lua: Do not force the HTTP analysers in use-services") fixed a bug in how services are used in Lua, but this fix broke the ability for Lua services to support keep-alive. The cause is that we branch to a service while we have not yet set the body analysers on the request nor the response, and when we start to deal with the response we don't have any request analyser anymore. This leads the response forward engine to detect an error and abort. It's very likely that this also causes some random truncation of responses though this has not been observed during the tests. The root cause is not the Lua part in fact, the commit above was correct, the problem is the implementation of the "use-service" action. When done in an HTTP request, it bypasses the load balancing decisions and the connect() phase. These ones are normally the ones preparing the request analysers to parse the body when keep-alive is set. This should be dealt with in the main process_use_service() function in fact. That's what this patch does. If process_use_service() is called from the http-request rule set, it enables the XFER_BODY analyser on the request (since the same is always set on the response). Note that it's exactly what is being done on the stats page which properly supports keep-alive and compression. This fix must be backported to 1.7 and 1.6 as the breakage appeared in 1.6.3. commit c9f4ea0f61665a840fcce6d64edbf64a3b7ab3e1 Author: Willy Tarreau Date: Wed Aug 23 09:32:06 2017 +0200 MINOR: lua: properly process the contents of the content-length field The header's value was parsed with atoi() then compared against -1, meaning that all the unparsable stuff returning zero was not considered and that all multiples of 2^32 + 0xFFFFFFFF would continue to emit a chunk. Now instead we parse the value using a long long, only accept positive values and consider all unparsable values as incorrect and switch to either close or chunked encoding. This is more in line with what a client (including haproxy's parser) would expect. This may be backported as a cleanup to stable versions, though it's really unlikely that Lua applications are facing side effects of this. commit 06c75fec1783f1641b11f627be06f7df26e9d282 Author: Willy Tarreau Date: Wed Aug 23 09:10:38 2017 +0200 BUG/MEDIUM: lua: HTTP services must take care of body-less status codes The following Lua code causes emission of a final chunk after the body, which is wrong : core.register_service("send204", "http", function(applet) applet:set_status(204) applet:start_response() end) Indeed, responses with status codes 1xx, 204 and 304 do not contain any body and the message ends immediately after the empty header (cf RFC7230) so by emitting a 0 we're disturbing keep-alive responses. There's a workaround against this for now which consists in always emitting "Content-length: 0" but it may not be cool with 304 when clients use the headers to update their cache. This fix must be backported to stable versions back to 1.6. commit d9587418867d86aa503c249272634139fe909cbf Author: Willy Tarreau Date: Wed Aug 23 16:07:33 2017 +0200 BUG/MAJOR: lua: fix the impact of the scheduler changes again Commit d1aa41f ("BUG/MAJOR: lua: properly dequeue hlua_applet_wakeup() for new scheduler") tried to address the side effects of the scheduler changes on Lua, but it was not enough. Having some Lua code send data in chunks separated by one second each clearly shows busy polling being done. The issue was tracked down to hlua_applet_wakeup() being woken up on timer expiration, and returning itself without clearing the timeout, causing the task to be re-inserted with an expiration date in the past, thus firing again. In the past it was not a problem, as returning NULL was enough to clear the timer. Now we can't rely on this anymore so it's important to clear this timeout. No backport is needed, this issue is specific to 1.8-dev and results from an incomplete fix in the commit above. commit 0c219be3dffa85b936048cc6b0ca45f3cf684d23 Author: Willy Tarreau Date: Tue Aug 22 12:01:26 2017 +0200 BUG/MEDIUM: dns: fix accepted_payload_size parser to avoid integer overflow Since commit 9d8dbbc ("MINOR: dns: Maximum DNS udp payload set to 8192") it's possible to specify a packet size, but passing too large a size or a negative size is not detected and results in memset() being performed over a 2GB+ area upon receipt of the first DNS response, causing runtime crashes. We now check that the size is not smaller than the smallest packet which is the DNS header size (12 bytes). No backport is needed. commit f5f71304b009f6f00fdd521a10295a5314d2ffa4 Author: Baptiste Assmann Date: Mon Aug 21 13:21:48 2017 +0200 BUG/MINOR: dns: wrong resolution interval lead to 100% CPU Since the DNS layer split and the use of obj_type structure, we did not updated propoerly the code used to compute the interval between 2 resolutions. A nasty loop was then created when: - resolver's hold.valid is shorter than servers' check.inter - a valid response is available in the DNS cache A task was woken up for a server's resolution. The servers pick up the IP in the cache and returns without updating the 'last update' timestamp of the resolution (which is normal...). Then the task is woken up again for the same server. The fix simply computes now properly the interval between 2 resolutions and the cache is used properly while a new resolution is triggered if the data is not fresh enough. commit 8d112362343dfc7848b3838fe070358b38002afa Author: Baptiste Assmann Date: Mon Aug 21 16:11:32 2017 +0200 CLEANUP: dns: remove duplicated code in dns_validate_dns_response() a reader pointer comparison to the end of the buffer was performed twice while once is obviously enough. backport status: this patch can be backported into HAProxy 1.6 (with some modification. Please contact me) commit 60e9c264c961bacf95ad4bd20f2df60f77330ace Author: Baptiste Assmann Date: Mon Aug 21 16:09:59 2017 +0200 CLEANUP: dns: remove duplicated code in dns_resolve_recv() by mistake, I duplicated a block when introductiing the list_for_each loop on a resolution's requester list. commit e2d03d2a43473a637491c0a8c9e10d5caed8e63b Author: Baptiste Assmann Date: Fri Aug 18 23:36:07 2017 +0200 MINOR: dns: make SRV record processing more verbose For troubleshooting purpose, it may be important to know when a server got its fqdn updated by a SRV record. This patch makes HAProxy to report such events through stderr and logs. commit e70bc05b3a62ca84b5ce4440d340440f00411886 Author: Baptiste Assmann Date: Mon Aug 21 16:51:09 2017 +0200 MINOR: dns: automatic reduction of DNS accpeted payload size RFC 6891 states that if a DNS client announces "big" payload size and doesn't receive a response (because some equipments on the path may block/drop UDP fragmented packets), then it should try asking for smaller responses. commit 9d8dbbc56b73a4473b4e312317692a3d13d1da61 Author: Baptiste Assmann Date: Fri Aug 18 23:35:08 2017 +0200 MINOR: dns: Maximum DNS udp payload set to 8192 Following up DNS extension introduction, this patch aims at making the computation of the maximum number of records in DNS response dynamic. This computation is based on the announced payload size accepted by HAProxy. commit 747359eeca03716558063ddb3e7a9ba8627282e3 Author: Baptiste Assmann Date: Mon Aug 14 10:37:46 2017 +0200 BUG/MINOR: dns: server set by SRV records stay in "no resolution" status This patch fixes a bug where some servers managed by SRV record query types never ever recover from a "no resolution" status. The problem is due to a wrong function called when breaking the server/resolution (A/AAAA) relationship: this is performed when a server's SRV record disappear from the SRV response. commit 6ca71a9297b0f9e5a59b8adf3bc8204de5c308a4 Author: Frédéric Lécaille Date: Tue Aug 22 10:33:14 2017 +0200 BUG/MINOR: Wrong type used as argument for spoe_decode_buffer(). Contrary to 64-bits libCs where size_t type size is 8, on systems with 32-bits size of size_t is 4 (the size of a long) which does not equal to size of uint64_t type. This was revealed by such GCC warnings on 32bits systems: src/flt_spoe.c:2259:40: warning: passing argument 4 of spoe_decode_buffer from incompatible pointer type if (spoe_decode_buffer(&p, end, &str, &sz) == -1) ^ As the already existing code using spoe_decode_buffer() already use such pointers to uint64_t, in place of pointer to size_t ;), most of this code is in contrib directory, this simple patch modifies the prototype of spoe_decode_buffer() so that to use a pointer to uint64_t in place of a pointer to size_t, uint64_t type being the type finally required for decode_varint(). commit a5480694bfdff33e6327573cd7a97ff112ea4806 Author: Willy Tarreau Date: Wed Jul 26 08:07:15 2017 +0200 MINOR: http: export some of the HTTP parser macros The two macros EXPECT_LF_HERE and EAT_AND_JUMP_OR_RETURN were exported for use outside the HTTP parser. They now take extra arguments to avoid implicit pointers and jump labels. These will be used to reimplement a minimalist HTTP/1 parser in the H1->H2 gateway. commit b6b3f660d01747c7d816534a01a04cc98ace9bff Author: Willy Tarreau Date: Wed Aug 9 23:36:48 2017 +0200 TESTS: ist: add a test file for the functions This test file covers the various functions provided by ist.h. It allows both to test them for absence of regression, and to observe the code emitted at different optimization levels. commit e11f727c95f86143f44a3cc58d6e12024f8802cb Author: Willy Tarreau Date: Tue May 30 17:49:36 2017 +0200 MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important. commit f92a73d2fcdfe4afa974aed1c94a9ad75b694b41 Author: Willy Tarreau Date: Tue Jun 27 15:25:14 2017 +0200 MEDIUM: session: do not free a session until no stream references it We now refrain from clearing a session's variables, counters, and from releasing it as long as at least one stream references it. For now it never happens but with H2 this will be mandatory to avoid double frees. commit 2bfd35885ee1fb73461477e3237b28f4d6c7f2f8 Author: Willy Tarreau Date: Tue Jun 27 15:20:05 2017 +0200 MINOR: stream: link the stream to its session Now each stream is added to the session's list of streams, so that it will be possible to know all the streams belonging to a session, and to know if any stream is still attached to a sessoin. commit bcb86abaca2607a6658a3fc5a8b41b2710eae102 Author: Willy Tarreau Date: Tue Jun 27 15:13:43 2017 +0200 MINOR: session: add a streams field to the session struct This will be used to hold the list of streams belonging to a given session. commit 82032f12236b5d1cb7c7a3196193bf869ba57096 Author: Willy Tarreau Date: Thu Jul 27 13:35:34 2017 +0200 MINOR: chunks: add chunk_memcpy() and chunk_memcat() These two functions respectively copy a memory area onto the chunk, and append the contents of a memory area over a chunk. They are convenient to prepare binary output data to be sent and will be used for HTTP/2. commit 686408bb6ccf5d8f3a21877c9a3ba120e09fd88f Author: Baptiste Assmann Date: Fri Aug 18 10:15:42 2017 +0200 MINOR: dns: default "hold obsolete" timeout set to 0 The "hold obsolete" timer is used to prevent HAProxy from moving a server to an other IP or from considering the server as DOWN if the IP currently affected to this server has not been seen for this period of time in DNS responses. That said, historically, HAProxy used to update servers as soon as the IP has disappeared from the response. Current default timeout break this historical behavior and may change HAProxy's behavior when people will upgrade to 1.8. This patch changes the default value to 0 to keep backward compatibility. commit 2af08fe3de1a816bed2ec7b68064b6361a76bcce Author: Baptiste Assmann Date: Mon Aug 14 00:13:01 2017 +0200 MINOR: dns: enabled edns0 extension and make accpeted payload size tunable Edns extensions may be used to negotiate some settings between a DNS client and a server. For now we only use it to announce the maximum response payload size accpeted by HAProxy. This size can be set through a configuration parameter in the resolvers section. If not set, it defaults to 512 bytes. commit 572ab8b26990900f8bd722d54bb53af2507ff480 Author: Baptiste Assmann Date: Mon Aug 14 00:04:58 2017 +0200 MINOR: dns: new dns record type (RTYPE) for OPT DNS record type OPT is required to send additional records. OPT has been assigned ID 41. commit 6fb8192b28a90d75cb1c40f8ad472c72ff335a66 Author: Baptiste Assmann Date: Mon Aug 14 16:35:45 2017 +0200 MINOR: dns: enable caching of responses for server set by a SRV record The function srv_set_fqdn() is used to update a server's fqdn and set accordingly its DNS resolution. Current implementation prevents a server whose update is triggered by a SRV record from being linked to an existing resolution in the cache (if applicable). This patch aims at fixing this. commit 7d2a12662bec18e4a5b2b59c6174da07d2faa29f Author: Baptiste Assmann Date: Mon Aug 14 16:38:29 2017 +0200 MINOR: dns: ability to use a SRV resolution for multiple backends Current code implementation prevents multiple backends from relying on the same SRV resolution. Actually, only the first backend which triggers the resolution gets updated. This patch makes HAProxy to process the whole list of the 'curr' requesters to apply the changes everywhere (hence, the cache also applies to SRV records...) commit a644aa8e395a9955c19ac56c9dbcbbd49e2db30c Author: Baptiste Assmann Date: Sat Aug 12 11:16:55 2017 +0200 MINOR: dns: make debugging function dump_dns_config() compatible with SRV records This function is particularly useful when debugging DNS resolution at run time in HAProxy. SRV records must be read differently, hence we have to update this function. commit 20433271709517c5988ef28d6145f2cfe0e61a74 Author: Baptiste Assmann Date: Sat Aug 12 09:03:06 2017 +0200 MINOR: dns: duplicate entries in resolution wait queue for SRV records This loop is useless and duplicates entries into the resolution queue for nothing. Entries are already added a bit sooner in the same function. commit 63a28110771ecb18b84409dab4fa1b7a17253e52 Author: Baptiste Assmann Date: Fri Aug 11 10:37:20 2017 +0200 MINOR: dns: update dns response buffer reading pointer due to SRV record DNS SRV records uses "dns name compression" to store the target name. "dns compression" principle is simple. Let's take the name below: 3336633266663038.red.default.svc.cluster.local. It can be stored "as is" in the response or it can be compressed like this: 3336633266663038 and would point to the string '.red.default.svc.cluster.local.' availble in the question section for example. This mechanism allows storing much more data in a single DNS response. This means the flag "record->data_len" which stores the size of the record (hence the whole string, uncompressed) can't be used to move the pointer forward when reading responses. We must use the "offset" integer which means the real number of bytes occupied by the target name. If we don't do that, we can properly read the first SRV record, then we loose alignment and we start reading unrelated data (still in the response) leading to a false negative error treated as an "invalid" response... commit ddc8ce6d29eb131bcb8f864b5bda46b78e4e42e6 Author: Baptiste Assmann Date: Fri Aug 11 10:31:22 2017 +0200 MINOR: dns: update record dname matching for SRV query types DNS response for SRV queries look like this: - query dname looks like '_http._tcp.red.default.svc.cluster.local' - answer record dname looks like '3336633266663038.red.default.svc.cluster.local.' Of course, it never matches... and it triggers many false positive in the current code (which is suitable for A/AAAA/CNAME). This patch simply ignores this dname matching in the case of SRV query type. commit 251abb9cbe894411856d0f9da7637ea62c0740ba Author: Baptiste Assmann Date: Fri Aug 11 09:58:27 2017 +0200 MINOR: dns: Update analysis of TRUNCATED response for SRV records First implementation of the DNS parser used to consider TRUNCATED responses as errors and triggered a failover to an other query type (usually A to AAAA or vice-versa). When we query for SRV records, a TRUNCATED response still contains valid records we can exploit, so we shouldn't trigger a failover in such case. Note that we had to move the maching against the flag later in the response parsing (actually, until we can read the query type....) commit 97148f60b8feec39b76768d1bcfab6d755c12164 Author: Olivier Houchard Date: Wed Aug 16 17:29:11 2017 +0200 MINOR: init: Fix CPU affinity setting on FreeBSD. Use a cpuset_t instead of assuming the cpu mask is an unsigned long. This should fix setting the CPU affinity on FreeBSD >= 11. This patch should be backported to stable releases. commit 0d00593361b91017b894c4c7d5e24721a7838d6e Author: Olivier Houchard Date: Mon Aug 14 15:59:44 2017 +0200 CLEANUP: raw_sock: Use a better name for the constructor than __ssl_sock_deinit() I just noticed the raw socket constructor was called __ssl_sock_deinit, which is a bit confusing, and wrong twice, so the attached patch renames it to __raw_sock_init, which seems more correct. commit 7632548d9789e1588a1ad70ced3b87ef06d41b21 Author: Willy Tarreau Date: Thu Aug 17 15:54:46 2017 +0200 BUG/MAJOR: stream: in stream_free(), close the front endpoint and not the origin stream_free() used to close the front connection by using s->sess->origin, instead of using s->si[0].end. This is very visible in HTTP/2 where the front connection is abusively closed and causes all sort of issues including crashes caused by double closes due to the same origin being referenced many times. It's also suspected that it may have caused some of the early issues met during the Lua development. It's uncertain whether stable branches are affected. It might be worth backporting it once it has been confirmed not to create new impacts. commit 50bc31d4d6a6861eac3ca2298b93cd62a51310ed Author: Willy Tarreau Date: Wed Aug 16 15:35:19 2017 +0200 BUILD/MINOR: build without openssl still broken As mentionned in commit cf4e496c9 ("BUG/MEDIUM: build without openssl broken"), commit 872f9c213 ("MEDIUM: ssl: add basic support for OpenSSL crypto engine") broke the build without openssl support. But the former did only fix it when openssl is not enabled, but not when it's not installed on the system : In file included from src/haproxy.c:112: include/proto/ssl_sock.h:24:25: openssl/ssl.h: No such file or directory In file included from src/haproxy.c:112: include/proto/ssl_sock.h:45: error: syntax error before "SSL_CTX" include/proto/ssl_sock.h:75: error: syntax error before '*' token include/proto/ssl_sock.h:75: warning: type defaults to `int' in declaration of `ssl_sock_create_cert' include/proto/ssl_sock.h:75: warning: data definition has no type or storage class include/proto/ssl_sock.h:76: error: syntax error before '*' token include/proto/ssl_sock.h:76: warning: type defaults to `int' in declaration of `ssl_sock_get_generated_cert' include/proto/ssl_sock.h:76: warning: data definition has no type or storage class include/proto/ssl_sock.h:77: error: syntax error before '*' token Now we also surround the include with #ifdef USE_OPENSSL to fix this. No backport is needed since openssl async engines were not backported. commit 15969297af0371fec1523f7a4182dae87e19c782 Author: Emmanuel Hocdet Date: Fri Aug 11 10:56:00 2017 +0200 BUILD: ssl: replace SSL_CTX_get0_privatekey for openssl < 1.0.2 Commit 48a8332a introduce SSL_CTX_get0_privatekey in openssl-compat.h but SSL_CTX_get0_privatekey access internal structure and can't be a candidate to openssl-compat.h. The workaround with openssl < 1.0.2 is to use SSL_new then SSL_get_privatekey. commit 286ec68f8230853f6e19913022b97cd7b9dad0e8 Author: Willy Tarreau Date: Wed Aug 9 16:35:44 2017 +0200 BUILD/MINOR: cli: shut a minor gcc warning in "show fd" Recent commit 7a4a0ac ("MINOR: cli: add a new "show fd" command") introduced a warning when building at -O2 and above. The compiler doesn't know if a variable's value might have changed between two if blocks so warns that some values might be used uninitialized, which is not the case. Let's simply initialize them to shut the warning. commit ecfa18dda64db46f5e4285b9beef82ce09139bf2 Author: Olivier Houchard Date: Mon Aug 7 17:30:03 2017 +0200 MINOR: doc: Document SRV label usage. commit b68fda40d7f27842b1911662efb9e97a457c8e6e Author: Olivier Houchard Date: Fri Aug 4 18:39:01 2017 +0200 MINOR: check: Fix checks when using SRV records. When started, a server may not yet have an associated protocol, so don't bother trying to run the checks until it is there. commit 8da5f98fbe227f7fa619989867a942ac78e2a8f8 Author: Olivier Houchard Date: Fri Aug 4 18:35:36 2017 +0200 MINOR: dns: Handle SRV records. Make it so for each server, instead of specifying a hostname, one can use a SRV label. When doing so, haproxy will first resolve the SRV label, then use the resulting hostnames, as well as port and weight (priority is ignored right now), to each server using the SRV label. It is resolved periodically, and any server disappearing from the SRV records will be removed, and any server appearing will be added, assuming there're free servers in haproxy. commit e962fd880dbd33fee0657ff0a071923c2e563766 Author: Olivier Houchard Date: Mon Aug 7 19:20:04 2017 +0200 Add a few functions to do unaligned access. Add a few functions to read 16bits and 32bits integers that may be unaligned, both in host and network order. commit e2c222b12f161403a84a6f6f5c180f27ade18d20 Author: Olivier Houchard Date: Fri Aug 4 18:31:56 2017 +0200 MINOR: obj: Add a new type of object, OBJ_TYPE_SRVRQ. dns_srvrq will be objects used for dealing with SRV records. commit a8c6db8d2d97629b2734c1d2be0860b6b11e5709 Author: Olivier Houchard Date: Thu Jul 6 18:46:47 2017 +0200 MINOR: dns: Cache previous DNS answers. As DNS servers may not return all IPs in one answer, we want to cache the previous entries. Those entries are removed when considered obsolete, which happens when the IP hasn't been returned by the DNS server for a time defined in the "hold obsolete" parameter of the resolver section. The default is 30s. commit aa0d6372922882f58b54f851dfd7ca1f5a71b26d Author: Emmanuel Hocdet Date: Wed Aug 9 11:24:25 2017 +0200 MINOR: ssl: allow to start without certificate if strict-sni is set With strict-sni, ssl connection will fail if no certificate match. Have no certificate in bind line, fail on all ssl connections. It's ok with the behavior of strict-sni. When 'generate-certificates' is set 'strict-sni' is never used. When 'strict-sni' is set, default_ctx is never used. Allow to start without certificate only in this case. Use case is to start haproxy with ssl before customer start to use certificates. Typically with 'crt' on a empty directory and 'strict-sni' parameters. commit 3169471964fdc49963e63f68c1fd88686821a0c4 Author: Frédéric Lécaille Date: Tue Aug 1 08:47:19 2017 +0200 MINOR: Add server port field to server state file. This patch adds server ports to server state file at the end of each line for backward compatibility. commit 48a8332a4a82f151877bd6baf567031088845f2d Author: Christopher Faulet Date: Fri Jul 28 16:56:09 2017 +0200 BUG/MEDIUM: ssl: Fix regression about certificates generation Since the commit f6b37c67 ["BUG/MEDIUM: ssl: in bind line, ssl-options after 'crt' are ignored."], the certificates generation is broken. To generate a certificate, we retrieved the private key of the default certificate using the SSL object. But since the commit f6b37c67, the SSL object is created with a dummy certificate (initial_ctx). So to fix the bug, we use directly the default certificate in the bind_conf structure. We use SSL_CTX_get0_privatekey function to do so. Because this function does not exist for OpenSSL < 1.0.2 and for LibreSSL, it has been added in openssl-compat.h with the right #ifdef. commit 7a4a0ac71d51c592f20622d1b5530042ae79f3b9 Author: Willy Tarreau Date: Tue Jul 25 19:32:50 2017 +0200 MINOR: cli: add a new "show fd" command This one dumps the fdtab for all active FDs with some quickly interpretable characters to read the flags (like upper case=set, lower case=unset). It can probably be improved to report fdupdt[] and/or fdinfo[] but at least it provides a good start and allows to see how FDs are seen. When the fd owner is a connection, its flags are also reported as it can help compare with the polling status, and the target (fe/px/sv) as well. When it's a listener, the listener's state is reported as well as the frontend it belongs to. commit 6d0d3f65465c3d2693757faf2f7598a8c811dbc3 Author: Willy Tarreau Date: Tue Jul 25 19:46:06 2017 +0200 MINOR: listener: add a function to return a listener's state as a string This will be used in debugging output, so it's a short 3-character string. commit d6129fca8b541f50d28a0a7784b1ed6ced71510e Author: Willy Tarreau Date: Fri Jul 28 16:52:23 2017 +0200 DOC: fix alphabetical order of "show commands" in management.txt "show acl" and "show backend" were misplaced. commit ceee9338625a81811e8475fdca74511f9ae2e654 Author: Adis Nezirovic Date: Wed Jul 26 09:19:06 2017 +0200 BUG/MINOR: lua: Fix bitwise logic for hlua_server_check_* functions. The logical operations were inverted so enable/disable operations did the opposite. The bug is present since 1.7 so the fix should be backported there. commit 174dfe55a01bca1206d4107bddc2145a3bd2b594 Author: Emmanuel Hocdet Date: Fri Jul 28 15:01:05 2017 +0200 MINOR: ssl: add "no-ca-names" parameter for bind This option prevent to send CA names in server hello message when ca-file is used. This parameter is also available in "crt-list". commit 46d5b0872ada05952574f070b751c793ab6af4fc Author: Willy Tarreau Date: Wed Jul 26 20:13:37 2017 +0200 BUG/MEDIUM: stream: don't retry SSL connections which fail the SNI name check Commits 2ab8867 ("MINOR: ssl: compare server certificate names to the SNI on outgoing connections") and 96c7b8d ("BUG/MINOR: ssl: Fix check against SNI during server certificate verification") made it possible to check that the server's certificate matches the name presented in the SNI field. While it solves a class of problems, it opens another one which is that by failing such a connection, we'll retry it and put more load on the server. It can be a real problem if a user can trigger this issue, which is what will very often happen when the SNI is forwarded from the client to the server. This patch solves this by detecting that this very specific hostname verification failed and that the hostname was provided using SNI, and then it simply disables retries and the failure is immediate. At the time of writing this patch, the previous patches were not backported (yet), so no backport is needed for this one unless the aforementionned patches are backported as well. This patch requires previous patches "BUG/MINOR: ssl: make use of the name in SNI before verifyhost" and "MINOR: ssl: add a new error code for wrong server certificates". commit 71d058c288b4d108ce9352da712fd33fa5a434c7 Author: Willy Tarreau Date: Wed Jul 26 20:09:56 2017 +0200 MINOR: ssl: add a new error codes for wrong server certificates If a server presents an unexpected certificate to haproxy, that is, a certificate that doesn't match the expected name as configured in verifyhost or as requested using SNI, we want to store that precious information. Fortunately we have access to the connection in the verification callback so it's possible to store an error code there. For this purpose we use CO_ER_SSL_MISMATCH_SNI (for when the cert name didn't match the one requested using SNI) and CO_ER_SSL_MISMATCH for when it doesn't match verifyhost. commit ad92a9a7be1e249234143fb0c56ed51f7a446841 Author: Willy Tarreau Date: Fri Jul 28 11:38:41 2017 +0200 BUG/MINOR: ssl: make use of the name in SNI before verifyhost Commit 2ab8867 ("MINOR: ssl: compare server certificate names to the SNI on outgoing connections") introduced the ability to check server cert names against the name provided with in the SNI, but verifyhost was kept as a way to force the name to check against. This was a mistake, because : - if an SNI is used, any static hostname in verifyhost will be wrong ; worse, if it matches and doesn't match the SNI, the server presented the wrong certificate ; - there's no way to have a default name to check against for health checks anymore because the point above mandates the removal of the verifyhost directive This patch reverses the ordering of the check : whenever SNI is used, the name provided always has precedence (ie the server must always present a certificate that matches the requested name). And if no SNI is provided, then verifyhost is used, and will be configured to match the server's default certificate name. This will work both when SNI is not used and for health checks. If the commit 2ab8867 is backported in 1.7 and/or 1.6, this one must be backported too. commit 96c7b8dbd2d43017c66af098d97e8e99236a7517 Author: Christopher Faulet Date: Wed Jul 26 11:50:01 2017 +0200 BUG/MINOR: ssl: Fix check against SNI during server certificate verification This patch fixes the commit 2ab8867 ("MINOR: ssl: compare server certificate names to the SNI on outgoing connections") When we check the certificate sent by a server, in the verify callback, we get the SNI from the session (SSL_SESSION object). In OpenSSL, tlsext_hostname value for this session is copied from the ssl connection (SSL object). But the copy is done only if the "server_name" extension is found in the server hello message. This means the server has found a certificate matching the client's SNI. When the server returns a default certificate not matching the client's SNI, it doesn't set any "server_name" extension in the server hello message. So no SNI is set on the SSL session and SSL_SESSION_get0_hostname always returns NULL. To fix the problemn, we get the SNI directly from the SSL connection. It is always defined with the value set by the client. If the commit 2ab8867 is backported in 1.7 and/or 1.6, this one must be backported too. Note: it's worth mentionning that by making the SNI check work, we introduce another problem by which failed SNI checks can cause long connection retries on the server, and in certain cases the SNI value used comes from the client. So this patch series must not be backported until this issue is resolved. commit 9b82a588cd7ec1fb4d8016020d8022f962d20df9 Author: Thierry FOURNIER Date: Mon Jul 24 13:30:43 2017 +0200 MINOR: lua: Add lists of frontends and backends Adis Nezirovic reports: While playing with Lua API I've noticed that core.proxies attribute doesn't return all the proxies, more precisely the ones with same names (e.g. for frontend and backend with the same name it would only return the latter one). So, this patch fixes this problem without breaking the actual behaviour. We have two case of proxies with frontend/backend capabilities: The first case is the listen. This case is not a problem because the proxy object process these two entities as only one and it is the expected behavior. With these case the "proxies" list works fine. The second case is the frontend and backend with the same name. i think that this case is possible for compatibility with 'listen' declaration. These two proxes with same name and different capabilities must not processed with the same object (different statitics, differents orders). In fact, one the the two object crush the other one whoch is no longer accessible. To fix this problem, this patch adds two lists which are "frontends" and "backends", each of these list contains specialized proxy, but warning the "listen" proxy are declare in each list. commit 817e7598986b8a239e9e4d06395822ab3c572f5a Author: Thierry FOURNIER Date: Mon Jul 24 14:35:04 2017 +0200 DOC: lua: Proxy class doc update Following the patch adding the name of the proxy, this patch contains the associated doc update. commit f2bbe38242cb4c3801c87323efc0d4f0400b9ae6 Author: Thierry FOURNIER Date: Mon Jul 24 13:59:22 2017 +0200 MINOR: lua: Add proxy as member of proxy object. By Adis Nezirovic: This is just for convenience and uniformity, Proxy.servers/listeners returns a table/hash of objects with names as keys, but for example when I want to pass such object to some other Lua function I have to manually copy the name (or wrap the object), since the object itself doesn't expose name info. This patch simply adds the proxy name as member of the proxy object. commit d1aa41f83b75f874d9586facc0f93ae4b1b51783 Author: Willy Tarreau Date: Fri Jul 21 16:41:56 2017 +0200 BUG/MAJOR: lua: properly dequeue hlua_applet_wakeup() for new scheduler The recent scheduler change broke the Lua co-sockets due to hlua_applet_wakeup() returning NULL after waking the applet up. With the previous scheduler, returning NULL was a way to do nothing on return. With the new one it keeps TASK_RUNNING set, causing all new notifications to end up into t->pending_state instead of t->state, and prevents the task from being added into the run queue again, so and it's never woken up anymore. The applet keeps waking up, causing hlua_socket_handler() to do nothing new, then si_applet_wake_cb() calling stream_int_notify() to try to wake the task up, which it can't do due to the TASK_RUNNING flag, then decide that since the associated task is not in the run queue, it needs to call stream_int_update_applet() to propagate the update. This last one finds that the applet needs to be woken up to deal with the last reported events and calling appctx_wakeup() again. Previously, this situation didn't exist because the task was always added in the run queue despite the TASK_RUNNING flag. By returning the task instead in hlua_applet_wakeup(), we can ensure its flag is properly cleared and the task is requeued if needed or just sits waiting for new events to happen. This fix requires the previous ones ("BUG/MINOR: lua: always detach the tcp/http tasks before freeing them") and MINOR: task: always preinitialize the task's timeout in task_init(). Thanks to Thierry, Christopher and Emeric for the long head-scratching session! No backport is needed as the bug doesn't appear in older versions and it's unsure whether we'll not break something by backporting it. commit f1d33db10a3f726485f58b67e3cea30af250bbbc Author: Willy Tarreau Date: Mon Jul 24 17:55:20 2017 +0200 CLEANUP: task: remove all initializations to TICK_ETERNITY after task_new() This is now guaranteed by design, simply remove these unneeded parts to avoid confusion. commit f42199975c8ab0442935ecb8f3db608c051675b5 Author: Willy Tarreau Date: Mon Jul 24 17:52:58 2017 +0200 MINOR: task: always preinitialize the task's timeout in task_init() task_init() is called exclusively by task_new() which is the only way to create a task. Most callers set t->expire to TICK_ETERNITY, some set it to another value and a few like Lua don't set it at all as they don't need a timeout, causing random values to be used in case the task gets queued. Let's always set t->expire to TICK_ETERNITY in task_init() so that all tasks are now initialized in a clean state. This patch can be backported as it will definitely make the code more robust (at least the Lua code, possibly other places). commit bd7fc95edbce821f1d7b745a7b75deef4d6b1e27 Author: Willy Tarreau Date: Mon Jul 24 17:35:27 2017 +0200 BUG/MINOR: lua: always detach the tcp/http tasks before freeing them In hlua_{http,tcp}_applet_release(), a call to task_free() is performed to release the task, but no task_delete() is made on these tasks. Till now it wasn't much of a problem because this was normally not done with the task in the run queue, and the task was never put into the wait queue since it doesn't have any timer. But with threading it will become an issue. And not having this already prevents another bug from being fixed. Thanks to Christopher for spotting this one. A backport to 1.7 and 1.6 is preferred for safety. commit d02210cd30dad77711e58857e610cd9bd71c49ff Author: Christopher Faulet Date: Mon Jul 24 16:24:39 2017 +0200 MINOR: samples: Don't allocate memory for SMP_T_METH sample when method is known For known methods (GET,POST...), in samples, an enum is used instead of a chunk to reference the method. So there is no needs to allocate memory when a variable is stored with this kind of sample. commit 5db105e8b2dea551a217dfb9f8fdb00c5a769332 Author: Christopher Faulet Date: Mon Jul 24 16:07:12 2017 +0200 MINOR: samples: Handle the type SMP_T_METH in smp_is_safe and smp_is_rw For all known methods, samples are considered as safe and rewritable. For unknowns, we handle them like strings (SMP_T_STR). commit ec1005134995d5c9da0d8c8b20ab96856449d83c Author: Christopher Faulet Date: Mon Jul 24 15:38:41 2017 +0200 MINOR: samples: Handle the type SMP_T_METH when we duplicate a sample in smp_dup First, the type SMP_T_METH was not handled by smp_dup function. It was never called with this kind of samples, so it's not really a problem. But, this could be useful in future. For all known HTTP methods (GET, POST...), there is no extra space allocated for a sample of type SMP_T_METH. But for unkown methods, it uses a chunk. So, like for strings, we duplicate data, using a trash chunk. commit a9f040453acc09e888c3f2dc983f15dcf3fa66e3 Author: Nenad Merdanovic Date: Sun Jul 23 22:04:59 2017 -0400 BUG/MINOR: lua: Correctly use INET6_ADDRSTRLEN in Server.get_addr() The get_addr() method of the Lua Server class incorrectly used INET_ADDRSTRLEN for IPv6 addresses resulting in failing to convert longer IPv6 addresses to strings. This fix should be backported to 1.7. commit 3849473828f319829aff422d2fbbce0823e65d64 Author: Nenad Merdanovic Date: Sun Jul 23 22:04:58 2017 -0400 BUG/MINOR: lua: Fix Server.get_addr() port values The get_addr() method of the Lua Server class was using the 'sockaddr_storage addr' member to get the port value. HAProxy does not store ports in this member as it uses a separate member, called 'svc_port'. This fix should be backported to 1.7. commit b781dbede304ef44a8c6200868c1eddd3673cd26 Author: David Carlier Date: Fri Jul 21 08:44:40 2017 +0100 MINOR: memory: remove macros We finally get rid of the macros and use usual memory management functions directly. commit 56d260916f61e48c8b2f1fd2f9431afac776d160 Author: Christopher Faulet Date: Thu Jul 20 11:05:10 2017 +0200 BUG/MAJOR: http: Fix possible infinity loop in http_sync_(req|res)_state In commit "MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flags", it is possible to have an infinite loop on HTTP_MSG_CLOSING state. commit ac752ff68cd3ac88a7a27ce17daa5c3f0c839694 Author: ben51degrees Date: Wed Jul 19 16:22:04 2017 +0100 DOC: Updated 51Degrees git URL to point to a stable version. The previously documented location doesn't work anymore and must not be used. Warning for backports, different branches are in use depending on the version (v3.2.10 for 1.7, v3.2.5 for 1.6). commit abd9bb20b76818c9f461a82b72b10818736ff8b3 Author: Willy Tarreau Date: Wed Jul 19 19:08:48 2017 +0200 BUILD: lua: replace timegm() with my_timegm() to fix build on Solaris 10 Akhnin Nikita reported that Lua doesn't build on Solaris 10 because the code uses timegm() to parse a date, which is not provided there. The recommended way to implement timegm() is broken in the man page, as it is based on a change of the TZ environment variable at run time before calling the function (which is obviously not thread safe, and terribly inefficient). Here instead we rely on the new my_timegm() function, it should be sufficient for all known use cases. commit cb1949b8b30b8db7e05546da2939eff2b5973321 Author: Willy Tarreau Date: Wed Jul 19 19:05:29 2017 +0200 MINOR: tools: add a portable timegm() alternative timegm() is not provided everywhere and the documentation on how to replace it is bogus as it proposes an inefficient and non-thread safe alternative. Here we reimplement everything needed to compute the number of seconds since Epoch based on the broken down fields in struct tm. It is only guaranteed to return correct values for correct inputs. It was successfully tested with all possible 32-bit values of time_t converted to struct tm using gmtime() and back to time_t using the legacy timegm() and this function, and both functions always produced the same result. Thanks to Benoît Garnier for an instructive discussion and detailed explanations of the various time functions, leading to this solution. commit 8c2ddc20decbc927261089e330421251194cd865 Author: Emmanuel Hocdet Date: Wed Jul 19 16:04:05 2017 +0200 BUILD: ssl: fix compatibility with openssl without TLSEXT_signature_* In openssl < 1.0.1, TLSEXT_signature_* is undefined. Add TLSEXT signatures (RFC 5246) when TLSEXT_signature_anonymous is undefined. commit f80bc24ddef866e9a487cb9f67877a2c4759252c Author: Emmanuel Hocdet Date: Wed Jul 12 14:25:38 2017 +0200 MINOR: ssl: remove an unecessary SSL_OP_NO_* dependancy Use methodVersions table to display "OpenSSL library supports". commit 23877ab6533e97981edcb3d11bc74ab44b8531aa Author: Emmanuel Hocdet Date: Wed Jul 12 12:53:02 2017 +0200 BUG/MINOR: ssl: remove haproxy SSLv3 support when ssl lib have no SSLv3 The commit 5db33cbd "MEDIUM: ssl: ssl_methods implementation is reworked and factored for min/max tlsxx" drop the case when ssl lib have removed SSLv3. The commit 1e59fcc5 "BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0" fix build but it's false because haproxy think that ssl lib support SSLv3. SSL_OP_NO_* are flags to set in ssl_options and is the way haproxy do the link between ssl capabilities and haproxy configuration. (The mapping table is done via methodVersions). SSL_OP_NO_* is set to 0 when ssl lib doesn't support a new TLS version. Older version (like SSLv3) can be removed at build or unsupported (like libressl). In all case OPENSSL_NO_SSL3 is define. To keep the same logic, this patch alter SSL_OP_NO_SSLv3 to 0 when SSLv3 is not supported by ssl lib (when OPENSSL_NO_SSL3 is define). commit 80ebd30c9648b84495087f8e9c6e4d5f2d602f1f Author: David Carlier Date: Wed Jun 7 12:39:16 2017 +0100 BUG/MINOR: contrib/mod_defender: build fix In similar manner than modsecurity, making the build possible under *BSD flavors, the -lm for ceilf function. commit 0f4df640d20c7e939994c7ccda6fd4d6746bcedb Author: David Carlier Date: Tue Jun 6 10:20:51 2017 +0100 BUG/MINOR: contrib/modsecurity: BSD build fix previous version introduced in the last commit was not the correct one. commit a81ff60454e2cec95bb843034cf2ac62f791f6c1 Author: Christopher Faulet Date: Tue Jul 18 22:01:05 2017 +0200 BUG/MINOR: http: Fix bug introduced in previous patch in http_resync_states The previous patch ("MINOR: http: Rely on analyzers mask to end processing in forward_body functions") contains a bug for keep-alive transactions. For these transactions, AN_REQ_FLT_END and AN_RES_FLT_END analyzers must be removed only when all outgoing data was forwarded. commit 894da4c8ea1461d1be8f25aaa7f78556ff7ecc8a Author: Christopher Faulet Date: Tue Jul 18 11:29:07 2017 +0200 MINOR: http: Rely on analyzers mask to end processing in forward_body functions Instead of relying on request or response state, we use "chn->analysers" mask as all other analyzers. So now, http_resync_states does not return anything anymore. The debug message in http_resync_states has been improved. commit 1486b0ab6de744e14ae684af105951345534f9ec Author: Christopher Faulet Date: Tue Jul 18 11:42:08 2017 +0200 BUG/MEDIUM: http: Switch HTTP responses in TUNNEL mode when body length is undefined When the body length of a HTTP response is undefined, the HTTP parser is blocked in the body parsing. Before HAProxy 1.7, in this case, because AN_RES_HTTP_XFER_BODY is never set, there is no visible effect. When the server closes its connection to terminate the response, HAProxy catches it as a normal closure. Since 1.7, we always set this analyzer to enter at least once in http_response_forward_body. But, in the present case, when the server connection is closed, http_response_forward_body is called one time too many. The response is correctly sent to the client, but an error is catched and logged with "SD--" flags. To reproduce the bug, you can use the configuration "tests/test-fsm.cfg". The tests 3 and 21 hit the bug. Idea to fix the bug is to switch the response in TUNNEL mode without switching the request. This is possible because of previous patches. First, we need to detect responses with undefined body length during states synchronization. Excluding tunnelled transactions, when the response length is undefined, TX_CON_WANT_CLO is always set on the transaction. So, when states are synchronized, if TX_CON_WANT_CLO is set, the response is switched in TUNNEL mode and the request remains unchanged. Then, in http_msg_forward_body, we add a specific check to switch the response in DONE mode if the body length is undefined and if there is no data filter. This patch depends on following previous commits: * MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flags * MINOR: http: Reorder/rewrite checks in http_resync_states This patch must be backported in 1.7 with 2 previous ones. commit 4be9803914ae7156109c915659aad216e4a3c6c1 Author: Christopher Faulet Date: Tue Jul 18 10:48:24 2017 +0200 MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flags Today, the only way to have a request or a response in HTTP_MSG_TUNNEL state is to have the flag TX_CON_WANT_TUN set on the transaction. So this is a symmetric state. Both the request and the response are switch in same time in this state. This can be done only by checking transaction flags instead of relying on the other side state. This is the purpose of this patch. This way, if for any reason we need to switch only one side in TUNNEL mode, it will be possible. And to prepare asymmetric cases, we check channel flags in DONE _AND_ TUNNEL states. WARNING: This patch will be used to fix a bug. The fix will be commited in a very next commit. So if the fix is backported, this one must be backported too. commit f77bb539d4846ab278269b99a3165a5608ca0cf4 Author: Christopher Faulet Date: Tue Jul 18 11:18:46 2017 +0200 MINOR: http: Reorder/rewrite checks in http_resync_states The previous patch removed the forced symmetry of the TUNNEL mode during the state synchronization. Here, we take care to remove body analyzer only on the channel in TUNNEL mode. In fact, today, this change has no effect because both sides are switched in same time. But this way, with some changes, it will be possible to keep body analyzer on a side (to finish the states synchronization) with the other one in TUNNEL mode. WARNING: This patch will be used to fix a bug. The fix will be commited in a very next commit. So if the fix is backported, this one must be backported too. commit a3992e06a6e74142d9784d18d8cb3527fadb64d6 Author: Christopher Faulet Date: Tue Jul 18 10:35:55 2017 +0200 BUG/MINOR: http: Set the response error state in http_sync_res_state This is just typo. It may only report a wrong response message state in "show errors" on the CLI. This patch must be backported in 1.7. commit 7ab16868bc6e9d5ef879e1046effa035789835cc Author: Willy Tarreau Date: Tue Jul 18 06:58:16 2017 +0200 DOC: update the list of OpenSSL versions in the README 1.1.0 is also supported nowadays. Also mention the best effort support for derivatives. commit 9d84cd602f4adb3954209eb14c94eea9254d1b5b Author: Willy Tarreau Date: Tue Jul 18 06:56:40 2017 +0200 DOC: update CONTRIBUTING regarding optional parts and message format Make it clear that optional components must not break when disabled, that openssl is the only officially supported library and its support must not be broken, and that bug fixes must always be detailed. commit 6b546a604889e4b9a94891c31b1af11812bfc241 Author: Thierry FOURNIER Date: Sun Jul 16 16:35:53 2017 +0200 BUG/MINOR: Lua: variable already initialized The variable strm->hlua is already initilized by the function stream_new(). commit 7bd10d58d3aecf7cf1e5ee7df01193e07128a52d Author: Thierry FOURNIER Date: Mon Jul 17 00:44:40 2017 +0200 BUG/MEDIUM: lua: bad memory access We cannot perform garbage collection on unreferenced thread. This memory is now free and another Lua process can use it for other things. HAProxy is monothread, so this bug doesn't cause crash. This patch must be backported in 1.6 and 1.7 commit b13b20a19aacb039a33f886e38a181b00c9a6d41 Author: Thierry FOURNIER Date: Sun Jul 16 20:48:54 2017 +0200 BUG/MAJOR: lua/socket: resources not detroyed when the socket is aborted In some cases, the socket is misused. The user can open socket and never close it, or open the socket and close it without sending data. This causes resources leak on all resources associated to the stream (buffer, spoe, ...) This is caused by the stream_shutdown function which is called outside of the stream execution process. Sometimes, the shtudown is required while the stream is not started, so the cleanup is ignored. This patch change the shutdown mode of the session. Now if the session is no longer used and the Lua want to destroy it, it just set a destroy flag and the session kill itself. This patch should be backported in 1.6 and 1.7 commit 75d0208009c3189b5d10793e08f27dd62a76c3ae Author: Thierry FOURNIER Date: Wed Jul 12 13:41:33 2017 +0200 BUG/MINOR: lua: executes the function destroying the Lua session in safe mode When we destroy the Lua session, we manipulates Lua stack, so errors can raises. It will be better to catch these errors. This patch should be backported in 1.6 and 1.7 commit 0a97620c080232a21ad7fce2c859a2edc9d7147e Author: Thierry FOURNIER Date: Wed Jul 12 11:18:00 2017 +0200 BUG/MINOR: lua: In error case, the safe mode is not removed Just forgot of reset the safe mode. This have not consequences the safe mode just set a pointer on fucntion which is called only and initialises a longjmp. Out of lua execution, this longjmp is never executed and the function is never called. This patch should be backported in 1.6 and 1.7 commit be7b1ce4c1580a897d979a59c2d489f66dc151b0 Author: Olivier Houchard Date: Mon Jul 17 17:25:33 2017 +0200 BUG/MINOR: Prevent a use-after-free on error scenario on option "-x". This was introduced with recent commit f73629d ("MINOR: global: Add an option to get the old listening sockets."). No backport is needed. commit 106f63128079d8e4fdee396cb0a9bd725b932418 Author: Willy Tarreau Date: Mon Jul 17 20:46:05 2017 +0200 CLEANUP: hdr_idx: make some function arguments const where possible Functions hdr_idx_first_idx() and hdr_idx_first_pos() were missing a "const" qualifier on their arguments which are not modified, causing a warning in some experimental H2 code. commit ed2b4a6b793d062000518e51ed71e014c649c313 Author: Frédéric Lécaille Date: Thu Jul 13 09:07:09 2017 +0200 BUG/MINOR: peers: peer synchronization issue (with several peers sections). When several stick-tables were configured with several peers sections, only a part of them could be synchronized: the ones attached to the last parsed 'peers' section. This was due to the fact that, at least, the peer I/O handler refered to the wrong peer section list, in fact always the same: the last one parsed. The fact that the global peer section list was named "struct peers *peers" lead to this issue. This variable name is dangerous ;). So this patch renames global 'peers' variable to 'cfg_peers' to ensure that no such wrong references are still in use, then all the functions wich used old 'peers' variable have been modified to refer to the correct peer list. Must be backported to 1.6 and 1.7. commit 7784f1739c740ff5a9958906bf5ddd1d75b36f60 Author: Willy Tarreau Date: Tue Jul 11 14:38:39 2017 +0200 OPTIM: ssl: don't consider a small ssl_read() as an indication of end of buffer In ssl_sock_to_buf(), when we face a small read, we used to consider it as an indication for the end of incoming data, as is the case with plain text. The problem is that here it's quite different, SSL records are returned at once so doing so make us wake all the upper layers for each and every record. Given that SSL records are 16kB by default, this is rarely observed unless the protocol employs small records or the buffers are increased. But with 64kB buffers while trying to deal with HTTP/2 frames, the exchanges are obviously suboptimal as there are two messages per frame (one for the frame header and another one for the frame payload), causing the H2 parser to be woken up half of the times without being able to proceed : try=65536 ret=45 try=65536 ret=16384 try=49152 ret=9 try=49143 ret=16384 try=32759 ret=9 try=32750 ret=16384 try=16366 ret=9 try=32795 ret=27 try=49161 ret=9 try=49152 ret=16384 try=49116 ret=9 try=49107 ret=16384 try=32723 ret=9 try=32714 ret=16384 try=16330 ret=9 try=32831 ret=63 try=49161 ret=9 try=49152 ret=16384 try=49080 ret=9 try=49071 ret=2181 With this change, the buffer can safely be filled with all pending frames at once when they are available. commit a14ad72d307028457988bfd890bc70d3974b41a1 Author: Willy Tarreau Date: Fri Jul 7 11:36:32 2017 +0200 BUG/MINOR: http: properly handle all 1xx informational responses Only 100 was considered informational instead of all 1xx. This can be a problem when facing a 102 ("progress") or with the upcoming 103 for early hints. Let's properly handle all 1xx now, leaving a special case for 101 which is used for the upgrade. This fix should be backported to 1.7, 1.6 and 1.5. In 1.4 the code is different but the backport should be made there as well. commit 37a72546f6da720d99facc013ddc28a7f8f68ab9 Author: Frédéric Lécaille Date: Thu Jul 6 15:02:16 2017 +0200 MINOR: peers: Add additional information to stick-table definition messages. With this patch additional information are added to stick-table definition messages so that to make external application capable of learning peer stick-table configurations. First stick-table entries duration is added followed by the frequency counters type IDs and values. May be backported to 1.7 and 1.6. commit 570f79987756b1534d5601b66ba68602a847ec09 Author: Christopher Faulet Date: Thu Jul 6 15:53:02 2017 +0200 BUG/MEDIUM: filters: Be sure to call flt_end_analyze for both channels In the commit 2b553de5 ("BUG/MINOR: filters: Don't force the stream's wakeup when we wait in flt_end_analyze"), we removed a task_wakeup in flt_end_analyze to no consume too much CPU by looping in certain circumstances. But this fix was too drastic. For Keep-Alive transactions, flt_end_analyze is often called only for the response. Then the stream is paused until a timeout is hitted or the next request is received. We need first let a chance to both channels to call flt_end_analyze function. Then if a filter need to wait here, it is its responsibility to wake up the stream when needed. To fix the bug, and thanks to previous commits, we set the flag CF_WAKE_ONCE on channels to pretend there is an activity. On the current channel, the flag will be removed without any effect, but for the other side the analyzer will be called immediatly. Thanks for Lukas Tribus for his detailed analysis of the bug. This patch must be backported in 1.7 with the 2 previous ones: * a94fda3 ("BUG/MINOR: http: Don't reset the transaction if there are still data to send") * cdaea89 ("BUG/MINOR: stream: Don't forget to remove CF_WAKE_ONCE flag on response channel") commit a94fda30bd5638a77ea96a1c5fca7945fe736714 Author: Christopher Faulet Date: Thu Jul 6 15:51:35 2017 +0200 BUG/MINOR: http: Don't reset the transaction if there are still data to send To reset an HTTP transaction, we need to be sure all data were sent, for the request and the response. There are tests on request and response buffers for that in http_resync_states function. But the return code was wrong. We must return 0 to wait. This patch must be backported in 1.7 commit cdaea89a0c0eb8c028ce9220c1af7ea795845dd7 Author: Christopher Faulet Date: Thu Jul 6 15:49:30 2017 +0200 BUG/MINOR: stream: Don't forget to remove CF_WAKE_ONCE flag on response channel This flag can be set on a channel to pretend there is activity on it. This is a way to wake-up the corresponding stream and evaluate stream analyzers on the channel. It is correctly handled on both channels but removed only on the request channel. This patch is flagged as a bug but for now, CF_WAKE_ONCE is never set on the response channel. commit 2ab88675ecbf960a6f33ffe9c6a27f264150b201 Author: Willy Tarreau Date: Wed Jul 5 18:23:03 2017 +0200 MINOR: ssl: compare server certificate names to the SNI on outgoing connections When support for passing SNI to the server was added in 1.6-dev3, there was no way to validate that the certificate presented by the server would really match the name requested in the SNI, which is quite a problem as it allows other (valid) certificates to be presented instead (when hitting the wrong server or due to a man in the middle). This patch adds the missing check against the value passed in the SNI. The "verifyhost" value keeps precedence if set. If no SNI is used and no verifyhost directive is specified, then the certificate name is not checked (this is unchanged). In order to extract the SNI value, it was necessary to make use of SSL_SESSION_get0_hostname(), which appeared in openssl 1.1.0. This is a trivial function which returns the value of s->tlsext_hostname, so it was provided in the compat layer for older versions. After some refinements from Emmanuel, it now builds with openssl 1.0.2, openssl 1.1.0 and boringssl. A test file was provided to ease testing all cases. After some careful observation period it may make sense to backport this to 1.7 and 1.6 as some users rightfully consider this limitation as a bug. Cc: Emmanuel Hocdet Signed-off-by: Willy Tarreau commit 96fd926ccc61466caecf573aebad190dfd8b9a83 Author: Emeric Brun Date: Wed Jul 5 13:33:16 2017 +0200 BUG/MAJOR: http: fix buffer overflow on loguri buffer. The pool used to log the uri was created with a size of 0 because the configuration and 'tune.http.logurilen' were parsed too earlier. The fix consist to postpone the pool_create as it is done for cookie captures. Regression introduced with 'MINOR: log: Add logurilen tunable' commit 7d27f3c12d2aaa3514cdd02ed94ce4421c32fd6d Author: Emeric Brun Date: Mon Jul 3 17:54:23 2017 +0200 BUG/MEDIUM: map/acl: fix unwanted flags inheritance. The bug: Maps/ACLs using the same file/id can mistakenly inherit their flags from the last declared one. i.e. $ cat haproxy.conf listen mylistener mode http bind 0.0.0.0:8080 acl myacl1 url -i -f mine.acl acl myacl2 url -f mine.acl acl myacl3 url -i -f mine.acl redirect location / if myacl2 $ cat mine.acl foobar Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'. This fix should be backported on mainline branches v1.6 and v1.7. commit e0ee0be4e74af11f47a7bcf0b1e8cfd0e3092b2a Author: Jarno Huuskonen Date: Tue Jul 4 10:35:12 2017 +0300 DOC: fix references to the section about time format. Time format is documented in section 2.4, not 2.2. commit 2802b07d9742ca73e73dfcbe77d74ce09565b326 Author: Emeric Brun Date: Fri Jun 30 14:11:56 2017 +0200 BUG/MAJOR: applet: fix a freeze if data is immedately forwarded. Introduced regression with 'MAJOR: applet scheduler rework' (1.8-dev only). The fix consist to re-enable the appctx immediatly from the applet wake cb if the process_stream is not pending in runqueue and the applet want perform a put or a get and the WAIT_ROOM flag was removed by stream_int_notify. commit a03d4ada2611d47f4b8d4e281f1bb8b22c46f5dd Author: Christopher Faulet Date: Mon Jun 26 16:53:33 2017 +0200 MINOR: compression: Use a memory pool to allocate compression states Instead of doing a malloc/free to each HTTP transaction to allocate the compression state (when the HTTP compression is enabled), we use a memory pool. commit d60b3cf43191488b0dece7000e7da94be981ba1a Author: Christopher Faulet Date: Mon Jun 26 11:47:13 2017 +0200 BUG/MAJOR: compression: Be sure to release the compression state in all cases This patch fixes an obvious memory leak in the compression filter. The compression state (comp_state) is allocated when a HTTP transaction starts, in channel_start_analyze callback, Whether we are able to compression the response or not. So it must be released when the transaction ends, in channel_end_analyze callback. But there is a bug here. The state is released on the response side only. So, if a transaction ends before the response is started, it is never released. This happens when a connection is closed before the response is started. To fix the bug, statistics about the HTTP compression are now updated in http_end callback, when the response parsing ends. It happens only if no error is encountered and when the response is compressed. So, it is safe to release the compression state in channel_end_analyze callback, regardless the channel's type. This patch must be backported in 1.7. commit 8d85aa44dabb772cf99360c41143486875f3e383 Author: Emeric Brun Date: Thu Jun 29 15:40:33 2017 +0200 BUG/MAJOR: map: fix segfault during 'show map/acl' on cli. The reference of the current map/acl element to dump could be destroyed if map is updated from an 'http-request del-map' configuration rule or throught a 'del map/acl' on CLI. We use a 'back_refs' chaining element to fix this. As it is done to dump sessions. This patch needs also fix: 'BUG/MAJOR: cli: fix custom io_release was crushed by NULL.' To clean the back_ref and avoid a crash on a further del/clear map operation. Those fixes should be backported on mainline branches 1.7 and 1.6. This patch wont directly apply on 1.6. commit d6871f785f5a57a051cf593df64d79927180699f Author: Emeric Brun Date: Thu Jun 29 19:54:13 2017 +0200 BUG/MAJOR: cli: fix custom io_release was crushed by NULL. The io_release could be set into the parsing request handler and must not be crushed. This patch should be backported on mainline branches 1.7 and 1.6 commit 27f2dbbdfdb589bc5659cfdda2479c5fb47821fd Author: Willy Tarreau Date: Tue Jun 27 15:47:56 2017 +0200 BUG/MAJOR: frontend: don't dereference a null conn on outgoing connections Recently merged commit 0cfe388 ("MINOR: frontend: retrieve the ALPN name when available") assumed that the connection is always known in frontend_accept() which is not true for outgoing peers connections for example. No backport needed. commit c73060687919fcadc5f1d767842c1762153854c1 Author: Emeric Brun Date: Mon Jun 26 16:36:53 2017 +0200 MAJOR: applet: applet scheduler rework. In order to authorize call of appctx_wakeup on running task: - from within the task handler itself. - in futur, from another thread. The appctx is considered paused as default after running the handler. The handler should explicitly call appctx_wakeup to be re-called. When the appctx_free is called on a running handler. The real free is postponed at the end of the handler process. commit 57ec32fb993d4b426ccca7ed2322eacbf9218596 Author: Willy Tarreau Date: Tue Apr 11 19:59:33 2017 +0200 MINOR: connection: send data before receiving It's more efficient this way, as it allows to flush a send buffer before receiving data in the other one. This can lead to a slightly faster buffer recycling, thus slightly less memory and a small performance increase by using a hotter cache. commit d62b98c6e80bd69ff80d80e90e62ab51531bc250 Author: Willy Tarreau Date: Tue Dec 13 15:26:56 2016 +0100 MINOR: stream: don't set backend's nor response analysers on SF_TUNNEL In order to implement hot-pluggable applets like we'll need for HTTP/2 which will speak a different protocol than the expected one, it will be mandatory to be able to clear all analysers from the request and response channel and/or to keep only the ones the applet initializer installed. Unfortunately for now in sess_establish() we systematically place a number of analysers inherited from the frontend, backend and some hard-coded ones. This patch reuses the now unused SF_TUNNEL flag on the stream to indicate we're dealing with a tunnel and don't want to add more analysers anymore. It will be usable to install such a specific applet. Ideally over the long term it might be nice to be able to set the mode on the stream instead of the proxy so that we can decide to change a stream's mode (eg: TCP, HTTP, HTTP/2) at run time. But it would require many more changes for a gain which is not yet obvious. commit 9c26680eb90d4f84d9967203afda03a805d4e9e5 Author: Willy Tarreau Date: Sun Dec 4 19:05:24 2016 +0100 MINOR: frontend: report the connection's ALPN in the debug output Now the incoming connection will also report the ALPN field, truncated to 15 characters. commit 0cfe3887deaeb6761211a784a97a1f7daee04270 Author: Willy Tarreau Date: Sun Dec 4 18:59:29 2016 +0100 MINOR: frontend: retrieve the ALPN name when available Here we try to retrieve the negociated ALPN on the front connection. This will be used to decide whether or not we want to switch to H2. commit 8743f7e56763be2eef31443facbfd8afd0642166 Author: Willy Tarreau Date: Sun Dec 4 18:44:29 2016 +0100 MINOR: ssl: add a get_alpn() method to ssl_sock This is used to retrieve the TLS ALPN information from a connection. We also support a fallback to NPN if ALPN doesn't find anything or is not available on the existing implementation. It happens that depending on the library version, either one or the other is available. NPN was present in openssl 1.0.1 (very common) while ALPN is in 1.0.2 and onwards (still uncommon at the time of writing). Clients are used to send either one or the other to ensure a smooth transition. commit a9c1741820830ea47d4ebfbc7a87fac902e764bf Author: Willy Tarreau Date: Sun Dec 4 18:42:09 2016 +0100 MINOR: connection: add a .get_alpn() method to xprt_ops This will be used to retrieve the ALPN negociated over SSL (or possibly via the proxy protocol later). It's likely that this information should be stored in the connection itself, but it requires adding an extra pointer and an extra integer. Thus better rely on the transport layer to pass this info for now. commit 0a6bed2394dabf7f05a7a2af2fc9b8ba936073be Author: Willy Tarreau Date: Sun Dec 4 18:39:22 2016 +0100 MINOR: frontend: initialize HTTP layer after the debugging code For HTTP/2 we'll have to choose the upper layer based on the advertised protocol name here and we want to keep debugging, so let's move debugging earlier. commit 9b82d941c58183efe43a49da73927e537aacef90 Author: Willy Tarreau Date: Mon Dec 5 00:26:31 2016 +0100 MEDIUM: stream: make stream_new() always set the target and analysers It doesn't make sense that stream_new() doesn't sets the target nor analysers and that the caller has to do it even if it doesn't know about streams (eg: in session_accept_fd()). This causes trouble for H2 where the applet handling the protocol cannot properly change these information during its init phase. Let's ensure it's always set and that the callers don't set it anymore. Note: peers and lua don't use analysers and that's properly handled. commit f3a55dbd2235684f4e9711511e0c87353c0a3773 Author: Christopher Faulet Date: Fri Jun 9 14:26:38 2017 +0200 MINOR: queue: Change pendconn_from_srv/pendconn_from_px into private functions commit f0614e81115bb3f3f8a01f25022f5c4880afb87b Author: Christopher Faulet Date: Fri Jun 9 14:20:29 2017 +0200 MINOR: backends: Change get_server_sh/get_server_uh into private function commit 87566c923b07af3373be7526572e5a4473507617 Author: Christopher Faulet Date: Tue Jun 6 10:34:51 2017 +0200 MINOR: queue: Change pendconn_get_next_strm into private function commit 5f77fef34e76f20fe62f3e4102559c1e0fe8f5a1 Author: Emeric Brun Date: Mon May 29 15:26:51 2017 +0200 MINOR: task/stream: tasks related to a stream must be init by the caller. The task_wakeup was called on stream_new, but the task/stream wasn't fully initialized yet. The task_wakeup must be called explicitly by the caller once the task/stream is initialized. commit 0194897e540cec67d7d1e9281648b70efe403f08 Author: Emeric Brun Date: Thu Mar 30 15:37:25 2017 +0200 MAJOR: task: task scheduler rework. In order to authorize call of task_wakeup on running task: - from within the task handler itself. - in futur, from another thread. The lookups on runqueue and waitqueue are re-worked to prepare multithread stuff. If task_wakeup is called on a running task, the woken message flags are savec in the 'pending_state' attribute of the state. The real wakeup is postponed at the end of the handler process and the woken messages are copied from pending_state to the state attribute of the task. It's important to note that this change will cause a very minor (though measurable) performance loss but it is necessary to make forward progress on a multi-threaded scheduler. Most users won't ever notice. commit ff4491726f2879ea0ddd69f5c735058938b78476 Author: Emeric Brun Date: Fri Mar 31 12:04:09 2017 +0200 BUG/MINOR: stream: flag TASK_WOKEN_RES not set if task in runqueue Under certain circumstances, if a stream's task is first woken up (eg: I/O event) then notified of the availability of a buffer it was waiting for via stream_res_wakeup(), this second event is lost because the flags are only merged after seeing that the task is running. At the moment it seems that the TASK_WOKEN_RES event is not explicitly checked for, but better fix this before getting reports of lost events. This fix removes this "task running" test which is properly performed in task_wakeup(), while the flags are properly merged. It must be backported to 1.7 and 1.6. commit 1af20c7161e012136d5e19791c4ff2c9297c1b0e Author: Willy Tarreau Date: Fri Jun 23 16:01:14 2017 +0200 DOC: fix references to the section about the unix socket The unix socket is documented in 9.3, not 9.2 of the management guide. This should be backported to 1.7. commit d02286d6c866e5c0a7eb6fbb127fa57f3becaf16 Author: Willy Tarreau Date: Fri Jun 23 11:23:43 2017 +0200 BUG/MINOR: log: pin the front connection when front ip/ports are logged Mathias Weiersmueller reported an interesting issue with logs which Lukas diagnosed as dating back from commit 9b061e332 (1.5-dev9). When front connection information (ip, port) are logged in TCP mode and the log is emitted at the end of the connection (eg: because %B or any log tag requiring LW_BYTES is set), the log is emitted after the connection is closed, so the address and ports cannot be retrieved anymore. It could be argued that we'd make a special case of these to immediatly retrieve the source and destination addresses from the connection, but it seems cleaner to simply pin the front connection, marking it "tracked" by adding the LW_XPRT flag to mention that we'll need some of these elements at the last moment. Only LW_FRTIP and LW_CLIP are affected. Note that after this change, LW_FRTIP could simply be removed as it's not used anywhere. Note that the problem doesn't happen when using %[src] or %[dst] since all sample expressions set LW_XPRT. This must be backported to 1.7, 1.6 and 1.5. commit 50174f36003141297a4824dd0b98a2f0e6d4997d Author: Christopher Faulet Date: Wed Jun 21 16:31:35 2017 +0200 BUG/MINOR: cfgparse: Check if tune.http.maxhdr is in the range 1..32767 We cannot store more than 32K headers in the structure hdr_idx, because internaly we use signed short integers. To avoid any bugs (due to an integers overflow), a check has been added on tune.http.maxhdr to be sure to not set a value greater than 32767 and lower than 1 (because this is a nonsense to set this parameter to a value <= 0). The documentation has been updated accordingly. This patch can be backported in 1.7, 1.6 and 1.5. commit 5d6e5f86c51cef6c22c6aeacc6b0e528f8fe58e9 Author: Frédéric Lécaille Date: Mon May 29 13:47:16 2017 +0200 BUG/MINOR: Wrong peer task expiration handling during synchronization processing. When a peer task has sent a synchronization request to remote peers its next expiration date was updated based on a resynchronization timeout value which itself may have already expired leading the underlying poller to wait for 0ms during a fraction of second (consuming high CPU resources). With this patch we update such peer task expiration dates only if the resynchronization timeout is not already expired. Thanks to Patrick Hemmer who reported an issue with nice traces which helped in finding this one. This patch may be backported to 1.7 and 1.6. commit 8a361b594ea38fb1f89f0b00f095ece0c8613d10 Author: William Lallemand Date: Tue Jun 20 11:20:33 2017 +0200 BUG/MEDIUM: mworker: don't reuse PIDs passed to the master When starting the master worker with -sf or -st, the PIDs will be reused on the next reload, which is a problem if new processes on the system took those PIDs. This patch ensures that we don't register old PIDs in the reload system when launching the master worker. commit 2bf6d6291614537530958282da6a7a66c03bf5fe Author: William Lallemand Date: Tue Jun 20 11:20:23 2017 +0200 MINOR: mworker: don't copy -x argument anymore in copy_argv() Don't copy the -x argument anymore in copy_argv() since it's already allocated in mworker_reload(). Make the copy_argv() more consistent when used with multiple arguments to strip. It prevents multiple -x on reload, which is not supported. commit 4fc09693d6f466ddf6b0721b20841d056759755a Author: William Lallemand Date: Mon Jun 19 16:37:19 2017 +0200 MINOR: warning on multiple -x Multiple use of the -x option is useless, emit a warning. commit 45eff44e28a36be113a933ee27f16ff5e92775bb Author: William Lallemand Date: Mon Jun 19 15:57:55 2017 +0200 BUG/MEDIUM: fix segfault when no argument to -x option This patch fixes a segfault in the command line parser. When haproxy is launched with -x with no argument and -x is the latest option in argv it segfaults. Use usage() insteads of exit() on error. commit 4a5be934f121dd227dd22afb96dbf74b87d7dc65 Author: Willy Tarreau Date: Fri Jun 16 12:43:53 2017 +0200 SCRIPTS: create-release: enforce GIT_COMMITTER_{NAME|EMAIL} validity If it's not set, fall back to git config --get. commit 2c44cd8a3708053fef0d8529aa01f7f2ec41ffb4 Author: Willy Tarreau Date: Fri Jun 16 12:35:54 2017 +0200 scripts: create-release pass -n to tail Some versions of tail don't accept "tail +4". commit 68986abe93ea26e3507d3053d266d46dc6328b71 Author: Willy Tarreau Date: Fri Jun 16 10:34:20 2017 +0200 BUG/MEDIUM: unix: never unlink a unix socket from the file system James Brown reported some cases where a race condition happens between the old and the new processes resulting in the leaving process removing a newly bound unix socket. Jeff gave all the details he observed here : https://www.mail-archive.com/haproxy@formilux.org/msg25001.html The unix socket removal was an attempt at an optimal cleanup, which almost never works anyway since the process is supposed to be chrooted. And in the rare cases where it works it occasionally creates trouble. There was already a workaround in place to avoid removing this socket when it's been inherited from a parent's file descriptor. So let's finally kill this useless stuff now to definitely get rid of this persistent problem. This fix should be backported to all stable releases. commit 0bedb8ac90ffdf1498a999c44d1c91556fb726ee Author: Frédéric Lécaille Date: Thu Jun 15 14:09:10 2017 +0200 BUG/MAJOR: server: Segfault after parsing server state file. This patch makes the server state file parser ignore servers wich are not present in the configuration file. commit 5df119008a1485a297906299f2bbb13fb7e0de77 Author: Frédéric Lécaille Date: Tue Jun 13 16:39:57 2017 +0200 BUG/MEDIUM: peers: Peers CLOSE_WAIT issue. A peer session which has just been created upon reconnect timeout expirations, could be right after shutdown (at peer session level) because the remote side peer could also righ after have connected. In such a case the underlying TCP session was still running (connect()/accept()) and finally left in CLOSE_WAIT state after the remote side stopped writting (shutdown(SHUT_WR)). Now on, with this patch we never shutdown such peer sessions wich have just been created. We leave them connect to the remote peer which is already connected and must shutdown its own peer session. Thanks to Patric Hemmer and Yves Lafon at w3.org for reporting this issue, and for having tested this patch on the field. Thanks also to Willy and Yelp blogs which helped me a lot in fixing it (see https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ and https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.htmll). commit a33510b215b9886d95360f2d60e558a8c2a57ec2 Author: Christopher Faulet Date: Fri Mar 31 15:37:29 2017 +0200 BUG/MINOR: http/filters: Be sure to wait if a filter loops in HTTP_MSG_ENDING A filter can choose to loop when a HTTP message is in the state HTTP_MSG_ENDING. But the transaction is terminated with an error if the input is closed (CF_SHUTR set on the channel). At this step, we have received all data, so we can wait. So now, we also check the parser state before leaving. This fix only affects configs that use a filter that can wait in http_forward_data or http_end callbacks, when all data were parsed. commit 1e59fcc5885da5cab7ba779eee5f312b3c1d5d72 Author: Christopher Faulet Date: Thu Jun 8 22:18:52 2017 +0200 BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0 For openssl 1.0.2, SSLv3_server_method and SSLv3_client_method are undefined if OPENSSL_NO_SSL3_METHOD is set. So we must add a check on this macro before using these functions. commit 54ceb041d6372a70afdbdd76f4f2bac46a088d18 Author: Christopher Faulet Date: Wed Jun 14 14:41:33 2017 +0200 BUG/MINOR: acls: Set the right refflag when patterns are loaded from a map For an ACL, we can load patterns from a map using the flag -M. For example: acl test hdr(host) -M -f hosts.map The file is parsed as a map et the ACL will be executed as expected. But the reference flag is wrong. It is set to PAT_REF_ACL. So the map will never be listed by a "show map" on the stat socket. Setting the reference flag to PAT_REF_ACL|PAT_REF_MAP fixes the bug. commit a4d0361969a47e85d41738bdea4578773577fff7 Author: Frédéric Lécaille Date: Wed Jun 14 15:16:15 2017 +0200 CONTRIB: plug qdiscs: Plug queuing disciplines mini HOWTO. Add plug_qdisc.c source file which may help in how to programatically use plug queueing disciplines with its README file. Such code may be useful to reproduce painful network application bugs. commit a36b311b9f1eee33ac42b008858c1adc1763a791 Author: Christopher Faulet Date: Tue Jun 13 22:00:22 2017 +0200 BUG/MINOR: buffers: Fix bi/bo_contig_space to handle full buffers These functions was added in commit 637f8f2c ("BUG/MEDIUM: buffers: Fix how input/output data are injected into buffers"). This patch fixes hidden bugs. When a buffer is full (buf->i + buf->o == buf->size), instead of returning 0, these functions can return buf->size. Today, this never happens because callers already check if the buffer is full before calling bi/bo_contig_space. But to avoid possible bugs if calling conditions changed, we slightly refactored these functions. commit 6a0bca9e7862984b0edf8fc1e1edc54295a7a5e2 Author: Willy Tarreau Date: Sun Jun 11 17:56:27 2017 +0200 BUG/MAJOR: http: call manage_client_side_cookies() before erasing the buffer Jean Lubatti reported a crash on haproxy using a config involving cookies and tarpit rules. It just happens that since 1.7-dev3 with commit 83a2c3d ("BUG/MINOR : allow to log cookie for tarpit and denied request"), function manage_client_side_cookies() was called after erasing the request buffer in case of a tarpit action. The problem is that this function must absolutely not be called with an empty buffer since it moves parts of it. A typical reproducer consists in sending : "GET / HTTP/1.1\r\nCookie: S=1\r\n\r\n" On such a config : listen crash bind :8001 mode http reqitarpit . cookie S insert indirect server s1 127.0.0.1:8000 cookie 1 The fix simply consists in moving the call to the function before the call to buffer_erase(). Many thanks to Jean for testing instrumented code and providing a usable core. This fix must be backported to all stable versions since the fix introducing this bug was backported as well. commit bd6989152f977764f8bafe09e7a0265620da2919 Author: Willy Tarreau Date: Fri Jun 9 15:57:31 2017 +0200 BUILD: scripts: add a "quiet" mode to publish-release Option "-q" will make it silent in automatic mode when there is nothing to do. commit 7ca88159a18682e413258444b1c087203f598a46 Author: Willy Tarreau Date: Fri Jun 9 15:54:39 2017 +0200 BUILD: scripts: add an automatic mode for publish-release Passing "-a" will make it easier to automatically create archives from tagged repositories. It doesn't ask any question and doesn't return an error when the current branch is not tagged nor if the release already exists. commit 600cb57450e928b62d1a3c3f507de251084b434e Author: Willy Tarreau Date: Fri Jun 9 15:36:02 2017 +0200 BUILD: scripts: make publish-release support bare repositories First we must not report an error when "git diff HEAD" fails. Second, we don't want to "cd" to the home dir when "git rev-parse --show-toplevel" returns an empty string. Third, we definitely want to check that a master branch really exists in the current directory to avoid mistakes. commit b286fffa4281860ed606a5f18b6994748afa09b3 Author: Nan Liu Date: Fri Jun 9 15:57:45 2017 +0800 BUG/MINOR: Makefile: fix compile error with USE_LUA=1 in ubuntu16.04 include/types/hlua.h:6:17: fatal error: lua.h: No such file or directory commit 1499b9b7efedf6162d3e4744993558c28bdca77c Author: William Lallemand Date: Wed Jun 7 15:04:47 2017 +0200 BUG/MEDIUM: misplaced exit and wrong exit code Commit cb11fd2 ("MEDIUM: mworker: wait mode on reload failure") introduced a regression, when HAProxy is used in daemon mode, it exits 1 after forking its children. HAProxy should exit(0), the exit(EXIT_FAILURE) was expected to be use when the master fail in master-worker mode. Thanks to Emmanuel Hocdet for reporting this bug. No backport needed. commit cc9b94ac943caaa0292ca047a63f64a382d85d80 Author: William Lallemand Date: Thu Jun 8 19:30:39 2017 +0200 BUG/MINOR: warning: ‘need_resend’ may be used uninitialized The commit 201c07f68 ("MAJOR/REORG: dns: DNS resolution task and requester queues") introduces a warning during compilation: src/dns.c: In function ‘dns_resolve_recv’: src/dns.c:487:6: warning: ‘need_resend’ may be used uninitialized in this function [-Wmaybe-uninitialized] if (need_resend) { ^ This patch initialize the variable and remove the comment about it. commit cf4e496c9c80ad96aff47e4d8cf318be429165f7 Author: William Lallemand Date: Thu Jun 8 19:05:48 2017 +0200 BUG/MEDIUM: build without openssl broken The commit 872f9c213 ("MEDIUM: ssl: add basic support for OpenSSL crypto engine") broke the build without openssl support. The ssl_free_dh() function is not defined when USE_OPENSSL is not defined and leads to a compilation failure. commit 04919d53c591a7f3bfce4f19d9f0564f06a18bf9 Author: David Carlier Date: Wed Jun 7 20:02:43 2017 +0100 BUG/MINOR: haproxy/cli : fix for solaris/illumos distros for CMSG* macros control message sockets macros implies (SUS)XPG4V2 enabled under solaris based oses. commit bbc165447e5b800df668a1010419ef8e2c48c124 Author: Emeric Brun Date: Fri Jun 2 15:54:06 2017 +0000 BUG/MINOR: ssl: do not call directly the conn_fd_handler from async_fd_handler This patch modifies the way to re-enable the connection from the async fd handler calling conn_update_sock_polling instead of the conn_fd_handler. It also ensures that the polling is really stopped on the async fd. commit b5e42a817b4b687c90ad9db12fe2bcd1652a00fe Author: Emeric Brun Date: Tue Jun 6 12:35:14 2017 +0000 BUG/MAJOR: ssl: buffer overflow using offloaded ciphering on async engine The Openssl's ASYNC API does'nt support moving buffers on SSL_read/write This patch disables the ASYNC mode dynamically when the handshake is left and re-enables it on reneg. commit ce9e01c6744262ec2bf88aa71f72bd8042dcad89 Author: Emeric Brun Date: Wed May 31 10:02:53 2017 +0000 BUG/MAJOR: ssl: fix segfault on connection close using async engines. This patch ensure that the ASYNC fd handlers won't be wake up too early, disabling the event cache for this fd on connection close and when a WANT_ASYNC is rised by Openssl. The calls to SSL_read/SSL_write/SSL_do_handshake before rising a real read event from the ASYNC fd, generated an EAGAIN followed by a context switch for some engines, or a blocked read for the others. On connection close it resulted in a too early call to SSL_free followed by a segmentation fault. commit bd695fe024e8028fc96a5222c26dadf84f05402d Author: Emmanuel Hocdet Date: Mon May 15 15:53:41 2017 +0200 MEDIUM: ssl: disable SSLv3 per default for bind For security, disable SSLv3 on bind line must be the default configuration. SSLv3 can be enabled with "ssl-min-ver SSLv3". commit df701a2adbc57670bb6fffd4744f5deae23d11e4 Author: Emmanuel Hocdet Date: Thu May 18 12:46:50 2017 +0200 MINOR: ssl: support ssl-min-ver and ssl-max-ver with crt-list SSL/TLS version can be changed per certificat if and only if openssl lib support earlier callback on handshake and, of course, is implemented in haproxy. It's ok for BoringSSL. For Openssl, version 1.1.1 have such callback and could support it. commit 4aa615ff6bc5803ff4835f9e7f662a3437a52fd9 Author: Emmanuel Hocdet Date: Thu May 18 12:33:19 2017 +0200 MEDIUM: ssl: ctx_set_version/ssl_set_version func for methodVersions table This patch cleanup the usage of set_version func with a more suitable name: ctx_set_version. It introduce ssl_set_version func (unused for the moment). commit ecb0e234b9bf34a477d6d52c26d294608bdee85c Author: Emmanuel Hocdet Date: Thu May 18 11:56:58 2017 +0200 REORG: ssl: move defines and methodVersions table upper It will used in ssl_sock_switchctx_cbk. commit f57a29a1cd4c301986a895772312d70c2fedf955 Author: Willy Tarreau Date: Fri Jun 2 15:59:51 2017 +0200 [RELEASE] Released version 1.8-dev2 Released version 1.8-dev2 with the following main changes : - CLEANUP: server: moving netinet/tcp.h inclusion - DOC: changed "block"(deprecated) examples to http-request deny - DOC: add few comments to examples. - DOC: update sample code for PROXY protocol - DOC: mention lighttpd 1.4.46 implements PROXY - MINOR server: Restrict dynamic cookie check to the same proxy. - DOC: stick-table is available in frontend sections - BUG/MINOR: server : no transparent proxy for DragonflyBSD - BUILD/MINOR: stats: remove unexpected argument to stats_dump_json_header() - BUILD/MINOR: tools: fix build warning in debug_hexdump() - BUG/MINOR: dns: Wrong address family used when creating IPv6 sockets. - BUG/MINOR: config: missing goto out after parsing an incorrect ACL character - BUG/MINOR: arg: don't try to add an argument on failed memory allocation - MEDIUM: server: Inherit CLI weight changes and agent-check weight responses - BUG/MEDIUM: arg: ensure that we properly unlink unresolved arguments on error - BUG/MEDIUM: acl: don't free unresolved args in prune_acl_expr() - BUG/MEDIUM: servers: unbreak server weight propagation - MINOR: lua: ensure the memory allocator is used all the time - MINOR: cli: Add a command to send listening sockets. - MINOR: global: Add an option to get the old listening sockets. - MINOR: tcp: When binding socket, attempt to reuse one from the old proc. - MINOR: doc: document the -x flag - MINOR: proxy: Don't close FDs if not our proxy. - MINOR: socket transfer: Set a timeout on the socket. - MINOR: systemd wrapper: add support for passing the -x option. - BUG/MINOR: server: Fix a wrong error message during 'usesrc' keyword parsing. - BUG/MAJOR: Broken parsing for valid keywords provided after 'source' setting. - CLEANUP: logs: typo: simgle => single - BUG/MEDIUM: acl: proprely release unused args in prune_acl_expr() - MEDIUM: config: don't check config validity when there are fatal errors - BUG/MAJOR: Use -fwrapv. - BUG/MINOR: server: don't use "proxy" when px is really meant. - BUG/MEDIUM: http: Drop the connection establishment when a redirect is performed - BUG/MINOR: server: missing default server 'resolvers' setting duplication. - MINOR: server: Extract the code responsible of copying default-server settings. - MINOR: server: Extract the code which finalizes server initializations after 'server' lines parsing. - MINOR: server: Add 'server-template' new keyword supported in backend sections. - MINOR: server: Add server_template_init() function to initialize servers from a templates. - DOC: Add documentation for new "server-template" keyword. - DOC: add layer 4 links/cross reference to "block" keyword. - DOC: errloc/errorloc302/errorloc303 missing status codes. - BUG/MEDIUM: lua: memory leak - CLEANUP: lua: remove test - BUG/MINOR: hash-balance-factor isn't effective in certain circumstances - BUG/MINOR: change header-declared function to static inline - REORG: spoe: move spoe_encode_varint / spoe_decode_varint from spoe to common - MINOR: Add binary encoding request header sample fetch - MINOR: proto-http: Add sample fetch wich returns all HTTP headers - MINOR: Add ModSecurity wrapper as contrib - BUG/MINOR: ssl: fix warnings about methods for opensslv1.1. - DOC: update RFC references - CONTRIB: tcploop: add action "X" to execute a command - MINOR: server: cli: Add server FQDNs to server-state file and stats socket. - BUG/MINOR: contrib/mod_security: fix build on FreeBSD - BUG/MINOR: checks: don't send proxy protocol with agent checks - MINOR: ssl: add prefer-client-ciphers - MEDIUM: ssl: revert ssl/tls version settings relative to default-server. - MEDIUM: ssl: ssl_methods implementation is reworked and factored for min/max tlsxx - MEDIUM: ssl: calculate the real min/max TLS version and find holes - MINOR: ssl: support TLSv1.3 for bind and server - MINOR: ssl: show methods supported by openssl - MEDIUM: ssl: add ssl-min-ver and ssl-max-ver parameters for bind and server - MEDIUM: ssl: ssl-min-ver and ssl-max-ver compatibility. - CLEANUP: retire obsoleted USE_GETSOCKNAME build option - BUG/MAJOR: dns: Broken kqueue events handling (BSD systems). - MINOR: sample: Add b64dec sample converter - BUG/MEDIUM: lua: segfault if a converter or a sample doesn't return anything - MINOR: cli: add ACCESS_LVL_MASK to store the access level - MINOR: cli: add 'expose-fd listeners' to pass listeners FDs - MEDIUM: proxy: zombify proxies only when the expose-fd socket is bound - MEDIUM: ssl: add basic support for OpenSSL crypto engine - MAJOR: ssl: add openssl async mode support - MEDIUM: ssl: handle multiple async engines - MINOR: boringssl: basic support for OCSP Stapling - MEDIUM: mworker: replace systemd mode by master worker mode - MEDIUM: mworker: handle reload and signals - MEDIUM: mworker: wait mode on reload failure - MEDIUM: mworker: try to guess the next stats socket to use with -x - MEDIUM: mworker: exit-on-failure option - MEDIUM: mworker: workers exit when the master leaves - DOC: add documentation for the master-worker mode - MEDIUM: systemd: Type=forking in unit file - MAJOR: systemd-wrapper: get rid of the wrapper - MINOR: log: Add logurilen tunable. - CLEANUP: server.c: missing prototype of srv_free_dns_resolution - MINOR: dns: smallest DNS fqdn size - MINOR: dns: functions to manage memory for a DNS resolution structure - MINOR: dns: parse_server() now uses srv_alloc_dns_resolution() - REORG: dns: dns_option structure, storage of hostname_dn - MINOR: dns: new snr_check_ip_callback function - MAJOR: dns: save a copy of the DNS response in struct resolution - MINOR: dns: implement a LRU cache for DNS resolutions - MINOR: dns: make 'ancount' field to match the number of saved records - MINOR: dns: introduce roundrobin into the internal cache (WIP) - MAJOR/REORG: dns: DNS resolution task and requester queues - BUILD: ssl: fix build with OPENSSL_NO_ENGINE - MINOR: Add Mod Defender integration as contrib - CLEANUP: str2mask return code comment: non-zero -> zero. - MINOR: tools: make debug_hexdump() use a const char for the string - MINOR: tools: make debug_hexdump() take a string prefix - CLEANUP: connection: remove unused CO_FL_WAIT_DATA commit 2686dcad1ea8252ac4851bb02a1c5c734ced8115 Author: Willy Tarreau Date: Wed Apr 26 16:25:12 2017 +0200 CLEANUP: connection: remove unused CO_FL_WAIT_DATA Very early in the connection rework process leading to v1.5-dev12, commit 56a77e5 ("MEDIUM: connection: complete the polling cleanups") marked the end of use for this flag which since was never set anymore, but it continues to be tested. Let's kill it now. commit ed936c5d3715a1975b6d6f4e7d608cec10d55919 Author: Willy Tarreau Date: Thu Apr 27 18:03:20 2017 +0200 MINOR: tools: make debug_hexdump() take a string prefix When dumping data at various places in the code, it's hard to figure what is present where. To make this easier, this patch slightly modifies debug_hexdump() to take a prefix string which is prepended in front of each output line. commit 9faef1e391a90d8c69d6f5aeeacbc6b995cc12d8 Author: Willy Tarreau Date: Thu Apr 27 17:54:58 2017 +0200 MINOR: tools: make debug_hexdump() use a const char for the string There's no reason the string to be dumped should be a char *, it's a const. commit 577d5ac8ae6b6a6860037d62b5089e8ad6bc0599 Author: Jarno Huuskonen Date: Sun May 21 17:32:21 2017 +0300 CLEANUP: str2mask return code comment: non-zero -> zero. commit 59bb97a19279610b2cbae996bd726c8336cb1a02 Author: Dragan Dosen Date: Fri Jun 2 12:03:16 2017 +0200 MINOR: Add Mod Defender integration as contrib This is a service that talks SPOE protocol and uses the Mod Defender (a NAXSI clone) functionality to detect HTTP attacks. It returns a HTTP status code to indicate whether the request is suspicious or not, based on NAXSI rules. The value of the returned code can be used in HAProxy rules to determine if the HTTP request should be blocked/rejected. commit 9ac143b6070e78afa3086b676f1049c6eb93cc13 Author: Emmanuel Hocdet Date: Mon May 29 14:36:20 2017 +0200 BUILD: ssl: fix build with OPENSSL_NO_ENGINE Build is broken with openssl library without support of engin (like boringssl). Add OPENSSL_NO_ENGINE flag to fix that. commit 201c07f681fc38df6cc236e6744849e5bde1f276 Author: Baptiste Assmann Date: Mon May 22 15:17:15 2017 +0200 MAJOR/REORG: dns: DNS resolution task and requester queues This patch is a major upgrade of the internal run-time DNS resolver in HAProxy and it brings the following 2 main changes: 1. DNS resolution task Up to now, DNS resolution was triggered by the health check task. From now, DNS resolution task is autonomous. It is started by HAProxy right after the scheduler is available and it is woken either when a network IO occurs for one of its nameserver or when a timeout is matched. From now, this means we can enable DNS resolution for a server without enabling health checking. 2. Introduction of a dns_requester structure Up to now, DNS resolution was purposely made for resolving server hostnames. The idea, is to ensure that any HAProxy internal object should be able to trigger a DNS resolution. For this purpose, 2 things has to be done: - clean up the DNS code from the server structure (this was already quite clean actually) and clean up the server's callbacks from manipulating too much DNS resolution - create an agnostic structure which allows linking a DNS resolution and a requester of any type (using obj_type enum) 3. Manage requesters through queues Up to now, there was an uniq relationship between a resolution and it's owner (aka the requester now). It's a shame, because in some cases, multiple objects may share the same hostname and may benefit from a resolution being performed by a third party. This patch introduces the notion of queues, which are basically lists of either currently running resolution or waiting ones. The resolutions are now available as a pool, which belongs to the resolvers. The pool has has a default size of 64 resolutions per resolvers and is allocated at configuration parsing. commit 8ea0bcc911809e77560bdd937c02a0b832526ef7 Author: Baptiste Assmann Date: Thu May 4 08:24:11 2017 +0200 MINOR: dns: introduce roundrobin into the internal cache (WIP) This patch introduces a bit of roundrobin in the records stored in our local cache. Purpose is to allow some kind of distribution of the IPs found in a response. Note that distribution properly applies only when the IP used by many requesters disappear and is replaced by an other one. commit 69fce67b56f2fda613604e0520adb64d209041a8 Author: Baptiste Assmann Date: Thu May 4 08:37:45 2017 +0200 MINOR: dns: make 'ancount' field to match the number of saved records ancount is the number of answers available in a DNS response. Before this patch, HAProxy used to store the ancount found in the buffer (sent by the DNS server). Unfortunately, this is now inaccurate and does not correspond to the number of records effectively stored in our local version of the response. In Example, the CNAMEs are not stored. This patch updates ancount field in to make it match what is effectively stored in our version. commit fa4a6630950138cfcb8904855c7517d9fdd472f7 Author: Baptiste Assmann Date: Thu May 4 09:05:00 2017 +0200 MINOR: dns: implement a LRU cache for DNS resolutions Introduction of a DNS response LRU cache in HAProxy. When a positive response is received from a DNS server, HAProxy stores it in the struct resolution and then also populates a LRU cache with the response. For now, the key in the cache is a XXHASH64 of the hostname in the domain name format concatened to the query type in string format. commit 729c901c3fcca6f4950e2b0e086e86cc2a73bbde Author: Baptiste Assmann Date: Mon May 22 15:13:10 2017 +0200 MAJOR: dns: save a copy of the DNS response in struct resolution Prior this patch, the DNS responses were stored in a pre-allocated memory area (allocated at HAProxy's startup). The problem is that this memory is erased for each new DNS responses received and processed. This patch removes the global memory allocation (which was not thread safe by the way) and introduces a storage of the dns response in the struct resolution. The memory in the struct resolution is also reserved at start up and is thread safe, since each resolution structure will have its own memory area. For now, we simply store the response and use it atomically per response per server. commit fb7091e213e7651d7a0305a4cfec3b45499f43f4 Author: Baptiste Assmann Date: Wed May 3 15:43:12 2017 +0200 MINOR: dns: new snr_check_ip_callback function In the process of breaking links between dns_* functions and other structures (mainly server and a bit of resolution), the function dns_get_ip_from_response needs to be reworked: it now can call "callback" functions based on resolution's owner type to allow modifying the way the response is processed. For now, main purpose of the callback function is to check that an IP address is not already affected to an element of the same type. For now, only server type has a callback. commit 42746373eb1fc6e57e89982c653caccaa7500ea0 Author: Baptiste Assmann Date: Wed May 3 12:12:02 2017 +0200 REORG: dns: dns_option structure, storage of hostname_dn This patch introduces a some re-organisation around the DNS code in HAProxy. 1. make the dns_* functions less dependent on 'struct server' and 'struct resolution'. With this in mind, the following changes were performed: - 'struct dns_options' has been removed from 'struct resolution' (well, we might need it back at some point later, we'll see) ==> we'll use the 'struct dns_options' from the owner of the resolution - dns_get_ip_from_response(): takes a 'struct dns_options' instead of 'struct resolution' ==> so the caller can pass its own dns options to get the most appropriate IP from the response - dns_process_resolve(): struct dns_option is deduced from new resolution->requester_type parameter 2. add hostname_dn and hostname_dn_len into struct server In order to avoid recomputing a server's hostname into its domain name format (and use a trash buffer to store the result), it is safer to compute it once at configuration parsing and to store it into the struct server. In the mean time, the struct resolution linked to the server doesn't need anymore to store the hostname in domain name format. A simple pointer to the server one will make the trick. The function srv_alloc_dns_resolution() properly manages everything for us: memory allocation, pointer updates, etc... 3. move resolvers pointer into struct server This patch makes the pointer to struct dns_resolvers from struct dns_resolution obsolete. Purpose is to make the resolution as "neutral" as possible and since the requester is already linked to the resolvers, then we don't need this information anymore in the resolution itself. commit 4f91f7ea59786a9a7bd592aad26dc25b28d3c9b0 Author: Baptiste Assmann Date: Wed May 3 12:09:54 2017 +0200 MINOR: dns: parse_server() now uses srv_alloc_dns_resolution() In order to make DNS code more consistent, the function parse_server() now uses srv_alloc_dns_resolution() to set up a server and its resolution. commit 81ed1a0516820c31272059eabdcc41565d2c714e Author: Baptiste Assmann Date: Wed May 3 10:11:44 2017 +0200 MINOR: dns: functions to manage memory for a DNS resolution structure A couple of new functions to allocate and free memory for a DNS resolution structure. Main purpose is to to make the code related to DNS more consistent. They allocate or free memory for the structure itself. Later, if needed, they should also allocate / free the buffers, etc, used by this structure. They don't set/unset any parameters, this is the role of the caller. This patch also implement calls to these function eveywhere it is required. commit d0aa6d23996ee5413bbab21a7598afc6724fe4aa Author: Baptiste Assmann Date: Mon Apr 3 14:40:20 2017 +0200 MINOR: dns: smallest DNS fqdn size global variable used to define the size of the smallest fqdn possible. commit 9d41fe7f98d7a5bd38b642ffa4fcc27b51a8820e Author: Baptiste Assmann Date: Thu May 4 12:13:31 2017 +0200 CLEANUP: server.c: missing prototype of srv_free_dns_resolution Prototype for the function srv_free_dns_resolution() missing at the top of the file. commit 23e9e931284b44e9d06cca26ab13648873b4029b Author: Stéphane Cottin Date: Thu May 18 08:58:41 2017 +0200 MINOR: log: Add logurilen tunable. The default len of request uri in log messages is 1024. In some use cases, you need to keep the long trail of GET parameters. The only way to increase this len is to recompile with DEFINE=-DREQURI_LEN=2048. This commit introduces a tune.http.logurilen configuration directive, allowing to tune this at runtime. commit a6cfa9098e5ad4e7fdf0f557c4e5945461ba6baa Author: William Lallemand Date: Thu Jun 1 17:38:58 2017 +0200 MAJOR: systemd-wrapper: get rid of the wrapper The master worker mode obsoletes the systemd-wrapper, to ensure that nobody uses it anymore, the code has been removed. commit 6db884d96158aa25b95131986c6db8ad2d987734 Author: William Lallemand Date: Thu Jun 1 17:38:57 2017 +0200 MEDIUM: systemd: Type=forking in unit file Adding Type=forking in the unit file ensure better monitoring from systemd. During a systemctl start the tool is able to return an error if it didn't work with this option. commit e202b1e951777abed6ad80381fcb8543adf69ed1 Author: William Lallemand Date: Thu Jun 1 17:38:56 2017 +0200 DOC: add documentation for the master-worker mode commit e20b6a62f853d493c67945880d9f89b6f3b4adbf Author: William Lallemand Date: Thu Jun 1 17:38:55 2017 +0200 MEDIUM: mworker: workers exit when the master leaves This patch ensure that the children will exit when the master quits, even if the master didn't send any signal. The master and the workers are connected through a pipe, when the pipe closes the children leave. commit 69f9b3bfa4ecee87cf95a6a8dc18c4a62ec126f9 Author: William Lallemand Date: Thu Jun 1 17:38:54 2017 +0200 MEDIUM: mworker: exit-on-failure option This option exits every workers when one of the current workers die. It allows you to monitor the master process in order to relaunch everything on a failure. For example it can be used with systemd and Restart=on-failure in a spec file. commit 85b0bd9e54041501248eff73e0f9901ffc7ed140 Author: William Lallemand Date: Thu Jun 1 17:38:53 2017 +0200 MEDIUM: mworker: try to guess the next stats socket to use with -x In master worker mode, you can't specify the stats socket where you get your listeners FDs on a reload, because the command line of the re-exec is launched by the master. To solve the problem, when -x is found on the command line, its parameter is rewritten on a reexec with the first stats socket with the capability to send sockets. It tries to reuse the original parameter if it has this capability. commit cb11fd2c7a32f5476f76cc93ed4c712935222a23 Author: William Lallemand Date: Thu Jun 1 17:38:52 2017 +0200 MEDIUM: mworker: wait mode on reload failure In Master Worker mode, when the reloading of the configuration fail, the process is exiting leaving the children without their father. To handle this, we register an exit function with atexit(3), which is reexecuting the binary in a special mode. This particular mode of HAProxy don't reload the configuration, it only loops on wait(). commit 73b85e75b3963086be889e1fb40a59e7ef2ad63b Author: William Lallemand Date: Thu Jun 1 17:38:51 2017 +0200 MEDIUM: mworker: handle reload and signals The master-worker will reload itself on SIGUSR2/SIGHUP It's inherited from the systemd wrapper, when the SIGUSR2 signal is received, the master process will reexecute itself with the -sf flag followed by the PIDs of the children. In the systemd wrapper, the children were using a pipe to notify when the config has been parsed and when the new process is ready. The goal was to ensure that the process couldn't reload during the parsing of the configuration, before signals were send to old process. With the new mworker model, the master parses the configuration and is aware of all the children. We don't need a pipe, but we need to block those signals before the end of a reload, to ensure that the process won't be killed during a reload. The SIGUSR1 signal is forwarded to the children to soft-stop HAProxy. The SIGTERM and SIGINT signals are forwarded to the children in order to terminate them. commit 095ba4c2428ec8bcccb134b3d24f07de2aabbdcd Author: William Lallemand Date: Thu Jun 1 17:38:50 2017 +0200 MEDIUM: mworker: replace systemd mode by master worker mode This commit remove the -Ds systemd mode in HAProxy in order to replace it by a more generic master worker system. It aims to replace entirely the systemd wrapper in the near future. The master worker mode implements a new way of managing HAProxy processes. The master is in charge of parsing the configuration file and is responsible for spawning child processes. The master worker mode can be invoked by using the -W flag. It can be used either in background mode (-D) or foreground mode. When used in background mode, the master will fork to daemonize. In master worker background mode, chroot, setuid and setgid are done in each child rather than in the master process, because the master process will still need access to filesystem to reload the configuration. commit 2c32d8f3793b2413e09f85361116da441c9df446 Author: Emmanuel Hocdet Date: Mon May 22 14:58:00 2017 +0200 MINOR: boringssl: basic support for OCSP Stapling Use boringssl SSL_CTX_set_ocsp_response to set OCSP response from file with '.ocsp' extension. CLI update is not supported. commit 3854e0102b9ebc444b4a58ebd51b05721e9ce2ef Author: Emeric Brun Date: Wed May 17 20:42:48 2017 +0200 MEDIUM: ssl: handle multiple async engines This patch adds the support of a maximum of 32 engines in async mode. Some tests have been done using 2 engines simultaneously. This patch also removes specific 'async' attribute from the connection structure. All the code relies only on Openssl functions. commit fa6c7ee702f53ef91883551c034ec346fb0479e1 Author: Grant Zhang Date: Sat Jan 14 01:42:15 2017 +0000 MAJOR: ssl: add openssl async mode support ssl-mode-async is a global configuration parameter which enables asynchronous processing in OPENSSL for all SSL connections haproxy handles. With SSL_MODE_ASYNC set, TLS I/O operations may indicate a retry with SSL_ERROR_WANT_ASYNC with this mode set if an asynchronous capable engine is used to perform cryptographic operations. Currently async mode only supports one async-capable engine. This is the latest version of the patchset which includes Emeric's updates : - improved async fd cleaning when openssl reports an fd to delete - prevent conn_fd_handler from calling SSL_{read,write,handshake} until the async fd is ready, as these operations are very slow and waste CPU - postpone of SSL_free to ensure the async operation can complete and does not cause a dereference a released SSL. - proper removal of async fd from the fdtab and removal of the unused async flag. commit 872f9c2139ed0e8fadb6d48de10d66fe132a2cf1 Author: Grant Zhang Date: Sat Jan 21 01:10:18 2017 +0000 MEDIUM: ssl: add basic support for OpenSSL crypto engine This patch adds the global 'ssl-engine' keyword. First arg is an engine identifier followed by a list of default_algorithms the engine will operate. If the openssl version is too old, an error is reported when the option is used. commit 7f80eb2383bb54ddafecf0e7df6b3b3ef4b4f6e5 Author: William Lallemand Date: Fri May 26 18:19:55 2017 +0200 MEDIUM: proxy: zombify proxies only when the expose-fd socket is bound When HAProxy is running with multiple processes and some listeners arebound to processes, the unused sockets were not closed in the other processes. The aim was to be able to send those listening sockets using the -x option. However to ensure the previous behavior which was to close those sockets, we provided the "no-unused-socket" global option. This patch changes this behavior, it will close unused sockets which are not in the same process as an expose-fd socket, making the "no-unused-socket" option useless. The "no-unused-socket" option was removed in this patch. commit f6975e9f76112c375af8ff98a22d5886541faeb3 Author: William Lallemand Date: Fri May 26 17:42:10 2017 +0200 MINOR: cli: add 'expose-fd listeners' to pass listeners FDs This patch changes the stats socket rights for allowing the sending of listening sockets. The previous behavior was to allow any unix stats socket with admin level to send sockets. It's not possible anymore, you have to set this option to activate the socket sending. Example: stats socket /var/run/haproxy4.sock mode 666 expose-fd listeners level user process 4 commit 07a62f7a7ee1caf24283ba73df1467be2517ae88 Author: William Lallemand Date: Wed May 24 00:57:40 2017 +0200 MINOR: cli: add ACCESS_LVL_MASK to store the access level The current level variable use only 2 bits for storing the 3 access level (user, oper and admin). This patch add a bitmask which allows to use the remaining bits for other usage. commit fd80df11c37af2d5d3c9a8659d45905638de717c Author: Thierry FOURNIER Date: Fri May 12 16:32:20 2017 +0200 BUG/MEDIUM: lua: segfault if a converter or a sample doesn't return anything In the case of a Lua sample-fetch or converter doesn't return any value, an acces outside the Lua stack can be performed. This patch check the stack size before converting the top value to a HAProxy internal sample. A workaround consist to check that a value value is always returned with sample fetches and converters. This patch should be backported in the version 1.6 and 1.7 commit 1bfc24ba03e4ae05fc76ba16119f24a226cd79c2 Author: Holger Just Date: Sat May 6 00:56:53 2017 +0200 MINOR: sample: Add b64dec sample converter Add "b64dec" as a new converter which can be used to decode a base64 encoded string into its binary representation. It performs the inverse operation of the "base64" converter. commit 64920538fc19f3c7f94dfa1e84a9a6569b8c3d37 Author: Frédéric Lécaille Date: Fri May 12 09:57:15 2017 +0200 BUG/MAJOR: dns: Broken kqueue events handling (BSD systems). Some DNS related network sockets were closed without unregistering their file descriptors from their underlying kqueue event sets. This patch replaces calls to close() by fd_delete() calls to that to delete such events attached to DNS network sockets from the kqueue before closing the sockets. The bug was introduced by commit 26c6eb8 ("BUG/MAJOR: dns: restart sockets after fork()") which was backported in 1.7 so this fix has to be backported there as well. Thanks to Jim Pingle who reported it and indicated the faulty commit, and to Lukas Tribus for the trace showing the bad file descriptor. commit 047000a8bddfffece923d254c72148d1d3211c69 Author: Dmitry Sivachenko Date: Fri May 12 15:38:29 2017 +0200 CLEANUP: retire obsoleted USE_GETSOCKNAME build option The last user of this option disappeared in 1.5-dev10. commit abd323395fa528ad5c633fbe7180d9ccb2c26c64 Author: Emmanuel Hocdet Date: Fri May 5 18:06:12 2017 +0200 MEDIUM: ssl: ssl-min-ver and ssl-max-ver compatibility. In haproxy < 1.8, no-sslv3/no-tlsv1x are ignored when force-sslv3/force-tlsv1x is used (without warning). With this patch, no-sslv3/no-tlsv1x are ignored when ssl-min-ver or ssl-max-ver is used (with warning). When all SSL/TLS versions are disable: generate an error, not a warning. example: ssl-min-ver TLSV1.3 (or force-tlsv13) with a openssl <= 1.1.0. commit e1c722b5e8e34a3e752bcc59fff88e8ebc384785 Author: Emmanuel Hocdet Date: Fri Mar 31 15:02:54 2017 +0200 MEDIUM: ssl: add ssl-min-ver and ssl-max-ver parameters for bind and server 'ssl-min-ver' and 'ssl-max-ver' with argument SSLv3, TLSv1.0, TLSv1.1, TLSv1.2 or TLSv1.3 limit the SSL negotiation version to a continuous range. ssl-min-ver and ssl-max-ver should be used in replacement of no-tls* and no-sslv3. Warning and documentation are set accordingly. commit 50e25e1dbce265f4a40cb4495ec3a4fe7c3af023 Author: Emmanuel Hocdet Date: Fri Mar 24 15:20:03 2017 +0100 MINOR: ssl: show methods supported by openssl TLS v1.3 incoming, SSLv3 will disappears: it could be useful to list all methods supported by haproxy/openssl (with -vvv). commit 42fb980e53773b9b705f0aadb389a994e4dd6cd9 Author: Emmanuel Hocdet Date: Thu Mar 30 19:29:39 2017 +0200 MINOR: ssl: support TLSv1.3 for bind and server This patch add 'no-tlsv13' and 'force-tlsv13' configuration. This is only useful with openssl-dev and boringssl. commit b4e9ba4b3618fe49ceceb325d58b17c68aeb6fca Author: Emmanuel Hocdet Date: Thu Mar 30 19:25:07 2017 +0200 MEDIUM: ssl: calculate the real min/max TLS version and find holes Plan is to add min-tlsxx max-tlsxx configuration, more consistent than no-tlsxx. Find the real min/max versions (openssl capabilities and haproxy configuration) and generate warning with bad versions range. 'no-tlsxx' can generate 'holes': "The list of protocols available can be further limited using the SSL_OP_NO_X options of the SSL_CTX_set_options or SSL_set_options functions. Clients should avoid creating 'holes' in the set of protocols they support, when disabling a protocol, make sure that you also disable either all previous or all subsequent protocol versions. In clients, when a protocol version is disabled without disabling all previous protocol versions, the effect is to also disable all subsequent protocol versions." To not break compatibility, "holes" is authorized with warning, because openssl 1.1.0 and boringssl deal with it (keep the upper or lower range depending the case and version). commit 5db33cbdc4f2952cbd3c140edce0eda84e1447b4 Author: Emmanuel Hocdet Date: Thu Mar 30 19:19:37 2017 +0200 MEDIUM: ssl: ssl_methods implementation is reworked and factored for min/max tlsxx Plan is to add min-tlsxx max-tlsxx configuration, more consistent than no-tlsxx. This patch introduce internal min/max and replace force-tlsxx implementation. SSL method configuration is store in 'struct tls_version_filter'. SSL method configuration to openssl setting is abstract in 'methodVersions' table. With openssl < 1.1.0, SSL_CTX_set_ssl_version is used for force (min == max). With openssl >= 1.1.0, SSL_CTX_set_min/max_proto_version is used. commit 6cb2d1e9638f22f080d1391e6bb8260b6a0b1f95 Author: Emmanuel Hocdet Date: Thu Mar 30 14:43:31 2017 +0200 MEDIUM: ssl: revert ssl/tls version settings relative to default-server. Plan is to add min-tlsxx max-tlsxx configuration, more consistent than no-tlsxx. min-tlsxx and max-tlsxx can be overwrite on local definition. This directives should be the only ones needed in default-server. To simplify next patches (rework of tls versions settings with min/max) all ssl/tls version settings relative to default-server are reverted first: remove: 'sslv3', 'tls*', 'no-force-sslv3', 'no-force-tls*'. remove from default-server: 'no-sslv3', 'no-tls*'. Note: . force-tlsxx == min-tlsxx + max-tlsxx : would be ok in default-server. . no-tlsxx is keep for compatibility: should not be propagated to default-server. commit 53ae85c38e1d70634d06ea0958c17bb5b49d05b7 Author: Lukas Tribus Date: Thu May 4 15:45:40 2017 +0000 MINOR: ssl: add prefer-client-ciphers Currently we unconditionally set SSL_OP_CIPHER_SERVER_PREFERENCE [1], which may not always be a good thing. The benefit of server side cipher prioritization may not apply to all cases out there, and it appears that the various SSL libs are going away from this recommendation ([2], [3]), as insecure ciphers suites are properly blacklisted/removed and honoring the client's preference is more likely to improve user experience (for example using SW-friendly ciphers on devices without HW AES support). This is especially true for TLSv1.3, which will restrict the cipher suites to just AES-GCM and Chacha20/Poly1305. Apache [4], nginx [5] and others give admins full flexibility, we should as well. The initial proposal to change the current default and add a "prefer-server-ciphers" option (as implemented in e566ecb) has been declined due to the possible security impact. This patch implements prefer-client-ciphers without changing the defaults. [1] https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_options.html [2] https://github.com/openssl/openssl/issues/541 [3] https://github.com/libressl-portable/portable/issues/66 [4] https://httpd.apache.org/docs/2.0/en/mod/mod_ssl.html#sslhonorcipherorder [5] https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers commit f494977bc1a361c26f8cc0516366ef2662ac9502 Author: Willy Tarreau Date: Sat May 6 08:45:28 2017 +0200 BUG/MINOR: checks: don't send proxy protocol with agent checks James Brown reported that agent-check mistakenly sends the proxy protocol header when it's configured. This is obviously wrong as the agent is an independant servie and not a traffic port, let's disable this. This fix must be backported to 1.7 and possibly 1.6. commit 8abbd3daa7d61590edbd28afd6f50067ac956002 Author: David CARLIER Date: Wed May 3 10:23:08 2017 +0100 BUG/MINOR: contrib/mod_security: fix build on FreeBSD Fix linker flags settings since 3rd parties libraries are not in /usr/lib Plus libfuzzy needs to be added. undef LIST_HEAD from event2 which conflicts with haproxy's commit b418c1228cc13d3c26ff95ea1b617e7f35fa9d4e Author: Frédéric Lécaille Date: Wed Apr 26 11:24:02 2017 +0200 MINOR: server: cli: Add server FQDNs to server-state file and stats socket. This patch adds a new stats socket command to modify server FQDNs at run time. Its syntax: set server / fqdn This patch also adds FQDNs to server state file at the end of each line for backward compatibility ("-" if not present). commit b7a6d0d8d781f8917709e9c111fe18877193147d Author: Willy Tarreau Date: Tue May 2 22:14:59 2017 +0200 CONTRIB: tcploop: add action "X" to execute a command Sometimes it's convenient to be able to execute a command directly on the stream, whether we're connecting or accepting an incoming connection. New command 'X' makes this possible. It simply calls execvp() on the next arguments and branches stdin/stdout/stderr on the socket. Optionally it's possible to limit the passed FDs to any combination of them by appending 'i', 'o', 'e' after the X. In any case the program ends just after executing this command. Examples : - chargen server tcploop 8001 L A Xo cat /dev/zero - telnet server tcploop 8001 L W N A X /usr/sbin/in.telnetd commit 23953686da572fcbde697694cfbdb4b5df738a51 Author: Lukas Tribus Date: Fri Apr 28 13:24:30 2017 +0000 DOC: update RFC references A few doc and code comment updates bumping RFC references to the new ones. commit fa5c5c892d32fd5a52214f4c96cea54ed78d9f0d Author: Emeric Brun Date: Fri Apr 28 16:19:51 2017 +0200 BUG/MINOR: ssl: fix warnings about methods for opensslv1.1. This patch replaces the calls to TLSvX_X_client/server/_method by the new TLS_client/server_method and it uses the new functions SSL_set_min_proto_version and SSL_set_max_proto_version, setting them at the wanted protocol version using 'force-' statements. commit a5ec06de2c2bb71bbd16050512e08b5b05749163 Author: Thierry FOURNIER Date: Mon Apr 10 23:47:23 2017 +0200 MINOR: Add ModSecurity wrapper as contrib This patch contains a base for a modsecurity wrapper in HAProxy using SPOE. commit d7d8881543a1eb1be91207e7b811ef5f9df0eddd Author: Thierry FOURNIER Date: Wed Apr 19 15:15:14 2017 +0200 MINOR: proto-http: Add sample fetch wich returns all HTTP headers The sample fetch returns all headers including the last jump line. The last jump line is used to determine if the block of headers is truncated or not. commit 5617dce27d69f1b90130d38dc61d6940ff01fbcc Author: Thierry FOURNIER Date: Sun Apr 9 05:38:19 2017 +0200 MINOR: Add binary encoding request header sample fetch This sample fetch encodes the http request headers in binary format. This sample-fetch is useful with SPOE. commit 6ab2bae08499442ec22d2d43d5bfed0127d2db86 Author: Thierry FOURNIER Date: Wed Apr 19 11:49:44 2017 +0200 REORG: spoe: move spoe_encode_varint / spoe_decode_varint from spoe to common These encoding functions does general stuff and can be used in other context than spoe. This patch moves the function spoe_encode_varint and spoe_decode_varint from spoe to common. It also remove the prefix spoe. These functions will be used for encoding values in new binary sample fetch. commit f4128a9981de0b0fd5f5cea9e58942245eeef332 Author: Thierry FOURNIER Date: Sun Apr 9 05:41:27 2017 +0200 BUG/MINOR: change header-declared function to static inline When we include the header proto/spoe.h in other files in the same project, the compilator claim that the symbol have multiple definitions: src/flt_spoe.o: In function `spoe_encode_varint': ~/git/haproxy/include/proto/spoe.h:45: multiple definition of `spoe_encode_varint' src/proto_http.o:~/git/haproxy/include/proto/spoe.h:45: first defined here commit 18330ab17fb36a3eb8292c8c128751e0d230ec27 Author: Andrew Rodland Date: Wed Apr 26 02:57:03 2017 -0400 BUG/MINOR: hash-balance-factor isn't effective in certain circumstances in chash_get_server_hash, we find the nearest server entries both before and after the request hash. If the next and prev entries both point to the same server, the function would exit early and return that server, to save work. Before hash-balance-factor this was a valid optimization -- one of nsrv and psrv would definitely be chosen, so if they are the same there's no need to choose between them. But with hash-balance-factor it's possible that adding another request to that server would overload it (chash_server_is_eligible returns false) and we go further around the ring. So it's not valid to return before checking for that. This commit simply removes the early return, as it provides a minimal savings even when it's correct. commit e068b60605ad5bdf7ec3bde016b4f681171f5443 Author: Thierry FOURNIER Date: Wed Apr 26 13:27:05 2017 +0200 CLEANUP: lua: remove test The man of "luaL_unref" says "If ref is LUA_NOREF or LUA_REFNIL, luaL_unref does nothing.", so I remove the check. commit f3267677119430d07c0c9d75f2b5818430b99816 Author: Thierry FOURNIER Date: Wed Apr 26 14:25:58 2017 +0200 BUG/MEDIUM: lua: memory leak The priv context is not cleaned when we set a new priv context. This is caused by a stupid swap between two parameter of the luaL_unref() function. workaround: use set_priv only once when we process a stream. This patch should be backported in version 1.7 and 1.6 commit 013a84fe939cf393fbcf8deb9b4504941d382777 Author: Jarno Huuskonen Date: Sat Apr 22 11:26:50 2017 +0300 DOC: errloc/errorloc302/errorloc303 missing status codes. errorloc/errorloc302/errorloc303 was missing some status codes that haproxy can generate. commit 95b012bb9191ff37d4ed17b324499f914b206cf1 Author: Jarno Huuskonen Date: Thu Apr 6 13:59:14 2017 +0300 DOC: add layer 4 links/cross reference to "block" keyword. Idea from Aleksandar Lazic: add explanation/links about layer4 tcp-request connection or content reject to "block" keyword. Add http-request cross ref. to "tcp-request content". commit cb4502e3aa70e552eb83937508cd1ea501d9d7d0 Author: Frédéric Lécaille Date: Thu Apr 20 13:36:25 2017 +0200 DOC: Add documentation for new "server-template" keyword. commit 72ed4758d6c329404083e6681e383261c650f755 Author: Frédéric Lécaille Date: Fri Apr 14 13:28:00 2017 +0200 MINOR: server: Add server_template_init() function to initialize servers from a templates. This patch adds server_template_init() function used to initialize servers from server templates. It is called just after having parsed a 'server-template' line. commit b82f742b7879a37c12c7915aaef2392e2f48a0e0 Author: Frédéric Lécaille Date: Thu Apr 13 18:24:23 2017 +0200 MINOR: server: Add 'server-template' new keyword supported in backend sections. This patch makes backend sections support 'server-template' new keyword. Such 'server-template' objects are parsed similarly to a 'server' object by parse_server() function, but its first arguments are as follows: server-template : ... The remaining arguments are the same as for 'server' lines. With such server template declarations, servers may be allocated with IDs built from and arguments. For instance declaring: server-template foo 1-5 google.com:80 ... or server-template foo 5 google.com:80 ... would be equivalent to declare: server foo1 google.com:80 ... server foo2 google.com:80 ... server foo3 google.com:80 ... server foo4 google.com:80 ... server foo5 google.com:80 ... commit 759ea98db2cd90e4c9f81e46fbd88bc60a69e309 Author: Frédéric Lécaille Date: Thu Mar 30 17:32:36 2017 +0200 MINOR: server: Extract the code which finalizes server initializations after 'server' lines parsing. This patch moves the code which is responsible of finalizing server initializations after having fully parsed a 'server' line (health-check, agent check and SNI expression initializations) from parse_server() to new functions. commit 58b207cdd54353b9eacc46e9ce23d6ecbf70d68e Author: Frédéric Lécaille Date: Thu Mar 30 14:18:30 2017 +0200 MINOR: server: Extract the code responsible of copying default-server settings. This patch moves the code responsible of copying default server settings to a new server instance from parse_server() function to new defsrv_*_cpy() functions which may be used both during server lines parsing and during server templates initializations to come. These defsrv_*_cpy() do not make any reference to anything else than default server settings. commit daa2fe662116e0741d16c9deb6825773abb0bff3 Author: Frédéric Lécaille Date: Thu Apr 20 12:17:50 2017 +0200 BUG/MINOR: server: missing default server 'resolvers' setting duplication. 'resolvers' setting was not duplicated from default server setting to new server instances when parsing 'server' lines. This fix is simple: strdup() default resolvers string argument after having allocated a new server when parsing 'server' lines. This patch must be backported to 1.7 and 1.6. commit 9f724edbd8d1cf595d4177c3612607f395b4380e Author: Christopher Faulet Date: Thu Apr 20 14:16:13 2017 +0200 BUG/MEDIUM: http: Drop the connection establishment when a redirect is performed This bug occurs when a redirect rule is applied during the request analysis on a persistent connection, on a proxy without any server. This means, in a frontend section or in a listen/backend section with no "server" line. Because the transaction processing is shortened, no server can be selected to perform the connection. So if we try to establish it, this fails and a 503 error is returned, while a 3XX was already sent. So, in this case, HAProxy generates 2 replies and only the first one is expected. Here is the configuration snippet to easily reproduce the problem: listen www bind :8080 mode http timeout connect 5s timeout client 3s timeout server 6s redirect location / A simple HTTP/1.1 request without body will trigger the bug: $ telnet 0 8080 Trying 0.0.0.0... Connected to 0. Escape character is '^]'. GET / HTTP/1.1 HTTP/1.1 302 Found Cache-Control: no-cache Content-length: 0 Location: / HTTP/1.0 503 Service Unavailable Cache-Control: no-cache Connection: close Content-Type: text/html

503 Service Unavailable

No server is available to handle this request. Connection closed by foreign host. [wt: only 1.8-dev is impacted though the bug is present in older ones] commit 7d8e6889536f38c00e70c4187397d1c4d009aeae Author: Olivier Houchard Date: Thu Apr 20 18:21:17 2017 +0200 BUG/MINOR: server: don't use "proxy" when px is really meant. In server_parse_sni_expr(), we use the "proxy" global variable, when we should probably be using "px" given as an argument. It happens to work by accident right now, but may not in the future. [wt: better backport it] commit 73bdb325edfc12a58ff6a7ebec77d56d62dcabe0 Author: Olivier Houchard Date: Wed Apr 19 11:34:10 2017 +0200 BUG/MAJOR: Use -fwrapv. Haproxy relies on signed integer wraparound on overflow, however this is really an undefined behavior, so the C compiler is allowed to do whatever it wants, and clang does exactly that, and that causes problems when the timer goes from <= INT_MAX to > INT_MAX, and explains the various hangs reported on FreeBSD every 49.7 days. To make sure we get the intended behavior, use -fwrapv for now. A proper fix is to switch everything to unsigned, and it will happen later, but this is simpler, and more likely to be backported to the stable branches. Many thanks to David King, Mark S, Dave Cottlehuber, Slawa Olhovchenkov, Piotr Pawel Stefaniak, and any other I may have forgotten for reporting that and investigating. commit b83dc3d2ef5ffa882aed926ee4d6a82bd94024f0 Author: Willy Tarreau Date: Wed Apr 19 11:24:07 2017 +0200 MEDIUM: config: don't check config validity when there are fatal errors Overall we do have an issue with the severity of a number of errors. Most fatal errors are reported with ERR_FATAL (which prevents startup) and not ERR_ABORT (which stops parsing ASAP), but check_config_validity() is still called on ERR_FATAL, and will most of the time report bogus errors. This is what caused smp_resolve_args() to be called on a number of unparsable ACLs, and it also is what reports incorrect ordering or unresolvable section names when certain entries could not be properly parsed. This patch stops this domino effect by simply aborting before trying to further check and resolve the configuration when it's already know that there are fatal errors. A concrete example comes from this config : userlist users : user foo insecure-password bar listen foo bind :1234 mode htttp timeout client 10S timeout server 10s timeout connect 10s stats uri /stats stats http-request auth unless { http_auth(users) } http-request redirect location /index.html if { path / } It contains a colon after the userlist name, a typo in the client timeout value, another one in "mode http" which cause some other configuration elements not to be properly handled. Previously it would confusingly report : [ALERT] 108/114851 (20224) : parsing [err-report.cfg:1] : 'userlist' cannot handle unexpected argument ':'. [ALERT] 108/114851 (20224) : parsing [err-report.cfg:6] : unknown proxy mode 'htttp'. [ALERT] 108/114851 (20224) : parsing [err-report.cfg:7] : unexpected character 'S' in 'timeout client' [ALERT] 108/114851 (20224) : Error(s) found in configuration file : err-report.cfg [ALERT] 108/114851 (20224) : parsing [err-report.cfg:11] : unable to find userlist 'users' referenced in arg 1 of ACL keyword 'http_auth' in proxy 'foo'. [WARNING] 108/114851 (20224) : config : missing timeouts for proxy 'foo'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [WARNING] 108/114851 (20224) : config : 'stats' statement ignored for proxy 'foo' as it requires HTTP mode. [WARNING] 108/114851 (20224) : config : 'http-request' rules ignored for proxy 'foo' as they require HTTP mode. [ALERT] 108/114851 (20224) : Fatal errors found in configuration. The "requires HTTP mode" errors are just pollution resulting from the improper spelling of this mode earlier. The unresolved reference to the userlist is caused by the extra colon on the declaration, and the warning regarding the missing timeouts is caused by the wrong character. Now it more accurately reports : [ALERT] 108/114900 (20225) : parsing [err-report.cfg:1] : 'userlist' cannot handle unexpected argument ':'. [ALERT] 108/114900 (20225) : parsing [err-report.cfg:6] : unknown proxy mode 'htttp'. [ALERT] 108/114900 (20225) : parsing [err-report.cfg:7] : unexpected character 'S' in 'timeout client' [ALERT] 108/114900 (20225) : Error(s) found in configuration file : err-report.cfg [ALERT] 108/114900 (20225) : Fatal errors found in configuration. Despite not really a fix, this patch should be backported at least to 1.7, possibly even 1.6, and 1.5 since it hardens the config parser against certain bad situations like the recently reported use-after-free and the last null dereference. commit bcfe23a7ecb212ca07dfbe20423684944d55e086 Author: Willy Tarreau Date: Wed Apr 19 11:13:48 2017 +0200 BUG/MEDIUM: acl: proprely release unused args in prune_acl_expr() Stephan Zeisberg reported another dirty abort case which can be triggered with this simple config (where file "d" doesn't exist) : backend b1 stats auth a:b acl auth_ok http_auth(c) -f d This issue was brought in 1.5-dev9 by commit 34db108 ("MAJOR: acl: make use of the new argument parsing framework") when prune_acl_expr() started to release arguments. The arg pointer is set to NULL but not its length. Because of this, later in smp_resolve_args(), the argument is still seen as valid (since only a test on the length is made as in all other places), and the NULL pointer is dereferenced. This patch properly clears the lengths to avoid such tests. This fix needs to be backported to 1.7, 1.6, and 1.5. commit a2278c8bbbee63ee75c8e3e97efd6828c2083d89 Author: Jim Freeman Date: Sat Apr 15 08:01:59 2017 -0600 CLEANUP: logs: typo: simgle => single Typo in error message. Backport to 1.7. commit dfacd69b94f6f2358e252dfa27e279686bcee922 Author: Frédéric Lécaille Date: Sun Apr 16 17:14:14 2017 +0200 BUG/MAJOR: Broken parsing for valid keywords provided after 'source' setting. Any valid keyword could not be parsed anymore if provided after 'source' keyword. This was due to the fact that 'source' number of arguments is variable. So, as its parser srv_parse_source() is the only one who may know how many arguments was provided after 'source' keyword, it updates 'cur_arg' variable (the index in the line of the current arg to be parsed), this is a good thing. This variable is also incremented by one (to skip the 'source' keyword). This patch disable this behavior. Should have come with dba9707 commit. commit 8d083ed796af135be942594182f4fc83459ee43a Author: Frédéric Lécaille Date: Fri Apr 14 15:19:56 2017 +0200 BUG/MINOR: server: Fix a wrong error message during 'usesrc' keyword parsing. 'usesrc' setting is not permitted on 'server' lines if not provided after 'source' setting. This is now also the case on 'default-server' lines. Without this patch parse_server() parser displayed that 'usersrc' is an unknown keyword. Should have come with dba9707 commit. commit 2c9744fe56825c83d62fd94db52fbb489227f4cd Author: Olivier Houchard Date: Sun Apr 9 16:28:10 2017 +0200 MINOR: systemd wrapper: add support for passing the -x option. Make the systemd wrapper chech if HAPROXY_STATS_SOCKET if set. If set, it will use it as an argument to the "-x" option, which makes haproxy asks for any listening socket, on the stats socket, in order to achieve reloads with no new connection lost. commit 547408787ffe1c03b975e94f01b64492d15ca97d Author: Olivier Houchard Date: Thu Apr 6 14:45:14 2017 +0200 MINOR: socket transfer: Set a timeout on the socket. Make sure we're not stuck forever by setting a timeout on the socket. commit 1fc0516516defd57574efb677d7f698757e62d68 Author: Olivier Houchard Date: Thu Apr 6 01:05:05 2017 +0200 MINOR: proxy: Don't close FDs if not our proxy. When running with multiple process, if some proxies are just assigned to some processes, the other processes will just close the file descriptors for the listening sockets. However, we may still have to provide those sockets when reloading, so instead we just try hard to pretend those proxies are dead, while keeping the sockets opened. A new global option, no-reused-socket", has been added, to restore the old behavior of closing the sockets not bound to this process. commit d33fc3a7f5882add601725e71163539eb1163847 Author: Olivier Houchard Date: Wed Apr 5 22:50:59 2017 +0200 MINOR: doc: document the -x flag commit 153659f1ae69a1741109fcb95cac2c7d64f99a29 Author: Olivier Houchard Date: Wed Apr 5 22:39:56 2017 +0200 MINOR: tcp: When binding socket, attempt to reuse one from the old proc. Try to reuse any socket from the old process, provided by the "-x" flag, before binding a new one, assuming it is compatible. "Compatible" here means same address and port, same namspace if any, same interface if any, and that the following flags are the same : LI_O_FOREIGN, LI_O_V6ONLY and LI_O_V4V6. Also change tcp_bind_listener() to always enable/disable socket options, instead of just doing so if it is in the configuration file, as the option may have been removed, ie TCP_FASTOPEN may have been set in the old process, and removed from the new configuration, so we have to disable it. commit f73629d23a44995f84887859fcbc7d22a2227eec Author: Olivier Houchard Date: Wed Apr 5 22:33:04 2017 +0200 MINOR: global: Add an option to get the old listening sockets. Add the "-x" flag, that takes a path to a unix socket as an argument. If used, haproxy will connect to the socket, and asks to get all the listening sockets from the old process. Any failure is fatal. This is needed to get seamless reloads on linux. commit f886e3478def3afe645b086243a919246a4102a4 Author: Olivier Houchard Date: Wed Apr 5 22:24:59 2017 +0200 MINOR: cli: Add a command to send listening sockets. Add a new command that will send all the listening sockets, via the stats socket, and their properties. This is a first step to workaround the linux problem when reloading haproxy. commit 42ef75fb849daeda001aa6b7ad01d66091ef553a Author: Willy Tarreau Date: Wed Apr 12 21:40:29 2017 +0200 MINOR: lua: ensure the memory allocator is used all the time luaL_setstate() uses malloc() to initialize the first objects, and only after this we replace the allocator. This creates trouble when replacing the standard memory allocators during debugging sessions since the new allocator is used to realloc() an area previously allocated using the default malloc(). Lua provides lua_newstate() in addition to luaL_newstate(), which takes an allocator for the initial malloc. This is exactly what we need, and this patch does this and fixes the problem. The now useless call to lua_setallocf() could be removed. This has no impact outside of debugging sessions and there's no need to backport this. commit 04bf98149bacb4764ae9263fa30ab91dc7c595bc Author: Willy Tarreau Date: Thu Apr 13 15:09:26 2017 +0200 BUG/MEDIUM: servers: unbreak server weight propagation This reverts commit 266b1a8 ("MEDIUM: server: Inherit CLI weight changes and agent-check weight responses") from Michal Idzikowski, which is still broken. It stops propagating weights at the first error encountered, leaving servers in a random state depending on what LB algorithms are used on other servers tracking the one experiencing the weight change. It's unsure what the best way to address this is, but we cannot leave the servers in an inconsistent state between farms. For example : backend site1 mode http balance uri hash-type consistent server s1 127.0.0.1:8001 weight 10 track servers/s1 backend site2 mode http balance uri server s1 127.0.0.1:8001 weight 10 track servers/s1 backend site3 mode http balance uri hash-type consistent server s1 127.0.0.1:8001 weight 10 track servers/s1 backend servers server s1 127.0.0.1:8001 weight 10 check inter 1s The weight change is applied on "servers/s1". It tries to propagate to the servers tracking it, which are site1/s1, site2/s1 and site3/s1. Let's say that "weight 50%" is requested. The servers are linked in reverse-order, so the change is applied to "servers/s1", then to "site3/s1", then to "site2/s1" and this one fails and rejects the change. The change is aborted and never propagated to "site1/s1", which keeps the server in a different state from "site3/s1". At the very least, in case of error, the changes should probably be unrolled. Also the error reported on the CLI (when changing from the CLI) simply says : Backend is using a static LB algorithm and only accepts weights '0%' and '100%'. Without more indications what the faulty backend is. Let's revert this change for now, as initially feared it will definitely cause more harm than good and at least needs to be revisited. It was never backported to any stable branch so no backport is needed. commit 145325e59d2f2ac78cc3eb6eb85243846b67fb41 Author: Willy Tarreau Date: Wed Apr 12 23:03:31 2017 +0200 BUG/MEDIUM: acl: don't free unresolved args in prune_acl_expr() In case of error it's very difficult to properly unroll the list of unresolved args because the error can appear on any argument, and all of them share the same memory area, pointed to by one or multiple links from the global args list. The problem is that till now the arguments themselves were released and were not unlinked from the list, causing all forms of corruption in deinit() when quitting on the error path if an argument couldn't properly parse. A few attempts at trying to selectively spot the appropriate list entries to kill before releasing the shared area have only resulted in complicating the code and pushing the issue further. Here instead we use a simple conservative approach : prune_acl_expr() only tries to free the argument array if none of the arguments were unresolved, which means that none of them was added to the arg list. It's unclear what a better approach would be. We could imagine that args would point to their own location in the shared list but given that this extra cost and complexity would be added exclusively in order to cleanly release everything when we're exiting due to a config parse error, this seems quite overkill. This bug was noticed on 1.7 and likely affects 1.6 and 1.5, so the fix should be backported. It's not easy to reproduce it, as the reproducers randomly work depending on how memory is allocated. One way to do it is to use parsable and non-parsable patterns on an ACL making use of args. Big thanks to Stephan Zeisberg for reporting this problem with a working reproducer. commit 0622f02b5aac4ae104f809a6beb836ccecef9a60 Author: Willy Tarreau Date: Wed Apr 12 22:32:04 2017 +0200 BUG/MEDIUM: arg: ensure that we properly unlink unresolved arguments on error If make_arg_list() fails to process an argument after having queued an unresolvable one, it frees the allocated argument list but doesn't remove the referenced args from the arg list. This causes a use after free or a double free if the same location was reused, during the deinit phase upon exit after reporting the error. Since it's not easy to properly unlinked all elements, we only release the args block if none of them was queued in the list. commit 266b1a8336bcb9a2d5de772a14b9d84af25bcf81 Author: Michal Idzikowski Date: Mon Mar 27 14:45:49 2017 +0200 MEDIUM: server: Inherit CLI weight changes and agent-check weight responses When agent-check or CLI command executes relative weight change this patch propagates it to tracking server allowing grouping many backends running on same server underneath. Additionaly in case with many src IPs many backends can have shared state checker, so there won't be unnecessary health checks. [wt: Note: this will induce some behaviour change on some setups] commit a9e2e4b89922384755ac6a0fb4c24a1b159d43f2 Author: Willy Tarreau Date: Wed Apr 12 22:28:52 2017 +0200 BUG/MINOR: arg: don't try to add an argument on failed memory allocation Take care of arg_list_clone() returning NULL in arg_list_add() since the former does it too. It's only used during parsing so the impact is very low. Can be backported to 1.7, 1.6 and 1.5. commit 1822e8c35614fb01f25661ea541d9e0599b56a59 Author: Willy Tarreau Date: Wed Apr 12 18:54:00 2017 +0200 BUG/MINOR: config: missing goto out after parsing an incorrect ACL character The error doesn't prevent checking for other errors after an invalid character was detected in an ACL name. Better quit ASAP to avoid risking to emit garbled and confusing error messages if something else fails on the same line. This should be backported to 1.7, 1.6 and 1.5. commit 5e5bc9fc235fcce48236795777ef29ae63452056 Author: Frédéric Lécaille Date: Tue Apr 11 08:46:37 2017 +0200 BUG/MINOR: dns: Wrong address family used when creating IPv6 sockets. AF_INET address family was always used to create sockets to connect to name servers. This prevented any connection over IPv6 from working. This fix must be backported to 1.7 and 1.6. commit 73459797fdba710990d835cd018058e5e8c3a52f Author: Willy Tarreau Date: Tue Apr 11 07:58:08 2017 +0200 BUILD/MINOR: tools: fix build warning in debug_hexdump() Commit 0ebb511 ("MINOR: tools: add a generic hexdump function for debugging") introduced debug_hexdump() which is used to dump a memory area during debugging sessions. This function can start at an unaligned offset and uses a signed comparison to know where to start dumping from. But the operation mixes signed and unsigned, making the test incorrect and causing the following warnings to be emitted under Clang : src/standard.c:3775:14: warning: comparison of unsigned expression >= 0 is always true [-Wtautological-compare] if (b + j >= 0 && b + j < len) ~~~~~ ^ ~ Make "j" signed instead. At the moment this function is not used at all so there's no impact. Thanks to Dmitry Sivachenko for reporting it. No backport is needed. commit 9d7fb63e33535cbd81d3c3c5fb512ace19c03ba2 Author: Willy Tarreau Date: Tue Apr 11 07:53:04 2017 +0200 BUILD/MINOR: stats: remove unexpected argument to stats_dump_json_header() Commit 05ee213 ("MEDIUM: stats: Add JSON output option to show (info|stat)") used to pass argument "uri" to the aforementionned function which doesn't take any. It's probably a leftover from multiple iterations of the same patchset. Spotted by Dmitry Sivachenko. No backport is needed. commit 3a471935e6b83bc6172508eba9059e1262fd1ac3 Author: David Carlier Date: Fri Apr 7 20:48:00 2017 +0100 BUG/MINOR: server : no transparent proxy for DragonflyBSD IP*_BINDANY is not defined under this system thus it is necessary to make those fields access since CONFIG_HAP_TRANSPARENT is not defined. [wt: problem introduced late in 1.8-dev. The same fix was also reported by Steven Davidovitz] commit 68af3c1a2eacdf56a1d676d2371d915238a64bfc Author: Adam Spiers Date: Thu Apr 6 16:31:39 2017 +0100 DOC: stick-table is available in frontend sections Fix the proxy keywords matrix to reflect that it's permitted to use stick-table in frontend sections. Signed-off-by: Adam Spiers commit b4a2d5e19a06225fd5b07632354314bc8d35a2d6 Author: Olivier Houchard Date: Tue Apr 4 22:10:36 2017 +0200 MINOR server: Restrict dynamic cookie check to the same proxy. Each time we generate a dynamic cookie, we try to make sure the same cookie hasn't been generated for another server, it's very unlikely, but it may happen. We only have to check that for the servers in the same proxy, no, need to check in others, plus the code was buggy and would always check in the first proxy of the proxy list. commit c28bb55cdc554549a59f92997ebe7abf8d4612fe Author: Glenn Strauss Date: Wed Apr 5 01:51:37 2017 -0400 DOC: mention lighttpd 1.4.46 implements PROXY lighttpd 1.4.46 implements PROXY v1 and v2 for incoming connections. commit 91cc8081a2eff2e3d7d9f63411c2b6cf1e183923 Author: Glenn Strauss Date: Wed Apr 5 01:37:20 2017 -0400 DOC: update sample code for PROXY protocol lengths are in network byte order so use ntohs(hdr.v2.len) when calculating size of v2 header to recv() from head of stream. commit e5ae702222a02614612fa9d869bcfa761d29f9f0 Author: Jarno Huuskonen Date: Mon Apr 3 14:36:21 2017 +0300 DOC: add few comments to examples. - http-check expect rstring is missing html comment closing tag. - option redis-check: See also "option tcp-check", "tcp-check expect". - ignore-persist static url example ((from 7.2 Using ACLs to form conditions) - tcp-request content: hdr(x-forwarded-for) example: added hdr_ip(x-forwarded-for) version. - tcp-request content: added comments to sc0_inc_gpc0 stick table example. - timeout tarpit: mention http-request tarpit. commit 84c51ec9e7593aea1fc42e1cf64cb1dca37ba193 Author: Jarno Huuskonen Date: Mon Apr 3 14:20:34 2017 +0300 DOC: changed "block"(deprecated) examples to http-request deny commit 6f1820864b5d83a9773ed495a3d4802f1cfb2a3c Author: David Carlier Date: Mon Apr 3 21:58:04 2017 +0100 CLEANUP: server: moving netinet/tcp.h inclusion netinet/tcp.h needs sys/types.h for u_int* types usage, issue found while building on OpenBSD. commit 7b677265fdb132df0eb52e5ff5469043f1479f98 Author: Willy Tarreau Date: Mon Apr 3 09:27:49 2017 +0200 [RELEASE] Released version 1.8-dev1 Released version 1.8-dev1 with the following main changes : - BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos - BUG/MINOR: stats: make field_str() return an empty string on NULL - DOC: Spelling fixes - BUG/MEDIUM: http: Fix tunnel mode when the CONNECT method is used - BUG/MINOR: http: Keep the same behavior between 1.6 and 1.7 for tunneled txn - BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER - BUG/MINOR: filters: Invert evaluation order of HTTP_XFER_BODY and XFER_DATA analyzers - BUG/MINOR: http: Call XFER_DATA analyzer when HTTP txn is switched in tunnel mode - BUG/MAJOR: stream: fix session abort on resource shortage - OPTIM: stream-int: don't disable polling anymore on DONT_READ - BUG/MINOR: cli: allow the backslash to be escaped on the CLI - BUG/MEDIUM: cli: fix "show stat resolvers" and "show tls-keys" - DOC: Fix map table's format - DOC: Added 51Degrees conv and fetch functions to documentation. - BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect - DOC: mention that req_tot is for both frontends and backends - BUG/MEDIUM: variables: some variable name can hide another ones - MINOR: lua: Allow argument for actions - BUILD: rearrange target files by build time - CLEANUP: hlua: just indent functions - MINOR: lua: give HAProxy variable access to the applets - BUG/MINOR: stats: fix be/sessions/max output in html stats - MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id - DOC: lua: Documentation about some entry missing - DOC: lua: Add documentation about variable manipulation from applet - MINOR: Do not forward the header "Expect: 100-continue" when the option http-buffer-request is set - DOC: Add undocumented argument of the trace filter - DOC: Fix some typo in SPOE documentation - MINOR: cli: Remove useless call to bi_putchk - BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is full - MINOR: applet: Count number of (active) applets - MINOR: task: Rename run_queue and run_queue_cur counters - BUG/MEDIUM: stream: Save unprocessed events for a stream - BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled - BUILD/MEDIUM: Fixing the build using LibreSSL - BUG/MEDIUM: lua: In some case, the return of sample-fetches is ignored (2) - SCRIPTS: git-show-backports: fix a harmless typo - SCRIPTS: git-show-backports: add -H to use the hash of the commit message - BUG/MINOR: stream-int: automatically release SI_FL_WAIT_DATA on SHUTW_NOW - CLEANUP: applet/lua: create a dedicated ->fcn entry in hlua_cli context - CLEANUP: applet/table: add an "action" entry in ->table context - CLEANUP: applet: remove the now unused appctx->private field - DOC: lua: documentation about time parser functions - DOC: lua: improve links - DOC: lua: section declared twice - MEDIUM: cli: 'show cli sockets' list the CLI sockets - BUG/MINOR: cli: "show cli sockets" wouldn't list all processes - BUG/MINOR: cli: "show cli sockets" would always report process 64 - CLEANUP: lua: rename one of the lua appctx union - BUG/MINOR: lua/cli: bad error message - MEDIUM: lua: use memory pool for hlua struct in applets - MINOR: lua/signals: Remove Lua part from signals. - DOC: cli: show cli sockets - MINOR: cli: automatically enable a CLI I/O handler when there's no parser - CLEANUP: memory: remove the now unused cli_parse_show_pools() function - CLEANUP: applet: group all CLI contexts together - CLEANUP: stats: move a misplaced stats context initialization - MINOR: cli: add two general purpose pointers and integers in the CLI struct - MINOR: appctx/cli: remove the cli_socket entry from the appctx union - MINOR: appctx/cli: remove the env entry from the appctx union - MINOR: appctx/cli: remove the "be" entry from the appctx union - MINOR: appctx/cli: remove the "dns" entry from the appctx union - MINOR: appctx/cli: remove the "server_state" entry from the appctx union - MINOR: appctx/cli: remove the "tlskeys" entry from the appctx union - CONTRIB: tcploop: add limits.h to fix build issue with some compilers - MINOR/DOC: lua: just precise one thing - DOC: fix small typo in fe_id (backend instead of frontend) - BUG/MINOR: Fix the sending function in Lua's cosocket - BUG/MINOR: lua: memory leak executing tasks - BUG/MINOR: lua: bad return code - BUG/MINOR: lua: memleak when Lua/cli fails - MEDIUM: lua: remove Lua struct from session, and allocate it with memory pools - CLEANUP: haproxy: statify unexported functions - MINOR: haproxy: add a registration for build options - CLEANUP: wurfl: use the build options list to report it - CLEANUP: 51d: use the build options list to report it - CLEANUP: da: use the build options list to report it - CLEANUP: namespaces: use the build options list to report it - CLEANUP: tcp: use the build options list to report transparent modes - CLEANUP: lua: use the build options list to report it - CLEANUP: regex: use the build options list to report the regex type - CLEANUP: ssl: use the build options list to report the SSL details - CLEANUP: compression: use the build options list to report the algos - CLEANUP: auth: use the build options list to report its support - MINOR: haproxy: add a registration for post-check functions - CLEANUP: checks: make use of the post-init registration to start checks - CLEANUP: filters: use the function registration to initialize all proxies - CLEANUP: wurfl: make use of the late init registration - CLEANUP: 51d: make use of the late init registration - CLEANUP: da: make use of the late init registration code - MINOR: haproxy: add a registration for post-deinit functions - CLEANUP: wurfl: register the deinit function via the dedicated list - CLEANUP: 51d: register the deinitialization function - CLEANUP: da: register the deinitialization function - CLEANUP: wurfl: move global settings out of the global section - CLEANUP: 51d: move global settings out of the global section - CLEANUP: da: move global settings out of the global section - MINOR: cfgparse: add two new functions to check arguments count - MINOR: cfgparse: move parsing of "ca-base" and "crt-base" to ssl_sock - MEDIUM: cfgparse: move all tune.ssl.* keywords to ssl_sock - MEDIUM: cfgparse: move maxsslconn parsing to ssl_sock - MINOR: cfgparse: move parsing of ssl-default-{bind,server}-ciphers to ssl_sock - MEDIUM: cfgparse: move ssl-dh-param-file parsing to ssl_sock - MEDIUM: compression: move the zlib-specific stuff from global.h to compression.c - BUG/MEDIUM: ssl: properly reset the reused_sess during a forced handshake - BUG/MEDIUM: ssl: avoid double free when releasing bind_confs - BUG/MINOR: stats: fix be/sessions/current out in typed stats - MINOR: tcp-rules: check that the listener exists before updating its counters - MEDIUM: spoe: don't create a dummy listener for outgoing connections - MINOR: listener: move the transport layer pointer to the bind_conf - MEDIUM: move listener->frontend to bind_conf->frontend - MEDIUM: ssl: remote the proxy argument from most functions - MINOR: connection: add a new prepare_bind_conf() entry to xprt_ops - MEDIUM: ssl_sock: implement ssl_sock_prepare_bind_conf() - MINOR: connection: add a new destroy_bind_conf() entry to xprt_ops - MINOR: ssl_sock: implement ssl_sock_destroy_bind_conf() - MINOR: server: move the use_ssl field out of the ifdef USE_OPENSSL - MINOR: connection: add a minimal transport layer registration system - CLEANUP: connection: remove all direct references to raw_sock and ssl_sock - CLEANUP: connection: unexport raw_sock and ssl_sock - MINOR: connection: add new prepare_srv()/destroy_srv() entries to xprt_ops - MINOR: ssl_sock: implement and use prepare_srv()/destroy_srv() - CLEANUP: ssl: move tlskeys_finalize_config() to a post_check callback - CLEANUP: ssl: move most ssl-specific global settings to ssl_sock.c - BUG/MINOR: backend: nbsrv() should return 0 if backend is disabled - BUG/MEDIUM: ssl: for a handshake when server-side SNI changes - BUG/MINOR: systemd: potential zombie processes - DOC: Add timings events schemas - BUILD: lua: build failed on FreeBSD. - MINOR: samples: add xx-hash functions - MEDIUM: regex: pcre2 support - BUG/MINOR: option prefer-last-server must be ignored in some case - MINOR: stats: Support "select all" for backend actions - BUG/MINOR: sample-fetches/stick-tables: bad type for the sample fetches sc*_get_gpt0 - BUG/MAJOR: channel: Fix the definition order of channel analyzers - BUG/MINOR: http: report real parser state in error captures - BUILD: scripts: automatically update the branch in version.h when releasing - MINOR: tools: add a generic hexdump function for debugging - BUG/MAJOR: http: fix risk of getting invalid reports of bad requests - MINOR: http: custom status reason. - MINOR: connection: add sample fetch "fc_rcvd_proxy" - BUG/MINOR: config: emit a warning if http-reuse is enabled with incompatible options - BUG/MINOR: tools: fix off-by-one in port size check - BUG/MEDIUM: server: consider AF_UNSPEC as a valid address family - MEDIUM: server: split the address and the port into two different fields - MINOR: tools: make str2sa_range() return the port in a separate argument - MINOR: server: take the destination port from the port field, not the addr - MEDIUM: server: disable protocol validations when the server doesn't resolve - BUG/MEDIUM: tools: do not force an unresolved address to AF_INET:0.0.0.0 - BUG/MINOR: ssl: EVP_PKEY must be freed after X509_get_pubkey usage - BUG/MINOR: ssl: assert on SSL_set_shutdown with BoringSSL - MINOR: Use "500 Internal Server Error" for 500 error/status code message. - MINOR: proto_http.c 502 error txt typo. - DOC: add deprecation notice to "block" - MINOR: compression: fix -vv output without zlib/slz - BUG/MINOR: Reset errno variable before calling strtol(3) - MINOR: ssl: don't show prefer-server-ciphers output - OPTIM/MINOR: config: Optimize fullconn automatic computation loading configuration - BUG/MINOR: stream: Fix how backend-specific analyzers are set on a stream - MAJOR: ssl: bind configuration per certificat - MINOR: ssl: add curve suite for ECDHE negotiation - MINOR: checks: Add agent-addr config directive - MINOR: cli: Add possiblity to change agent config via CLI/socket - MINOR: doc: Add docs for agent-addr configuration variable - MINOR: doc: Add docs for agent-addr and agent-send CLI commands - BUILD: ssl: fix to build (again) with boringssl - BUILD: ssl: fix build on OpenSSL 1.0.0 - BUILD: ssl: silence a warning reported for ERR_remove_state() - BUILD: ssl: eliminate warning with OpenSSL 1.1.0 regarding RAND_pseudo_bytes() - BUILD: ssl: kill a build warning introduced by BoringSSL compatibility - BUG/MEDIUM: tcp: don't poll for write when connect() succeeds - BUG/MINOR: unix: fix connect's polling in case no data are scheduled - MINOR: server: extend the flags to 32 bits - BUG/MINOR: lua: Map.end are not reliable because "end" is a reserved keyword - MINOR: dns: give ability to dns_init_resolvers() to close a socket when requested - BUG/MAJOR: dns: restart sockets after fork() - MINOR: chunks: implement a simple dynamic allocator for trash buffers - BUG/MEDIUM: http: prevent redirect from overwriting a buffer - BUG/MEDIUM: filters: Do not truncate HTTP response when body length is undefined - BUG/MEDIUM: http: Prevent replace-header from overwriting a buffer - BUG/MINOR: http: Return an error when a replace-header rule failed on the response - BUG/MINOR: sendmail: The return of vsnprintf is not cleanly tested - BUG/MAJOR: ssl: fix a regression in ssl_sock_shutw() - BUG/MAJOR: lua segmentation fault when the request is like 'GET ?arg=val HTTP/1.1' - BUG/MEDIUM: config: reject anything but "if" or "unless" after a use-backend rule - MINOR: http: don't close when redirect location doesn't start with "/" - MEDIUM: boringssl: support native multi-cert selection without bundling - BUG/MEDIUM: ssl: fix verify/ca-file per certificate - BUG/MEDIUM: ssl: switchctx should not return SSL_TLSEXT_ERR_ALERT_WARNING - MINOR: ssl: removes SSL_CTX_set_ssl_version call and cleanup CTX creation. - BUILD: ssl: fix build with -DOPENSSL_NO_DH - MEDIUM: ssl: add new sample-fetch which captures the cipherlist - MEDIUM: ssl: remove ssl-options from crt-list - BUG/MEDIUM: ssl: in bind line, ssl-options after 'crt' are ignored. - BUG/MINOR: ssl: fix cipherlist captures with sustainable SSL calls - MINOR: ssl: improved cipherlist captures - BUG/MINOR: spoe: Fix soft stop handler using a specific id for spoe filters - BUG/MINOR: spoe: Fix parsing of arguments in spoe-message section - MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents - MINOR: spoe: Add support for pipelining/async capabilities in the SPOA example - MINOR: spoe: Remove SPOE details from the appctx structure - MINOR: spoe: Add status code in error variable instead of hardcoded value - MINOR: spoe: Send a log message when an error occurred during event processing - MINOR: spoe: Check the scope of sample fetches used in SPOE messages - MEDIUM: spoe: Be sure to wakeup the good entity waiting for a buffer - MINOR: spoe: Use the min of all known max_frame_size to encode messages - MAJOR: spoe: Add support of payload fragmentation in NOTIFY frames - MINOR: spoe: Add support for fragmentation capability in the SPOA example - MAJOR: spoe: refactor the filter to clean up the code - MINOR: spoe: Handle NOTIFY frames cancellation using ABORT bit in ACK frames - REORG: spoe: Move struct and enum definitions in dedicated header file - REORG: spoe: Move low-level encoding/decoding functions in dedicated header file - MINOR: spoe: Improve implementation of the payload fragmentation - MINOR: spoe: Add support of negation for options in SPOE configuration file - MINOR: spoe: Add "pipelining" and "async" options in spoe-agent section - MINOR: spoe: Rely on alertif_too_many_arg during configuration parsing - MINOR: spoe: Add "send-frag-payload" option in spoe-agent section - MINOR: spoe: Add "max-frame-size" statement in spoe-agent section - DOC: spoe: Update SPOE documentation to reflect recent changes - MINOR: config: warn when some HTTP rules are used in a TCP proxy - BUG/MEDIUM: ssl: Clear OpenSSL error stack after trying to parse OCSP file - BUG/MEDIUM: cli: Prevent double free in CLI ACL lookup - BUG/MINOR: Fix "get map " CLI command - MINOR: Add nbsrv sample converter - CLEANUP: Replace repeated code to count usable servers with be_usable_srv() - MINOR: Add hostname sample fetch - CLEANUP: Remove comment that's no longer valid - MEDIUM: http_error_message: txn->status / http_get_status_idx. - MINOR: http-request tarpit deny_status. - CLEANUP: http: make http_server_error() not set the status anymore - MEDIUM: stats: Add JSON output option to show (info|stat) - MEDIUM: stats: Add show json schema - BUG/MAJOR: connection: update CO_FL_CONNECTED before calling the data layer - MINOR: server: Add dynamic session cookies. - MINOR: cli: Let configure the dynamic cookies from the cli. - BUG/MINOR: checks: attempt clean shutw for SSL check - CONTRIB: tcploop: make it build on FreeBSD - CONTRIB: tcploop: fix time format to silence build warnings - CONTRIB: tcploop: report action 'K' (kill) in usage message - CONTRIB: tcploop: fix connect's address length - CONTRIB: tcploop: use the trash instead of NULL for recv() - BUG/MEDIUM: listener: do not try to rebind another process' socket - BUG/MEDIUM server: Fix crash when dynamic is defined, but not key is provided. - CLEANUP: config: Typo in comment. - BUG/MEDIUM: filters: Fix channels synchronization in flt_end_analyze - TESTS: add a test configuration to stress handshake combinations - BUG/MAJOR: stream-int: do not depend on connection flags to detect connection - BUG/MEDIUM: connection: ensure to always report the end of handshakes - MEDIUM: connection: don't test for CO_FL_WAKE_DATA - CLEANUP: connection: completely remove CO_FL_WAKE_DATA - BUG: payload: fix payload not retrieving arbitrary lengths - BUILD: ssl: simplify SSL_CTX_set_ecdh_auto compatibility - BUILD: ssl: fix OPENSSL_NO_SSL_TRACE for boringssl and libressl - BUG/MAJOR: http: fix typo in http_apply_redirect_rule - MINOR: doc: 2.4. Examples should be 2.5. Examples - BUG/MEDIUM: stream: fix client-fin/server-fin handling - MINOR: fd: add a new flag HAP_POLL_F_RDHUP to struct poller - BUG/MINOR: raw_sock: always perfom the last recv if RDHUP is not available - OPTIM: poll: enable support for POLLRDHUP - MINOR: kqueue: exclusively rely on the kqueue returned status - MEDIUM: kqueue: take care of EV_EOF to improve polling status accuracy - MEDIUM: kqueue: only set FD_POLL_IN when there are pending data - DOC/MINOR: Fix typos in proxy protocol doc - DOC: Protocol doc: add checksum, TLV type ranges - DOC: Protocol doc: add SSL TLVs, rename CHECKSUM - DOC: Protocol doc: add noop TLV - MEDIUM: global: add a 'hard-stop-after' option to cap the soft-stop time - MINOR: dns: improve DNS response parsing to use as many available records as possible - BUG/MINOR: cfgparse: loop in tracked servers lists not detected by check_config_validity(). - MINOR: server: irrelevant error message with 'default-server' config file keyword. - MINOR: server: Make 'default-server' support 'backup' keyword. - MINOR: server: Make 'default-server' support 'check-send-proxy' keyword. - CLEANUP: server: code alignement. - MINOR: server: Make 'default-server' support 'non-stick' keyword. - MINOR: server: Make 'default-server' support 'send-proxy' and 'send-proxy-v2 keywords. - MINOR: server: Make 'default-server' support 'check-ssl' keyword. - MINOR: server: Make 'default-server' support 'force-sslv3' and 'force-tlsv1[0-2]' keywords. - CLEANUP: server: code alignement. - MINOR: server: Make 'default-server' support 'no-ssl*' and 'no-tlsv*' keywords. - MINOR: server: Make 'default-server' support 'ssl' keyword. - MINOR: server: Make 'default-server' support 'send-proxy-v2-ssl*' keywords. - CLEANUP: server: code alignement. - MINOR: server: Make 'default-server' support 'verify' keyword. - MINOR: server: Make 'default-server' support 'verifyhost' setting. - MINOR: server: Make 'default-server' support 'check' keyword. - MINOR: server: Make 'default-server' support 'track' setting. - MINOR: server: Make 'default-server' support 'ca-file', 'crl-file' and 'crt' settings. - MINOR: server: Make 'default-server' support 'redir' keyword. - MINOR: server: Make 'default-server' support 'observe' keyword. - MINOR: server: Make 'default-server' support 'cookie' keyword. - MINOR: server: Make 'default-server' support 'ciphers' keyword. - MINOR: server: Make 'default-server' support 'tcp-ut' keyword. - MINOR: server: Make 'default-server' support 'namespace' keyword. - MINOR: server: Make 'default-server' support 'source' keyword. - MINOR: server: Make 'default-server' support 'sni' keyword. - MINOR: server: Make 'default-server' support 'addr' keyword. - MINOR: server: Make 'default-server' support 'disabled' keyword. - MINOR: server: Add 'no-agent-check' server keyword. - DOC: server: Add docs for "server" and "default-server" new "no-*" and other settings. - MINOR: doc: fix use-server example (imap vs mail) - BUG/MEDIUM: tcp: don't require privileges to bind to device - BUILD: make the release script use shortlog for the final changelog - BUILD: scripts: fix typo in announce-release error message - CLEANUP: time: curr_sec_ms doesn't need to be exported - BUG/MEDIUM: server: Wrong server default CRT filenames initialization. - BUG/MEDIUM: peers: fix buffer overflow control in intdecode. - BUG/MEDIUM: buffers: Fix how input/output data are injected into buffers - BUG/MINOR: http: Fix conditions to clean up a txn and to handle the next request - CLEANUP: http: Remove channel_congested function - CLEANUP: buffers: Remove buffer_bounce_realign function - CLEANUP: buffers: Remove buffer_contig_area and buffer_work_area functions - MINOR: http: remove useless check on HTTP_MSGF_XFER_LEN for the request - MINOR: http: Add debug messages when HTTP body analyzers are called - BUG/MEDIUM: http: Fix blocked HTTP/1.0 responses when compression is enabled - BUG/MINOR: filters: Don't force the stream's wakeup when we wait in flt_end_analyze - DOC: fix parenthesis and add missing "Example" tags - DOC: update the contributing file - DOC: log-format/tcplog/httplog update - MINOR: config parsing: add warning when log-format/tcplog/httplog is overriden in "defaults" sections commit ea5b0e6fb7fe8763a98c90d768aca633eb49e0d6 Author: Guillaume de Lafond Date: Fri Mar 31 19:54:09 2017 +0200 MINOR: config parsing: add warning when log-format/tcplog/httplog is overriden in "defaults" sections Add a warning when "log-format" or "tcplog" or "httplog" is overriden in "defaults" sections. commit 29f4560b905ef6f4f2d8e48477dcff5168d0b2e7 Author: Guillaume de Lafond Date: Fri Mar 31 19:52:15 2017 +0200 DOC: log-format/tcplog/httplog update "log-format"/"tcplog"/"httplog" overrides any previous "log-format"/"tcplog"/"httplog" config variables. commit 138544f4b4773d3f77b76ad38091af30d62316a9 Author: Willy Tarreau Date: Fri Mar 31 16:24:44 2017 +0200 DOC: update the contributing file Bring more details, examples of good/bad messages, and avoid confusion between commit message and message subject. commit 676f6224ac3ea237c94c528b6883838a0a438f39 Author: Jarno Huuskonen Date: Thu Mar 30 09:19:45 2017 +0300 DOC: fix parenthesis and add missing "Example" tags - urlp_val had unbalanced parenthesis / square brackets - src_clr_gpc0,src_inc_gpc0,sc2_clr_gpc0,sc2_inc_gpc0,ssl_c_sha1 had examples not tagged as such. commit 2b553de5951b1a60e0c01775a2e7ef84b6472cb3 Author: Christopher Faulet Date: Thu Mar 30 11:13:22 2017 +0200 BUG/MINOR: filters: Don't force the stream's wakeup when we wait in flt_end_analyze In flt_end_analyze, we wait that the anlayze is finished for both the request and the response. In this case, because of a task_wakeup, some streams can consume too much CPU to do nothing. So now, this is the filter's responsibility to know if this wakeup is needed. This fix should be backported in 1.7. commit 69744d92a308f22936f4854447a85ed4d2b09dec Author: Christopher Faulet Date: Thu Mar 30 10:54:35 2017 +0200 BUG/MEDIUM: http: Fix blocked HTTP/1.0 responses when compression is enabled When the compression filter is enabled, if a HTTP/1.0 response is received (no content-length and no transfer-encoding), no data are forwarded to the client because of a bug and the transaction is blocked indefinitly. The bug comes from the fact we need to synchronize the end of the request and the response because of the compression filter. This inhibits the infinite forwarding of data. But for these responses, the compression is not activated. So the response body is not analyzed. This leads to a deadlock. The solution is to enable the analyze of the response body in all cases and handle this one to enable the infinite forwarding. All other cases should already by handled. This fix should be backported to 1.7. commit 814d2703607fd994505ecd8e2e6480e83e20162e Author: Christopher Faulet Date: Thu Mar 30 11:33:44 2017 +0200 MINOR: http: Add debug messages when HTTP body analyzers are called Only DPRINTF() for developers. commit be821b9f40698ab937e7e58875db30ac80399ff3 Author: Christopher Faulet Date: Thu Mar 30 11:21:53 2017 +0200 MINOR: http: remove useless check on HTTP_MSGF_XFER_LEN for the request The flag HTTP_MSGF_XFER_LEN is always set for an HTTP request because we always now the body length. So there is no need to do check on it. commit a545569f1e5be49587b88cf78a2e2db6ca760843 Author: Christopher Faulet Date: Wed Mar 29 10:49:49 2017 +0200 CLEANUP: buffers: Remove buffer_contig_area and buffer_work_area functions Not used anymore since last commit. commit aaf4a325ca5339c09e7c1a08b857162fc1817be8 Author: Christopher Faulet Date: Tue Mar 28 11:53:34 2017 +0200 CLEANUP: buffers: Remove buffer_bounce_realign function Not used anymore since last commit. commit 533182f1c82645e63a2b45d966ff1b0b281bf68f Author: Christopher Faulet Date: Tue Mar 28 11:52:37 2017 +0200 CLEANUP: http: Remove channel_congested function Not used anymore since last commit. commit c0c672a2abd599cf01b4e940675ecf6601265864 Author: Christopher Faulet Date: Tue Mar 28 11:51:33 2017 +0200 BUG/MINOR: http: Fix conditions to clean up a txn and to handle the next request To finish a HTTP transaction and to start the new one, we check, among other things, that there is enough space in the reponse buffer to eventually inject a message during the parsing of the next request. Because these messages can reach the maximum buffers size, it is mandatory to have an empty response buffer. Remaining input data are trimmed during the txn cleanup (in http_reset_txn), so we just need to check that the output data were flushed. The current implementation depends on channel_congested, which does check the reserved area is available. That's not of course good enough. There are other tests on the reponse buffer is http_wait_for_request. But conditions to move on are almost the same. So, we can imagine some scenarii where some output data remaining in the reponse buffer during the request parsing prevent any messages injection. To fix this bug, we just wait that output data were flushed before cleaning up the HTTP txn (ie. s->res.buf->o == 0). In addition, in http_reset_txn we realign the response buffer (note the buffer is empty at this step). Thanks to this changes, there is no more need to set CF_EXPECT_MORE on the response channel in http_end_txn_clean_session. And more important, there is no more need to check the response buffer state in http_wait_for_request. This remove a workaround on response analysers to handle HTTP pipelining. This patch can be backported in 1.7, 1.6 and 1.5. commit 637f8f2ca710c0f11d52fc91f00285fe6eee0e08 Author: Christopher Faulet Date: Wed Mar 29 11:58:28 2017 +0200 BUG/MEDIUM: buffers: Fix how input/output data are injected into buffers The function buffer_contig_space is buggy and could lead to pernicious bugs (never hitted until now, AFAIK). This function should return the number of bytes that can be written into the buffer at once (without wrapping). First, this function is used to inject input data (bi_putblk) and to inject output data (bo_putblk and bo_inject). But there is no context. So it cannot decide where contiguous space should placed. For input data, it should be after bi_end(buf) (ie, buf->p + buf->i modulo wrapping calculation). For output data, it should be after bo_end(buf) (ie, buf->p) and input data are assumed to not exist (else there is no space at all). Then, considering we need to inject input data, this function does not always returns the right value. And when we need to inject output data, we must be sure to have no input data at all (buf->i == 0), else the result can also be wrong (but this is the caller responsibility, so everything should be fine here). The buffer can be in 3 different states: 1) no wrapping <---- o ----><----- i -----> +------------+------------+-------------+------------+ | |oooooooooooo|iiiiiiiiiiiii|xxxxxxxxxxxx| +------------+------------+-------------+------------+ ^ p ^ ^ l r 2) input wrapping ...---> <---- o ----><-------- i -------... +-----+------------+------------+--------------------+ |iiiii|xxxxxxxxxxxx|oooooooooooo|iiiiiiiiiiiiiiiiiiii| +-----+------------+------------+--------------------+ ^ ^ ^ p l r 3) output wrapping ...------ o ------><----- i -----> <----... +------------------+-------------+------------+------+ |oooooooooooooooooo|iiiiiiiiiiiii|xxxxxxxxxxxx|oooooo| +------------------+-------------+------------+------+ ^ p ^ ^ l r buffer_contig_space returns (l - r). The cases 1 and 3 are correctly handled. But for the second case, r is wrong. It points on the buffer's end (buf->data + buf->size). It should be bo_end(buf) (ie, buf->p - buf->o). To fix the bug, the function has been splitted. Now, bi_contig_space and bo_contig_space should be used to know the contiguous space available to insert, respectively, input data and output data. For bo_contig_space, input data are assumed to not exist. And the right version is used, depending what we want to do. In addition, to clarify the buffer's API, buffer_realign does not return value anymore. So it has the same API than buffer_slow_realign. This patch can be backported in 1.7, 1.6 and 1.5. commit 18928af3a3e3692cc2f086f8f271538436fcbe2e Author: Emeric Brun Date: Wed Mar 29 16:32:53 2017 +0200 BUG/MEDIUM: peers: fix buffer overflow control in intdecode. A buffer overflow could happen if an integer is badly encoded in the data part of a msg received from a peer. It should not happen with authenticated peers (the handshake do not use this function). This patch makes the code of the 'intdecode' function more robust. It also adds some comments about the intencode function. This bug affects versions >= 1.6. commit acd4827ecabebccc9e57fe091de23a1ea5ec95b7 Author: Frédéric Lécaille Date: Wed Mar 29 14:58:09 2017 +0200 BUG/MEDIUM: server: Wrong server default CRT filenames initialization. This patch fixes a bug which came with 5e57643 commit where server default CRT filenames were initialized to the same value as server default CRL filenames. commit 351b3a17809be828b3841283405acce23aa923ef Author: Willy Tarreau Date: Wed Mar 29 15:24:33 2017 +0200 CLEANUP: time: curr_sec_ms doesn't need to be exported It's not used anywhere outside of tv_update_date(). commit 827385fb7c2b537153123ca2ea14b51b1b9fc770 Author: Willy Tarreau Date: Mon Mar 27 19:36:45 2017 +0200 BUILD: scripts: fix typo in announce-release error message It used to say the tag already existed instead of the opposite. commit 6a375ef73064177ef963eea64ff0a55156ace4af Author: Willy Tarreau Date: Mon Mar 27 19:32:24 2017 +0200 BUILD: make the release script use shortlog for the final changelog It used to reuse the same command producting the list for the changelog, requiring to run shortlog manually. commit 19060a302033b70fbe29eb825617b03ea5e9e71d Author: Willy Tarreau Date: Mon Mar 27 16:22:59 2017 +0200 BUG/MEDIUM: tcp: don't require privileges to bind to device Ankit Malp reported a bug that we've had since binding to devices was implemented. Haproxy wrongly checks that the process stays privileged after startup when a binding to a device is specified via the bind keyword "interface". This is wrong, because after startup we're not binding any socket anymore, and during startup if there's a permission issue it will be immediately reported ("permission denied"). More importantly there's no way around it as the process exits on startup when facing such an option. This fix should be backported to 1.7, 1.6 and 1.5. commit 98a3e3f998e2803c368a9431a39cf957781f4657 Author: Lukas Tribus Date: Sun Mar 26 12:55:35 2017 +0000 MINOR: doc: fix use-server example (imap vs mail) Another minor doc issue in the use-server example, use-server refers to server "imap", but the server below is actually called "mail". Renames the server from "mail" to "imap". commit d237627d3b9bd36de5bd2abe405fe15d01948151 Author: Frédéric Lécaille Date: Tue Mar 21 18:52:12 2017 +0100 DOC: server: Add docs for "server" and "default-server" new "no-*" and other settings. New boolean settings have been added to disable others. Most of them have "no-" as prefix. "enabled" disables "disabled" setting, "no-agent-check" disables "agent-check", "no-backup" disables "backup", "no-check" disables "check", "no-check-ssl" disables "check-ssl", "no-force-sslv3" disables "force-sslv3", "no-force-tlsv10" disables "force-tlsv10", "no-force-tlsv11" disables "force-tlsv11", "no-force-tlsv12" disables "force-tlsv12, "no-send-proxy" disables "send-proxy", "no-send-proxy-v2" disables "send-proxy-v2", "no-send-proxy-v2-ssl" disables "send-proxy-v2-ssl", "no-send-proxy-v2-ssl-cn" disables "send-proxy-v2-ssl-cn", "no-ssl" disables "ssl", "no-verifyhost" disables "verifyhost", "sslv3" disables "no-sslv3", "ssl-reuse" disables "no-ssl-reuse", "stick" disables "non-stick", "tlsv10" disables "no-tlsv10", "tlsv11" disables "no-tlsv11", "tlsv12" disables "no-tlsv12", "tls-tickets" disables "no-tls-tickets". Settings with arguments are now supported on "default-server" lines: "addr", "ca-file", "ciphers", "crl-file", "crt", "cookie", "namespace", "observe", "redir", "sni", "source", "tcp-ut" and "track". From now on, all server "settings" including the new ones above are supported by "default-server" except "id" which is only supported on "server" lines. commit 6e0843c0e09fd8737cb377b5387b9946292d7378 Author: Frédéric Lécaille Date: Tue Mar 21 16:39:15 2017 +0100 MINOR: server: Add 'no-agent-check' server keyword. This patch adds 'no-agent-check' setting supported both by 'default-server' and 'server' directives to disable an agent check for a specific server which would have 'agent-check' set as default value (inherited from 'default-server' 'agent-check' setting), or, on 'default-server' lines, to disable 'agent-check' setting as default value for any further 'server' declarations. For instance, provided this configuration: default-server agent-check server srv1 server srv2 no-agent-check server srv3 default-server no-agent-check server srv4 srv1 and srv3 would have an agent check enabled contrary to srv2 and srv4. We do not allocate anymore anything when parsing 'default-server' 'agent-check' setting. commit 2a0d061a6001d631a6d82d50ad7120ced2bd11d5 Author: Frédéric Lécaille Date: Tue Mar 21 11:53:54 2017 +0100 MINOR: server: Make 'default-server' support 'disabled' keyword. Before this patch, only 'server' directives could support 'disabled' setting. This patch makes also 'default-server' directives support this setting. It is used to disable a list of servers declared after a 'defaut-server' directive. 'enabled' new keyword has been added, both supported as 'default-server' and 'server' setting, to enable again a list of servers (so, declared after a 'default-server enabled' directive) or to explicitly enable a specific server declared after a 'default-server disabled' directive. For instance provided this configuration: default-server disabled server srv1... server srv2... server srv3... enabled server srv4... enabled srv1 and srv2 are disabled and srv3 and srv4 enabled. This is equivalent to this configuration: default-server disabled server srv1... server srv2... default-server enabled server srv3... server srv4... even if it would have been preferable/shorter to declare: server srv3... server srv4... default-server disabled server srv1... server srv2... as 'enabled' is the default server state. commit 6e5e0d8f9edcc963dda0ea5980ea58e908f5dfb1 Author: Frédéric Lécaille Date: Mon Mar 20 16:30:18 2017 +0100 MINOR: server: Make 'default-server' support 'addr' keyword. This patch makes 'default-server' support 'addr' setting. The code which was responsible of parsing 'server' 'addr' setting has moved from parse_server() to implement a new parser callable both as 'default-server' and 'server' 'addr' setting parser. Should not break anything. commit 9a146de934e80f27d2bb54c62c8de472987dcb63 Author: Frédéric Lécaille Date: Mon Mar 20 14:54:41 2017 +0100 MINOR: server: Make 'default-server' support 'sni' keyword. This patch makes 'default-server' directives support 'sni' settings. A field 'sni_expr' has been added to 'struct server' to temporary stores SNI expressions as strings during both 'default-server' and 'server' lines parsing. So, to duplicate SNI expressions from 'default-server' 'sni' setting for new 'server' instances we only have to "strdup" these strings as this is often done for most of the 'server' settings. Then, sample expressions are computed calling sample_parse_expr() (only for 'server' instances). A new function has been added to produce the same error output as before in case of any error during 'sni' settings parsing (display_parser_err()). Should not break anything. commit dba9707713eb49a39b218f331c252fb09494c566 Author: Frédéric Lécaille Date: Fri Mar 17 15:33:50 2017 +0100 MINOR: server: Make 'default-server' support 'source' keyword. Before this patch, only 'server' directives could support 'source' setting. This patch makes also 'default-server' directives support this setting. To do so, we had to extract the code responsible of parsing 'source' setting arguments from parse_server() function and make it callable both as 'default-server' and 'server' 'source' setting parser. So, the code is mostly the same as before except that before allocating anything for 'struct conn_src' members, we must free the memory previously allocated. Should not break anything. commit 22f41a2d23d8a2fb0a48f0f9c9d97438d9e9a905 Author: Frédéric Lécaille Date: Thu Mar 16 17:17:36 2017 +0100 MINOR: server: Make 'default-server' support 'namespace' keyword. Before this patch, 'namespace' setting was only supported by 'server' directive. This patch makes 'default-server' directive support this setting. commit 5c3cd97550e892b3552111de7adf4550b00bffce Author: Frédéric Lécaille Date: Wed Mar 15 16:36:09 2017 +0100 MINOR: server: Make 'default-server' support 'tcp-ut' keyword. This patch makes 'default-server' directive support 'tcp-ut' keyword. commit bcaf1d7397c7c27081821fb35153776eccf12ef2 Author: Frédéric Lécaille Date: Wed Mar 15 16:20:02 2017 +0100 MINOR: server: Make 'default-server' support 'ciphers' keyword. This patch makes 'default-server' directive support 'ciphers' setting. commit 9d1b95b5917fe0778f8e63fd3aeb8f7185f429d1 Author: Frédéric Lécaille Date: Wed Mar 15 09:13:33 2017 +0100 MINOR: server: Make 'default-server' support 'cookie' keyword. Before this patch, 'cookie' setting was only supported by 'server' directives. This patch makes 'default-server' directive also support 'cookie' setting. Should not break anything. commit 547356e48474f6117512c104fa67ef6cae0a0ba4 Author: Frédéric Lécaille Date: Wed Mar 15 08:55:39 2017 +0100 MINOR: server: Make 'default-server' support 'observe' keyword. Before this path, 'observe' setting was only supported by 'server' directives. This patch makes 'default-server' directives also support 'observe' setting. Should not break anything. commit 16186236dd14a9783e39aad93ed4cb90f1bc33f6 Author: Frédéric Lécaille Date: Tue Mar 14 16:42:49 2017 +0100 MINOR: server: Make 'default-server' support 'redir' keyword. Before this patch only 'server' directives could support 'redir' setting. This patch makes also 'default-server' directives support 'redir' setting. Should not break anything. commit 5e57643e092d71706d480e15ecfdcbd156801fcf Author: Frédéric Lécaille Date: Tue Mar 14 15:52:04 2017 +0100 MINOR: server: Make 'default-server' support 'ca-file', 'crl-file' and 'crt' settings. This patch makes 'default-server' directives support 'ca-file', 'crl-file' and 'crt' settings. commit 67e0e61316e5a7cd2ff53f6806a5203f3f4f0a64 Author: Frédéric Lécaille Date: Tue Mar 14 15:21:31 2017 +0100 MINOR: server: Make 'default-server' support 'track' setting. Before this patch only 'server' directives could support 'track' setting. This patch makes 'default-server' directives also support this setting. Should not break anything. commit 65aa356c0b945820c3728b87650021800382a181 Author: Frédéric Lécaille Date: Tue Mar 14 11:20:13 2017 +0100 MINOR: server: Make 'default-server' support 'check' keyword. Before this patch 'check' setting was only supported by 'server' directives. This patch makes also 'default-server' directives support this setting. A new 'no-check' keyword parser has been implemented to disable this setting both in 'default-server' and 'server' directives. Should not break anything. commit 273f321404210c6c49f295ee00249d8c4cae21ae Author: Frédéric Lécaille Date: Mon Mar 13 15:52:01 2017 +0100 MINOR: server: Make 'default-server' support 'verifyhost' setting. This patch makes 'default-server' directive support 'verifyhost' setting. Note: there was a little memory leak when several 'verifyhost' arguments were supplied on the same 'server' line. commit 7c8cd587c2e6873a5bde38a1ce0fc6e08e1c4330 Author: Frédéric Lécaille Date: Mon Mar 13 13:41:16 2017 +0100 MINOR: server: Make 'default-server' support 'verify' keyword. This patch makes 'default-server' directive support 'verify' keyword. commit 18388c910c6d308cb5f068860468df19146eb508 Author: Frédéric Lécaille Date: Mon Mar 13 13:10:59 2017 +0100 CLEANUP: server: code alignement. Code alignement again. commit e892c4c6702bf2e5ab4b0683c63343e42c8b818c Author: Frédéric Lécaille Date: Mon Mar 13 12:08:01 2017 +0100 MINOR: server: Make 'default-server' support 'send-proxy-v2-ssl*' keywords. This patch makes 'default-server' directive support 'send-proxy-v2-ssl' (resp. 'send-proxy-v2-ssl-cn') setting. A new keyword 'no-send-proxy-v2-ssl' (resp. 'no-send-proxy-v2-ssl-cn') has been added to disable 'send-proxy-v2-ssl' (resp. 'send-proxy-v2-ssl-cn') setting both in 'server' and 'default-server' directives. commit e381d7699eae5ec77c4ee9f9f2f77bd5f8c15fa7 Author: Frédéric Lécaille Date: Mon Mar 13 11:54:17 2017 +0100 MINOR: server: Make 'default-server' support 'ssl' keyword. This patch makes 'default-server' directive support 'ssl' setting. A new keyword 'no-ssl' has been added to disable this setting both in 'server' and 'default-server' directives. commit 2cfcdbe58d515129a96de25e0c272d73aa68a682 Author: Frédéric Lécaille Date: Mon Mar 13 11:32:20 2017 +0100 MINOR: server: Make 'default-server' support 'no-ssl*' and 'no-tlsv*' keywords. This patch makes 'default-server' directive support 'no-sslv3' (resp. 'no-ssl-reuse', 'no-tlsv10', 'no-tlsv11', 'no-tlsv12', and 'no-tls-tickets') setting. New keywords 'sslv3' (resp. 'ssl-reuse', 'tlsv10', 'tlsv11', 'tlsv12', and 'tls-no-tickets') have been added to disable these settings both in 'server' and 'default-server' directives. commit ec16f0300a99142accbcc03d92c8ea3712c8c655 Author: Frédéric Lécaille Date: Mon Mar 13 11:02:01 2017 +0100 CLEANUP: server: code alignement. Code alignement. commit 9698092ab6fb94b29f68ba8d727cce95a64ef855 Author: Frédéric Lécaille Date: Mon Mar 13 10:54:52 2017 +0100 MINOR: server: Make 'default-server' support 'force-sslv3' and 'force-tlsv1[0-2]' keywords. This patch makes 'default-server' directive support 'force-sslv3' and 'force-tlsv1[0-2]' settings. New keywords 'no-force-sslv3' (resp. 'no-tlsv1[0-2]') have been added to disable 'force-sslv3' (resp. 'force-tlsv1[0-2]') setting both in 'server' and 'default-server' directives. commit 340ae606af030b031725cc2556987890a64b4c99 Author: Frédéric Lécaille Date: Mon Mar 13 10:38:04 2017 +0100 MINOR: server: Make 'default-server' support 'check-ssl' keyword. This patch makes 'default-server' directive support 'check-ssl' setting to enable SSL for health checks. A new keyword 'no-check-ssl' has been added to disable this setting both in 'server' and 'default-server' directives. commit 31045e4c106867eba0bc0b6f747b1206a5c9f6bb Author: Frédéric Lécaille Date: Fri Mar 10 16:40:00 2017 +0100 MINOR: server: Make 'default-server' support 'send-proxy' and 'send-proxy-v2 keywords. This patch makes 'default-server' directive support 'send-proxy' (resp. 'send-proxy-v2') setting. A new keyword 'no-send-proxy' (resp. 'no-send-proxy-v2') has been added to disable 'send-proxy' (resp. 'send-proxy-v2') setting both in 'server' and 'default-server' directives. commit f9bc1d6a13623ac9944983df2233735f9807d940 Author: Frédéric Lécaille Date: Fri Mar 10 15:50:49 2017 +0100 MINOR: server: Make 'default-server' support 'non-stick' keyword. This patch makes 'default-server' directive support 'non-stick' setting. A new keyword 'stick' has been added so that to disable 'non-stick' setting both in 'server' and 'default-server' directives. commit 1502cfd1a3f087b1c8fc77aa07032bb9d3a2774c Author: Frédéric Lécaille Date: Fri Mar 10 15:36:14 2017 +0100 CLEANUP: server: code alignement. Code alignement. commit 25df89066bd59bcf6dd48b0ef3dd0489425523e5 Author: Frédéric Lécaille Date: Fri Mar 10 14:04:31 2017 +0100 MINOR: server: Make 'default-server' support 'check-send-proxy' keyword. This patch makes 'default-server' directive support 'check-send-proxy' setting. A new keyword 'no-check-send-proxy' has been added so that to disable 'check-send-proxy' setting both in 'server' and 'default-server' directives. commit f5bf903be63064c9fac767cbce95b98b07fad076 Author: Frédéric Lécaille Date: Fri Mar 10 11:51:05 2017 +0100 MINOR: server: Make 'default-server' support 'backup' keyword. At this time, only 'server' supported 'backup' keyword. This patch makes also 'default-server' directive support this keyword. A new keyword 'no-backup' has been added so that to disable 'backup' setting both in 'server' and 'default-server' directives. For instance, provided the following sequence of directives: default-server backup server srv1 server srv2 no-backup default-server no-backup server srv3 server srv4 backup srv1 and srv4 are declared as backup servers, srv2 and srv3 are declared as non-backup servers. commit 8065b6d4f21dc14212cd3c413e82be0ef43984fc Author: Frédéric Lécaille Date: Thu Mar 9 14:01:02 2017 +0100 MINOR: server: irrelevant error message with 'default-server' config file keyword. There is no reason to emit such an error message: "'default-server' expects and [:] as arguments." if less than two arguments are provided on 'default-server' lines. This is a 'server' specific error message. commit 2efc64944726cec78d87210cd5fba67b24d686b6 Author: Frédéric Lécaille Date: Tue Mar 14 14:32:17 2017 +0100 BUG/MINOR: cfgparse: loop in tracked servers lists not detected by check_config_validity(). There is a silly case where a loop is not detected in tracked servers lists: when a server tracks itself. Ex: server srv1 127.0.0.1:8000 track srv1 Well, this never happens and this does not prevent haproxy from working. But with this next following configuration: server srv1 127.0.0.1:8000 track srv2 server srv2 127.0.0.1:8000 track srv2 server srv3 127.0.0.1:8000 track srv2 the code in charge of detecting such loops never returns (without any error message). haproxy becomes stuck in an infinite loop because of this statement found in check_config_validity(): for (loop = srv->track; loop && loop != newsrv; loop = loop->track); Again, such a configuration is never accidentally used I guess. This latter example seems silly, but as several 'default-server' directives may be used in the same proxy section, and as 'default-server' settings are not resetted each a new 'default-server' line is created, it will match the following configuration, in the future, when 'track' setting will be supported by 'default-server': default-server track srv3 server srv1 127.0.0.1:8000 server srv2 127.0.0.1:8000 . . . default-server check server srv3 127.0.0.1:8000 (cherry picked from commit 6528fc93d3c065fdac841f24e55cfe9674a67414) commit fc725905443dffd08a335b2a71a4f370118797d5 Author: Baptiste Date: Mon Dec 26 23:21:08 2016 +0100 MINOR: dns: improve DNS response parsing to use as many available records as possible A "weakness" exist in the first implementation of the parsing of the DNS responses: HAProxy always choses the first IP available matching family preference, or as a failover, the first IP. It should be good enough, since most DNS servers do round robin on the records they send back to clients. That said, some servers does not do proper round robin, or we may be unlucky too and deliver the same IP to all the servers sharing the same hostname. Let's take the simple configuration below: backend bk srv s1 www:80 check resolvers R srv s2 www:80 check resolvers R The DNS server configured with 2 IPs for 'www'. If you're unlucky, then HAProxy may apply the same IP to both servers. Current patch improves this situation by weighting the decision algorithm to ensure we'll prefer use first an IP found in the response which is not already affected to any server. The new algorithm does not guarantee that the chosen IP is healthy, neither a fair distribution of IPs amongst the servers in the farm, etc... It only guarantees that if the DNS server returns many records for a hostname and that this hostname is being resolved by multiple servers in the same backend, then we'll use as many records as possible. If a server fails, HAProxy won't pick up an other record from the response. commit 203ec5a2b566bfd731538ad4ea21a849b79412ad Author: Cyril Bonté Date: Thu Mar 23 22:44:13 2017 +0100 MEDIUM: global: add a 'hard-stop-after' option to cap the soft-stop time When SIGUSR1 is received, haproxy enters in soft-stop and quits when no connection remains. It can happen that the instance remains alive for a long time, depending on timeouts and traffic. This option ensures that soft-stop won't run for too long. Example: global hard-stop-after 30s # Once in soft-stop, the instance will remain # alive for at most 30 seconds. commit 1a943c48fb7bca16cb0c1d24609b2a0168fa6932 Author: Andriy Palamarchuk Date: Thu Mar 23 16:30:24 2017 -0400 DOC: Protocol doc: add noop TLV Add definition of the PP2_TYPE_NOOP TLV which can be used for data padding and alignment. commit 01105ac3a88dc0707f976235e1bef199c6758196 Author: Andriy Palamarchuk Date: Tue Mar 14 18:59:09 2017 -0400 DOC: Protocol doc: add SSL TLVs, rename CHECKSUM Add SSL-related TLV types PP2_SUBTYPE_SSL_CIPHER, PP2_SUBTYPE_SSL_SIG_ALG and PP2_SUBTYPE_SSL_KEY_ALG. Rename PP2_TYPE_CHECKSUM to PP2_TYPE_CRC32C to make it easier to add checksums using other algorithms. Clarified encoding of the string fields. Renamed ASCII to US-ASCII as recommended by https://www.iana.org/assignments/character-sets/character-sets.xhtml. commit ceae85ba4a6032d30a8736d97acb76201f6b4715 Author: Andriy Palamarchuk Date: Tue Jan 24 13:48:27 2017 -0500 DOC: Protocol doc: add checksum, TLV type ranges Add the CRC32c checksum TLV PP2_TYPE_CHECKSUM. Reserve TLV type ranges CUSTOM, EXPERIMENT and FUTURE. Clarify that only UNSPEC protocol byte is mandatory to implement on the receiver. commit f1eae4ec38294020f28ef53ba7be2f9143feb179 Author: Andriy Palamarchuk Date: Tue Jan 24 13:34:08 2017 -0500 DOC/MINOR: Fix typos in proxy protocol doc commit fa0617660a9f288caade3c00382fbe3b3c2ffa6d Author: Willy Tarreau Date: Mon Mar 13 20:49:56 2017 +0100 MEDIUM: kqueue: only set FD_POLL_IN when there are pending data Let's avoid setting FD_POLL_IN when there's no pending data. It will save a useless recv() syscall on pure closes. commit 19c4ab97c10b75b69d02bf7ebb6c2c3331d7bd83 Author: Willy Tarreau Date: Mon Mar 13 20:36:48 2017 +0100 MEDIUM: kqueue: take care of EV_EOF to improve polling status accuracy kevent() always sets EV_EOF with EVFILT_READ to notify of a read shutdown and EV_EOF with EVFILT_WRITE to notify of a write error. Let's check this flag to properly update the FD's polled status (FD_POLL_HUP and FD_POLL_ERR respectively). It's worth noting that this one can be coupled with a regular read event to notify about a pending read followed by a shutdown, but for now we only use this to set the relevant flags (HUP and ERR). The poller now exhibits the flag HAP_POLL_F_RDHUP to indicate this new capability. An improvement may consist in not setting FD_POLL_IN when the "data" field is null since it normally only reflects the amount of pending data. commit dd437d9a4c4ff6aa2d4003ab121790ae08020784 Author: Willy Tarreau Date: Mon Mar 13 20:30:12 2017 +0100 MINOR: kqueue: exclusively rely on the kqueue returned status After commit e852545 ("MEDIUM: polling: centralize polled events processing") all pollers stopped to explicitly check the FD's polled status, except the kqueue poller which is constructed a bit differently. It doesn't seem possible to cause any issue such as missing an event however, but anyway it's better to definitely get rid of this since the event filter already provides the event direction. commit 3c8a89642d6d8b3f098daaff24edcb5aa23e58a0 Author: Willy Tarreau Date: Mon Mar 13 17:14:51 2017 +0100 OPTIM: poll: enable support for POLLRDHUP On Linux since 2.6.17 poll() supports POLLRDHUP to notify of an upcoming hangup after pending data. Making use of it allows us to avoid a useless recv() after short responses on keep-alive connections. Note that we automatically enable the feature once this flag has been met first in a poll() status. Till now it was only enabled on epoll. commit 68128710d00c871495317cfb84e177ef34584fc2 Author: Willy Tarreau Date: Mon Mar 13 12:04:34 2017 +0100 BUG/MINOR: raw_sock: always perfom the last recv if RDHUP is not available Curu Wong reported a case where haproxy used to send RST to a server after receiving its FIN. The problem is caused by the fact that being a server connection, its fd is marked with linger_risk=1, and that the poller didn't report POLLRDHUP, making haproxy unaware of a pending shutdown that came after the data, so it used to resort to nolinger for closing. However when pollers support RDHUP we're pretty certain whether or not a shutdown comes after the data and we don't need to perform that extra recv() call. Similarly when we're dealing with an inbound connection we don't care and don't want to perform this extra recv after a request for a very unlikely case, as in any case we'll have to deal with the client-facing TIME_WAIT socket. So this patch ensures that only when it's known that there's no risk with lingering data, as well as in cases where it's known that the poller would have detected a pending RDHUP, we perform the fd_done_recv() otherwise we continue, trying a subsequent recv() to try to detect a pending shutdown. This effectively results in an extra recv() call for keep-alive sockets connected to a server when POLLRDHUP isn't known to be supported, but it's the only way to know whether they're still alive or closed. This fix should be backported to 1.7, 1.6 and 1.5. It relies on the previous patch bringing support for the HAP_POLL_F_RDHUP flag. commit 5a767693b5c068b1142ddb540b9aeb26748fcdfc Author: Willy Tarreau Date: Mon Mar 13 11:38:28 2017 +0100 MINOR: fd: add a new flag HAP_POLL_F_RDHUP to struct poller We'll need to differenciate between pollers which can report hangup at the same time as read (POLL_RDHUP) from the other ones, because only these ones may benefit from the fd_done_recv() optimization. Epoll has had support for EPOLLRDHUP since Linux 2.6.17 and has always been used this way in haproxy, so now we only set the flag once we've observed it once in a response. It means that some initial requests may try to perform a second recv() call, but after the first closed connection it will be enough to know that the second call is not needed anymore. Later we may extend these flags to designate event-triggered pollers. commit e39683c4d4c527d1b561c3ba3983d26cc3e7f42d Author: Hongbo Long Date: Fri Mar 10 18:41:51 2017 +0100 BUG/MEDIUM: stream: fix client-fin/server-fin handling A tcp half connection can cause 100% CPU on expiration. First reproduced with this haproxy configuration : global tune.bufsize 10485760 defaults timeout server-fin 90s timeout client-fin 90s backend node2 mode tcp timeout server 900s timeout connect 10s server def 127.0.0.1:3333 frontend fe_api mode tcp timeout client 900s bind :1990 use_backend node2 Ie timeout server-fin shorter than timeout server, the backend server sends data, this package is left in the cache of haproxy, the backend server continue sending fin package, haproxy recv fin package. this time the session information is as follows: time the session information is as follows: 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 srv=def ts=08 age=1s calls=3 rq[f=848000h,i=0,an=00h,rx=14m58s,wx=,ax=] rp[f=8004c020h,i=0,an=00h,rx=,wx=14m58s,ax=] s0=[7,0h,fd=6,ex=] s1=[7,18h,fd=7,ex=] exp=14m58s rp has set the CF_SHUTR state, next, the client sends the fin package, session information is as follows: 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 srv=def ts=08 age=38s calls=4 rq[f=84a020h,i=0,an=00h,rx=,wx=,ax=] rp[f=8004c020h,i=0,an=00h,rx=1m11s,wx=14m21s,ax=] s0=[7,0h,fd=6,ex=] s1=[9,10h,fd=7,ex=] exp=1m11s After waiting 90s, session information is as follows: 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 srv=def ts=04 age=4m11s calls=718074391 rq[f=84a020h,i=0,an=00h,rx=,wx=,ax=] rp[f=8004c020h,i=0,an=00h,rx=?,wx=10m49s,ax=] s0=[7,0h,fd=6,ex=] s1=[9,10h,fd=7,ex=] exp=? run(nice=0) cpu information: 6899 root 20 0 112224 21408 4260 R 100.0 0.7 3:04.96 haproxy Buffering is set to ensure that there is data in the haproxy buffer, and haproxy can receive the fin package, set the CF_SHUTR flag, If the CF_SHUTR flag has been set, The following code does not clear the timeout message, causing cpu 100%: stream.c:process_stream: if (unlikely((res->flags & (CF_SHUTR|CF_READ_TIMEOUT)) == CF_READ_TIMEOUT)) { if (si_b->flags & SI_FL_NOHALF) si_b->flags |= SI_FL_NOLINGER; si_shutr(si_b); } If you have closed the read, set the read timeout does not make sense. With or without cf_shutr, read timeout is set: if (tick_isset(s->be->timeout.serverfin)) { res->rto = s->be->timeout.serverfin; res->rex = tick_add(now_ms, res->rto); } After discussion on the mailing list, setting half-closed timeouts the hard way here doesn't make sense. They should be set only at the moment the shutdown() is performed. It will also solve a special case which was already reported of some half-closed timeouts not working when the shutw() is performed directly at the stream-interface layer (no analyser involved). Since the stream interface layer cannot know the timeout values, we'll have to store them directly in the stream interface so that they are used upon shutw(). This patch does this, fixing the problem. An easier reproducer to validate the fix is to keep the huge buffer and shorten all timeouts, then call it under tcploop server and client, and wait 3 seconds to see haproxy run at 100% CPU : global tune.bufsize 10485760 listen px bind :1990 timeout client 90s timeout server 90s timeout connect 1s timeout server-fin 3s timeout client-fin 3s server def 127.0.0.1:3333 $ tcploop 3333 L W N20 A P100 F P10000 & $ tcploop 127.0.0.1:1990 C S10000000 F commit aa83a3144b8eaccfc1d630bc564e294c7c9ebf66 Author: Lukas Tribus Date: Tue Mar 21 09:25:09 2017 +0000 MINOR: doc: 2.4. Examples should be 2.5. Examples Guillaume Michaud reported against Cyril's haproxy-dconv project that the index for the Examples section should be 2.5 instead of 2.4. Should be backported to 1.7 and 1.6, so that the example section can be linked to: https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#2.5 commit 014e39c0b6ec5abb3fc4dabf4e80bb758b3548dd Author: Christopher Faulet Date: Fri Mar 10 13:52:30 2017 +0100 BUG/MAJOR: http: fix typo in http_apply_redirect_rule Because of this typo, AN_RES_FLT_END was never called when a redirect rule is applied on a keep-alive connection. In almost all cases, this bug has no effect. But, it leads to a memory leak if a redirect is done on a http-response rule when the HTTP compression is enabled. This patch should be backported in 1.7. commit 9490cedb4eee0d3c72d4cb6eb57520beabaabe83 Author: Emmanuel Hocdet Date: Mon Mar 20 11:39:57 2017 +0100 BUILD: ssl: fix OPENSSL_NO_SSL_TRACE for boringssl and libressl "sample-fetch which captures the cipherlist" patch introduce #define do deal with trace functions only available in openssl > 1.0.2. Add this #define to libressl and boringssl environment. Thanks to Piotr Kubaj for postponing and testing with libressl. commit a52bb15cc78ecf07f2c4be3f6e5aba417dafd098 Author: Emmanuel Hocdet Date: Mon Mar 20 11:11:49 2017 +0100 BUILD: ssl: simplify SSL_CTX_set_ecdh_auto compatibility SSL_CTX_set_ecdh_auto is declared (when present) with #define. A simple #ifdef avoid to list all cases of ssllibs. It's a placebo in new ssllibs. It's ok with openssl 1.0.1, 1.0.2, 1.1.0, libressl and boringssl. Thanks to Piotr Kubaj for postponing and testing with libressl. commit 00f55524e0b215e17d82a00c0faa58efc8eafd81 Author: Felipe Guerreiro Barbosa Ruiz Date: Thu Mar 16 17:01:41 2017 -0300 BUG: payload: fix payload not retrieving arbitrary lengths This fixes a regression introduced in d7bdcb874bcb, that removed the ability to use req.payload(0,0) to read the whole buffer content. The offending commit is present starting in version 1.6, so the patch should be backported to versions 1.6 and 1.7. commit de40d798decc8d6a24fe6b4609b6bdcb14d7ba54 Author: Willy Tarreau Date: Sat Mar 18 17:40:22 2017 +0100 CLEANUP: connection: completely remove CO_FL_WAKE_DATA Since it's only set and never tested anymore, let's remove it. commit 9fa1ee61cc6a7aef2d7cd7bba41869f682921dcb Author: Willy Tarreau Date: Sat Mar 18 15:39:57 2017 +0100 MEDIUM: connection: don't test for CO_FL_WAKE_DATA This flag is always set when we end up here, for each and every data layer (idle, stream-interface, checks), and continuing to test it leaves a big risk of forgetting to set it as happened once already before 1.5-dev13. It could make sense to backport this into stable branches as part of the connection flag fixes, after some cool down period. commit 3c0cc49d30968cf839a1d3a747de6adda18d26db Author: Willy Tarreau Date: Sun Mar 19 07:54:28 2017 +0100 BUG/MEDIUM: connection: ensure to always report the end of handshakes Despite the previous commit working fine on all tests, it's still not sufficient to completely address the problem. If the connection handler is called with an event validating an L4 connection but some handshakes remain (eg: accept-proxy), it will still wake the function up, which will not report the activity, and will not detect a change once the handshake it complete so it will not notify the ->wake() handler. In fact the only reason why the ->wake() handler is still called here is because after dropping the last handshake, we try to call ->recv() and ->send() in turn and change the flags in order to detect a data activity. But if for any reason the data layer is not interested in reading nor writing, it will not get these events. A cleaner way to address this is to call the ->wake() handler only on definitive status changes (shut, error), on real data activity, and on a complete connection setup, measured as CONNECTED with no more handshake pending. It could be argued that the handshake flags have to be made part of the condition to set CO_FL_CONNECTED but that would currently break a part of the health checks. Also a handshake could appear at any moment even after a connection is established so we'd lose the ability to detect a second end of handshake. For now the situation around CO_FL_CONNECTED is not clean : - session_accept() only sets CO_FL_CONNECTED if there's no pending handshake ; - conn_fd_handler() will set it once L4 and L6 are complete, which will do what session_accept() above refrained from doing even if an accept_proxy handshake is still pending ; - ssl_sock_infocbk() and ssl_sock_handshake() consider that a handshake performed with CO_FL_CONNECTED set is a renegociation ; => they should instead filter on CO_FL_WAIT_L6_CONN - all ssl_fc_* sample fetch functions wait for CO_FL_CONNECTED before accepting to fetch information => they should also get rid of any pending handshake - smp_fetch_fc_rcvd_proxy() uses !CO_FL_CONNECTED instead of CO_FL_ACCEPT_PROXY - health checks (standard and tcp-checks) don't check for HANDSHAKE and may report a successful check based on CO_FL_CONNECTED while not yet done (eg: send buffer full on send_proxy). This patch aims at solving some of these side effects in a backportable way before this is reworked in depth : - we need to call ->wake() to report connection success, measure connection time, notify that the data layer is ready and update the data layer after activity ; this has to be done either if we switch from pending {L4,L6}_CONN to nothing with no handshakes left, or if we notice some handshakes were pending and are now done. - we document that CO_FL_CONNECTED exactly means "L4 connection setup confirmed at least once, L6 connection setup confirmed at least once or not necessary, all this regardless of any possibly remaining handshakes or future L6 negociations". This patch also renames CO_FL_CONN_STATUS to the more explicit CO_FL_NOTIFY_DATA, and works around the previous flags trick consiting in setting an impossible combination of flags to notify the data layer, by simply clearing the current flags. This fix should be backported to 1.7, 1.6 and 1.5. commit 52821e27376f89b41167565b01d975f47266284c Author: Willy Tarreau Date: Sat Mar 18 17:11:37 2017 +0100 BUG/MAJOR: stream-int: do not depend on connection flags to detect connection Recent fix 7bf3fa3 ("BUG/MAJOR: connection: update CO_FL_CONNECTED before calling the data layer") marked an end to a fragile situation where the absence of CO_FL_{CONNECTED,L4,L6}* flags is used to mark the completion of a connection setup. The problem is that by setting the CO_FL_CONNECTED flag earlier, we can indeed call the ->wake() function from conn_fd_handler but the stream-interface's wake function needs to see CO_FL_CONNECTED unset to detect that a connection has just been established, so if there's no pending data in the buffer, the connection times out. The other ->wake() functions (health checks and idle connections) don't do this though. So instead of trying to detect a subtle change in connection flags, let's simply rely on the stream-interface's state and validate that the connection is properly established and that handshakes are completed before reporting the WRITE_NULL indicating that a pending connection was just completed. This patch passed all tests of handshake and non-handshake combinations, with synchronous and asynchronous connect() and should be safe for backport to 1.7, 1.6 and 1.5 when the fix above is already present. commit d78c0fa0fb32b27c49279c8fdb6e76bb6a01602f Author: Willy Tarreau Date: Sun Mar 19 06:59:29 2017 +0100 TESTS: add a test configuration to stress handshake combinations This config tries to involve the various possible combinations of connection handshakes, on the accept side and on the connect side. It also produces logs indicating the handshake time. May be tested with tcploop as the server, both for TCP and HTTP mode : - accept new connection - pause 100ms - send what looks like an HTTP response - wait 500ms and close Starting log server (mainly to check timers) : $ socat udp-recvfrom:5514,fork - Starting server : $ tcploop 8000 L N A W P100 S:"HTTP/1.0 200 OK\r\nConnection: close\r\n\r\n" P500 Testing all combinations with server-speaks-first (tcp) : $ nc 0 8007 Testing all combinations with client-speaks-first (tcp) : $ (printf "GET / HTTP/1.0\r\n\r\n";sleep 1) | nc 0 8007 Testing all combinations with client-speaks-first after pause (tcp) : $ (usleep 0.05 ; printf "GET / HTTP/1.0\r\n\r\n";sleep 1) | nc 0 8007 Testing all combinations with client-speaks-first (http) : $ (printf "GET / HTTP/1.0\r\n\r\n";sleep 1) | nc 0 8017 Testing all combinations with client-speaks-first after pause (http) : $ (usleep 0.05 ; printf "GET / HTTP/1.0\r\n\r\n";sleep 1) | nc 0 8017 Same tests must be redone after surrounding connect() in tcp_connect_server() with fcntl(fd, F_SETFL, 0) and fcntl(fd, F_SETFL, O_NONBLOCK) for sycnhronous connect(). commit e6006245de214fa6689bbddb5463867452575b35 Author: Christopher Faulet Date: Fri Mar 10 11:52:44 2017 +0100 BUG/MEDIUM: filters: Fix channels synchronization in flt_end_analyze When a filter is used, there are 2 channel's analyzers to surround all the others, flt_start_analyze and flt_end_analyze. This is the good place to acquire and release resources used by filters, when needed. In addition, the last one is used to synchronize the both channels, especially for HTTP streams. We must wait that the analyze is finished for the both channels for an HTTP transaction before restarting it for the next one. But this part was buggy, leading to unexpected behaviours. First, depending on which channel ends first, the request or the response can be switch in a "forward forever" mode. Then, the HTTP transaction can be cleaned up too early, while a processing is still in progress on a channel. To fix the bug, the flag CF_FLT_ANALYZE has been added. It is set on channels in flt_start_analyze and is kept if at least one filter is still analyzing the channel. So, we can trigger the channel syncrhonization if this flag was removed on the both channels. In addition, the flag TX_WAIT_CLEANUP has been added on the transaction to know if the transaction must be cleaned up or not during channels syncrhonization. This way, we are sure to reset everything once all the processings are finished. This patch should be backported in 1.7. commit a5938f71e4b621e78ca44bd9191d695d5c7fbb05 Author: Olivier Houchard Date: Wed Mar 15 15:12:06 2017 +0100 CLEANUP: config: Typo in comment. This is for the recently merged dynamic cookie patch set. commit 2cb49ebbc4f0bb39ef98bc1e59fa32affdb7c570 Author: Olivier Houchard Date: Wed Mar 15 15:11:06 2017 +0100 BUG/MEDIUM server: Fix crash when dynamic is defined, but not key is provided. Wait until we're sure we have a key before trying to calculate its length. [wt: no backport needed, was just merged] commit 3569df3fcf770d6785b32a39c0ff0c3d2a180f44 Author: Willy Tarreau Date: Wed Mar 15 12:47:46 2017 +0100 BUG/MEDIUM: listener: do not try to rebind another process' socket When the "process" setting of a bind line limits the processes a listening socket is enabled on, a "disable frontend" operation followed by an "enable frontend" triggers a bug because all declared listeners are attempted to be bound again regardless of their assigned processes. This can at minima create new sockets not receiving traffic, and at worst prevent from re-enabling a frontend if it's bound to a privileged port. This bug was introduced by commit 1c4b814 ("MEDIUM: listener: support rebinding during resume()") merged in 1.6-dev1, trying to perform the bind() before checking the process list instead of after. Just move the process check before the bind() operation to fix this. This fix must be backported to 1.7 and 1.6. Thanks to Pavlos for reporting this one. commit a84a2dba946bd49676f5222d49eca8d40346e53c Author: Willy Tarreau Date: Tue Mar 14 14:50:52 2017 +0100 CONTRIB: tcploop: use the trash instead of NULL for recv() NULL is Linux-centric and we're not focused on performance here but portability and reproducibility. Don't use NULL and use the trash instead. It may lead to multiple recv() calls for large blocks but as a benefit it will be possible to see the contents with strace. commit 24d41b9b4d009a9ae1aaae54de0be9a0281ba198 Author: Willy Tarreau Date: Tue Mar 14 14:50:05 2017 +0100 CONTRIB: tcploop: fix connect's address length FreeBSD wants the address size to be correct, so let's pass the size of a sockaddr_in struct, not the sockaddr_storage. commit bcd817e6699af7ae3d146110174ae0e241b0759c Author: Willy Tarreau Date: Tue Mar 14 14:44:06 2017 +0100 CONTRIB: tcploop: report action 'K' (kill) in usage message It was missing from the initial import. commit 752cc49728a2dc1e640aa0353ff8f66cfbf988e5 Author: Willy Tarreau Date: Tue Mar 14 14:37:13 2017 +0100 CONTRIB: tcploop: fix time format to silence build warnings timeval doesn't necessarily have ints for tv_sec/tv_usec, let's cast them. commit 0c0c0a6c26c31355f0913a971077c7d764a01b7a Author: Willy Tarreau Date: Tue Mar 14 14:36:26 2017 +0100 CONTRIB: tcploop: make it build on FreeBSD A few changes : - SOL_TCP must be replaced with IPPROTO_TCP - TCP_NOQUICKACK is not defined - MSG_MORE can be ignored and replaced with 0 commit 544d481516024e13c25b5673b8d6b3b647898bae Author: Steven Davidovitz Date: Wed Mar 8 11:06:20 2017 -0800 BUG/MINOR: checks: attempt clean shutw for SSL check Strict interpretation of TLS can cause SSL sessions to be thrown away when the socket is shutdown without sending a "close notify", resulting in each check to go through the complete handshake, eating more CPU on the servers. [wt: strictly speaking there's no guarantee that the close notify will be delivered, it's only best effort, but that may be enough to ensure that once at least one is received, next checks will be cheaper. This should be backported to 1.7 and possibly 1.6] commit 614f8d7d56dc558d433f55d4c710599f61a87bab Author: Olivier Houchard Date: Tue Mar 14 20:08:46 2017 +0100 MINOR: cli: Let configure the dynamic cookies from the cli. This adds 3 new commands to the cli : enable dynamic-cookie backend that enables dynamic cookies for a specified backend disable dynamic-cookie backend that disables dynamic cookies for a specified backend set dynamic-cookie-key backend that lets one change the dynamic cookie secret key, for a specified backend. commit 4e694049fad08fc7afccac227d547c32bb537454 Author: Olivier Houchard Date: Tue Mar 14 20:01:29 2017 +0100 MINOR: server: Add dynamic session cookies. This adds a new "dynamic" keyword for the cookie option. If set, a cookie will be generated for each server (assuming one isn't already provided on the "server" line), from the IP of the server, the TCP port, and a secret key provided. To provide the secret key, a new keyword as been added, "dynamic-cookie-key", for backends. Example : backend bk_web balance roundrobin dynamic-cookie-key "bla" cookie WEBSRV insert dynamic server s1 127.0.0.1:80 check server s2 192.168.56.1:80 check This is a first step to be able to dynamically add and remove servers, without modifying the configuration file, and still have all the load balancers redirect the traffic to the right server. Provide a way to generate session cookies, based on the IP address of the server, the TCP port, and a secret key provided. commit 7bf3fa3c23f6a1b7ed1212783507ac50f7e27544 Author: Willy Tarreau Date: Tue Mar 14 20:19:29 2017 +0100 BUG/MAJOR: connection: update CO_FL_CONNECTED before calling the data layer Matthias Fechner reported a regression in 1.7.3 brought by the backport of commit 819efbf ("BUG/MEDIUM: tcp: don't poll for write when connect() succeeds"), causing some connections to fail to establish once in a while. While this commit itself was a fix for a bad sequencing of connection events, it in fact unveiled a much deeper bug going back to the connection rework era in v1.5-dev12 : 8f8c92f ("MAJOR: connection: add a new CO_FL_CONNECTED flag"). It's worth noting that in a lab reproducing a similar environment as Matthias' about only 1 every 19000 connections exhibit this behaviour, making the issue not so easy to observe. A trick to make the problem more observable consists in disabling non-blocking mode on the socket before calling connect() and re-enabling it later, so that connect() always succeeds. Then it becomes 100% reproducible. The problem is that this CO_FL_CONNECTED flag is tested after deciding to call the data layer (typically the stream interface but might be a health check as well), and that the decision to call the data layer relies on a change of one of the flags covered by the CO_FL_CONN_STATE set, which is made of CO_FL_CONNECTED among others. Before the fix above, this bug couldn't appear with TCP but it could appear with Unix sockets. Indeed, connect() was always considered blocking so the CO_FL_WAIT_L4_CONN connection flag was always set, and polling for write events was always enabled. This used to guarantee that the conn_fd_handler() could detect a change among the CO_FL_CONN_STATE flags. Now with the fix above, if a connect() immediately succeeds for non-ssl connection with send-proxy enabled, and no data in the buffer (thus TCP mode only), the CO_FL_WAIT_L4_CONN flag is not set, the lack of data in the buffer doesn't enable polling flags for the data layer, the CO_FL_CONNECTED flag is not set due to send-proxy still being pending, and once send-proxy is done, its completion doesn't cause the data layer to be woken up due to the fact that CO_FL_CONNECT is still not present and that the CO_FL_SEND_PROXY flag is not watched in CO_FL_CONN_STATE. Then no progress is made when data are received from the client (and attempted to be forwarded), because a CF_WRITE_NULL (or CF_WRITE_PARTIAL) flag is needed for the stream-interface state to turn from SI_ST_CON to SI_ST_EST, allowing ->chk_snd() to be called when new data arrive. And the only way to set this flag is to call the data layer of course. After the connect timeout, the connection gets killed and if in the mean time some data have accumulated in the buffer, the retry will succeed. This patch fixes this situation by simply placing the update of CO_FL_CONNECTED where it should have been, before the check for a flag change needed to wake up the data layer and not after. This fix must be backported to 1.7, 1.6 and 1.5. Versions not having the patch above are still affected for unix sockets. Special thanks to Matthias Fechner who provided a very detailed bug report with a bisection designating the faulty patch, and to Olivier Houchard for providing full access to a pretty similar environment where the issue could first be reproduced. commit 6f6bb380ef78ad500c38886d30dea5eee2f653aa Author: Simon Horman Date: Wed Jan 4 09:37:26 2017 +0100 MEDIUM: stats: Add show json schema This may be used to output the JSON schema which describes the output of show info json and show stats json. The JSON output is without any extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. e.g.: $ echo "show schema json" | socat /var/run/haproxy.stat stdio | \ python -m json.tool The implementation does not generate the schema. Some consideration could be given to integrating the output of the schema with the output of typed and json info and stats. In particular the types (u32, s64, etc...) and tags. A sample verification of show info json and show stats json using the schema is as follows. It uses the jsonschema python module: cat > jschema.py << __EOF__ import json from jsonschema import validate from jsonschema.validators import Draft3Validator with open('schema.txt', 'r') as f: schema = json.load(f) Draft3Validator.check_schema(schema) with open('instance.txt', 'r') as f: instance = json.load(f) validate(instance, schema, Draft3Validator) __EOF__ $ echo "show schema json" | socat /var/run/haproxy.stat stdio > schema.txt $ echo "show info json" | socat /var/run/haproxy.stat stdio > instance.txt python ./jschema.py $ echo "show stats json" | socat /var/run/haproxy.stat stdio > instance.txt python ./jschema.py Signed-off-by: Simon Horman commit 05ee213f8b27855d0c6578ced12127e2e1196266 Author: Simon Horman Date: Wed Jan 4 09:37:25 2017 +0100 MEDIUM: stats: Add JSON output option to show (info|stat) Add a json parameter to show (info|stat) which will output information in JSON format. A follow-up patch will add a JSON schema which describes the format of the JSON output of these commands. The JSON output is without any extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. e.g.: $ echo "show info json" | socat /var/run/haproxy.stat stdio | \ python -m json.tool STAT_STARTED has bee added in order to track if show output has begun or not. This is used in order to allow the JSON output routines to only insert a "," between elements when needed. I would value any feedback on how this might be done better. Signed-off-by: Simon Horman commit 2019f95997d6747d6df010739476e5dccca13548 Author: Willy Tarreau Date: Tue Mar 14 11:07:31 2017 +0100 CLEANUP: http: make http_server_error() not set the status anymore Given that all call places except one had to set txn->status prior to calling http_server_error(), it's simpler to make this function rely on txn->status than have it store it from an argument. commit 800d1761d0bb2ee2d0951d2e39b8b163479158cd Author: Jarno Huuskonen Date: Mon Mar 6 14:56:36 2017 +0200 MINOR: http-request tarpit deny_status. Implements deny_status for http-request tarpit rule (allows setting custom http status code). This commit depends on: MEDIUM: http_error_message: txn->status / http_get_status_idx. commit 9e6906b9ecdf8e4c40c3194f7bea83ebbfdd995d Author: Jarno Huuskonen Date: Mon Mar 6 14:21:49 2017 +0200 MEDIUM: http_error_message: txn->status / http_get_status_idx. This commit removes second argument(msgnum) from http_error_message and changes http_error_message to use s->txn->status/http_get_status_idx for mapping status code from 200..504 to HTTP_ERR_200..HTTP_ERR_504(enum). This is needed for http-request tarpit deny_status commit. commit 50c8044423c384d858b24463d9ff6f4591b03beb Author: Nenad Merdanovic Date: Sun Mar 12 22:00:29 2017 +0100 CLEANUP: Remove comment that's no longer valid Code was deleted in ad63582eb, but the comment remained. Signed-off-by: Nenad Merdanovic commit 807a6e7856eb5fa88131e34e5a259684917c6d3f Author: Nenad Merdanovic Date: Sun Mar 12 22:00:00 2017 +0100 MINOR: Add hostname sample fetch It adds "hostname" as a new sample fetch. It does exactly the same as "%H" in a log format except that it can be used outside of log formats. Signed-off-by: Nenad Merdanovic commit 2754fbcfd6ce7f812a1c1b59dc8ddb0a44721f95 Author: Nenad Merdanovic Date: Sun Mar 12 21:56:56 2017 +0100 CLEANUP: Replace repeated code to count usable servers with be_usable_srv() 2 places were using an open-coded implementation of this function to count available servers. Note that the avg_queue_size() fetch didn't check that the proxy was in STOPPED state so it would possibly return a wrong server count here but that wouldn't impact the returned value. Signed-off-by: Nenad Merdanovic commit b7e7c4720ad06aba630c92a7d94b891b93f66df6 Author: Nenad Merdanovic Date: Sun Mar 12 21:56:55 2017 +0100 MINOR: Add nbsrv sample converter This is like the nbsrv() sample fetch function except that it works as a converter so it can count the number of available servers of a backend name retrieved using a sample fetch or an environment variable. Signed-off-by: Nenad Merdanovic commit 96c15719434a4eb46e191fcf1dafcb051065a76e Author: Nenad Merdanovic Date: Sun Mar 12 22:01:36 2017 +0100 BUG/MINOR: Fix "get map " CLI command The said form of the CLI command didn't return anything since commit ad8be61c7. This fix needs to be backported to 1.7. Signed-off-by: Nenad Merdanovic commit 24f45d8e34797ed9c16ac3fa6d89f3eed435e706 Author: Nenad Merdanovic Date: Sun Mar 12 22:01:35 2017 +0100 BUG/MEDIUM: cli: Prevent double free in CLI ACL lookup The memory is released by cli_release_mlook, which also properly sets the pointer to NULL. This was introduced with a big code reorganization involving moving to the new keyword registration form in commit ad8be61c7. This fix needs to be backported to 1.7. Signed-off-by: Nenad Merdanovic commit 8d7104982e1c41f7dc4d75ae7f7d2bbb96052d40 Author: Janusz Dziemidowicz Date: Wed Mar 8 16:59:41 2017 +0100 BUG/MEDIUM: ssl: Clear OpenSSL error stack after trying to parse OCSP file Invalid OCSP file (for example empty one that can be used to enable OCSP response to be set dynamically later) causes errors that are placed on OpenSSL error stack. Those errors are not cleared so anything that checks this stack later will fail. Following configuration: bind :443 ssl crt crt1.pem crt crt2.pem With following files: crt1.pem crt1.pem.ocsp - empty one crt2.pem.rsa crt2.pem.ecdsa Will fail to load. This patch should be backported to 1.7. commit de7dc88c517e22d3ae02ce8a8a23046ab2c62238 Author: Willy Tarreau Date: Fri Mar 10 11:49:21 2017 +0100 MINOR: config: warn when some HTTP rules are used in a TCP proxy Surprizingly, http-request, http-response, block, redirect, and capture rules did not cause a warning to be emitted when used in a TCP proxy, so let's fix this. This patch may be backported to older versions as it helps spotting configuration issues. commit d1307cea8417b1f1e8be4d1484dd1b59b0eb219d Author: Christopher Faulet Date: Mon Feb 27 21:59:39 2017 +0100 DOC: spoe: Update SPOE documentation to reflect recent changes commit 2eca6b50a770328a50ccad15775918486427aee7 Author: Christopher Faulet Date: Mon Feb 27 09:40:34 2017 +0100 MINOR: spoe: Add "max-frame-size" statement in spoe-agent section As its named said, this statement customize the maximum allowed size for frames exchanged between HAProxy and SPOAs. It should be greater than or equal to 256 and less than or equal to (tune.bufsize - 4) (4 bytes are reserved to the frame length). commit cecd8527b3d2c9c625b2697367b0893dc9ab5bc8 Author: Christopher Faulet Date: Fri Feb 24 22:11:21 2017 +0100 MINOR: spoe: Add "send-frag-payload" option in spoe-agent section This option can be used to enable or to disable (prefixing the option line with the "no" keyword) the sending of fragmented payload to agents. By default, this option is enabled. commit ecc537a8b9a790f15b6e703345a34326af4aa21f Author: Christopher Faulet Date: Thu Feb 23 22:52:39 2017 +0100 MINOR: spoe: Rely on alertif_too_many_arg during configuration parsing commit 305c6079d4ad5fc8659e99d335907acb854c7354 Author: Christopher Faulet Date: Thu Feb 23 16:17:53 2017 +0100 MINOR: spoe: Add "pipelining" and "async" options in spoe-agent section These options can be used to enable or to disable (prefixing the option line with the "no" keyword), respectively, pipelined and asynchronous exchanged between HAproxy and agents. By default, pipelining and async options are enabled. commit 6a2940c5f52c956bb880de0b3a450828f8bfd8ac Author: Christopher Faulet Date: Thu Feb 23 15:06:26 2017 +0100 MINOR: spoe: Add support of negation for options in SPOE configuration file For now, no options support negation (using "no" keyword). So it always returns an error. commit f032c3ec0995bac7d0c59b49edb67748722f5b31 Author: Christopher Faulet Date: Fri Feb 17 15:18:35 2017 +0100 MINOR: spoe: Improve implementation of the payload fragmentation Now, when a payload is fragmented, the first frame must define the frame type and the followings must use the special type SPOE_FRM_T_UNSET. This way, it is easy to know if a fragment is the first one or not. Of course, all frames must still share the same stream-id and frame-id. Update SPOA example accordingly. commit 4ff3e574ac1f5a48edf9e193ea35525ed4ff8995 Author: Christopher Faulet Date: Fri Feb 24 14:31:11 2017 +0100 REORG: spoe: Move low-level encoding/decoding functions in dedicated header file So, it will be easier to anyone to develop external services using these functions. SPOA example has been updated accordingly. commit 1f40b91a83bb4436f0cf1ce7b2e20951b4fc18a5 Author: Christopher Faulet Date: Fri Feb 17 09:32:19 2017 +0100 REORG: spoe: Move struct and enum definitions in dedicated header file SPOA example has been Updated accordingly commit 8eda93f30f1889fb3f0679b48143f5eb65466b11 Author: Christopher Faulet Date: Thu Feb 9 09:44:33 2017 +0100 MINOR: spoe: Handle NOTIFY frames cancellation using ABORT bit in ACK frames If an agent want to abort the processing a fragmented NOTIFY frame before receiving all fragments, it can send an ACK frame at any time with ABORT bit set (and of course, the FIN bit too). Beside this change, SPOE_FRM_ERR_FRAMEID_NOTFOUND error flag has been added. It is set when a unknown ACK frame is received. commit 8ef75251e359f62d6f114b6887cf33581d24e67d Author: Christopher Faulet Date: Mon Feb 20 22:56:03 2017 +0100 MAJOR: spoe: refactor the filter to clean up the code The SPOE code is now pretty big and it was the good time to clean it up. It is not perfect, some parts remains a bit ugly. But it is far better now. commit 850103546c29febc2aa56d668ecc9ffda71dee2f Author: Christopher Faulet Date: Thu Feb 2 10:14:36 2017 +0100 MINOR: spoe: Add support for fragmentation capability in the SPOA example This is just an example. So be careful to not send really huge payload because it would eat all your memory. commit f51f5fa56c9eb8924e323482e650d373fd901666 Author: Christopher Faulet Date: Thu Jan 19 10:01:12 2017 +0100 MAJOR: spoe: Add support of payload fragmentation in NOTIFY frames Now, agents can announce the support for the "fragmentation" capability during the HELLO handshake. HAProxy will never announce it because fragmented frame decoding is not implemented yet. But it can send such kind of frames. So, if an agent supports this capability, payloads exceeding the frame size will be split. A fragemented payload consists of several frames with the FIN bit clear and terminated by a single frame with the FIN bit set. All these frames must share the same STREAM-ID and FRAME-ID. Note that an unfragemnted payload consists of a single frame with the FIN bit set. And HELLO and DISCONNECT frames cannot be fragmented. This means that only NOTIFY frames can transport fragmented payload for now. commit 7aa0b2b0ddc7c98b8abe3f666c12f31a11b17b42 Author: Christopher Faulet Date: Fri Jan 13 11:30:50 2017 +0100 MINOR: spoe: Use the min of all known max_frame_size to encode messages The max_frame_size value is negociated between HAProxy and SPOE agents during the HELLO handshake. It is a per-connection value. Different SPOE agents can choose to use different max_frame_size values. So, now, we keep the minimum of all known max_frame_size. This minimum is updated when a new connection to a SPOE agent is opened and when a connection is closed. We use this value as a limit to encode messages in NOTIFY frames. commit 4596fb7056ae59e918537af046aed287f169226c Author: Christopher Faulet Date: Wed Jan 11 14:05:19 2017 +0100 MEDIUM: spoe: Be sure to wakeup the good entity waiting for a buffer This happens when buffer allocation failed. In the SPOE context, buffers are allocated by streams and SPOE applets at different time. First, by streams, when messages need to be encoded before sending them in a NOTIFY frame. Then, by SPOE applets, when a ACK frame is received. The first case works as expected, we wake up the stream. But for the second one, we must wake up the waiting SPOE applet. commit a21b064f819d69c988b5d3c2a289b81f2776d9fb Author: Christopher Faulet Date: Mon Jan 9 16:56:23 2017 +0100 MINOR: spoe: Check the scope of sample fetches used in SPOE messages If an error is triggered, the corresponding message is ignored and a warning is emitted. commit 72bcc4724f8ebdbfe6a67b3f9615cff434ccdc63 Author: Christopher Faulet Date: Wed Jan 4 16:39:41 2017 +0100 MINOR: spoe: Send a log message when an error occurred during event processing commit b067b06fc79a7dc08708ce1fbe4cda172c33bf87 Author: Christopher Faulet Date: Wed Jan 4 16:39:11 2017 +0100 MINOR: spoe: Add status code in error variable instead of hardcoded value Now, when option "set-on-error" is enabled, we set a status code representing the error occurred instead of "true". For values under 256, it represents an error coming from the engine. Below 256, it reports a SPOP error. In this case, to retrieve the right SPOP status code, you must remove 256 to this value. Here are possible values: * 1: a timeout occurred during the event processing. * 2: an error was triggered during the ressources allocation. * 255: an unknown error occurred during the event processing. * 256+N: a SPOP error occurred during the event processing. commit 42bfa46234f4920be22e0dbd0107c6120b1f7437 Author: Christopher Faulet Date: Wed Jan 4 14:14:19 2017 +0100 MINOR: spoe: Remove SPOE details from the appctx structure Now, as for peers, we use an opaque pointer to store information related to the SPOE filter in appctx structure. These information are now stored in a dedicated structure (spoe_appctx) and allocated, using a pool, when the applet is created. This removes the dependency between applets and the SPOE filter and avoids to eventually inflate the appctx structure. commit f95b111ddeba3be9c7e7f7e0e79961140ccabba3 Author: Christopher Faulet Date: Wed Dec 21 08:58:16 2016 +0100 MINOR: spoe: Add support for pipelining/async capabilities in the SPOA example Now, we can use the option '-c' to enable the support of a capability. By default, all capabilities are disabled. For example: $> ./spoa -c async -c pipelining In addition, it is also possible to set the maximum frame size supported by your agent (-m) and to add a delay in frames processing (-t). commit a1cda029958da17f4f467ab4316a8bf4bac668f1 Author: Christopher Faulet Date: Wed Dec 21 08:58:06 2016 +0100 MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents Now, HAProxy and agents can announce the support for "pipelining" and/or "async" capabilities during the HELLO handshake. For now, HAProxy always announces the support of both. In addition, in its HELLO frames. HAproxy adds the "engine-id" key. It is a uniq string that identify a SPOE engine. The "pipelining" capability is the ability for a peer to decouple NOTIFY and ACK frames. This is a symmectical capability. To be used, it must be supported by HAproxy and agents. Unlike HTTP pipelining, the ACK frames can be send in any order, but always on the same TCP connection used for the corresponding NOTIFY frame. The "async" capability is similar to the pipelining, but here any TCP connection established between HAProxy and the agent can be used to send ACK frames. if an agent accepts connections from multiple HAProxy, it can use the "engine-id" value to group TCP connections. commit b0b42388259224be54bf504fa559803b39d6453c Author: Christopher Faulet Date: Thu Feb 23 22:41:09 2017 +0100 BUG/MINOR: spoe: Fix parsing of arguments in spoe-message section The array of pointers passed to sample_parse_expr was not really an array but a pointer to pointer. So it can easily lead to a segfault during the configuration parsing. commit 3b386a318f699def6f37b4d3199cd1d72c481a0f Author: Christopher Faulet Date: Thu Feb 23 10:17:15 2017 +0100 BUG/MINOR: spoe: Fix soft stop handler using a specific id for spoe filters During a soft stop, we need to wakeup all SPOE applets to stop them. So we loop on all proxies, and for each proxy, on all filters. But we must be sure to only handle SPOE filters here. To do so, we use a specific id. commit e38047423da5a123ef21fa0ffac1e36c30c2b686 Author: Emmanuel Hocdet Date: Wed Mar 8 11:07:10 2017 +0100 MINOR: ssl: improved cipherlist captures Alloc capture buffer later (when filling), parse client-hello after heartbeat check and remove capture->conn (unused). commit aaee75088a5c85d77cd87aa91169705215f152d2 Author: Emmanuel Hocdet Date: Tue Mar 7 18:34:58 2017 +0100 BUG/MINOR: ssl: fix cipherlist captures with sustainable SSL calls Use SSL_set_ex_data/SSL_get_ex_data standard API call to store capture. We need to avoid internal structures/undocumented calls usage to try to control the beast and limit painful compatibilities. commit f6b37c67be277b5f0ae60438d796ff29ef19be40 Author: Emmanuel Hocdet Date: Mon Mar 6 15:34:44 2017 +0100 BUG/MEDIUM: ssl: in bind line, ssl-options after 'crt' are ignored. Bug introduced with "removes SSL_CTX_set_ssl_version call and cleanup CTX creation": ssl_sock_new_ctx is called before all the bind line is parsed. The fix consists of separating the use of default_ctx as the initialization context of the SSL connection via bind_conf->initial_ctx. Initial_ctx contains all the necessary parameters before performing the selection of the CTX: default_ctx is processed as others ctx without unnecessary parameters. commit 4608ed9511a0bc7f96e22376953d1b05de466cfd Author: Emmanuel Hocdet Date: Fri Jan 20 13:06:27 2017 +0100 MEDIUM: ssl: remove ssl-options from crt-list ssl-options are link to the initial negotiation environnement worn by default_ctx. Remove it from crt-list to avoid any confusion. commit 5bf77329b6d1577bf88db1adb59a436645a664be Author: Thierry FOURNIER Date: Sat Feb 25 12:45:22 2017 +0100 MEDIUM: ssl: add new sample-fetch which captures the cipherlist This new sample-fetches captures the cipher list offer by the client SSL connection during the client-hello phase. This is useful for fingerprint the SSL connection. commit cc6c2a2cb7116ec2e9893db0e7126d0a50d78067 Author: Emmanuel Hocdet Date: Fri Mar 3 17:04:14 2017 +0100 BUILD: ssl: fix build with -DOPENSSL_NO_DH commit 4de1ff1fd6973a8c206806bd092581eda09d3cfb Author: Emmanuel Hocdet Date: Fri Mar 3 12:21:32 2017 +0100 MINOR: ssl: removes SSL_CTX_set_ssl_version call and cleanup CTX creation. BoringSSL doesn't support SSL_CTX_set_ssl_version. To remove this call, the CTX creation is cleanup to clarify what is happening. SSL_CTX_new is used to match the original behavior, in order: force- according the method version then the default method with no- options. OPENSSL_NO_SSL3 error message is now in force-sslv3 parsing (as force-tls*). For CTX creation in bind environement, all CTX set related to the initial ctx are aggregate to ssl_sock_new_ctx function for clarity. Tests with crt-list have shown that server_method, options and mode are linked to the initial CTX (default_ctx): all ssl-options are link to each bind line and must be removed from crt-list. commit d3850603933c9319528375088a9b28b9b345246b Author: Emmanuel Hocdet Date: Fri Mar 3 15:21:26 2017 +0100 BUG/MEDIUM: ssl: switchctx should not return SSL_TLSEXT_ERR_ALERT_WARNING Extract from RFC 6066: "If the server understood the ClientHello extension but does not recognize the server name, the server SHOULD take one of two actions: either abort the handshake by sending a fatal-level unrecognized_name(112) alert or continue the handshake. It is NOT RECOMMENDED to send a warning-level unrecognized_name(112) alert, because the client's behavior in response to warning-level alerts is unpredictable. If there is a mismatch between the server name used by the client application and the server name of the credential chosen by the server, this mismatch will become apparent when the client application performs the server endpoint identification, at which point the client application will have to decide whether to proceed with the communication." Thanks Roberto Guimaraes for the bug repport, spotted with openssl-1.1.0. This fix must be backported. commit 530141f7479704be9d8f6c8f18fc9f71a6da2b3c Author: Emmanuel Hocdet Date: Wed Mar 1 18:54:56 2017 +0100 BUG/MEDIUM: ssl: fix verify/ca-file per certificate SSL verify and client_CA inherits from the initial ctx (default_ctx). When a certificate is found, the SSL connection environment must be replaced by the certificate configuration (via SSL_set_verify and SSL_set_client_CA_list). commit 0594211987351eaf521577b798a3a461b043710c Author: Emmanuel Hocdet Date: Mon Feb 20 16:11:50 2017 +0100 MEDIUM: boringssl: support native multi-cert selection without bundling This patch used boringssl's callback to analyse CLientHello before any handshake to extract key signature capabilities. Certificat with better signature (ECDSA before RSA) is choosed transparenty, if client can support it. RSA and ECDSA certificates can be declare in a row (without order). This makes it possible to set different ssl and filter parameter with crt-list. commit 19b1412e021451d4c7ac39750b556efaaf8639bf Author: Willy Tarreau Date: Tue Feb 28 09:48:11 2017 +0100 MINOR: http: don't close when redirect location doesn't start with "/" In 1.4-dev5 when we started to implement keep-alive, commit a9679ac ("[MINOR] http: make the conditional redirect support keep-alive") added a specific check was added to support keep-alive on redirect rules but only when the location would start with a "/" indicating the client would come back to the same server. But nowadays most applications put http:// or https:// in front of each and every location, and continuing to perform a close there is counter-efficient, especially when multiple objects are fetched at once from a same origin which redirects them to the correct origin (eg: after an http to https forced upgrade). It's about time to get rid of this old trick as it causes more harm than good at an era where persistent connections are omnipresent. Special thanks to Ciprian Dorin Craciun for providing convincing arguments with a pretty valid use case and proposing this draft patch which addresses the issue he was facing. This change although not exactly a bug fix should be backported to 1.7 to adapt better to existing infrastructure. commit 4f86264bae1e8bec4bf0f0596975597f6fec48ef Author: Willy Tarreau Date: Tue Feb 28 09:34:39 2017 +0100 BUG/MEDIUM: config: reject anything but "if" or "unless" after a use-backend rule Adrian Fitzpatrick reported that since commit f51658d ("MEDIUM: config: relax use_backend check to make the condition optional"), typos like "of" instead of "if" on use_backend rules are not properly detected. The reason is that the parser only checks for "if" or "unless" otherwise it considers there's no keyword, making the rule inconditional. This patch fixes it. It may reveal some rare config bugs for some people, but will not affect valid configurations. This fix must be backported to 1.7, 1.6 and 1.5. commit 7d388635526cffa79f4a62906f17afa5990f6092 Author: Thierry FOURNIER Date: Wed Feb 22 02:06:16 2017 +0100 BUG/MAJOR: lua segmentation fault when the request is like 'GET ?arg=val HTTP/1.1' Error in the HTTP parser. The function http_get_path() can return NULL and this case is not catched in the code. So, we try to dereference NULL pointer, and a segfault occurs. These two lines are useful to prevent the bug. acl prevent_bug path_beg / http-request deny if !prevent_bug This bug fix should be backported in 1.6 and 1.7 commit e3cc3a302651e905209b96e07203d03cf7e46ccb Author: Willy Tarreau Date: Mon Feb 13 11:12:29 2017 +0100 BUG/MAJOR: ssl: fix a regression in ssl_sock_shutw() Commit 405ff31 ("BUG/MINOR: ssl: assert on SSL_set_shutdown with BoringSSL") introduced a regression causing some random crashes apparently due to memory corruption. The issue is the use of SSL_CTX_set_quiet_shutdown() instead of SSL_set_quiet_shutdown(), making it use a different structure and causing the flag to be put who-knows-where. Many thanks to Jarno Huuskonen who reported this bug early and who bisected the issue to spot this patch. No backport is needed, this is 1.8-specific. commit 62c8a21c10c41629d63e93bee904514f456def74 Author: Thierry FOURNIER Date: Thu Feb 9 12:19:27 2017 +0100 BUG/MINOR: sendmail: The return of vsnprintf is not cleanly tested The string formatted by vsnprintf may be bigger than the size of the buffer "buf". This case is not tested. This sould be backported to 1.6 and 1.7 commit cdade94cf52886167979195af64b403360539db4 Author: Christopher Faulet Date: Wed Feb 8 12:41:31 2017 +0100 BUG/MINOR: http: Return an error when a replace-header rule failed on the response Historically, http-response rules couldn't produce errors generating HTTP responses during their evaluation. This possibility was "implicitly" added with http-response redirect rules (51d861a4). But, at the time, replace-header rules were kept untouched. When such a rule failed, the rules processing was just stopped (like for an accept rule). Conversely, when a replace-header rule fails on the request, it generates a HTTP response (400 Bad Request). With this patch, errors on replace-header rule are now handled in the same way for HTTP requests and HTTP responses. This patch should be backported in 1.7 and 1.6. commit 07a0fecced4a194c79c564f12fedfe9ca732713d Author: Christopher Faulet Date: Wed Feb 8 12:17:07 2017 +0100 BUG/MEDIUM: http: Prevent replace-header from overwriting a buffer This is the same fix as which concerning the redirect rules (0d94576c). The buffer used to expand the argument must be protected to prevent it being overwritten during build_logline() execution (the function used to expand the format string). This patch should be backported in 1.7, 1.6 and 1.5. It relies on commit b686afd ("MINOR: chunks: implement a simple dynamic allocator for trash buffers") for the trash allocator, which has to be backported as well. commit f1cc5d0eaf8ac56ef574d7349e34a255b149869a Author: Christopher Faulet Date: Wed Feb 8 09:45:13 2017 +0100 BUG/MEDIUM: filters: Do not truncate HTTP response when body length is undefined Some users have experienced some troubles using the compression filter when the HTTP response body length is undefined. They complained about receiving truncated responses. In fact, the bug can be triggered if there is at least one filter attached to the stream but none registered to analyze the HTTP response body. In this case, when the body length is undefined, data should be forwarded without any parsing. But, because of a wrong check, we were starting to parse them. Because it was not expected, the end of response was not correctly detected and the response could be truncted. So now, we rely on HAS_DATA_FILTER macro instead of HAS_FILTER one to choose to parse HTTP response body or not. Furthermore, in http_response_forward_body, the test to not forward the server closure to the client has been updated to reflect conditions listed in the associated comment. And finally, in http_msg_forward_body, when the body length is undefined, we continue the parsing it until the server closes the connection without any on filters. So filters can safely stop to filter data during their parsing. This fix should be backported in 1.7 commit 0d94576c74a4e6cef050b6cddb513fd9a363cf6c Author: Thierry FOURNIER Date: Sat Jan 28 07:39:53 2017 +0100 BUG/MEDIUM: http: prevent redirect from overwriting a buffer See 4b788f7d349ddde3f70f063b7394529eac6ab678 If we use the action "http-request redirect" with a Lua sample-fetch or converter, and the Lua function calls one of the Lua log function, the header name is corrupted, it contains an extract of the last loggued data. This is due to an overwrite of the trash buffer, because his scope is not respected in the "add-header" function. The scope of the trash buffer must be limited to the function using it. The build_logline() function can execute a lot of other function which can use the trash buffer. This patch fix the usage of the trash buffer. It limits the scope of this global buffer to the local function, we build first the header value using build_logline, and after we store the header name. Thanks Jesse Schulman for the bug repport. This patch must be backported in 1.7, 1.6 and 1.5 version, and it relies on commit b686afd ("MINOR: chunks: implement a simple dynamic allocator for trash buffers") for the trash allocator, which has to be backported as well. commit b686afd56817316a42529330a6b59c07708c2a37 Author: Willy Tarreau Date: Wed Feb 8 11:06:11 2017 +0100 MINOR: chunks: implement a simple dynamic allocator for trash buffers The trash buffers are becoming increasingly complex to deal with due to the code's modularity allowing some functions to be chained and causing the same chunk buffers to be used multiple times along the chain, possibly corrupting each other. In fact the trash were designed from scratch for explicitly not surviving a function call but string manipulation makes this impossible most of the time while not fullfilling the need for reliable temporary chunks. Here we introduce the ability to allocate a temporary trash chunk which is reserved, so that it will not conflict with the trash chunks other functions use, and will even support reentrant calls (eg: build_logline). For this, we create a new pool which is exactly the size of a usual chunk buffer plus the size of the chunk struct so that these chunks when allocated are exactly the same size as the ones returned by get_trash_buffer(). These chunks may fail so the caller must check them, and the caller is also responsible for freeing them. The code focuses on minimal changes and ease of reliable backporting because it will be needed in stable versions in order to support next patch. commit 26c6eb838311c31db0002c7d3c93a81297012d44 Author: Baptiste Assmann Date: Thu Feb 2 23:14:51 2017 +0100 BUG/MAJOR: dns: restart sockets after fork() UDP sockets used to send DNS queries are created before fork happens and this is a big problem because all the processes (in case of a configuration starting multiple processes) share the same socket. Some processes may consume responses dedicated to an other one, some servers may be disabled, some IPs changed, etc... As a workaround, this patch close the existing socket and create a new one after the fork() has happened. [wt: backport this to 1.7] commit 5cd1b9222eabcd73f387fd65b075b2af59014558 Author: Baptiste Assmann Date: Thu Feb 2 22:44:15 2017 +0100 MINOR: dns: give ability to dns_init_resolvers() to close a socket when requested The function dns_init_resolvers() is used to initialize socket used to send DNS queries. This patch gives the function the ability to close a socket before re-opening it. [wt: this needs to be backported to 1.7 for next fix] commit 4dc7197338e83610290e78d8a9e579e8e9013391 Author: Thierry FOURNIER Date: Sat Jan 28 08:33:08 2017 +0100 BUG/MINOR: lua: Map.end are not reliable because "end" is a reserved keyword This patch change the names prefixing it by a "_". So "end" becomes "_end". The backward compatibility with names without the prefix "_" is assured. In other way, another the keyword "end" can be used like this: Map['end']. Thanks Robin H. Johnson for the bug repport This should be backported in version 1.6 and 1.7 commit c0752565fe13fb040a5437f6c326a6f78a3b9d2b Author: Willy Tarreau Date: Mon Jan 23 21:38:57 2017 +0100 MINOR: server: extend the flags to 32 bits Right now not only we're limited to 8 bits, but it's mentionned nowhere and the limit was already reached. In addition, pp_opts (proxy protocol options) were set to 32 bits while only 3 are needed. So let's swap these two and group them together to avoid leaving two holes in the structure, saving 64 bits on 64-bit machines. commit 9484179f328bc7c7ae7c6c2fb404b761b256b73e Author: Willy Tarreau Date: Wed Jan 25 14:27:38 2017 +0100 BUG/MINOR: unix: fix connect's polling in case no data are scheduled There's a test after a successful synchronous connect() consisting in waking the data layer up asap if there's no more handshake. Unfortunately this test is run before setting the CO_FL_SEND_PROXY flag and before the transport layer adds its own flags, so it can indicate a willingness to send data while it's not the case and it will have to be handled later. This has no visible effect except a useless call to a function in case of health checks making use of the proxy protocol for example. Additionally a corner case where EALREADY was returned and considered equivalent to EISCONN was fixed so that it's considered equivalent to EINPROGRESS given that the connection is not complete yet. But this code should never return on the first call anyway so it's mostly a cleanup. This fix should be backported to 1.7 and 1.6 at least to avoid headaches during some debugging. commit 819efbf4b532d718abeb5e5aa6b2521ed725fe17 Author: Willy Tarreau Date: Wed Jan 25 14:12:22 2017 +0100 BUG/MEDIUM: tcp: don't poll for write when connect() succeeds While testing a tcp_fastopen related change, it appeared that in the rare case where connect() can immediately succeed, we still subscribe to write notifications on the socket, causing the conn_fd_handler() to immediately be called and a second call to connect() to be attempted to double-check the connection. In fact this issue had already been met with unix sockets (which often respond immediately) and partially addressed but incorrect so another patch will follow. But for TCP nothing was done. The fix consists in removing the WAIT_L4_CONN flag if connect() succeeds and to subscribe for writes only if some handshakes or L4_CONN are still needed. In addition in order not to fail raw TCP health checks, we have to continue to enable polling for data when nothing is scheduled for leaving and the connection is already established, otherwise the caller will never be notified. This fix should be backported to 1.7 and 1.6. commit e3e326d9f048a001bc848ca0638faf3d806e8b04 Author: Willy Tarreau Date: Thu Jan 19 17:25:20 2017 +0100 BUILD: ssl: kill a build warning introduced by BoringSSL compatibility A recent patch to support BoringSSL caused this warning to appear on OpenSSL 1.1.0 : src/ssl_sock.c:3062:4: warning: statement with no effect [-Wunused-value] It's caused by SSL_CTX_set_ecdh_auto() which is now only a macro testing that the last argument is zero, and the result is not used here. Let's just kill it for both versions. Tested with 0.9.8, 1.0.0, 1.0.1, 1.0.2, 1.1.0. This fix may be backported to 1.7 if the boringssl fix is as well. commit 77d88da7e1be29ff1ba5cd4634e2997060d30b2b Author: Willy Tarreau Date: Thu Jan 19 17:10:54 2017 +0100 BUILD: ssl: eliminate warning with OpenSSL 1.1.0 regarding RAND_pseudo_bytes() This function was deprecated in 1.1.0 causing this warning : src/ssl_sock.c:551:3: warning: 'RAND_pseudo_bytes' is deprecated (declared at /opt/openssl-1.1.0/include/openssl/rand.h:47) [-Wdeprecated-declarations] The man suggests to use RAND_bytes() instead. While the return codes differ, it turns out that the function was already misused and was relying on RAND_bytes() return code instead. The patch was tested on 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0. This fix must be backported to 1.7 and the return code check should be backported to earlier versions if relevant. commit a4fb8ed1f2969a79f2965b49a6f0e45536f01795 Author: Willy Tarreau Date: Thu Jan 19 16:50:25 2017 +0100 BUILD: ssl: silence a warning reported for ERR_remove_state() In 1.0.0, this function was replaced with ERR_remove_thread_state(). As of openssl 1.1.0, both are now deprecated and do nothing at all. Thus we simply make this call do nothing in 1.1.0 to silence the warning. The change was tested with 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0. This kills the following warning on 1.1.0 : src/ssl_sock.c:7266:9: warning: 'ERR_remove_state' is deprecated (declared at /dev/shm/openssl-1.1.0b/include/openssl/err.h:247) [-Wdeprecated-declarations] This fix should be backported to 1.7. commit 2b3205b5fa427376dd364f0ce0d6e0dff257b6f6 Author: Willy Tarreau Date: Thu Jan 19 17:04:02 2017 +0100 BUILD: ssl: fix build on OpenSSL 1.0.0 After the code was ported to support 1.1.0, this one broke on 1.0.0 : src/shctx.c:406: undefined reference to `SSL_SESSION_set1_id_context' The function was indeed introduced only in 1.0.1. The build was validated with 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0. This fix must be backported to 1.7. commit fdec7897fd2e8fba8f0d27e48b1e600553621c86 Author: Emmanuel Hocdet Date: Fri Jan 13 17:48:18 2017 +0100 BUILD: ssl: fix to build (again) with boringssl Limitations: . disable force-ssl/tls (need more work) should be set earlier with SSL_CTX_new (SSL_CTX_set_ssl_version is removed) . disable generate-certificates (need more work) introduce SSL_NO_GENERATE_CERTIFICATES to disable generate-certificates. Cleanup some #ifdef and type related to boringssl env. commit 4397290f27f09d1987e7d73a09a2c01ab7a924c7 Author: Misiek Date: Mon Jan 9 09:53:06 2017 +0100 MINOR: doc: Add docs for agent-addr and agent-send CLI commands commit 768d8608aa8d907f5ff052d22120f737e0d0fc3e Author: Misiek Date: Mon Jan 9 09:52:43 2017 +0100 MINOR: doc: Add docs for agent-addr configuration variable commit 2da082d732d9bf60619b6793a6000bd86deed404 Author: Misiek Date: Mon Jan 9 09:40:42 2017 +0100 MINOR: cli: Add possiblity to change agent config via CLI/socket This change adds possibility to change agent-addr and agent-send directives by CLI/socket. Now you can replace server's and their configuration without reloading/restarting whole haproxy, so it's a step in no-reload/no-restart direction. Depends on #e9602af - agent-addr is implemented there. Can be backported to 1.7. commit ea849333ca86af5fee3609fc7e7e698002d89898 Author: Misiek Date: Mon Jan 9 09:39:51 2017 +0100 MINOR: checks: Add agent-addr config directive This directive add possibility to set different address for agent-checks. With this you can manage server status and weight from central place. Can be backported to 1.7. commit e7f2b7301c0a6625654056356cca56853a14cd68 Author: Emmanuel Hocdet Date: Mon Jan 9 16:15:54 2017 +0100 MINOR: ssl: add curve suite for ECDHE negotiation Add 'curves' parameter on 'bind' and for 'crt-list' to set curve suite. (ex: curves X25519:P-256) commit 98263291cc3395a26a87408a900a93242f3f042d Author: Emmanuel Hocdet Date: Thu Dec 29 18:26:15 2016 +0100 MAJOR: ssl: bind configuration per certificat crt-list is extend to support ssl configuration. You can now have such line in crt-list : mycert.pem [npn h2,http/1.1] Support include "npn", "alpn", "verify", "ca_file", "crl_file", "ecdhe", "ciphers" configuration and ssl options. "crt-base" is also supported to fetch certificates. commit 70e2f272127a931a7b245a95e3a022879145e1dd Author: Christopher Faulet Date: Mon Jan 9 16:33:19 2017 +0100 BUG/MINOR: stream: Fix how backend-specific analyzers are set on a stream When the stream's backend was defined, the request's analyzers flag was always set to 0 if the stream had no listener. This bug was introduced with the filter API but never triggered (I think so). Because of the commit 5820a366, it is now possible to encountered it. For example, this happens when the trace filter is enabled on a SPOE backend. The fix is pretty trivial. This fix must be backported to 1.7. commit 3f78357066288c386709fc9439d56b5d36fe3f35 Author: Emeric Brun Date: Thu Jan 12 11:21:28 2017 +0100 OPTIM/MINOR: config: Optimize fullconn automatic computation loading configuration The previous version used an O(number of proxies)^2 algo to get the sum of the number of maxconns of frontends which reference a backend at least once. This new version adds the frontend's maxconn number to the backend's struct proxy member 'tot_fe_maxconn' when the backend name is resolved for switching rules or default_backend statment. At the end, the final backend's fullconn is computed looping only one time for all on proxies O(n). The load of a configuration using a large amount of backends (10 thousands) without configured fullconn was reduced from several minutes to few seconds. commit 3eb5b3fdd314a086cfcc04e3160dbc82e03d834a Author: Lukas Tribus Date: Wed Jan 11 22:47:18 2017 +0000 MINOR: ssl: don't show prefer-server-ciphers output The output of whether prefer-server-ciphers is supported by OpenSSL actually always show yes in 1.8, because SSL_OP_CIPHER_SERVER_PREFERENCE is redefined before the actual check in src/ssl_sock.c, since it was moved from here from src/haproxy.c. Since this is not really relevant anymore as we don't support OpenSSL < 0.9.7 anyway, this change just removes this output. commit 77ee7526deab920a2846d35365392127596ccd8f Author: Ryabin Sergey Date: Wed Jan 11 19:39:55 2017 +0400 BUG/MINOR: Reset errno variable before calling strtol(3) Sometimes errno != 0 before calling strtol(3) [wt: this needs to be backported to 1.7] commit b732321d03ea0bb3a27bf96fd353d12c1485a4bb Author: Lukas Tribus Date: Wed Jan 11 14:24:35 2017 +0000 MINOR: compression: fix -vv output without zlib/slz When haproxy is compiled without zlib or slz, the output of haproxy -vv shows (null). Make haproxy -vv output great again by providing the proper information (which is what we did before). This is for 1.8 only. commit 8c8c3497c0fbaf9c168eecbac5ebe89f7232f937 Author: Jarno Huuskonen Date: Wed Dec 28 18:50:29 2016 +0200 DOC: add deprecation notice to "block" [wt: this one is in fact emulated using http-request deny. This patch can thus be backported to 1.7, 1.6 and 1.5 so that users of older versions do not add this keyword in their configs] commit 59af2df102ccbbc62ab6855cc69dcf5933d1698f Author: Jarno Huuskonen Date: Wed Dec 28 10:49:01 2016 +0200 MINOR: proto_http.c 502 error txt typo. [wt: should be backported to 1.7 and 1.6 as it was introduced in 1.6-dev4] commit 16ad94adf6c3226c011ec5f77cce81096cd4d05c Author: Jarno Huuskonen Date: Mon Jan 9 14:17:10 2017 +0200 MINOR: Use "500 Internal Server Error" for 500 error/status code message. Internal Server Error is what is in RFC 2616/7231. commit 405ff31e31eb1cbdc76ba0d93c6db4c7a3fd497a Author: Emmanuel Hocdet Date: Sun Jan 8 14:07:39 2017 +0100 BUG/MINOR: ssl: assert on SSL_set_shutdown with BoringSSL With BoringSSL: SSL_set_shutdown: Assertion `(SSL_get_shutdown(ssl) & mode) == SSL_get_shutdown(ssl)' failed. "SSL_set_shutdown causes ssl to behave as if the shutdown bitmask (see SSL_get_shutdown) were mode. This may be used to skip sending or receiving close_notify in SSL_shutdown by causing the implementation to believe the events already happened. It is an error to use SSL_set_shutdown to unset a bit that has already been set. Doing so will trigger an assert in debug builds and otherwise be ignored. Use SSL_CTX_set_quiet_shutdown instead." Change logic to not notify on SSL_shutdown when connection is not clean. commit b7a4c34aac42a981d0fb8e733c0efb521d4e003e Author: Emmanuel Hocdet Date: Fri Jan 6 12:57:46 2017 +0100 BUG/MINOR: ssl: EVP_PKEY must be freed after X509_get_pubkey usage "X509_get_pubkey() attempts to decode the public key for certificate x. If successful it returns the public key as an EVP_PKEY pointer with its reference count incremented: this means the returned key must be freed up after use." commit 7b760c9c80f9f1be426d011f8228eb58343dc420 Author: Willy Tarreau Date: Fri Jan 6 19:23:20 2017 +0100 BUG/MEDIUM: tools: do not force an unresolved address to AF_INET:0.0.0.0 This prevents DNS from resolving IPv6-only servers in 1.7. Note, this patch depends on the previous series : 1. BUG/MINOR: tools: fix off-by-one in port size check 2. BUG/MEDIUM: server: consider AF_UNSPEC as a valid address family 3. MEDIUM: server: split the address and the port into two different fields 4. MINOR: tools: make str2sa_range() return the port in a separate argument 5. MINOR: server: take the destination port from the port field, not the addr 6. MEDIUM: server: disable protocol validations when the server doesn't resolve This fix (hence the whole series) must be backported to 1.7. commit 9698f4b295a5429a5341bfe633a6eb8d386f6182 Author: Willy Tarreau Date: Fri Jan 6 18:42:57 2017 +0100 MEDIUM: server: disable protocol validations when the server doesn't resolve When a server doesn't resolve we don't know the address family so we can't perform the basic protocol validations. However we know that we'll ultimately resolve to AF_INET4 or AF_INET6 so the controls are OK. It is important to proceed like this otherwise it will not be possible to start with unresolved addresses. commit 6ecb10aec7c12ddf8bf1813ec983bd6f2ed4c2df Author: Willy Tarreau Date: Fri Jan 6 18:36:06 2017 +0100 MINOR: server: take the destination port from the port field, not the addr Next patch will cause the port to disappear from the address field when servers do not resolve so we need to take it from the separate field provided by str2sa_range(). commit 48ef4c95b6b68c44f1d9bd72552a8644ec94b1b6 Author: Willy Tarreau Date: Fri Jan 6 18:32:38 2017 +0100 MINOR: tools: make str2sa_range() return the port in a separate argument This will be needed so that we're don't have to extract it from the returned address where it will not always be anymore (eg: for unresolved servers). commit 04276f3d6e146794edcb15fd599cade29b924129 Author: Willy Tarreau Date: Fri Jan 6 17:41:29 2017 +0100 MEDIUM: server: split the address and the port into two different fields Keeping the address and the port in the same field causes a lot of problems, specifically on the DNS part where we're forced to cheat on the family to be able to keep the port. This causes some issues such as some families not being resolvable anymore. This patch first moves the service port to a new field "svc_port" so that the port field is never used anymore in the "addr" field (struct sockaddr_storage). All call places were adapted (there aren't that many). commit 3acfcd1aa10b933053e6398baba2eae2d32ae7a6 Author: Willy Tarreau Date: Fri Jan 6 19:18:32 2017 +0100 BUG/MEDIUM: server: consider AF_UNSPEC as a valid address family The DNS code is written so as to support AF_UNSPEC to decide on the server family based on responses, but unfortunately snr_resolution_cb() considers it as invalid causing a DNS storm to happen when a server arrives with this family. This situation is not supposed to happen as long as unresolved addresses are forced to AF_INET, but this will change with the upcoming fixes and it's possible that it's not granted already when changing an address on the CLI. This fix must be backported to 1.7 and 1.6. commit d7dad1bc49176b9e2c6bc93d5918313b8054524e Author: Willy Tarreau Date: Fri Jan 6 16:46:22 2017 +0100 BUG/MINOR: tools: fix off-by-one in port size check port_to_str() checks that the port size is at least 5 characters instead of at least 6. While in theory it could permit a buffer overflow, it's harmless because all callers have at least 6 characters here. This fix needs to be backported to 1.7, 1.6 and 1.5. commit 4c18346c0f18e8ac470bd0cb40b1b895e9345e26 Author: Willy Tarreau Date: Fri Jan 6 12:21:38 2017 +0100 BUG/MINOR: config: emit a warning if http-reuse is enabled with incompatible options http-reuse should normally not be used in conjunction with the proxy protocol or with "usesrc clientip". While there's nothing fundamentally wrong with this, whenever these options are used, the server expects the IP address to be the source address for all requests, which doesn't make sense with http-reuse. commit 4f60301235ca1d35d51a49f15981706dd1ab67a0 Author: Emeric Brun Date: Thu Jan 5 15:11:44 2017 +0100 MINOR: connection: add sample fetch "fc_rcvd_proxy" fc_rcvd_proxy : boolean Returns true if the client initiated the connection with a PROXY protocol header. A flag is added on the struct connection if a PROXY header is successfully parsed. commit 52f5db2a44d153797ffd547887a3568e3218c4fc Author: Robin H. Johnson Date: Sun Jan 1 13:10:52 2017 -0800 MINOR: http: custom status reason. The older 'rsprep' directive allows modification of the status reason. Extend 'http-response set-status' to take an optional string of the new status reason. http-response set-status 418 reason "I'm a coffeepot" Matching updates in Lua code: - AppletHTTP.set_status - HTTP.res_set_status Signed-off-by: Robin H. Johnson commit 2afff9c2d6ce6d8f55b1664e76b1d30085540c7b Author: Willy Tarreau Date: Wed Jan 4 14:44:46 2017 +0100 BUG/MAJOR: http: fix risk of getting invalid reports of bad requests Commits 5f10ea3 ("OPTIM: http: improve parsing performance of long URIs") and 0431f9d ("OPTIM: http: improve parsing performance of long header lines") introduced a bug in the HTTP parser : when a partial request is read, the first part ends up on a 8-bytes boundary (or 4-byte on 32-bit machines), the end lies in the header field value part, and the buffer used to contain a CR character exactly after the last block, then the parser could be confused and read this CR character as being part of the current request, then switch to a new state waiting for an LF character. Then when the next part of the request appeared, it would read the character following what was erroneously mistaken for a CR, see that it is not an LF and fail on a bad request. In some cases, it can even be worse and the header following the hole can be improperly indexed causing all sort of unexpected behaviours like a content-length being ignored or a header appended at the wrong position. The reason is that there's no control of and of parsing just after breaking out of the loop. One way to reproduce it is with this config : global stats socket /tmp/sock1 mode 666 level admin stats timeout 1d frontend px bind :8001 mode http timeout client 10s redirect location / And sending requests this way : $ tcploop 8001 C P S:"$(dd if=/dev/zero bs=16384 count=1 2>/dev/null | tr '\000' '\r')" $ tcploop 8001 C P S:"$(dd if=/dev/zero bs=16384 count=1 2>/dev/null | tr '\000' '\r')" $ tcploop 8001 C P \ S:"GET / HTTP/1.0\r\nX-padding: 0123456789.123456789.123456789.123456789.123456789.123456789.1234567" P \ S:"89.123456789\r\n\r\n" P Then a "show errors" on the socket will report : $ echo "show errors" | socat - /tmp/sock1 Total events captured on [04/Jan/2017:15:09:15.755] : 32 [04/Jan/2017:15:09:13.050] frontend px (#2): invalid request backend (#-1), server (#-1), event #31 src 127.0.0.1:59716, session #91, session flags 0x00000080 HTTP msg state 17, msg flags 0x00000000, tx flags 0x00000000 HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00808002, out 0 bytes, total 111 bytes pending 111 bytes, wrapping at 16384, error at position 107: 00000 GET / HTTP/1.0\r\n 00016 X-padding: 0123456789.123456789.123456789.123456789.123456789.12345678 00086+ 9.123456789.123456789\r\n 00109 \r\n This fix must be backported to 1.7. Many thanks to Aleksey Gordeev and Axel Reinhold for providing detailed network captures and configurations exhibiting the issue. commit 0ebb511b3ea1b4b00410cedfec5c6bef47d4831a Author: Willy Tarreau Date: Mon Dec 5 00:10:57 2016 +0100 MINOR: tools: add a generic hexdump function for debugging debug_hexdump() prints to the requested output stream (typically stdout or stderr) an hex dump of the blob passed in argument. This is useful to help debug binary protocols. commit 990397ed2e7673f14fc635f8764fe86f41bd4d71 Author: Willy Tarreau Date: Thu Jan 5 19:58:24 2017 +0100 BUILD: scripts: automatically update the branch in version.h when releasing The stats page proudly displays "Updates (v1.5)". This version is inherited from version.h which has not been updated since 1.5, so let's teach the create-release script about it. This must be backported to 1.7. 1.6 now uses the same script (externally) for the release and will automatically benefit from it. commit 10e61cbf4134e2adf083ac960622a08fedf44c6a Author: Willy Tarreau Date: Wed Jan 4 14:51:22 2017 +0100 BUG/MINOR: http: report real parser state in error captures Error captures almost always report a state 26 (MSG_ERROR) making it very hard to know what the parser was expecting. The reason is that we have to switch to MSG_ERROR to trigger the dump, and then during the dump we capture the current state which is already MSG_ERROR. With this change we now copy the current state into an err_state field that will be reported as the faulty state. This patch looks a bit large because the parser doesn't update the current state until it runs out of data so the current state is never known when jumping to ther error label! Thus the code had to be updated to take copies of the current state before switching to MSG_ERROR based on the switch/case values. As a bonus, it now shows the current state in human-readable form and not only in numeric form ; in the past it was not an issue since it was always 26 (MSG_ERROR). At least now we can get exploitable invalid request/response reports : [05/Jan/2017:19:28:57.095] frontend f (#2): invalid request backend (#-1), server (#-1), event #1 src 127.0.0.1:39894, session #4, session flags 0x00000080 HTTP msg state MSG_RQURI(4), msg flags 0x00000000, tx flags 0x00000000 HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00908002, out 0 bytes, total 20 bytes pending 20 bytes, wrapping at 16384, error at position 5: 00000 GET /\e HTTP/1.0\r\n 00017 \r\n 00019 \n [05/Jan/2017:19:28:33.827] backend b (#3): invalid response frontend f (#2), server s1 (#1), event #0 src 127.0.0.1:39718, session #0, session flags 0x000004ce HTTP msg state MSG_HDR_NAME(17), msg flags 0x00000000, tx flags 0x08300000 HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x80008002, out 0 bytes, total 59 bytes pending 59 bytes, wrapping at 16384, error at position 31: 00000 HTTP/1.1 200 OK\r\n 00017 Content-length : 10\r\n 00038 \r\n 00040 0a\r\n 00044 0123456789\r\n 00056 0\r\n This should be backported to 1.7 and 1.6 at least to help with bug reports. commit 0184ea71a68a760a8d4dd2d0cf1b4411ecdf7623 Author: Christopher Faulet Date: Thu Jan 5 14:06:34 2017 +0100 BUG/MAJOR: channel: Fix the definition order of channel analyzers It is important to defined analyzers (AN_REQ_* and AN_RES_*) in the same order they are evaluated in process_stream. This order is really important because during analyzers evaluation, we run them in the order of the lower bit to the higher one. This way, when an analyzer adds/removes another one during its evaluation, we know if it is located before or after it. So, when it adds an analyzer which is located before it, we can switch to it immediately, even if it has already been called once but removed since. With the time, and introduction of new analyzers, this order was broken up. the main problems come from the filter analyzers. We used values not related with their evaluation order. Furthermore, we used same values for request and response analyzers. So, to fix the bug, filter analyzers have been splitted in 2 distinct lists to have different analyzers for the request channel than those for the response channel. And of course, we have moved them to the right place. Some other analyzers have been reordered to respect the evaluation order: * AN_REQ_HTTP_TARPIT has been moved just before AN_REQ_SRV_RULES * AN_REQ_PRST_RDP_COOKIE has been moved just before AN_REQ_STICKING_RULES * AN_RES_STORE_RULES has been moved just after AN_RES_WAIT_HTTP Note today we have 29 analyzers, all stored into a 32 bits bitfield. So we can still add 4 more analyzers before having a problem. A good way to fend off the problem for a while could be to have a different bitfield for request and response analyzers. [wt: all of this must be backported to 1.7, and part of it must be backported to 1.6 and 1.5] commit 401c64bfe4dee304c7025f7412ec111dd677eaa8 Author: Thierry FOURNIER Date: Thu Jan 5 11:44:09 2017 +0100 BUG/MINOR: sample-fetches/stick-tables: bad type for the sample fetches sc*_get_gpt0 The registered output type for the sample fetches sc*_get_gpt0 is a boolean, but the value returned is an integer. This patch fixs the default type to SINT in place of BOOL. This patch should be backported in 1.6 and 1.7 commit d3db35a1d179c7dd3d3f5154465672af203cecd5 Author: David Harrigan Date: Fri Dec 30 12:12:49 2016 +0000 MINOR: stats: Support "select all" for backend actions Allow the user to quickly select all servers within a group before invoking an action. commit 1ca1b6fe3c7bea92bf1844577303eb94fbdeef4c Author: Olivier Doucet Date: Mon Jan 2 11:48:57 2017 +0100 BUG/MINOR: option prefer-last-server must be ignored in some case when using "option prefer-last-server", we may not always stay on the same backend if option balance told us otherwise. For example, backend may change in the following cases: balance hdr() balance rdp-cookie balance source balance uri balance url_param [wt: backport this to 1.7 and 1.6] commit f2592b29f13907ddf2bba42d00bc41cb8ee5b69b Author: David Carlier Date: Mon Nov 21 21:25:58 2016 +0000 MEDIUM: regex: pcre2 support this adds a support of the newest pcre2 library, more secure than its older sibling in a cost of a more complex API. It works pretty similarly to pcre's part to keep the overall change smooth, except : - we define the string class supported at compile time. - after matching the ovec data is properly sized, althought we do not take advantage of it here. - the lack of jit support is treated less 'dramatically' as pcre2_jit_compile in this case is 'no-op'. commit 01e0974b5ae18c620d8aa8ca71f5965f90f41b63 Author: Thierry FOURNIER Date: Mon Dec 26 11:46:11 2016 +0100 MINOR: samples: add xx-hash functions This patch adds the support of xx-hash 32 and 64-bits functions. commit de6925eccfaba85f1402c5e5cf700c420792802a Author: Thierry FOURNIER Date: Fri Dec 23 17:03:25 2016 +0100 BUILD: lua: build failed on FreeBSD. s6_addr* fields are not available in the userland on BSD systems in general. bug reported by David Carlier needs backport to 1.7.x commit f27cddcf7d42cad6e1972b5075da348192ee4105 Author: Guillaume de Lafond Date: Fri Dec 23 17:32:43 2016 +0100 DOC: Add timings events schemas [wt: this could be backported to 1.7] commit 1e4fc43630eb194f8e8dd98197cf47c7e9912371 Author: William Lallemand Date: Fri Dec 23 15:44:15 2016 +0100 BUG/MINOR: systemd: potential zombie processes In systemd mode (-Ds), the master haproxy process is waiting for each child to exit in a specific order. If a process die when it's not his turn, it will become a zombie process until every processes exit. The master is now waiting for any process to exit in any order. This patch should be backported to 1.7, 1.6 and 1.5. commit 119a4084bf88418bce74d8af686576e371700c20 Author: Willy Tarreau Date: Thu Dec 22 21:58:38 2016 +0100 BUG/MEDIUM: ssl: for a handshake when server-side SNI changes Calling SSL_set_tlsext_host_name() on the current SSL ctx has no effect if the session is being resumed because the hostname is already stored in the session and is not advertised again in subsequent connections. It's visible when enabling SNI and health checks at the same time because checks do not send an SNI and regular traffic reuses the same connection, resulting in no SNI being sent. The only short-term solution is to reset the reused session when the SNI changes compared to the previous one. It can make the server-side performance suffer when SNIs are interleaved but it will work. A better long-term solution would be to keep a small cache of a few contexts for a few SNIs. Now with SSL_set_session(ctx, NULL) it works. This needs to be double- checked though. The man says that SSL_set_session() frees any previously existing context. Some people report a bit of breakage when calling SSL_set_session(NULL) on openssl 1.1.0a (freed session not reusable at all though it's not an issue for now). This needs to be backported to 1.7 and 1.6. commit 57b877147d726a743a038e7101b6d2e6922116c7 Author: Marcin Deranek Date: Thu Dec 22 16:21:08 2016 +0100 BUG/MINOR: backend: nbsrv() should return 0 if backend is disabled According to nbsrv() documentation this fetcher should return "an integer value corresponding to the number of usable servers". In case backend is disabled none of servers is usable, so I believe fetcher should return 0. This patch should be backported to 1.7, 1.6, 1.5. commit ef934603c04d6c6cc7e9e537048a099427b19b10 Author: Willy Tarreau Date: Thu Dec 22 23:12:01 2016 +0100 CLEANUP: ssl: move most ssl-specific global settings to ssl_sock.c Historically a lot of SSL global settings were stored into the global struct, but we've reached a point where there are 3 ifdefs in it just for this, and others in haproxy.c to initialize it. This patch moves all the private fields to a new struct "global_ssl" stored in ssl_sock.c. This includes : char *crt_base; char *ca_base; char *listen_default_ciphers; char *connect_default_ciphers; int listen_default_ssloptions; int connect_default_ssloptions; int tune.sslprivatecache; /* Force to use a private session cache even if nbproc > 1 */ unsigned int tune.ssllifetime; /* SSL session lifetime in seconds */ unsigned int tune.ssl_max_record; /* SSL max record size */ unsigned int tune.ssl_default_dh_param; /* SSL maximum DH parameter size */ int tune.ssl_ctx_cache; /* max number of entries in the ssl_ctx cache. */ The "tune" part was removed (useless here) and the occasional "ssl" prefixes were removed as well. Thus for example instead of global.tune.ssl_default_dh_param we now have : global_ssl.default_dh_param A few initializers were present in the constructor, they could be brought back to the structure declaration. A few other entries had to stay in global for now. They concern memory calculationn (used in haproxy.c) and stats (used in stats.c). The code is already much cleaner now, especially for global.h and haproxy.c which become readable. commit d1c5750370255e25774cbd672f00b50c4652156e Author: Willy Tarreau Date: Thu Dec 22 22:46:15 2016 +0100 CLEANUP: ssl: move tlskeys_finalize_config() to a post_check callback tlskeys_finalize_config() was the only reason for haproxy.c to still require ifdef and includes for ssl_sock. This one fits perfectly well in the late initializers so it was changed to be registered with hap_register_post_check(). commit 17d4538044fc37b9b4cccdc21543b183d5b69a0b Author: Willy Tarreau Date: Thu Dec 22 21:16:08 2016 +0100 MINOR: ssl_sock: implement and use prepare_srv()/destroy_srv() Now we can simply check the transport layer at run time and decide whether or not to initialize or destroy these entries. This removes other ifdefs and includes from cfgparse.c, haproxy.c and hlua.c. commit d84dab733a9d7e30737ab62a4911241089f029af Author: Willy Tarreau Date: Thu Dec 22 21:13:18 2016 +0100 MINOR: connection: add new prepare_srv()/destroy_srv() entries to xprt_ops These one will be used by the SSL layer to prepare and destroy a server-side SSL context. commit d9f5cca3d5c152fddb5151081a6d7349c2bddd87 Author: Willy Tarreau Date: Thu Dec 22 21:08:52 2016 +0100 CLEANUP: connection: unexport raw_sock and ssl_sock This way we're sure not to reuse them by accident. commit a261e9b0944649bfaeccb7a3ab7ffa469c08e81d Author: Willy Tarreau Date: Thu Dec 22 20:44:00 2016 +0100 CLEANUP: connection: remove all direct references to raw_sock and ssl_sock Now we exclusively use xprt_get(XPRT_RAW) instead of &raw_sock or xprt_get(XPRT_SSL) for &ssl_sock. This removes a bunch of #ifdef and include spread over a number of location including backend, cfgparse, checks, cli, hlua, log, server and session. commit 13e1410f8ad514c7660a9b7edf74e7928c7d6379 Author: Willy Tarreau Date: Thu Dec 22 20:25:26 2016 +0100 MINOR: connection: add a minimal transport layer registration system There are still a lot of #ifdef USE_OPENSSL in the code (still 43 occurences) because we never know if we can directly access ssl_sock or not. This patch attacks the problem differently by providing a way for transport layers to register themselves and for users to retrieve the pointer. Unregistered transport layers will point to NULL so it will be easy to check if SSL is registered or not. The mechanism is very inexpensive as it relies on a two-entries array of pointers, so the performance will not be affected. commit 141ad85d107dba3c511005a7d80350bcc8bd9995 Author: Willy Tarreau Date: Thu Dec 22 18:38:00 2016 +0100 MINOR: server: move the use_ssl field out of the ifdef USE_OPENSSL Having it in the ifdef complicates certain operations which require additional ifdefs just to access a member which could remain zero in non-ssl cases. Let's move it out, it will not even increase the struct size on 64-bit machines due to alignment. commit 795cdabb570f8829f306b13b773f204915b70d62 Author: Willy Tarreau Date: Thu Dec 22 17:30:54 2016 +0100 MINOR: ssl_sock: implement ssl_sock_destroy_bind_conf() Instead of hard-coding all SSL destruction in cfgparse.c and haproxy.c, we now register this new function as the transport layer's destroy_bind_conf() and call it only when defined. This removes some non-obvious SSL-specific code and #ifdefs from cfgparse.c and haproxy.c commit fa983d3caaac134db52778ce5ad597e485e1d4b0 Author: Willy Tarreau Date: Thu Dec 22 17:30:20 2016 +0100 MINOR: connection: add a new destroy_bind_conf() entry to xprt_ops This one will be set by the transport layers which want to destroy a bind_conf. It will typically be used by SSL to release certificates, CAs and so on. commit 55d3791b4633de3942f4b63b89937a8ec97dc2d4 Author: Willy Tarreau Date: Wed Dec 21 23:38:39 2016 +0100 MEDIUM: ssl_sock: implement ssl_sock_prepare_bind_conf() Instead of hard-coding all SSL preparation in cfgparse.c, we now register this new function as the transport layer's prepare_bind_conf() and call it only when definied. This removes some non-obvious SSL-specific code from cfgparse.c as well as a #ifdef. commit 5aacf78a346f2498bccb46c7eb36d53ed1c64a8d Author: Willy Tarreau Date: Thu Dec 22 17:19:24 2016 +0100 MINOR: connection: add a new prepare_bind_conf() entry to xprt_ops This one will be set by the transport layers which want to initialize a bind_conf. It will typically be used by SSL to load certificates, CAs and so on. commit 0320934f7e2de603a3de6416d4aa510f905e0347 Author: Willy Tarreau Date: Thu Dec 22 17:08:28 2016 +0100 MEDIUM: ssl: remote the proxy argument from most functions Most of the SSL functions used to have a proxy argument which was mostly used to be able to emit clean errors using Alert(). First, many of them were converted to memprintf() and don't require this pointer anymore. Second, the rare which still need it also have either a bind_conf argument or a server argument, both of which carry a pointer to the relevant proxy. So let's now get rid of it, it needlessly complicates the API and certain functions already have many arguments. commit c95bad5013a288f9e5f94baab91f5cebbc68b237 Author: Willy Tarreau Date: Thu Dec 22 00:13:31 2016 +0100 MEDIUM: move listener->frontend to bind_conf->frontend Historically, all listeners have a pointer to the frontend. But since the introduction of SSL, we now have an intermediary layer called bind_conf corresponding to a "bind" line. It makes no sense to have the frontend on each listener given that it's the same for all listeners belonging to a same bind_conf. Also certain parts like SSL can only operate on bind_conf and need the frontend. This patch fixes this by moving the frontend pointer from the listener to the bind_conf. The extra indirection is quite cheap given and the places were this is used are very scarce. commit 71a8c7c49e912f4c6117b62bdf24cbe21916c2ae Author: Willy Tarreau Date: Wed Dec 21 22:04:54 2016 +0100 MINOR: listener: move the transport layer pointer to the bind_conf A mistake was made when the socket layer was cut into proto and transport, the transport was attached to the listener while all listeners in a single "bind" line always have exactly the same transport. It doesn't seem obvious but this is the reason why there are so many #ifdefs USE_OPENSSL in cfgparse : a lot of operations have to be open-coded because cfgparse only manipulates bind_conf and we don't have the information of the transport layer here. Very little code makes use of the transport layer, mainly session setup and log. These places can afford an extra pointer indirection (the listener points to the bind_conf). This change is thus very small, it saves a little bit of memory (8B per listener) and makes the code more flexible. commit 5820a3669031b2f4de58577953ed3135aaaa1f62 Author: Willy Tarreau Date: Thu Dec 22 15:59:02 2016 +0100 MEDIUM: spoe: don't create a dummy listener for outgoing connections The code currently creates a listener only to ensure that sess->li is properly populated, and to retrieve the frontend (which is also available directly from the session). It turns out that the current infrastructure (for a large part) already supports not having any listener on a session (since Lua does the same), except for the following places which were not yet converted : - session_count_new() : used by session_accept_fd, ie never for spoe - session_accept_fd() : never used here, an applet initiates the session - session_prepare_log_prefix() : embryonic sessions only, thus unused - session_kill_embryonic() : same - conn_complete_session() : same - build_log_line() for fields %cp, %fp and %ft : unused here but may change - http_wait_for_request() and subsequent functions : unused here Thus for now it's as safe to run SPOE without listener as it is for Lua, and this was an obstacle against some cleanups of the listener code. The places above should be plugged so that it becomes save over the long term as well. An alternative in the future might be to create a dummy listener that outgoing connections could use just to avoid keeping a null here. commit a12dde04e00330e309aefe7822b8429eb2843944 Author: Willy Tarreau Date: Thu Dec 22 18:14:41 2016 +0100 MINOR: tcp-rules: check that the listener exists before updating its counters The tcp rules may be applied to a TCP stream initiated by applets (spoe, lua, peers, later H2). These ones do not necessarily have a valid listener so we must verify the field is not null before updating the stats. For now there's no way to trigger this bug because lua and peers don't have analysers, h2 is not implemented and spoe has a dummy listener. But this threatens to break at any instant. commit 0ff98a4758a511f573e50175992984dc5a3db050 Author: Thierry FOURNIER Date: Mon Dec 19 16:50:42 2016 +0100 BUG/MINOR: stats: fix be/sessions/current out in typed stats "scur" was typed as "limit" (FO_CONFIG) and "config value" (FN_LIMIT). The real types of "scur" are "metric" (FO_METRIC) and "gauge" (FN_GAUGE). FO_METRIC and FN_GAUGE are the value 0. commit 94ff03af84ee0c4a2b6cfb92332fcafbcdc48765 Author: Willy Tarreau Date: Thu Dec 22 17:57:46 2016 +0100 BUG/MEDIUM: ssl: avoid double free when releasing bind_confs ssl_sock functions don't mark pointers as NULL after freeing them. So if a "bind" line specifies some SSL settings without the "ssl" keyword, they will get freed at the end of check_config_validity(), then freed a second time on exit. Simply mark the pointers as NULL to fix this. This fix needs to be backported to 1.7 and 1.6. commit 30fd4bd8446dd7104b7d1cae9e762c7d1405171a Author: Willy Tarreau Date: Thu Dec 22 21:54:21 2016 +0100 BUG/MEDIUM: ssl: properly reset the reused_sess during a forced handshake We have a bug when SSL reuse is disabled on the server side : we reset the context but do not set it to NULL, causing a multiple free of the same entry. It seems like this bug cannot appear as-is with the current code (or the conditions to get it are not obvious) but it did definitely strike when trying to fix another bug with the SNI which forced a new handshake. This fix should be backported to 1.7, 1.6 and 1.5. commit 368780334cb1b4c46bcd666af353c177619f4774 Author: Willy Tarreau Date: Thu Dec 22 19:46:17 2016 +0100 MEDIUM: compression: move the zlib-specific stuff from global.h to compression.c This finishes to clean up the zlib-specific parts. It also unbreaks recent commit b97c6fb ("CLEANUP: compression: use the build options list to report the algos") which broke USE_ZLIB due to MAXWBITS not being defined anymore in haproxy.c. commit 14e36a101c40617cf9adaa1bec051ea197aa1c40 Author: Willy Tarreau Date: Wed Dec 21 23:28:13 2016 +0100 MEDIUM: cfgparse: move ssl-dh-param-file parsing to ssl_sock This one was missing an arg count check which was added in the operation. commit f22e9683e9bf5f7eb73ed89d21257946d4c7b4cc Author: Willy Tarreau Date: Wed Dec 21 23:23:19 2016 +0100 MINOR: cfgparse: move parsing of ssl-default-{bind,server}-ciphers to ssl_sock These ones are pretty similar, just an strdup. Contrary to ca-base and crt-base they support being changed. commit 0bea58d641ea6b5ae76d957faf1de4edd7127161 Author: Willy Tarreau Date: Wed Dec 21 23:17:25 2016 +0100 MEDIUM: cfgparse: move maxsslconn parsing to ssl_sock This one simply reuses the existing integer parser. It implicitly adds a control against negative numbers. commit 9ceda384e931500adf4c2f79b6d02c7afe33af50 Author: Willy Tarreau Date: Wed Dec 21 23:13:03 2016 +0100 MEDIUM: cfgparse: move all tune.ssl.* keywords to ssl_sock The following keywords were still parsed in cfgparse and were moved to ssl_sock to remove some #ifdefs : "tune.ssl.cachesize", "tune.ssl.default-dh-param", "tune.ssl.force-private-cache", "tune.ssl.lifetime", "tune.ssl.maxrecord", "tune.ssl.ssl-ctx-cache-size". It's worth mentionning that some of them used to have incorrect sign checks possibly resulting in some negative values being used. All of them are now checked for being positive. commit 8c3b0fd273bc26a8ef71b53052f7a295431adf0b Author: Willy Tarreau Date: Wed Dec 21 22:44:46 2016 +0100 MINOR: cfgparse: move parsing of "ca-base" and "crt-base" to ssl_sock This removes 2 #ifdefs and makes the code much cleaner. The controls are still there and the two parsers have been merged into a single function ssl_parse_global_ca_crt_base(). It's worth noting that there's still a check to prevent a change when the value was already specified. This test seems useless and possibly counter-productive, it may have to be revisited later, but for now it was implemented identically. commit ece9b07c7190b58dbaf1bc36c695a2ca8e4ec0cc Author: Willy Tarreau Date: Wed Dec 21 22:41:44 2016 +0100 MINOR: cfgparse: add two new functions to check arguments count We already had alertif_too_many_args{,_idx}(), but these ones are specifically designed for use in cfgparse. Outside of it we're trying to avoid calling Alert() all the time so we need an equivalent using a pointer to an error message. These new functions called too_many_args{,_idx)() do exactly this. They don't take the file name nor the line number which they have no use for but instead they take an optional pointer to an error message and the pointer to the error code is optional as well. With (NULL, NULL) they'll simply check the validity and return a verdict. They are quite convenient for use in isolated keyword parsers. These two new functions as well as the previous ones have all been exported. commit bee9dde31fa34a36fb1b6701042658295b744259 Author: Willy Tarreau Date: Wed Dec 21 21:25:06 2016 +0100 CLEANUP: da: move global settings out of the global section We replaced global.deviceatlas with global_deviceatlas since there's no need to store all this into the global section. This removes the last #ifdefs, and now the code is 100% self-contained in da.c. The file da.h was now removed because it was only used to load dac.h, which is more easily loaded directly from da.c. It provides another good example of how to integrate code in the future without touching the core parts. commit b7a671477fcef9d53baea026af1b198f60106d8e Author: Willy Tarreau Date: Wed Dec 21 21:18:44 2016 +0100 CLEANUP: 51d: move global settings out of the global section We replaced global._51degrees with global_51degrees since there's no need to store all this into the global section. This removes the last #ifdefs, and now the code is 100% self-contained in 51d.c. The file 51d.h was now removed because it was only used to load 51Degrees.h, which is more easily loaded from 51d.c. It provides a good example of how to integrate code in the future without touching the core parts. commit 350c1c6886f102100e86731840ae249bcdf49687 Author: Willy Tarreau Date: Wed Dec 21 14:57:34 2016 +0100 CLEANUP: wurfl: move global settings out of the global section We replaced global.wurfl with global_wurfl since there's no need to store all this into the global section. This removes the last #ifdefs, and now the code is 100% self-contained in wurfl.c. It provides a good example of how to integrate code in the future without touching the core parts. commit b149eedd5aa4b770f940d608865672b0d418f526 Author: Willy Tarreau Date: Wed Dec 21 21:03:49 2016 +0100 CLEANUP: da: register the deinitialization function deinit_deviceatlas() is not called anymore from haproxy.c, removing 2 still includes other parts of the Deviceatlas library so it was not touched. commit 7ac4c205098f6fe2358607c5ea738773525b9d3d Author: Willy Tarreau Date: Wed Dec 21 20:59:01 2016 +0100 CLEANUP: 51d: register the deinitialization function deinit_51degrees() is not called anymore from haproxy.c, removing 2 #ifdefs and one include. The function was made static. The include file still includes 51Degrees.h which is needed by global.h and 51d.c so it was not touched beyond this last function removal. commit 800f93f3751db98f0d147fa773c97f07d6f471f9 Author: Willy Tarreau Date: Wed Dec 21 20:52:38 2016 +0100 CLEANUP: wurfl: register the deinit function via the dedicated list By registering the deinit function we avoid another #ifdef in haproxy.c. The ha_wurfl_deinit() function has been made static and unexported. Now proto/wurfl.h is totally empty, the code being self-contained in wurfl.c, so the useless .h has been removed. commit 05554e6bf187584fe653adf4791de3181b5a5c9c Author: Willy Tarreau Date: Wed Dec 21 20:46:26 2016 +0100 MINOR: haproxy: add a registration for post-deinit functions The 3 device detection engines stop at the same place in deinit() with the usual #ifdefs. Similar to the other functions we can have some late deinitialization functions. These functions do not return anything however so we have to use a different type. commit 876054df96a15e820f7fa91d8d0f20c8f56e337b Author: Willy Tarreau Date: Wed Dec 21 20:39:16 2016 +0100 CLEANUP: da: make use of the late init registration code Instead of having a #ifdef in the main init code we now use the registered init functions. Doing so also enables error checking as errors were previously reported as alerts but ignored. Also they were incorrect as the 'status' variable was hidden by a second one and was always reporting DA_SYS (which is apparently an error) in every case including the case where no file was loaded. The init_deviceatlas() function was unexported since it's not used outside of this place anymore. commit 9f3f2549fb023215ba1e0582ed97a9f373d13721 Author: Willy Tarreau Date: Wed Dec 21 20:30:05 2016 +0100 CLEANUP: 51d: make use of the late init registration This removes some #ifdefs from the main haproxy code path. Function init_51degrees() now returns ERR_* instead of exit(1) on error, and this function was made static and is not exported anymore. commit dc2ed471633e0a0dd2f2285232d018641ef56b51 Author: Willy Tarreau Date: Wed Dec 21 20:20:17 2016 +0100 CLEANUP: wurfl: make use of the late init registration This removes some #ifdefs from the main haproxy code path and enables error checking. The current code only makes use of warnings even for some errors that look serious. While this choice is questionnable, it has been kept as-is, and only the return codes were adapted to ERR_WARN to at least report that some warnings were emitted. ha_wurfl_init() was unexported as it's not needed anymore. commit 64bca599d984be67949a199d01ddfefec96a57df Author: Willy Tarreau Date: Wed Dec 21 20:13:11 2016 +0100 CLEANUP: filters: use the function registration to initialize all proxies Function flt_init() was called in the main init code path, now we move it to the list of initializers and we can unexport flt_init(). commit 865c5148e6dbb8fc228a4b7e438f2fe5e3da00f3 Author: Willy Tarreau Date: Wed Dec 21 20:04:48 2016 +0100 CLEANUP: checks: make use of the post-init registration to start checks Instead of calling the checks directly from the init code, we now register the start_checks() function to be run at this point. This also allows to unexport the check init function and to remove one include from haproxy.c. commit e694573fa03480da5312313b9ce8a402ee5762e8 Author: Willy Tarreau Date: Wed Dec 21 19:57:00 2016 +0100 MINOR: haproxy: add a registration for post-check functions There's a significant amount of late initialization calls which are performed after the point where we exit in check mode. These calls are used to allocate resource and perform certain slow operations. Let's have a way to register some functions which need to be called there instead of having this multitude of #ifdef in the init path. commit e8692b41e5fb5a658fad3f0c81d6e2babb91eeb7 Author: Willy Tarreau Date: Wed Dec 21 19:36:25 2016 +0100 CLEANUP: auth: use the build options list to report its support This removes 1 #ifdef from haproxy.c. commit b97c6fb59e124c352bcc300140fe72a8685dd657 Author: Willy Tarreau Date: Wed Dec 21 19:30:30 2016 +0100 CLEANUP: compression: use the build options list to report the algos This removes 2 #ifdef, an include, an ugly construct and a wild "extern" declaration from haproxy.c. The message indicating that compression is *not* enabled is not there anymore. commit c2c0b612746cc2c85b29f865f270f60767c1307a Author: Willy Tarreau Date: Wed Dec 21 19:23:20 2016 +0100 CLEANUP: ssl: use the build options list to report the SSL details This removes 7 #ifdef from haproxy.c. The message indicating that openssl is *not* enabled is not there anymore. commit 7a9ac6dac6709707fb774af0fd9b83ed1717415a Author: Willy Tarreau Date: Wed Dec 21 19:13:14 2016 +0100 CLEANUP: regex: use the build options list to report the regex type This removes 3 #ifdef from haproxy.c. commit bb57d94a9629c7a1723c88f9173fe7c32ddf98de Author: Willy Tarreau Date: Wed Dec 21 19:04:56 2016 +0100 CLEANUP: lua: use the build options list to report it This removes 1 #ifdef from haproxy.c. The "build without" version is not reported anymore now. commit ba9629160054cc67a3044fc5e35be61c058430cb Author: Willy Tarreau Date: Wed Dec 21 18:55:02 2016 +0100 CLEANUP: tcp: use the build options list to report transparent modes This removes 6 #ifdef from haproxy.c. commit dba5002c4c1488266058c390e75635915cdb88d4 Author: Willy Tarreau Date: Wed Dec 21 18:51:45 2016 +0100 CLEANUP: namespaces: use the build options list to report it This removes one #ifdef from haproxy.c. commit 3dd483e727aa863fd4535b015dd61262a01ac0a9 Author: Willy Tarreau Date: Wed Dec 21 18:50:22 2016 +0100 CLEANUP: da: use the build options list to report it This removes one #ifdef from haproxy.c. commit b5e58d6ba1714e08d7470ce5f8a57de8450c6359 Author: Willy Tarreau Date: Wed Dec 21 18:49:29 2016 +0100 CLEANUP: 51d: use the build options list to report it This removes one #ifdef from haproxy.c. commit 770042d3c642292947545696cb5a16c0df839ef4 Author: Willy Tarreau Date: Wed Dec 21 18:47:13 2016 +0100 CLEANUP: wurfl: use the build options list to report it This removes one #ifdef from haproxy.c. commit cdb737e5a2ac74f481f380cb1bcbb9fe6d733538 Author: Willy Tarreau Date: Wed Dec 21 18:43:10 2016 +0100 MINOR: haproxy: add a registration for build options Many extensions now report some build options to ease debugging, but this is now being done at the expense of code maintainability. Let's provide a registration function to do this so that we can start to remove most of the #ifdefs from haproxy.c (18 currently just for a single function). commit 1b5af7cd4242066b4a04cbbb0280ec2582e16daa Author: Willy Tarreau Date: Wed Dec 21 18:19:57 2016 +0100 CLEANUP: haproxy: statify unexported functions haproxy.c is a real mess. Let's start to clean it up by declaring static all functions which are not exported (ie almost all of them). commit 2c8b54e7be1aab92da91a15fa5b39c4ddca3eb1b Author: Thierry FOURNIER Date: Sat Dec 17 12:45:32 2016 +0100 MEDIUM: lua: remove Lua struct from session, and allocate it with memory pools This patch use memory pools for allocating the Lua struct. This save 128B of memory in the session if the Lua is unused. commit 1be34152da64cad9f4737ddb17d51a6003e850f2 Author: Thierry FOURNIER Date: Sat Dec 17 12:09:51 2016 +0100 BUG/MINOR: lua: memleak when Lua/cli fails If the memory allocator fails, it return a bad code, and the execution continue. If the Lua/cli initializer fails, the allocated struct is not released. commit 33558c4a3f2521bb33fa71ebb312babbf1361c13 Author: Thierry FOURNIER Date: Sat Dec 17 12:05:56 2016 +0100 BUG/MINOR: lua: bad return code If the lua/cli fails during initialization, it returns an ok status, an the execution continue. This will probably occur a segfault. Thiw patch should be backported in 1.7 commit 4e7c708612730d79f4cc2ec7617fce01665ff807 Author: Thierry FOURNIER Date: Sat Dec 17 11:46:06 2016 +0100 BUG/MINOR: lua: memory leak executing tasks The struct hlua isn't freed when the task is complete. This patch should be backported in 1.6 and 1.7 commit 33834b15dce4658ee98fa85668162d78abf32e18 Author: Christopher Faulet Date: Mon Dec 19 09:29:06 2016 +0100 BUG/MINOR: Fix the sending function in Lua's cosocket This is a regression from the commit a73e59b6901a164d19b1145e8511602d9814f28f. When data are sent from a cosocket, the action is done in the context of the applet running a lua stack and not in the context of the applet owning the cosocket. So we must take care, explicitly, that this last applet have a buffer (the req buffer of the cosocket). This patch must be backported in 1.7 commit 6e413ed1ed33e65740bfa1f209383252eb472256 Author: Marcin Deranek Date: Tue Dec 13 12:40:01 2016 +0100 DOC: fix small typo in fe_id (backend instead of frontend) Needs to be backported to 1.7 and 1.6 at least. commit 3b0a6d480b7e875741e68f4c725b21738186b50b Author: Thierry FOURNIER Date: Fri Dec 16 08:48:32 2016 +0100 MINOR/DOC: lua: just precise one thing In the case of applet, the Lua context is taken from session when we get the private values. This patch just update comments associated to this action because it is not obvious. commit 5cd60670a6bc03b6895611c2ed46b56d6cf60354 Author: Willy Tarreau Date: Fri Dec 16 08:02:21 2016 +0100 CONTRIB: tcploop: add limits.h to fix build issue with some compilers Just got this while cross-compiling : tcploop.c: In function 'tcp_recv': tcploop.c:444:48: error: 'INT_MAX' undeclared (first use in this function) tcploop.c:444:48: note: each undeclared identifier is reported only once for each function it appears in commit f5f26e824a45c28110ad7e84306565fbf3df5e84 Author: Willy Tarreau Date: Fri Dec 16 18:47:27 2016 +0100 MINOR: appctx/cli: remove the "tlskeys" entry from the appctx union This one now migrates to the general purpose cli.p0 for the ref pointer, cli.i0 for the dump_all flag and cli.i1 for the dump_keys_index. A few comments were added. The applet.h file doesn't depend on openssl anymore. It's worth noting that the previous dependency was accidental and only used to work because all files including this one used to have openssl included prior to loading this file. commit 3c92f2aca45683c07b75dc19a133a744b1a59781 Author: Willy Tarreau Date: Fri Dec 16 18:23:39 2016 +0100 MINOR: appctx/cli: remove the "server_state" entry from the appctx union This one now migrates to the general purpose cli.p0 for the proxy pointer, cli.p1 for the server pointer, and cli.i0 for the proxy's instance if only one has to be dumped. commit 777b560d04e90743236d293b4a9e7bab58b5046d Author: Willy Tarreau Date: Fri Dec 16 18:06:26 2016 +0100 MINOR: appctx/cli: remove the "dns" entry from the appctx union This one now migrates to the general purpose cli.p0. commit 608ea5921abc82eabf6796b168b4b05baa948072 Author: Willy Tarreau Date: Fri Dec 16 18:01:15 2016 +0100 MINOR: appctx/cli: remove the "be" entry from the appctx union This one now migrates to the general purpose cli.p0. The parsing function was removed since it was only used to set the pointer to NULL. commit f6710f8811cb1dc17e1b94cdfb77150f3660c112 Author: Willy Tarreau Date: Fri Dec 16 17:45:44 2016 +0100 MINOR: appctx/cli: remove the env entry from the appctx union This one now migrates to the general purpose cli.p0. commit 3af9d832e8b6848ce538f52b317d0d14ca9dc09e Author: Willy Tarreau Date: Fri Dec 16 12:58:09 2016 +0100 MINOR: appctx/cli: remove the cli_socket entry from the appctx union This one now migrates to the general purpose cli.p0. commit a2d58722976c365a1beb2ef01c977f3cd15b4fb4 Author: Willy Tarreau Date: Fri Dec 16 12:37:03 2016 +0100 MINOR: cli: add two general purpose pointers and integers in the CLI struct Most of the keywords don't need to have their own entry in the appctx union, they just need to reuse some generic pointers like we've been used to do in the appctx with st{0,1,2}. This patch adds p0, p1, i0, i1 and initializes them to zero before calling the parser. This way some of the simplest existing keywords will be able to disappear from the union. It's worth noting that this is an extension to what was initially attempted via the "private" member that I removed a few patches ago by not understanding how it was supposed to be used. Here the fact that we share the same union will force us to be stricter: the code either uses the general purpose variables or it uses its own fields but not both. commit d25fc79d72ad5fa33217cc2f1c232647d456b60c Author: Willy Tarreau Date: Fri Dec 16 12:33:47 2016 +0100 CLEANUP: stats: move a misplaced stats context initialization This is a leftover from the cleanup campaign, the stats scope was still initialized by the CLI instead of being initialized by the stats keyword parsers. This should probably be backported to 1.7 to make the code more consistent. commit ebec3519b00eac6eb38558933db226af1d27e1ad Author: Willy Tarreau Date: Fri Dec 16 12:14:12 2016 +0100 CLEANUP: applet: group all CLI contexts together The appctx storage became a real mess along the years. It now contains mostly CLI-specific parts that share the same storage as the "cli" part which in fact only contains the fields needed to pass an error message to the caller, and it also has room a few other regular applets which may become more and more common. This first patch moves the parts around in the union so that all standard applet parts are grouped together and the CLI-specific ones are grouped together. It also adds a few comments to indicate what certain parts are used for since it's sometimes a bit confusing. commit e9ecec8935da4c003f786381b7bba0a6b91f861b Author: Willy Tarreau Date: Fri Dec 16 18:55:23 2016 +0100 CLEANUP: memory: remove the now unused cli_parse_show_pools() function We don't need this empty parser anymore since previous commit. commit eaffde38c8667668b0ffa76694cb78464a5b4f0b Author: Willy Tarreau Date: Fri Dec 16 17:59:25 2016 +0100 MINOR: cli: automatically enable a CLI I/O handler when there's no parser Sometimes a registered keyword will not need any specific parsing nor initialization, so it's annoying to have to write an empty parsing function returning zero just for this. This patch makes it possible to automatically call a keyword's I/O handler of when the parsing function is not defined, while still allowing a parser to set the I/O handler itself. commit 511321658317b079eee5c2e380cc904959a53653 Author: William Lallemand Date: Fri Dec 16 16:38:58 2016 +0100 DOC: cli: show cli sockets Add documentation for 'show cli sockets' on the CLI. commit 847ca66815c52c60ccc43e1fa6ee48e594e918ce Author: Thierry FOURNIER Date: Fri Dec 16 13:07:22 2016 +0100 MINOR: lua/signals: Remove Lua part from signals. The signals system embedded in Lua can be tranformed in general purpose signals code. To reach this goal, this path removes the Lua part of the signals. This is an easy job, because Lua is useles with signal. I change just two prototypes. commit ebed6e908a7dc7e9dbf12cefe81951e467156a28 Author: Thierry FOURNIER Date: Fri Dec 16 11:54:07 2016 +0100 MEDIUM: lua: use memory pool for hlua struct in applets The struct hlua size is 128 bytes. The size is the biggest of all the elements of the union embedded in the appctx struct. With HTTP2, it is possible that this appctx struct will be use many times for each connection, so the 128 bytes are a little bit heavy for the global memory consomation. This patch replace the embbeded hlua struct by a pointer and an associated memory pool. Now, the memory for lua is allocated only if it is required. [wt: the appctx is now down to 160 bytes] commit ffbf569edba2a10a3d2316cb6d7435944f591cce Author: Thierry FOURNIER Date: Fri Dec 16 11:14:06 2016 +0100 BUG/MINOR: lua/cli: bad error message Error message inherited from lua_appelet_tcp copy/paste. Should be backported in 1.7 commit 18d0990a5def998a7bebe8f3b42af50faddad558 Author: Thierry FOURNIER Date: Fri Dec 16 09:25:38 2016 +0100 CLEANUP: lua: rename one of the lua appctx union It is named hlua, which does not represent the usage of this variable. this patch renames this one to "hlua_cosocket". commit 4305ac7f1d49ed998a4f227e8cecd867c547da48 Author: Willy Tarreau Date: Fri Dec 16 12:56:31 2016 +0100 BUG/MINOR: cli: "show cli sockets" would always report process 64 Another small bug in "show cli sockets" made the last fix always report process 64 due to a signedness issue in the shift operation when building the mask. commit 20c5e52ac773a83e74442a0abe1cd717f34a0a69 Author: Willy Tarreau Date: Fri Dec 16 12:50:55 2016 +0100 BUG/MINOR: cli: "show cli sockets" wouldn't list all processes A small bug in "show cli sockets" made it limit the output to the first 8 processes only. commit eceddf72253600b3a8360e0c399ed3888709b06b Author: William Lallemand Date: Thu Dec 15 18:06:44 2016 +0100 MEDIUM: cli: 'show cli sockets' list the CLI sockets 'show cli sockets' from the CLI socket displays the list of CLI sockets available, with their level and process number. commit d7f2eb6f7eb7218b9f85fb59e24d1d414d6b9fae Author: Thierry FOURNIER Date: Wed Dec 14 19:43:08 2016 +0100 DOC: lua: section declared twice This patch remove the second section. This patch should be backported in versions 1.6 and 1.7 commit 12a865dc24ef111650e47593400baea6e7119b28 Author: Thierry FOURNIER Date: Wed Dec 14 19:40:37 2016 +0100 DOC: lua: improve links Sphinx provide a method for generating hyperlink between some references. This patch uses these methods for internal links. commit a78f037505c3176aef3ce80f6aaac2fdcc92ca27 Author: Thierry FOURNIER Date: Wed Dec 14 19:04:41 2016 +0100 DOC: lua: documentation about time parser functions This patch must be backported in version 1.7 commit 90f19e3c6806e70193434772b0a1820f1593b2d2 Author: Willy Tarreau Date: Wed Dec 14 15:58:39 2016 +0100 CLEANUP: applet: remove the now unused appctx->private field Nobody uses it anymore. This shrinks the appctx size by 8 bytes. commit a24bc78ad4148cf4ce6d44c7e9771298386253af Author: Willy Tarreau Date: Wed Dec 14 15:50:35 2016 +0100 CLEANUP: applet/table: add an "action" entry in ->table context Just like previous patch, this was the only other user of the "private" field of the applet. It used to store a copy of the keyword's action. Let's just put it into ->table->action and use it from there. It also slightly simplifies the code by removing a few pointer to integer casts. commit 8ae4f7533d5dfe01c2fa15c4b1b803bc3aa7beaf Author: Willy Tarreau Date: Wed Dec 14 15:41:45 2016 +0100 CLEANUP: applet/lua: create a dedicated ->fcn entry in hlua_cli context We have very few users of the appctx's private field which was introduced prior to the split of the CLI. Unfortunately it was not removed after the end. This commit simply introduces hlua_cli->fcn which is the pointer to the Lua function that the Lua code used to store in this private pointer. commit 8cf9c8e663fa468fa2380eddecd671172ff63868 Author: Willy Tarreau Date: Tue Dec 13 15:21:25 2016 +0100 BUG/MINOR: stream-int: automatically release SI_FL_WAIT_DATA on SHUTW_NOW While developing an experimental applet performing only one read per full line, it appeared that it would be woken up for the client's close, not read all data (missing LF), then wait for a subsequent call, and would only be woken up on client timeout to finish the read. The reason is that we preset SI_FL_WAIT_DATA in the stream-interface's flags to avoid a fast loop, but there's nothing which can remove this flag until there's a read operation. We must definitely remove it in stream_int_notify() each time we're called with CF_SHUTW_NOW because we know there will be no more subsequent read and we don't want an applet which keeps the WANT_GET flag to block on this. This fix should be backported to 1.7 and 1.6 though it's uncertain whether cli, peers, lua or spoe really are affected there. commit 5e637e556d12cd4d8b97b4896448ef6f3396224b Author: Willy Tarreau Date: Wed Dec 14 16:44:45 2016 +0100 SCRIPTS: git-show-backports: add -H to use the hash of the commit message Sometimes certain commits don't contain useful tracking information but we'd still like to be able to report them. Here we implement a hash on the author's name, e-mail and date, the subject and the body before the first s-o-b or cherry-picked line. These parts are supposed to be reasonable invariant across backports and are usable to compute an invariant hash of a given commit. When we don't find ancestry in a commit, we try this method (if -H is specified) to compare commits hashes and we can report a match. The equivalent commit is reported as "XXXX+?" to indicate that it's an apparent backport but we don't know how far it goes. commit b684cd4642186fd64c9335c316aeca16cc87d9de Author: Willy Tarreau Date: Wed Dec 14 16:43:23 2016 +0100 SCRIPTS: git-show-backports: fix a harmless typo There was a double output redirection in this script while dumping the current branch's refs which could cause either an error or an empty file. commit 11cfb3daecd789416103837001e30e9644b4c722 Author: Thierry FOURNIER Date: Tue Dec 13 13:06:23 2016 +0100 BUG/MEDIUM: lua: In some case, the return of sample-fetches is ignored (2) This problem is already detected here: 8dc7316a6fa8cc6f3a60456376c8a13a6902a5be Another case raises. Now HAProxy sends a final message (typically with "http-request deny"). Once the the message is sent, the response channel flags are not modified. HAProxy executes a Lua sample-fecthes for building logs, and the result is ignored because the response flag remains set to the value HTTP_MSG_RPBEFORE. So the Lua function hlua_check_proto() want to guarantee the valid state of the buffer and ask for aborting the request. The function check_proto() is not the good way to ensure request consistency. The real question is not "Are the message valid ?", but "Are the validity of message unchanged ?" This patch memorize the parser state before entering int the Lua code, and perform a check when it go out of the Lua code. If the parser state change for down, the request is aborted because the HTTP message is degraded. This patch should be backported in version 1.6 and 1.7 commit 578b169dcb653feb74d827238b879217b7220b89 Author: Luca Pizzamiglio Date: Mon Dec 12 10:56:56 2016 +0100 BUILD/MEDIUM: Fixing the build using LibreSSL Fixing the build using LibreSSL as OpenSSL implementation. Currently, LibreSSL 2.4.4 provides the same API of OpenSSL 1.0.1x, but it redefine the OpenSSL version number as 2.0.x, breaking all checks with OpenSSL 1.1.x. The patch solves the issue checking the definition of the symbol LIBRESSL_VERSION_NUMBER when Openssl 1.1.x features are requested. commit a73e59b6901a164d19b1145e8511602d9814f28f Author: Christopher Faulet Date: Fri Dec 9 17:30:18 2016 +0100 BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called ) and wait for an available buffer. Historically, the list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in , instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, is independant from streams. And we really add the waiting entity in . To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6] commit 9d810cae111991f1cd38d6ad37b4046c508884b8 Author: Christopher Faulet Date: Thu Dec 8 22:33:52 2016 +0100 BUG/MEDIUM: stream: Save unprocessed events for a stream A stream can be awakened for different reasons. During its processing, it can be early stopped if no buffer is available. In this situation, the reason why the stream was awakened is lost, because we rely on the task state, which is reset after each processing loop. In many cases, that's not a big deal. But it can be useful to accumulate the task states if the stream processing is interrupted, especially if some filters need to be called. To be clearer, here is an simple example: 1) A stream is awakened with the reason TASK_WOKEN_MSG. 2) Because no buffer is available, the processing is interrupted, the stream is back to sleep. And the task state is reset. 3) Some buffers become available, so the stream is awakened with the reason TASK_WOKEN_RES. At this step, the previous reason (TASK_WOKEN_MSG) is lost. Now, the task states are saved for a stream and reset only when the stream processing is not interrupted. The correspoing bitfield represents the pending events for a stream. And we use this one instead of the task state during the stream processing. Note that TASK_WOKEN_TIMER and TASK_WOKEN_RES are always removed because these events are always handled during the stream processing. [wt: backport to 1.7 and 1.6] commit 34c5cc98da7cad2c7e0733a7b86c8369d1dea5bd Author: Christopher Faulet Date: Tue Dec 6 09:15:30 2016 +0100 MINOR: task: Rename run_queue and run_queue_cur counters is used to track the number of task in the run queue and is a copy used for the reporting purpose. These counters has been renamed, respectively, and . So the naming is consistent between tasks and applets. [wt: needed for next fixes, backport to 1.7 and 1.6] commit 1cbe74cd837b460d59b31a534d20f1ffd980b8f0 Author: Christopher Faulet Date: Tue Dec 6 09:13:22 2016 +0100 MINOR: applet: Count number of (active) applets As for tasks, 2 counters has been added to track : * the total number of applets : nb_applets * the number of active applets : applets_active_queue [wt: needed for next fixes, to backport to 1.7 and 1.6] commit 90b5abe46ea507ae7831221555f5183d16864d4a Author: Christopher Faulet Date: Mon Dec 5 14:25:08 2016 +0100 BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is full [wt: may only strike if CLI commands are pipelined. Must be backported to 1.7 and 1.6, where it's a bit different and in dumpstats.c] commit 1821d3c25e80e1f687dc809901094b49aa670d98 Author: Christopher Faulet Date: Mon Dec 5 14:23:37 2016 +0100 MINOR: cli: Remove useless call to bi_putchk [wt: while it could seem suspicious, the preceeding call to dump_servers_state() indeed flushes the trash in case anything is emitted. No backport needed though.] commit 1002aac161df5d418d18f8b16f38dfbcd2fa71a7 Author: Christopher Faulet Date: Fri Dec 9 17:41:54 2016 +0100 DOC: Fix some typo in SPOE documentation [wt: backport to 1.7] commit 31bfe1f262c218574686ffdabef19cb40ae65ac3 Author: Christopher Faulet Date: Fri Dec 9 17:42:38 2016 +0100 DOC: Add undocumented argument of the trace filter [wt: to backport to 1.7] commit 43ad11dc75d2f6513f1d884c031e789b419b3b72 Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 15:19:58 2016 +0100 MINOR: Do not forward the header "Expect: 100-continue" when the option http-buffer-request is set When the option "http-buffer-request" is set, HAProxy send itself the "HTTP/1.1 100 Continue" response in order to retrieve the post content. When HAProxy forward the request, it send the body directly after the headers. The header "Expect: 100-continue" was sent with the headers. This header is useless because the body will be sent in all cases, and the server reponse is not removed by haproxy. This patch removes the header "Expect: 100-continue" if HAProxy sent it itself. commit c1edafe4a9ca1c1f6daaac0b83d80d24aefda9c0 Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 16:25:30 2016 +0100 DOC: lua: Add documentation about variable manipulation from applet This patch adds documentation about set_var, unset_var and get_var functions added in the Class AppletHTTP and AppletTCP. commit b210bcc559ffcbe691e9bd0eb9e0c91a94574045 Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 16:24:16 2016 +0100 DOC: lua: Documentation about some entry missing The parameter "value" of the function TXN.set_var() was not documented. This is a regression from the commit 85d79c94a93211b156ec644e164b178f5d7101fa. This patch must be backported in 1.7 commit d2471c2bdc98e2c014624ecff9747528397aee57 Author: Marcin Deranek Date: Mon Dec 12 14:08:05 2016 +0100 MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id These 2 patches add ability to fetch frontend/backend name in your logic, so they can be used later to make routing decisions (fe_name) or taking some actions based on backend which responded to request (be_name). In our case we needed a fetcher to be able to extract information we needed from frontend name. commit 8e0f17543e4d6d5b1e8b87e05341d0d9fb53c203 Author: Willy Tarreau Date: Mon Dec 12 15:07:29 2016 +0100 BUG/MINOR: stats: fix be/sessions/max output in html stats "Tadas / XtGem" reported that the max value was wrong and would report the current value instead. This needs to be backported to 1.7. commit 4394a2cc87f41fd7623d44b0dca99c7629527c30 Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 12:31:54 2016 +0100 MINOR: lua: give HAProxy variable access to the applets This patch give function for manipulating variables inside the applet HTTP and applet TCP functions. commit 3e1d791a4a97bdecf03e6553ae97c48276e13d0f Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 12:29:34 2016 +0100 CLEANUP: hlua: just indent functions Function indentation. The code is not modified. This is done in the goal of better integration of the next patch commit a9cf315b00bed165cac3c6393cb965a9bf12f316 Author: Willy Tarreau Date: Sun Dec 11 22:12:33 2016 +0100 BUILD: rearrange target files by build time When doing a parallel build on multiple CPUs it's common that at the end a few CPUs only are busy compiling very large files while the other ones have finished. By placing the largest files first, we can ensure that in the worst case they are present from the beginning to the end, and that other processes are free to take smaller files. This ordering was made based on a measurement consisting in counting the number of times a given file appears in the build. The top ten looks like this : 145 src/cfgparse.c 131 src/proto_http.c 83 src/ssl_sock.c 74 src/stats.c 73 src/stream.c 55 src/flt_spoe.c 48 src/server.c 46 src/pattern.c 43 src/checks.c 42 src/flt_http_comp.c Only a few files were moved, ssl_sock would need to be moved as well but that would not be a convenient thing to do in the makefile. This new order allows to save about 10-15% of build time on 4 CPUs, which is nice. commit 4b123bebe4c22d62dde88ce89dc46fdffc9ba4d2 Author: Thierry FOURNIER / OZON.IO Date: Fri Dec 9 18:03:31 2016 +0100 MINOR: lua: Allow argument for actions (http|tcp)-(request|response) action cannot take arguments from the configuration file. Arguments are useful for executing the action with a special context. This patch adds the possibility of passing arguments to an action. It runs exactly like sample fetches and other Lua wrappers. Note that this patch implements a 'TODO'. commit d2f6f47597844e07c393bf9224c99bc31538512c Author: Thierry FOURNIER / OZON.IO Date: Mon Dec 12 12:42:14 2016 +0100 BUG/MEDIUM: variables: some variable name can hide another ones The variable are compared only using text, the final '\0' (or the string length) are not checked. So, the variable name "txn.internal" matchs other one call "txn.int". This patch fix this behavior It must be backported ni 1.6 and 1.7 commit fb981bd7d1ca8c23097b0e70f3d4169820abd04e Author: Willy Tarreau Date: Mon Dec 12 14:31:46 2016 +0100 DOC: mention that req_tot is for both frontends and backends Thierry reported that only frontends were mentionned while the stats are for both. To be backported to 1.7. commit 35088f960de9e3331573d118adbbe956c7fbbf7d Author: Matthieu Guegan Date: Mon Dec 5 11:35:54 2016 +0100 BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect By investigating a keep-alive issue with CloudFlare, we[1] found that when using the 'set-cookie' option in a redirect (302) HAproxy is adding an extra `\r\n`. Triggering rule : `http-request redirect location / set-cookie Cookie=value if [...]` Expected result : ``` HTTP/1.1 302 Found Cache-Control: no-cache Content-length: 0 Location: / Set-Cookie: Cookie=value; path=/; Connection: close ``` Actual result : ``` HTTP/1.1 302 Found Cache-Control: no-cache Content-length: 0 Location: / Set-Cookie: Cookie=value; path=/; Connection: close ``` This extra `\r\n` seems to be harmless with another HAproxy instance in front of it (sanitizing) or when using a browser. But we confirm that the CloudFlare NGINX implementation is not able to handle this. It seems that both 'Content-length: 0' and extra carriage return broke RFC (to be confirmed). When looking into the code, this carriage-return was already present in 1.3.X versions but just before closing the connection which was ok I think. Then, with 1.4.X the keep-alive feature was added and this piece of code remains unchanged. [1] all credit for the bug finding goes to CloudFlare Support Team [wt: the bug was indeed present since the Set-Cookie was introduced in 1.3.16, by commit 0140f25 ("[MINOR] redirect: add support for "set-cookie" and "clear-cookie"") so backporting to all supported versions is desired] commit f25e8e56b4c591df1b2dfb2421fdcd94de31c41f Author: Ben Shillito Date: Fri Dec 2 14:25:37 2016 +0000 DOC: Added 51Degrees conv and fetch functions to documentation. Definitions and examples for 51d.single and 51d.all have been added to configuration.txt so it now appears in online documentation in addition to the README, The 51degrees-property-name-list entry has also been updated to make it clear that multiple properties can be added. commit 3c5e374142b173d7309f18220f126b11de9c2018 Author: Ruoshan Huang Date: Fri Dec 2 16:25:31 2016 +0800 DOC: Fix map table's format so the HTML doc parser can parse the table correctly commit 3067bfa815afce9217c5f7cf0812e2575583a2fa Author: Willy Tarreau Date: Mon Dec 5 14:50:15 2016 +0100 BUG/MEDIUM: cli: fix "show stat resolvers" and "show tls-keys" The recent CLI reorganization managed to break these two commands by having their parser return 1 (indicating an end of processing) instead of 0 to indicate new calls to the io handler were needed. Namely the faulty commits are : 69e9644 ("REORG: cli: move show stat resolvers to dns.c") 32af203 ("REORG: cli: move ssl CLI functions to ssl_sock.c") The fix is trivial and there is no other loss of functionality. Thanks to Dragan Dosen for reporting the issue and the faulty commits. The backport is needed in 1.7. commit a1c35ab68d59f877b54dab7f296d3fdd6edff6e9 Author: Dragan Dosen Date: Thu Nov 24 11:33:12 2016 +0100 BUG/MINOR: cli: allow the backslash to be escaped on the CLI In 1.5-dev20, commit 48bcfda ("MEDIUM: dumpstat: make the CLI parser understand the backslash as an escape char") introduced support for backslash on the CLI, but it strips all backslashes in all arguments instead of only unescaping them, making it impossible to pass a backslash in an argument. This will allow us to use a backslash in a command over the socket, eg. "add acl #0 ABC\\XYZ". [wt: this should be backported to 1.7 and 1.6] commit 796c5b79974f5b692da721385e849a56849b127d Author: Willy Tarreau Date: Tue Nov 29 21:47:02 2016 +0100 OPTIM: stream-int: don't disable polling anymore on DONT_READ Commit 5fddab0 ("OPTIM: stream_interface: disable reading when CF_READ_DONTWAIT is set") improved the connection layer's efficiency back in 1.5-dev13 by avoiding successive read attempts on an active FD. But by disabling this on a polled FD, it causes an unpleasant side effect which is that the FD that was subscribed to polling is suddenly stopped and may need to be re-enabled once the kernel starts to slow down on data eviction (eg: saturated server at the other end, bursty traffic caused by too large maxpollevents). This behaviour is observable with persistent connections when there is a large enough connection count so that there's no data in the early connection and polling is required, because there are then up to 4 epoll_ctl() calls per request. It's important that the server is slower than haproxy to cause some delays when reading response. The current connection layer as designed in 1.6 with the FD cache doesn't require this trick anymore, though it still benefits from it when it saves an FD from being uselessly polled. But compared to the increased cost of enabling and disabling poll all the time, it's still better to disable it. In some cases it's possible to observe a performance increase as high as 30% by avoiding this epoll_ctl() dance. In the end we only want to disable it when the FD is speculatively read and not when it's polled. For this we introduce a new function __conn_data_done_recv() which is used to indicate that we're done with recv() and not interested in new attempts. If/when we later support event-triggered epoll, this function will have to change a bit to do the same even in the polled case. A quick test with keep-alive requests run on a dual-core / dual- thread Atom shows a significant improvement : single process, 0 bytes : before: Requests per second: 12243.20 [#/sec] (mean) after: Requests per second: 13354.54 [#/sec] (mean) single process, 4k : before: Requests per second: 9639.81 [#/sec] (mean) after: Requests per second: 10991.89 [#/sec] (mean) dual process, 0 bytes (unstable) : before: Requests per second: 16900-19800 ~ 17600 [#/sec] (mean) after: Requests per second: 18600-21400 ~ 20500 [#/sec] (mean) commit 92b10c954defc82ceabdc9bba923320a157c664d Author: Willy Tarreau Date: Sun Dec 4 20:05:16 2016 +0100 BUG/MAJOR: stream: fix session abort on resource shortage In 1.6-dev2, commit 32990b5 ("MEDIUM: session: remove the task pointer from the session") introduced a bug which can sometimes crash the process on resource shortage. When stream_complete() returns -1, it has already reattached the connection to the stream, then kill_mini_session() is called and still expects to find the task in conn->owner. Note that since this commit, the code has moved a bit and is now in stream_new() but the problem remains the same. Given that we already know the task around these places, let's simply pass the task to kill_mini_session(). The conditions currently at risk are : - failure to initialize filters for the new stream (lack of memory or any filter returning < 0 on attach()) - failure to attach filters (any filter returning < 0 on stream_start()) - frontend's accept() returning < 0 (allocation failure) This fix is needed in 1.7 and 1.6. commit 6962f4e0d6dd1f1e976f815c1c8afaabfadc93fa Author: Christopher Faulet Date: Mon Nov 28 14:21:27 2016 +0100 BUG/MINOR: http: Call XFER_DATA analyzer when HTTP txn is switched in tunnel mode This allow a filter to start to analyze data in HTTP and to fallback in TCP when data are tunneled. [wt: backport desired in 1.7 - no impact right now but may impact the ability to backport future fixes] commit d47a1bd1d7311f70c368ccb016553a0d30fb123d Author: Christopher Faulet Date: Mon Nov 28 12:36:26 2016 +0100 BUG/MINOR: filters: Invert evaluation order of HTTP_XFER_BODY and XFER_DATA analyzers These 2 analyzers are responsible of the data forwarding in, respectively, HTTP mode and TCP mode. Now, the analyzer responsible of the HTTP data forwarding is called before the one responsible of the TCP data forwarding. This will allow the filtering of tunneled data in HTTP. [wt: backport desired in 1.7 - no impact right now but may impact the ability to backport future fixes] commit 4aad833924d9462ec8b00d26c9a831ebd4dbde14 Author: Christopher Faulet Date: Mon Nov 28 10:01:32 2016 +0100 BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER [wt: backport needed in 1.7] commit 3235957685f2eb4797a302a773a0f8776f068463 Author: Christopher Faulet Date: Tue Nov 29 14:49:54 2016 +0100 BUG/MINOR: http: Keep the same behavior between 1.6 and 1.7 for tunneled txn In HAProxy 1.6, When "http-tunnel" option is enabled, HTTP transactions are tunneled as soon as possible after the headers parsing/forwarding. When the transfer length of the response can be determined, this happens when all data are forwarded. But for responses with an undetermined transfer length this happens when headers are forwarded. This behavior is questionable, but this is not the purpose of this fix... In HAProxy 1.7, the first use-case works like in 1.6. But the second one not because of the data filtering. HAProxy was always trying to forward data until the server closes the connection. So the transaction was never switched in tunnel mode. This is the expected behavior when there is a data filter. But in the default case (no data filter), it should work like in 1.6. This patch fixes the bug. We analyze response data until the server closes the connection only when there is a data filter. [wt: backport needed in 1.7] commit d1cd209b21865c69761e4d5005a9f36423a38479 Author: Christopher Faulet Date: Mon Nov 28 10:14:03 2016 +0100 BUG/MEDIUM: http: Fix tunnel mode when the CONNECT method is used When a 2xx response to a CONNECT request is returned, the connection must be switched in tunnel mode immediatly after the headers, and Transfer-Encoding and Content-Length headers must be ignored. So from the HTTP parser point of view, there is no body. The bug comes from the fact the flag HTTP_MSGF_XFER_LEN was not set on the response (This flag means that the body size can be determined. In our case, it can, it is 0). So, during data forwarding, the connection was never switched in tunnel mode and we were blocked in a state where we were waiting that the server closes the connection to ends the response. Setting the flag HTTP_MSGF_XFER_LEN on the response fixed the bug. The code of http_wait_for_response has been slightly updated to be more readable. [wt: 1.7-only, this is not needed in 1.6] commit 4896c440b3e270a9acd41d2692ecc8bf5d7636fa Author: Tim Düsterhus Date: Tue Nov 29 02:15:19 2016 +0100 DOC: Spelling fixes [wt: this contains spelling fixes for both doc and code comments, should be backported, ignoring the parts which don't apply] commit ddc3e9e55de965432946c7d13c52756f71eb06b5 Author: Willy Tarreau Date: Sat Nov 26 15:55:59 2016 +0100 BUG/MINOR: stats: make field_str() return an empty string on NULL It already returns an empty string when the field is empty, but as a preventive measure we should do the same when the string itself is a NULL. While it is not supposed to happen, it will make the code more resistant against failed allocations and unexpected results. This fix should be backported to 1.7. commit b3e111b4fd99137b6b8c4d687282a497ed082111 Author: Willy Tarreau Date: Sat Nov 26 15:52:04 2016 +0100 BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos When a backend doesn't use any known LB algorithm, backend_lb_algo_str() returns NULL. It used to cause "nil" to be printed in the stats dump since version 1.4 but causes 1.7 to try to parse this NULL to encode it as a CSV string, causing a crash on "show stat" in this case. The only situation where this can happen is when "transparent" or "dispatch" are used in a proxy, in which case the LB algorithm is BE_LB_ALGO_NONE. Thus now we explicitly report "none" when this situation is detected, and we preventively report "unknown" if any unknown algorithm is detected, which may happen if such an algo is added in the future and the function is not updated. This fix must be backported to 1.7 and may be backported as far as 1.4, though it has less impact there. commit 0e658fb0518da7c3292da99a301e4405affec8fa Author: Willy Tarreau Date: Fri Nov 25 16:55:50 2016 +0100 [RELEASE] Released version 1.8-dev0 Released version 1.8-dev0 with the following main changes : - exact copy of 1.7.0 --- CHANGELOG | 52 ++++++++++++++++ Makefile | 19 +----- README | 2 +- VERDATE | 2 +- VERSION | 2 +- doc/configuration.txt | 103 +++++++++++++++++++------------ examples/haproxy.spec | 8 ++- include/common/xref.h | 5 ++ include/proto/session.h | 3 +- include/proto/stream.h | 6 +- include/proto/stream_interface.h | 6 ++ include/types/applet.h | 41 +++++++++++- include/types/connection.h | 1 + include/types/global.h | 40 ------------ include/types/proxy.h | 4 -- include/types/stream_interface.h | 36 ++++++----- scripts/announce-release | 2 + src/backend.c | 18 +++--- src/cfgparse.c | 68 +++++++++++++++++--- src/checks.c | 7 ++- src/flt_spoe.c | 87 ++++++++++++++++++-------- src/haproxy.c | 47 ++++++++------ src/hlua.c | 6 +- src/mux_h2.c | 66 ++++++++++++++------ src/proto_http.c | 5 +- src/sample.c | 7 +++ src/server.c | 48 +++++++------- src/ssl_sock.c | 51 +++++++++++++++ src/stream.c | 6 +- src/stream_interface.c | 8 +++ src/tcp_rules.c | 3 + 31 files changed, 517 insertions(+), 242 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 6a8554b00..d489531bc 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,6 +1,58 @@ ChangeLog : =========== +2019/02/11 : 1.8.19 + - DOC: ssl: Clarify when pre TLSv1.3 cipher can be used + - DOC: ssl: Stop documenting ciphers example to use + - BUG/MINOR: spoe: do not assume agent->rt is valid on exit + - BUG/MINOR: lua: initialize the correct idle conn lists for the SSL sockets + - BUG/MEDIUM: spoe: initialization depending on nbthread must be done last + - BUG/MEDIUM: server: initialize the idle conns list after parsing the config + - BUG/MAJOR: spoe: Don't try to get agent config during SPOP healthcheck + - BUG/MAJOR: stream: avoid double free on unique_id + - BUG/MINOR: config: Reinforce validity check when a process number is parsed + +2019/02/06 : 1.8.18 + - DOC: http-request cache-use / http-response cache-store expects cache name + - BUG/MAJOR: cache: fix confusion between zero and uninitialized cache key + - BUG/MEDIUM: ssl: Disable anti-replay protection and set max data with 0RTT. + - DOC: Be a bit more explicit about allow-0rtt security implications. + - BUG/MEDIUM: ssl: missing allocation failure checks loading tls key file + - BUG/MINOR: backend: don't use url_param_name as a hint for BE_LB_ALGO_PH + - BUG/MINOR: backend: balance uri specific options were lost across defaults + - BUG/MINOR: backend: BE_LB_LKUP_CHTREE is a value, not a bit + - BUG/MINOR: stick_table: Prevent conn_cur from underflowing + - BUG/MINOR: server: don't always trust srv_check_health when loading a server state + - BUG/MINOR: check: Wake the check task if the check is finished in wake_srv_chk() + - BUG/MEDIUM: ssl: Fix handling of TLS 1.3 KeyUpdate messages + - DOC: mention the effect of nf_conntrack_tcp_loose on src/dst + - MINOR: h2: add a bit-based frame type representation + - MINOR: h2: declare new sets of frame types + - BUG/MINOR: mux-h2: CONTINUATION in closed state must always return GOAWAY + - BUG/MINOR: mux-h2: headers-type frames in HREM are always a connection error + - BUG/MINOR: mux-h2: make it possible to set the error code on an already closed stream + - BUG/MINOR: hpack: return a compression error on invalid table size updates + - DOC: nbthread is no longer experimental. + - BUG/MINOR: spoe: corrected fragmentation string size + - BUG/MINOR: deinit: tcp_rep.inspect_rules not deinit, add to deinit + - SCRIPTS: add the slack channel URL to the announce script + - SCRIPTS: add the issue tracker URL to the announce script + - BUG/MINOR: stream: don't close the front connection when facing a backend error + - MINOR: xref: Add missing barriers. + - BUG/MEDIUM: mux-h2: wake up flow-controlled streams on initial window update + - BUG/MEDIUM: mux-h2: fix two half-closed to closed transitions + - BUG/MEDIUM: mux-h2: make sure never to send GOAWAY on too old streams + - BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection + - MINOR: stream-int: expand the flags to 32-bit + - MINOR: stream-int: add a new flag to mention that we want the connection to be killed + - MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection + - BUG/MEDIUM: mux-h2: do not close the connection on aborted streams + - BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). + - BUG/MINOR: config: fix bind line thread mask validation + - BUG/MAJOR: config: verify that targets of track-sc and stick rules are present + - BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes + - BUG/MINOR: config: make sure to count the error on incorrect track-sc/stick rules + 2019/01/08 : 1.8.17 - BUG/MAJOR: stream-int: Update the stream expiration date in stream_int_notify() - MINOR: mux-h2: only increase the connection window with the first update diff --git a/Makefile b/Makefile index e98848e32..9bd5d6847 100644 --- a/Makefile +++ b/Makefile @@ -632,14 +632,6 @@ endif endif endif -# For nuster -ifeq ($(USE_OPENSSL),) -ifneq ($(USE_PTHREAD_PSHARED),) -OPTIONS_CFLAGS += -DNUSTER_USE_PTHREAD -OPTIONS_LDFLAGS += -lpthread -endif -endif - ifneq ($(USE_LUA),) check_lua_lib = $(shell echo "int main(){}" | $(CC) -o /dev/null -x c - $(2) -l$(1) 2>/dev/null && echo $(1)) check_lua_inc = $(shell if [ -d $(2)$(1) ]; then echo $(2)$(1); fi;) @@ -893,15 +885,7 @@ OBJS = src/proto_http.o src/cfgparse.o src/server.o src/stream.o \ src/sha1.o src/hpack-tbl.o src/hpack-enc.o src/uri_auth.o \ src/time.o src/proto_udp.o src/arg.o src/signal.o \ src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o \ - src/mailers.o src/h2.o src/base64.o src/hash.o \ - \ - src/nuster/cache/dict.o src/nuster/cache/filter.o \ - src/nuster/cache/stats.o src/nuster/cache/manager.o \ - src/nuster/cache/engine.o \ - src/nuster/nosql/filter.o src/nuster/nosql/dict.o \ - src/nuster/nosql/stats.o src/nuster/nosql/engine.o \ - src/nuster/memory.o src/nuster/parser.o src/nuster/http.o \ - src/nuster/nuster.o + src/mailers.o src/h2.o src/base64.o src/hash.o EBTREE_OBJS = $(EBTREE_DIR)/ebtree.o $(EBTREE_DIR)/eb32sctree.o \ $(EBTREE_DIR)/eb32tree.o $(EBTREE_DIR)/eb64tree.o \ @@ -990,7 +974,6 @@ clean: for dir in . src include/* doc ebtree; do rm -f $$dir/*~ $$dir/*.rej $$dir/core; done rm -f haproxy-$(VERSION).tar.gz haproxy-$(VERSION)$(SUBVERS).tar.gz rm -f haproxy-$(VERSION) haproxy-$(VERSION)$(SUBVERS) nohup.out gmon.out - rm -f src/nuster/*.[oas] src/nuster/*/*.[oas] tags: find src include \( -name '*.c' -o -name '*.h' \) -print0 | \ diff --git a/README b/README index 07e095a09..e66b5fded 100644 --- a/README +++ b/README @@ -3,7 +3,7 @@ ---------------------- version 1.8 willy tarreau - 2019/01/08 + 2019/02/11 1) How to build it diff --git a/VERDATE b/VERDATE index 944f4907e..ddc77271d 100644 --- a/VERDATE +++ b/VERDATE @@ -1,2 +1,2 @@ $Format:%ci$ -2019/01/08 +2019/02/11 diff --git a/VERSION b/VERSION index f49e8ed74..c8f955aa6 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.8.17 +1.8.19 diff --git a/doc/configuration.txt b/doc/configuration.txt index 4f999e259..9026c0777 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -4,7 +4,7 @@ ---------------------- version 1.8 willy tarreau - 2019/01/08 + 2019/02/11 This document covers the configuration language as implemented in the version @@ -917,7 +917,14 @@ nbproc mode. By default, only one process is created, which is the recommended mode of operation. For systems limited to small sets of file descriptors per process, it may be needed to fork multiple daemons. USING MULTIPLE PROCESSES - IS HARDER TO DEBUG AND IS REALLY DISCOURAGED. See also "daemon". + IS HARDER TO DEBUG AND IS REALLY DISCOURAGED. See also "daemon" and + "nbthread". + +nbthread + This setting is only available when support for threads was built in. It + creates threads for each created processes. It means if HAProxy is + started in foreground, it only creates threads for the first + process. See also "nbproc". nbthread This setting is only available when support for threads was built in. It @@ -986,12 +993,14 @@ setenv ssl-default-bind-ciphers This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of cipher algorithms ("cipher suite") - that are negotiated during the SSL/TLS handshake except for TLSv1.3 for all + that are negotiated during the SSL/TLS handshake up to TLSv1.2 for all "bind" lines which do not explicitly define theirs. The format of the string - is defined in "man 1 ciphers" from OpenSSL man pages, and can be for instance - a string such as "AES:ALL:!aNULL:!eNULL:+RC4:@STRENGTH" (without quotes). For - TLSv1.3 cipher configuration, please check the "ssl-default-bind-ciphersuites" - keyword. Please check the "bind" keyword for more information. + is defined in "man 1 ciphers" from OpenSSL man pages. For background + information and recommendations see e.g. + (https://wiki.mozilla.org/Security/Server_Side_TLS) and + (https://mozilla.github.io/server-side-tls/ssl-config-generator/). For TLSv1.3 + cipher configuration, please check the "ssl-default-bind-ciphersuites" keyword. + Please check the "bind" keyword for more information. ssl-default-bind-ciphersuites This setting is only available when support for OpenSSL was built in and @@ -999,11 +1008,9 @@ ssl-default-bind-ciphersuites describing the list of cipher algorithms ("cipher suite") that are negotiated during the TLSv1.3 handshake for all "bind" lines which do not explicitly define theirs. The format of the string is defined in - "man 1 ciphers" from OpenSSL man pages under the section "ciphersuites", and can - be for instance a string such as - "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256" - (without quotes). For cipher configuration for TLSv1.2 and earlier, please check - the "ssl-default-bind-ciphers" keyword. Please check the "bind" keyword for more + "man 1 ciphers" from OpenSSL man pages under the section "ciphersuites". For + cipher configuration for TLSv1.2 and earlier, please check the + "ssl-default-bind-ciphers" keyword. Please check the "bind" keyword for more information. ssl-default-bind-options [