Skip to content

HTTP Resiliency

Jump to bottom
Johnny Boursiquot edited this page Mar 2, 2023 · 6 revisions

In this section, we learn how to build resilient HTTP servers and clients in Go.

We'll cover:

  1. Leveraging the context package to limit time spent waiting on clients and requests
  2. Understanding throttling and implementing retry/backoff
  3. Explore the various mechanisms available to control timeouts

Context

You are bound to come across the use of the context package in most professional-grade Go programs. The basic premise behind using Context is to create resilient servers that support requests that are cancellable, have deadlines, and possibly carry values across package and API boundaries.

To help us make sense of the context package, let's first look at an example that does not have the capabilities it provides.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"
)

func greetHandler(w http.ResponseWriter, r *http.Request) {
	log.Println("Handling greeting request")
	defer log.Println("Handled greeting request")

	completeAfter := time.After(5 * time.Second)

	for {
		select {
		case <-completeAfter:
			fmt.Fprintln(w, "Hello Gopher!")
			return
		default:
			time.Sleep(1 * time.Second)
			log.Println("Greetings are hard. Thinking...")
		}
	}
}

func main() {
	http.HandleFunc("/", greetHandler)
	log.Println("Server listening on :8080...")
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}

Action Steps

  1. Copy the program above locally and run it.
  2. Use curl to issue a request to http://localhost:8080. This will cause the server to start emitting log statements indicating that it's thinking on the request (to simulate a resource-intensive request).
  3. Before the server can send a response, cancel your curl request (CTRL-C should do it).
  4. Note that the server still carries on thinking about the request even though the client has cancelled the request.

If you followed the Action Steps above, you quickly start to realize the implications of having a server carry on working on requests that clients no longer care about. Network connections are often unreliable, especially for mobile devices. Though smart clients try to maintain a connection to the server they're talking to until the request comes back, they may be disconnected and connect again at any time. Your servers need to have resilience to this usage pattern and that's where the context package comes in.

Let's look at the same program from above but this time note the use of the context.Context type that is obtained from the http.Request object passed into the greetHandler function.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"
)

func greetHandler(w http.ResponseWriter, r *http.Request) {
	log.Println("Handling greeting request")
	defer log.Println("Handled greeting request")

	completeAfter := time.After(5 * time.Second)
	ctx := r.Context()

	for {
		select {
		case <-completeAfter:
			fmt.Fprintln(w, "Hello Gopher!")
			return
		case <-ctx.Done():
			err := ctx.Err()
			log.Printf("Context Error: %s", err.Error())
			return
		default:
			time.Sleep(1 * time.Second)
			log.Println("Greetings are hard. Thinking...")
		}
	}
}

func main() {
	http.HandleFunc("/", greetHandler)
	log.Println("Server listening on :8080...")
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}

Action Steps

  1. Copy the program above locally and run it.
  2. Use curl to issue a request to http://localhost:8080. This will cause the server to start emitting log statements indicating that it's thinking on the request (to simulate a resource-intensive request).
  3. Before the server can send a response, cancel your curl request (CTRL-C should do it).
  4. This time, note that the server immediately stops processing the request as well due to the fact that the context was canceled, thereby allowing a read on the select case for ctx.Done() when the client dropped the request.

Exercise 1

You've seen how you can use the request context from the server side to handled canceled connections by clients. In this exercise, you will implement cancelation from the client side. Your task is to write a client application that is resilient against a server hanging on to the request indefinitely.

Imagine your client makes a request to a third-party API that's having issues internally but takes forever to response to your API requests. For this task, you may reuse and modify the examples from above or use the net/http/httptest to create a fake server within your code to demonstrate the cancelation process.

Objectives

  1. Use the context package with a timeout setting to ensure your requests to the server take no more than 2 or so seconds.
  2. Create a fake server using the net/http/httptest package or simply repurpose an example from above.

You may find a solution to Exercise 1 here BUT only after you've given it your best shot at solving it.

Rate Limits, Retries, and Backoff

It's common for servers to be overwhelmed with requests from one or more clients. In such cases, the server may want to throttle the number of incoming requests that are processed. Thankfully, Go has a number of community packages that provide a mechanism for throttling requests. We'll rely on the standard library's golang.org/x/time/rate package to implement throttling.

golang.org/x/time/rate is itself used by many of the community packages that offer rate limiting capbilities!

Implementing Rate Limiting in your HTTP Servers

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"

	"golang.org/x/time/rate"
)

var limiter *rate.Limiter

// limit is middleware that rate limits requests
func limit(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if !limiter.Allow() {
			log.Printf("Rate limit exceeded (Request ID: %v)", r.Header.Get("X-Request-Id"))
			http.Error(w, http.StatusText(http.StatusTooManyRequests), http.StatusTooManyRequests)
			return
		}
		next.ServeHTTP(w, r)
	})
}

const (
	defaultPort  = 8080
	defaultRate  = 1
	defaultBurst = 3
)

func main() {
	port := fmt.Sprintf(":%d", *flag.Int("port", defaultPort, "port (int)"))
	r := flag.Float64("rate", defaultRate, "rate limit (float)")
	b := flag.Int("burst", defaultBurst, "burst limit (int)")
	flag.Parse()
	limiter = rate.NewLimiter(rate.Limit(*r), *b)

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte(http.StatusText(http.StatusOK)))
	})

	log.Printf("Server ready on %s with allowed rate of %v req/s and burst of %v reqs...", port, *r, *b)
	http.ListenAndServe(port, limit(mux))
}

In the code above, we make use of the golang.org/x/time/rate package to initialize a Limiter that allows for a maximum of 1 request per second and a burst of 3 requests. These values can be changed by passing in different values to the -rate and -burst flags. Leveraging limiters through middleware is a simple and common pattern in HTTP servers.

To see the rate limiting in action, run the following command from within the folder containing this example code (06-http-resiliency/server/backoff-example):

$ make
2021/10/19 12:54:15 Server ready on :8080 with allowed rate of 1 req/s and burst of 3 reqs...
HTTP/1.1 200 OK
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

OK
---
HTTP/1.1 200 OK
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

OK
---
HTTP/1.1 200 OK
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

OK
---
2021/10/19 12:54:16 Rate limit exceeded (Request ID: )
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 18

Too Many Requests

---
2021/10/19 12:54:16 Rate limit exceeded (Request ID: )
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 18

Too Many Requests

---
2021/10/19 12:54:16 Rate limit exceeded (Request ID: )
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 19 Oct 2021 16:54:16 GMT
Content-Length: 18

Too Many Requests

---

The Makefile target will take care of running the program and issuing multiple curl commands in quick succession. The server, in this case, limits a number of the requests following the first three.

Managing Timeouts

Consider the following example:

package main

import (
	"fmt"
	"log"
	"math/rand"
	"net/http"
	"net/http/httptest"
	"time"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Printf("Server >> Request received...[%s] %s", r.Method, r.RequestURI)
		rnd := rand.New(rand.NewSource(time.Now().UnixNano()))
		time.Sleep(time.Duration(rnd.Intn(60)) * time.Second) // Simulate a slow server
		msg := "Hello Gopher!"
		log.Printf("Server >> Sending %q", msg)
		fmt.Fprintln(w, msg)
	}))
	defer ts.Close()

	log.Println("Making request...")
	res, err := http.Get(ts.URL)
	if err != nil {
		log.Fatalln(err)
	}
	log.Println(res.Status)
	defer res.Body.Close()
}

We use the httptest package to create a simple HTTP server that sleeps for a random amount of time between 0 and 60 seconds before responding to a request. We then make a request to the server and print the response status.

The default HTTP client used by the http.Get function will timeout after 60 seconds, a, perhaps, surprisingly long amout of time. To understand why, let's explore the http.DefaultClient and the http.DefaultTransport that it uses.

There are several ways to manage client timeouts in Go. Let's walk through a few examples in the repository.

Summary

This section packed a lot of information on how to build resilient HTTP servers and clients, including leveraging the context package, timeouts, and rate limiting.

Clone this wiki locally