Add initial Sphinx docs

.flake8

@@ -230,9 +230,19 @@ pytest-parametrize-values-row-type = tuple
 pytest-mark-no-parentheses = true
 
 # flake8-rst-docstrings
+rst-directives =
+  spelling
 rst-roles =
-    # Built-in Sphinx roles:
-    py:class,
-    py:meth,
-    # Sphinx's internal role:
-    event,
+  # Built-in Sphinx roles:
+  class,
+  data,
+  exc,
+  meth,
+  term,
+  py:class,
+  py:data,
+  py:exc,
+  py:meth,
+  py:term,
+  # Sphinx's internal role:
+  event,

.github/workflows/test-library.yml

@@ -305,6 +305,7 @@ jobs:
         --parallel auto
         --parallel-live
         --skip-missing-interpreters false
+        --skip-pkg-install
     - name: Verify that the artifacts with expected names got created
       run: >-
         ls -1
@@ -335,6 +336,10 @@ jobs:
         toxenv:
         - lint
         - metadata-validation
+        - build-docs
+        - doctest-docs
+        - linkcheck-docs
+        - spellcheck-docs
       fail-fast: false
 
     env:
@@ -417,6 +422,7 @@ jobs:
         --parallel auto
         --parallel-live
         --skip-missing-interpreters false
+        --skip-pkg-install
 
   test:
     name: 🐍${{ matrix.python }} @ ${{ matrix.os }}

.pre-commit-config.yaml

@@ -74,7 +74,7 @@ repos:
       $
   - id: requirements-txt-fixer
     exclude: >-
-      ^requirements(|-(release|testing|tunnel))\.txt$
+      ^(docs/requirements|requirements(|-(release|testing|tunnel)))\.txt$
   # Non-modifying checks:
   - id: name-tests-test
     args:

.readthedocs.yml

@@ -0,0 +1,41 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html
+# for details
+
+---
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  builder: dirhtml
+  configuration: docs/conf.py
+  fail_on_warning: true
+
+# Optionally build your docs in additional formats
+# such as PDF and ePub
+formats: []
+
+submodules:
+  include: all  # []
+  exclude: []
+  recursive: true
+
+build:
+  os: ubuntu-20.04
+  tools:
+    python: >-  # PyYAML parses it as float `3.1` it it's not an explicit string
+      3.10
+
+# Optionally set the version of Python and requirements required
+# to build docs
+python:
+  install:
+  - method: pip
+    path: .
+  - requirements: requirements-tunnel.txt
+  - requirements: docs/requirements.txt
+  system_packages: false
+
+...

README.md

@@ -4,6 +4,8 @@
 ⚡ Fast • 🪶 Lightweight • 0️⃣ Dependency • 🔌 Pluggable • 😈 TLS interception • 🔒 DNS-over-HTTPS • 🔥 Poor Man's VPN • ⏪ Reverse & ⏩ Forward • 👮🏿 "Proxy Server" framework • 🌐 "Web Server" framework • ➵ ➶ ➷ ➠ "PubSub" framework • 👷 "Work" acceptor & executor framework
 </p>
 
+[//]: # (DO-NOT-REMOVE-docs-badges-START)
+
 [![PyPi Monthly](https://img.shields.io/pypi/dm/proxy.py?style=for-the-badge&color=darkgreen)](https://pypi.org/project/proxy.py/)
 [![Docker Pulls](https://img.shields.io/docker/pulls/abhinavsingh/proxy.py?style=for-the-badge&color=darkgreen)](https://hub.docker.com/r/abhinavsingh/proxy.py)
 [![No Dependencies](https://img.shields.io/static/v1?label=dependencies&message=0&style=for-the-badge&color=darkgreen)](https://github.com/abhinavsingh/proxy.py)
@@ -82,9 +84,9 @@
   - [Ephemeral Port](#ephemeral-port)
   - [Loading Plugins](#loading-plugins)
 - [Unit testing with proxy.py](#unit-testing-with-proxypy)
-  - [proxy.TestCase](#proxytestcase)
+  - [`proxy.TestCase`](#proxytestcase)
   - [Override Startup Flags](#override-startup-flags)
-  - [With unittest.TestCase](#with-unittesttestcase)
+  - [With `unittest.TestCase`](#with-unittesttestcase)
 - [Utilities](#utilities)
   - [TCP](#tcp-sockets)
     - [new_socket_connection](#new_socket_connection)
@@ -105,7 +107,7 @@
   - [Unable to connect with proxy.py from remote host](#unable-to-connect-with-proxypy-from-remote-host)
   - [Basic auth not working with a browser](#basic-auth-not-working-with-a-browser)
   - [Docker image not working on MacOS](#docker-image-not-working-on-macos)
-  - [ValueError: filedescriptor out of range in select](#valueerror-filedescriptor-out-of-range-in-select)
+  - [`ValueError: filedescriptor out of range in select`](#valueerror-filedescriptor-out-of-range-in-select)
   - [None:None in access logs](#nonenone-in-access-logs)
   - [OSError when wrapping client for TLS Interception](#oserror-when-wrapping-client-for-tls-interception)
 - [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
@@ -122,6 +124,8 @@
   - [v1.x](#v1x)
   - [v0.x](#v0x)
 
+[//]: # (DO-NOT-REMOVE-docs-badges-END)
+
 # Features
 - Fast & Scalable
 
@@ -170,7 +174,7 @@
   - Optionally, enable builtin [Web Server Plugins](#http-web-server-plugins). Example:
     - `--plugins proxy.plugin.ReverseProxyPlugin`
   - Plugin API is currently in development phase, expect breaking changes
-- Realtime Dashboard
+- Real-time Dashboard
   - Optionally, enable [proxy.py dashboard](#run-dashboard).
     - Available at `http://localhost:8899/dashboard`.
   - [Inspect, Monitor, Control and Configure](#inspect-traffic) `proxy.py` at runtime
@@ -423,7 +427,7 @@ You can override flag from command line when starting the docker container. For
 
 Add support for short links in your favorite browsers / applications.
 
-[![Shortlink Plugin](https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/shortlink.gif)](https://github.com/abhinavsingh/proxy.py#shortlinkplugin)
+[![Shortlink Plugin](https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/shortlink.gif)](https://github.com/abhinavsingh/proxy.py#user-content-shortlinkplugin)
 
 Start `proxy.py` as:
 
@@ -438,17 +442,17 @@ across all browsers.
 
 Following short links are enabled by default:
 
-| Short Link | Destination URL  |
-| :--------: | :--------------: |
-|     a/     |    amazon.com    |
-|     i/     |  instagram.com   |
-|     l/     |   linkedin.com   |
-|     f/     |   facebook.com   |
-|     g/     |    google.com    |
-|     t/     |   twitter.com    |
-|     w/     | web.whatsapp.com |
-|     y/     |   youtube.com    |
-|   proxy/   |  localhost:8899  |
+| Short Link |  Destination URL   |
+| :--------: |  :--------------:  |
+|     a/     |    `amazon.com`    |
+|     i/     |  `instagram.com`   |
+|     l/     |   `linkedin.com`   |
+|     f/     |   `facebook.com`   |
+|     g/     |    `google.com`    |
+|     t/     |   `twitter.com`    |
+|     w/     | `web.whatsapp.com` |
+|     y/     |   `youtube.com`    |
+|   proxy/   |  `localhost:8899`  |
 
 ### ModifyPostDataPlugin
 
@@ -461,7 +465,7 @@ Start `proxy.py` as:
     --plugins proxy.plugin.ModifyPostDataPlugin
 ```
 
-By default plugin replaces POST body content with hardcoded `b'{"key": "modified"}'`
+By default plugin replaces POST body content with hard-coded `b'{"key": "modified"}'`
 and enforced `Content-Type: application/json`.
 
 Verify the same using `curl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post`
@@ -763,7 +767,7 @@ Modify plugin to your taste e.g. Allow specific IP addresses only.
 
 ### ModifyChunkResponsePlugin
 
-This plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin uses `proxy.py` core to parse the chunked encoded response. Then we reconstruct the response using custom hardcoded chunks, ignoring original chunks received from upstream server.
+This plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin uses `proxy.py` core to parse the chunked encoded response. Then we reconstruct the response using custom hard-coded chunks, ignoring original chunks received from upstream server.
 
 Start `proxy.py` as:
 
@@ -784,14 +788,14 @@ plugin
 * Closing connection 0
 ```
 
-Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hardcoded chunks, parse and modify the original `JSON` chunks received from the upstream server.
+Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hard-coded chunks, parse and modify the original `JSON` chunks received from the upstream server.
 
 ### CloudflareDnsResolverPlugin
 
 This plugin uses `Cloudflare` hosted `DNS-over-HTTPS` [API](https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-over-https/make-api-requests/dns-json) (json).
 
 `DoH` mandates a HTTP2 compliant client. Unfortunately `proxy.py`
-doesn't provide that yet, so we use a dependency. Install it:
+does not provide that yet, so we use a dependency. Install it:
 
 ```console
 ❯ pip install "httpx[http2]"
@@ -811,7 +815,7 @@ Use `--cloudflare-dns-mode family` to also enable adult content protection too.
 
 This plugin demonstrate how to use a custom DNS resolution implementation with `proxy.py`.
 This example plugin currently uses Python's in-built resolution mechanism.  Customize code
-to your taste.  Example, query your custom DNS server, implement DoH or other mechanisms.
+to your taste.  Example, query your custom DNS server, implement `DoH` or other mechanisms.
 
 Start `proxy.py` as:
 
@@ -969,7 +973,7 @@ response from the server. Start `proxy.py` as:
     --ca-signing-key-file ca-signing-key.pem
 ```
 
-[![NOTE](https://img.shields.io/static/v1?label=MacOS&message=note&color=yellow)](https://github.com/abhinavsingh/proxy.py#flags) Also provide explicit CA bundle path needed for validation of peer certificates. See `--ca-file` flag.
+[![NOTE](https://img.shields.io/static/v1?label=MacOS&message=note&color=yellow)](https://github.com/abhinavsingh/proxy.py#user-content-flags) Also provide explicit CA bundle path needed for validation of peer certificates. See `--ca-file` flag.
 
 Verify TLS interception using `curl`
 
@@ -1042,7 +1046,8 @@ Important notes about TLS Interception with Docker container:
 - Since `v2.2.0`, `proxy.py` docker container also ships with `openssl`. This allows `proxy.py`
   to generate certificates on the fly for TLS Interception.
 
-- For security reasons, `proxy.py` docker container doesn't ship with CA certificates.
+- For security reasons, `proxy.py` docker container does not ship with
+  CA certificates.
 
 Here is how to start a `proxy.py` docker container
 with TLS Interception:
@@ -1329,9 +1334,9 @@ if __name__ == '__main__':
 
 # Unit testing with proxy.py
 
-## proxy.TestCase
+## `proxy.TestCase`
 
-To setup and teardown `proxy.py` for your Python `unittest` classes,
+To setup and tear down `proxy.py` for your Python `unittest` classes,
 simply use `proxy.TestCase` instead of `unittest.TestCase`.
 Example:
 
@@ -1346,10 +1351,10 @@ class TestProxyPyEmbedded(proxy.TestCase):
 
 Note that:
 
-1. `proxy.TestCase` overrides `unittest.TestCase.run()` method to setup and teardown `proxy.py`.
+1. `proxy.TestCase` overrides `unittest.TestCase.run()` method to setup and tear down `proxy.py`.
 2. `proxy.py` server will listen on a random available port on the system.
    This random port is available as `self.PROXY.acceptors.flags.port` within your test cases.
-3. Only a single acceptor and worker is started by default (`--num-workers 1 --num-acceptors 1`) for faster setup and teardown.
+3. Only a single acceptor and worker is started by default (`--num-workers 1 --num-acceptors 1`) for faster setup and tear down.
 4. Most importantly, `proxy.TestCase` also ensures `proxy.py` server
    is up and running before proceeding with execution of tests. By default,
    `proxy.TestCase` will wait for `10 seconds` for `proxy.py` server to start,
@@ -1373,13 +1378,15 @@ class TestProxyPyEmbedded(TestCase):
         self.assertTrue(True)
 ```
 
-See [test_embed.py](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/test_embed.py)
-for full working example.
+See [test_embed.py] for full working example.
+
+[test_embed.py]:
+https://github.com/abhinavsingh/proxy.py/blob/develop/tests/testing/test_embed.py
 
-## With unittest.TestCase
+## With `unittest.TestCase`
 
 If for some reasons you are unable to directly use `proxy.TestCase`,
-then simply override `unittest.TestCase.run` yourself to setup and teardown `proxy.py`.
+then simply override `unittest.TestCase.run` yourself to setup and tear down `proxy.py`.
 Example:
 
 ```python
@@ -1400,7 +1407,7 @@ class TestProxyPyEmbedded(unittest.TestCase):
             super().run(result)
 ```
 
-or simply setup / teardown `proxy.py` within
+or simply setup / tear down `proxy.py` within
 `setUpClass` and `teardownClass` class methods.
 
 # Utilities
@@ -1438,7 +1445,7 @@ As a decorator:
 >>>   ... [ use connection ] ...
 ```
 
-## Http Client
+## HTTP Client
 
 ### build_http_request
 
@@ -1482,7 +1489,7 @@ build_http_response(
 
 ### API Usage
 
-- gen_private_key
+- `gen_private_key`
 
   ```python
   gen_private_key(
@@ -1492,7 +1499,7 @@ build_http_response(
       timeout: int = 10) -> bool
   ```
 
-- gen_public_key
+- `gen_public_key`
 
   ```python
   gen_public_key(
@@ -1506,7 +1513,7 @@ build_http_response(
       timeout: int = 10) -> bool
   ```
 
-- remove_passphrase
+- `remove_passphrase`
 
   ```python
   remove_passphrase(
@@ -1516,7 +1523,7 @@ build_http_response(
       timeout: int = 10) -> bool
   ```
 
-- gen_csr
+- `gen_csr`
 
   ```python
   gen_csr(
@@ -1527,7 +1534,7 @@ build_http_response(
       timeout: int = 10) -> bool
   ```
 
-- sign_csr
+- `sign_csr`
 
   ```python
   sign_csr(
@@ -1640,7 +1647,7 @@ Visit dashboard:
 
 Wait for embedded `Chrome Dev Console` to load.  Currently, detail about all traffic flowing
 through `proxy.py` is pushed to the `Inspect Traffic` tab.  However, received payloads are not
-yet integrated with the embedded dev console.
+yet integrated with the embedded developer console.
 
 Current functionality can be verified by opening the `Dev Console` of dashboard and inspecting
 the websocket connection that dashboard established with the `proxy.py` server.
@@ -1782,7 +1789,7 @@ for some background.
 
 ## GCE log viewer integration for proxy.py
 
-A starter [fluentd.conf](https://github.com/abhinavsingh/proxy.py/blob/develop/fluentd.conf)
+A starter [fluentd.conf](https://github.com/abhinavsingh/proxy.py/blob/develop/helper/fluentd.conf)
 template is available.
 
 1. Copy this configuration file as `proxy.py.conf` under
@@ -1798,7 +1805,7 @@ template is available.
 Now `proxy.py` logs can be browsed using
 [GCE log viewer](https://console.cloud.google.com/logs/viewer).
 
-## ValueError: filedescriptor out of range in select
+## `ValueError: filedescriptor out of range in select`
 
 `proxy.py` is made to handle thousands of connections per second
 without any socket leaks.
@@ -1915,7 +1922,7 @@ the incoming client connections.
 
 Each `Acceptor` process delegates the accepted client connection
 to a threadless process via `Work` class.  Currently, `HttpProtocolHandler`
-is the default work klass.
+is the default work class.
 
 `HttpProtocolHandler` simply assumes that incoming clients will follow
 HTTP specification.  Specific HTTP proxy and HTTP server implementations