Documentation; JSON catalogs; cli.py cleanup [some] (#200)

Add catalog.json, replacing ad-hoc lists of providers in cli.py & setup.py.
catalog.py for easy use of the catalog, including loading known providers.
Add sewer.json info replaces __version__.py; sewer_about in lib to access it.
Rework GET, HEAD and POST for better mypy compatibility with less hacks.

docs/CHANGELOG.md

@@ -13,6 +13,7 @@ Features and Improvements:
 - added `--p_opt <name>=<value>` for passing kwargs to drivers
 - Added optional parameters accepted by base class for DNS drivers:
   - `alias=<alias_domain>` specifies a separate domain for DNS challenges
+    (requires driver support, see [Aliasing](Aliasing))
   - `prop_delay=<seconds>` gives a fixed delay (sleep) after challenge setup
 - gandi (legacy DNS driver) fixed internal bugs that broke common wildcard
   use cases (eg., `*.domain.tld`) as well as the "wildcard plus" pattern
@@ -21,6 +22,12 @@ Features and Improvements:
   could be useful to someone, maybe.
 
 Internals:
+- added [catalog.py](catalog) to manage provider catalogs; includes
+  get_provider(name) method to replace `import ......{name.}ClassName`
+- replace __version__.py with sewer.json; setup.py converted; add sewer_about()
+  in lib.py; cli.py converted; client.py converted
+- added catalog.json defining known drivers and their interfaces; also
+  information about dependencies for setup.py
 - added `**kwargs` to all legacy providers to allow new options that are
   handled in a parent class to pass through (for `alias`, `prop_delay`, etc.)
 - removed imports that were in `sewer/__init__` and
@@ -31,6 +38,8 @@ Internals:
 - fixed imports in client.py that didn't actually import the parts of
   OpenSSL and cryptography that we use (worked because we import requests?)
 
+See also [release notes](notes/0.8.3-notes).
+
 ## **version:** 0.8.2
 Feature additions:
 

docs/catalog.md

@@ -0,0 +1,142 @@
+## Driver Catalog
+
+The driver catalog, `sewer/catalog.json`, replaces scattered facilities that
+were used to stitch things together.  The import farms in `sewer.__init__`
+and `sewer.dns_providers.__init__` have already been removed; with the
+catalog in place, redundant lists in cli.py and setup.py are also removed,
+replaced by use of the catalog's data and a few lines of code.  The
+`dns_provider_name` is deprecated in favor of the catalog as well.
+
+`catalog.py` wraps the catalog in a class, adding some convenience methods
+for listing the known drivers, looking up a driver's data by name, and
+loading a driver's implementation class by name.  But using the catalog
+without `catalog.py` is as easy as loading it using the standard lib's json
+facilities - it's all lists & dicts (see eg. setup.py which loads the
+catalog this way to avoid potential issues with trying to call into the
+package's code before it's installed).
+
+### Catalog structure
+
+The catalog resides in a JSON file that loads as an array of dictionaries,
+one element for each registered driver.  The per-driver record contains the
+following items (some optional):
+
+- **name** The name used to identify this driver, eg., `--provider <name>`
+  to `sewer-cli`.  These names need to be unique, but are not required to
+  match the module or implementing class names.  (legacy DNS drivers usually
+  matched the module name, but not always)
+- **desc** A brief description of the driver, intended for display to humans
+  to help them understand what each driver is, eg. in --known_providers output
+- **chals** list of strings for the type of challenge(s) this driver
+  handles.  (if more than one type, in order of preference?)
+- **args** A list of the driver-specific [parameters](#args-parameter-descriptors)
+- **path** The path to use to import the driver's Python module.
+  _Default_ is `sewer.providers.{name}`
+- **cls** Name of module attribute which is called with parameters to get a
+  working instance of the driver.  Usually a class, but a factory function
+  may be used.  _Default_ is `Provider`.
+- **features** - a list of strings that name the optional features that this
+  driver supports.  _Default_ is an empty list.
+- **memo** Additional text/comments about the driver, the descriptor, etc.
+- **deps** list of additional projects this driver requires (for setup)
+
+### args - parameter desciptors
+
+This is a bit of a mess due to legacy drivers that ignored the established
+conventions.  To be fair to them, those conventions weren't clearly
+documented (then - see below).  This adds some complications to preserve
+compatibility, as usual.  Let's begin with a minimal descriptor for a driver
+that conforms to the new convention (hint: it's imaginary at this time):
+
+    {
+      "name": "well_behaved",
+      "desc": "made-up example driver that's mostly defaults",
+      "chals": ["dns-01"],
+      "args": [
+        { "name": "api_id", "req": 1 },
+        { "name": "api_key", "req": 1},
+      ],
+      "features": [ "alias" ],
+    }
+
+This describes a dns-01 challenge driver that is found in the module
+`sewer.providers.well_behaved`, constructed from a class named `Provider`.
+The constructor takes two required arguments, `api_id` and `api_key`, which
+the program should accept from environment variables `WELL_BEHAVED_API_ID"
+and "WELL_BEHAVED_API_KEY".  Since it is a `dns-01` challenge provider and
+up to date, it adds the claim that it supports the `alias` feature.  It
+doesn't support the "unpropagated" feature - perhaps the DNS service has no
+API to check the propagation of changes.
+
+If this had been a difficult old legacy driver, the descriptor might have
+looked more like this:
+
+    {
+      "name": "difficult",
+      "desc": "made-up example driver that's as non-default as can be",
+      "chals": ["dns-01"],
+      "args": [
+        { "name": "api_id",
+          "req": 1,
+          "param": "difficult_api_id",
+          "envvar": "DIFFICULT_DNS_API_ID",
+        },
+        { "name": "api_key",
+          "req": 1,
+          "param": "DIFFICULT_API_KEY",
+          "envvar": "DIFFICULT_DNS_API_KEY"},
+        },
+	{ "name": "api_base_url",
+	  "param": "API_BASE_URL",
+	  "envvar": "",
+	},
+      ],
+      "path": "sewer.dns_providers.difficultdns",
+      "cls": "DifficultDNSDns",
+      "features": [],
+      "memo", "difficult, indeed..."
+    }
+
+This driver has both parameter names and envvar names that defy convention,
+so both the parameter and envvar name must be given explicitly.  There is
+also an optional parameter that has never had an associated envvar that the
+implementation used.
+
+### driver parameter and environment variable names
+
+The convention is that the envvar name (if any) SHOULD be formed from the
+driver name and the individual args' names (see the first envvar rule
+below).  This gives envvar names similar to, sometimes identical to, the
+ones already used with legacy DNS drivers.  One thing that is changing is
+that the parameter names, which in the old convention were
+THE_SAME_AS_ENVVAR_NAMES, are changing to be lower case and losing
+driver-name prefixes, etc.  Where appropriate, the new names will use just a
+few shared names, viz., `id`, `key`, `token`.
+
+Obviously the drivers and envvar names are not so consistent among the
+legacy DNS drivers.  Therefore the descriptor has both `param` and `envvar`
+values, along with a set of rules for resolving the names to be used.
+
+#### parameter name rules
+
+1. `descriptor.args[n].name` is the "modern" name for the nth parameter
+2. if `param` is given, it overrides the "modern" name
+
+#### environment name rules
+
+1. f"{descriptor.name}_{descriptor.args[n].name}".upper() is the default
+2. if `envvar` is given, it overrides the default
+
+Two guidelines for the use of envvars:
+
+1. If `envvar` is given, is not the empty string, and the so-named envvar is
+   not found, the invoking code MAY also look for the default-named envvar
+   before reporting a missing envvar.
+
+2. If `envvar` is set to the empty string, then catalog using code will not
+   look for a matching envvar at all.
+
+### catalog representation in Python
+
+For now, see the brief implementation in sewer/catalog.py for the way the
+JSON structure is mapped into a ProviderDescriptor instance.

docs/dns-01.md

@@ -19,6 +19,7 @@ service's API is difficult.
 - [Hurricane Electric DNS](https://dns.he.net/)
 - [PowerDNS](https://doc.powerdns.com/authoritative/http-api/index.html)
 - [Rackspace](https://www.rackspace.com/cloud/dns)
+- [unbound_ssh]
 
 ### Add a driver for your DNS service
 
@@ -95,9 +96,12 @@ Three features that have varying support in the Legacy drivers.
 | hurricane | ? | no | no | test coverage 70% |
 | powerdns | NO | no | no | apparently not in 0.8.2; bug #195 |
 | rackspace | ? | no | no | test coverage 69% | 
-| route53 | OK | no | no | wildcard since pre-0.8.2 |
+| route53 (1) | OK | no | no | wildcard since pre-0.8.2 |
 | unbound_ssh | OK | yes | no | Working demonstrator model |
 
+> (1) route53 was never setup to be used from `sewer-cli`.  That will change,
+maybe for 0.8.3, but does anyone care?  No complaints have been heard...
+
 _wildcard_ is NOT the older issue - since 0.8.2, all drivers should be able
 to support creating certificates for simple `*.domain.tld` requests.
 There is a deeper problem when one wants a wildcard that _also_ covers plain

docs/drivers/route53.md

@@ -0,0 +1,17 @@
+## route53 - driver for AWS DNS service
+
+### Command line use
+
+route53 has never been wired into `sewer-cli`, and that hasn't really
+changed in 0.8.3.  It does appear in the list of "known providers", but it
+isn't usable, and raises an exception if named by `--provider`.
+
+Adding that integration is on the list, but seeing as no one has complained
+about this lack up to now it's nowhere near the top.  :-(
+
+### Programmatic use
+
+Apparently everyone using sewer's route53 has been rolling their own
+wrapper, since it has only been available for such use to date.  There is a
+patch to extend that Route53Dns.__init__ to allow additional AWS-specific
+methods of authentication which I expect will ship in 0.8.3.

docs/drivers/unbound_ssh.md

@@ -1,17 +1,21 @@
 ## unbound_ssh legacy DNS driver
 
-A working, if somewhat quirky, driver to setup challenges in the unbound
-server, using ssh to connect to an account able to run unbound-control
-commands.  The driver does NOT handle the login authorization, assuming that
-it is running interactively and ssh will prompt for your input, or that a
-key agent (eg., ssh-agent) is active to supply the cryptographic
-credentials.
+A working, if somewhat quirky, driver to setup challenges in local data of
+the [unbound](https://nlnetlabs.nl/projects/unbound/about/) caching
+resolver.  As the name suggests, it relies upon ssh to provide an
+authenticated connection the server; inside that connection the
+`unbound-control` program is used to add and remove the records.  The driver
+does NOT handle the login authorization, assuming that it is running
+interactively and ssh will prompt for your input, or that a key agent (eg.,
+ssh-agent) is active to supply the cryptographic credentials.  That's the
+_somewhat quirky_ part!
 
 ### `__init__(self, *, ssh_des, **kwargs)`
 
 There is one REQUIRED parameter, `ssh_des`, which is the login target, such
 as [email protected].  This is simply passed to the ssh command,
-with the unbound-control commands passed as the command to execute remotely.
+along with the `unbound-control` commands to be executed on the destination
+machine.
 
 ### Driver features
 
@@ -38,3 +42,7 @@ Sadly, This was written using the old paradigm where both the module name
 and the class name were more-or-less the same name aside from
 capitalization... and often less predictable changes.  Should have been
 unbound_ssh.Provider ...
+
+The `unbound-control` commands generated could be run locally with not very
+much change to the driver.  Perhaps that will become part of a demonstration
+of some different features in the future.

docs/notes/0.8.3-notes.md

@@ -1,4 +1,4 @@
-# Sewer 0.8.3 Release Notes
+## Sewer 0.8.3 Release Notes
 
 This will attempt to list all the changes that affect users of the
 `sewer-cli` program, including even cosmetic changes.  If you use sewer as a
@@ -7,7 +7,7 @@ library you may find internal changes not called out here.
 **New `sewer-cli` features are usually just mentioned, see
 [sewer-cli.md](sewer-cli) for more complete documentation.**
 
-## What's New
+### What's New
 
 - added many words in the /docs directory.  A lot of it is internals
   documentation; a lot of it was written to help me understand exactly how
@@ -34,7 +34,7 @@ library you may find internal changes not called out here.
 **NB: `--alias_domain` and the planned `--prop_*` options were only added
 during PRE-0.8.3, so they will just be dropped in the release.**
 
-## What's Changed
+### What's Changed
 
 Mostly I've tried to avoid changes that were likely to break things.  More
 so for `sewer-cli` than those who use the inner workings of Client, of
@@ -59,10 +59,34 @@ course.
   a legacy DNS driver.  Needs a rather specific environment to work, but I
   just renewed a handful of certificates using it the other day.
 
-## Breakage
+- JSON configuration has arrived.  sewer.json replaces __version__.py.
+  catalog.json adds a central description of known drivers, replacing the
+  mess of imports [removed], some non-DRY lists in setup.py and cli.py.
+
+- `sewer.catalog` provides loading of the JSON catalog as well as methods to
+  lookup the descriptor and load the module; replaces the mess of imports
+
+### Breakage
 
 - removed all the imports in __init__.py (both sewer & dns_providers).  This
   *will* affect you if you've just done `import sewer` and access especially
-  the provider classes as eg.  `sewer.ThatDNSDns`.  Using proper imports,
+  the provider classes as eg.  `sewer.ThatDNSDns`.  ~~Using proper imports,
   eg., `import sewer.dns_providers.thatdns.ThatDNSDns` is the current
-  workaround, sorry.  **This does NOT affect `sewer-cli` users.**
+  workaround, sorry.~~  Recommended use is now something like
+  ```
+  from sewer import catalog
+
+  pro_cls = catalog.ProviderCatalog().get_provider("route53")
+  provider = pro_cls(...arguments as needed...)
+
+  # use of route53 is ironic: it has no catalog entry because it has
+  # never been integrated into `sewer-cli`, which was what drove the
+  # creation of the catalog  [right, another FIX ME]
+  ```
+  **This does NOT affect `sewer-cli` users.**
+
+### Deprecated
+
+- `--action {run,renew}` option has never actually had any effect and is no
+  longer required (since 0.8.2?).  LOGS A WARNING IN 0.8.3.
+- ...