Access Control (Exclusion) System

[ikreymer]

Support a per-url exclusion system for pywb, including the following modes:

• A SURT-prefix- based white list of allowed content, and return a HTTP 451
  message if content is outside the white list.
• A SURT-based blacklist of disallowed content that should return a HTTP 404.

[ikreymer]

have a few questions regarding the exclusion system requirement (for @anjackson):

• Are the two modes used separately in different deployments, or within same deployment for different collections?
• Should the exclusion system also support date ranges?
• Is there any relation between the exclusion system and the concurrent-lock system (from Support Single-Lock Access for Legal Deposit Restrictions #6), eg. items that publicly excluded are internally lockable, or completely separate systems enabled independently as needed?

[anjackson]

• They are used within the same deployment.
• That would be a 'nice to have' - at the moment we just generate a SURT whitelist and blacklist, and that is sufficient.

The last question deserves a bit more detail. We have three deployments:

Deployment	Blacklist	Whitelist	Single-concurrent Use
NPLD Reading Room	Y	No (everything is whitelisted except the blacklist)	Y
Open Access	Y	Yes (except the blacklist)	N
QA Access	N	No (everything is whitelisted)	N

I imagine we will need three different pywb collections, in three different deployments.

Important: We generally expect to run all such services behind NGINX or Apache proxies. In the case of QA Wayback, it's actually proxied through a 'parent' app that performs authentication/authorisation. I've assumed this won't be a problem, but I shouldn't take that for granted.

[ikreymer]

Thanks for the clarifications, seems like best option is to configure these as 3 collections for integration testing using the above options:

• /reading-room
• /open-access
• /qa-access

To clarify, if the allow/disallow lists access checks work in this order?

• if disallow list is present, and SURT prefix in disallow list, return 404
• if allow list is present, and SURT prefix not in allow list, return 451
• return content

Does disallow always take precedence?

Or, is most specific match take precedence?
Eg. if com,example)/ is disallowed, but com,example)/allowed is allowed, com,example)/allowed/test.html would be allowed?

[ikreymer]

For maximum flexibility, considering the following approach:

Using CDXJ like format for ACL rules, with SURT key, but reverse sorted to facilitate longest prefix matching.

Each rule would have one of the following settings access settings:

• allow - CDX entry allowed
• block - CDX entry included, but content not allowed (served with a 451)
• exclude - CDX entry not included, as if it didn't exist (served with a 404)

An example rule set:

com,example)/path/allowed - {"access": "allow"}
com,example)/path - {"access": "exclude"}
com,example)/ - {"access": "block"}
com - {"access": "allow"}

ACL rules can be stored in .aclj or .adxj (a for access) files and can be merge sorted on lookup (like cdx).

There will also be a default rule, which can be configured to block to serve 451 if no other SURT matches.

[ikreymer]

The command-line wb-manager tool can also be extended to support management of ACL rules, to avoid having to manipulate the .acl files manually.

Add a rule:
wb-manager acl add <coll> <url_or_surt>

Remove rule:
wb-manager acl remove <coll> <url_or_surt>

Return matching rule:
wb-manager acl match <coll> <url_or_surt>

Import OpenWayback-style non-surt prefix based rules (eg. excludes.txt):
wb-manager acl importexcludes <coll> <path/to/excludes.txt>

[anjackson]

Thanks @ikreymer this looks good.

[ikreymer]

Added initial support for CLI command for operating on individual .aclj files.

For example, to list all rules in a file:
wb-manager acl list ./integration-test/pywb/acl/blocks.aclj

To add a new rule:
wb-manager acl list ./integration-test/pywb/acl/blocks.aclj http://test.example.com/ block

Also supports adding SURTS as well as url:
wb-manager acl add ./integration-test/pywb/acl/allows.aclj uk,org, allow

and removing surts/urls:
wb-manager acl list ./integration-test/pywb/acl/allows.aclj uk,org, remove

[ikreymer]

And also, example . command for importing OpenWayback-style exclusions (one url per line):

wb-manager acl importtxt ./integration-test/pywb/acl/blocks.aclj excludes.txt exclude

[ibnesayeed]

We are working towards a more flexible and efficient approach for archive profiling that contains aspects of ACLs as well. We were thinking along a similar CDXJ-style format, but more flexible than what is illustrated above. We have included the idea of wildcards in partial SURT keys to identify prefixed matches from exact matches. This enables us to more easily describe scenarios like having one rule for a domain (or path at certain depth), but other rules for other resources under that path.