Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Improve user/group permissions scheme #342

Closed
fmigneault opened this issue Jul 24, 2020 · 22 comments · Fixed by #353
Closed

[Feature] Improve user/group permissions scheme #342

fmigneault opened this issue Jul 24, 2020 · 22 comments · Fixed by #353
Assignees
Labels
api Something related to the API operations feature New feature to be developed ui Something related to the UI operations or display
Milestone

Comments

@fmigneault
Copy link
Collaborator

fmigneault commented Jul 24, 2020

Feature

When looking at permissions on user / group edit pages, a large matrix of checkboxes is listed.

With the eventual addition on 'deny' permission (#235), the list of applicable permissions can become quite large.
(every combination of deny/deny-match, <perm>/<perm>-match)
For example, if a resource has [read, write] "permissions category" we would then have 8 "permissions variants" (using the previous scheme "match" which means non-recursive while, default is recursive) :

read, read-match, write, write-match, deny-read, deny-read-match, deny-write, deny-write-match

This becomes worst the more permissions "category" we have.

API Changes

  • Migrate all [perm]-match/[perm] permissions to respective [perm]-allow-[match|recur]
    ([perm] without modifier remains the [recur] variant for backward compatibility).
  • Add new support for [perm]-deny-[match|recur] variants to existing sets of services.
  • Add some resolution method that converts these applied permission variants to a effective permission
    • saved permission in database must be a literal string for bw compat with ziggurat string permissions, which offers most of the auto-handling of relationships internally
    • since there would always be 3 parts ([perm]-[allow|deny]-[match|recur]), it will be easy to got from string to dict representation
  • Code that handles new permissions must handle use-cases were there are conflicts
    (how should we proceed with this??,
    eg:
    [perm]-allow-match applied when [perm]-deny-match already exists will not cause any problem from the db side as they are both unique strings. It wouldn't cause problem either when we resolve the effective permission (as deny has priority over allow).
    This will cause problem in the sense that when [perm]-allow-match is applied, if one forgets to check for [perm]-deny-match already applied on the same resource, it will still appear as forbidden access although the user just added the allow permission. It is not very intuitive.
  • (!) edit: see decisions according to following thread comments

For each permission we would have the following properties :

allow-match
allow-recur
deny-match
deny-recur

The behavior should look like this given resource "/root/dir/file" :

bool authorized = False
if deny(file):
    return 401
if allow(file):
    authorized = True
# Parsing in bottom up order
parents = [dir, root]
for resource in parents:
    if deny-recursive(resource):
        return 401
    if allow-recursive(resource):
        authorized = True
if authorized:
    return 200

UI Changes

We should change the checkboxes to a combobox of all applicable permissions to make it more succinct.

image

The currently existing checkbox view is still very relevant for overall viewing of effective permissions.
This could be the visual rendering of results from #341
Because on that view, we will want only the 'allowed' vs 'denied' effective results, the way this result is obtained (recursive or match) does not matter. Checkboxes would be read-only in that case.

Other

@fmigneault fmigneault added feature New feature to be developed ui Something related to the UI operations or display api Something related to the API operations labels Jul 24, 2020
@fmigneault
Copy link
Collaborator Author

@dbyrns I would like to draw attention on the last dot of "API Changes"
I'm still not sure how to handle this case precisely on a code-execution vs user-interaction point of view.

@fmigneault fmigneault added this to the 2.x.x milestone Jul 24, 2020
@tlvu
Copy link
Contributor

tlvu commented Jul 24, 2020

Related to Ouranosinc/pavics-sdi#111.

@dbyrns
Copy link
Contributor

dbyrns commented Jul 24, 2020

Code that handles new permissions must handle use-cases were there are conflicts (how should we proceed with this??)

I think we should first wipe any existing permutation of the permission being assigned. eg: [perm]-deny-match exists and I apply [perm]-allow-match on the same ressource. [perm]-deny-match should be removed and [perm]-allow-match applied. For a given permission we should never have more than one permutation from the set [allow-match, allow-recur, deny-match, deny-recur].

For inheritance, there is nothing to do. The doc should clearly state that deny has a higher priority than allow and thus even if you give access to something, a deny-recur above in the hierarchy will still be enforced. In the UI the effective permission should make things clear and avoid confusion.

@fmigneault
Copy link
Collaborator Author

@dbyrns
I've had a flash thinking about this a little bit more regarding conflicting combinations of permission "modifiers" ([allow|deny] and [match|recur]).

What we could do is use the PUT method instead of POST. From RESTful definitions, PUT is the proper way to do it since it specifically tells "replace whatever was there with this new value, or create it as needed".
This way, we could keep the current behavior of POST that is to raise conflict error (conflict being related to the [perm] part, regardless of other "modifiers") while providing the new way to auto-handle overrides with PUT.

@fmigneault
Copy link
Collaborator Author

fmigneault commented Aug 26, 2020

Another idea.

Response wise, we are lucky. Permissions are at the moment returned using format: "permission_names": [ "<perm-string>" ] .
This means we can support both the older format (string) and a new one (objects) quite nicely like follows without keyword conflict and with backward compatibility.

{
  "permission_names": [
     "[perm]-[allow|deny]-[match|recur]"
  ]
  "permissions:" [
    {
      "name": "[perm]", 
      "access": "[allow|deny]", 
      "scope": "[match|recur]", 
    }
  ] 
}

For actual 100% backward compatibility, we could add the plain [perm] variant to permission_names in responses when we detect that internally the database has [perm]-allow-recur (and we could even parse them during POST/PUT). The value would only be a post-processing value for output though, [perm] by itself would never be directly saved in the database anymore.

@dbyrns
Copy link
Contributor

dbyrns commented Aug 31, 2020

I'm a bit confused by all that!
First a bit of context, correct me if I'm wrong. Right now I see that adding a permission is effectively done by a post taking a string. So the only flag we have (match) is encoded in the name (Default behavior being recurse). Also, we can post as many permissions as we want on the same ressource, but an exception is raised if the same permission already exist regardless of the match name-encoded "flag".

What we could do is use the PUT method instead of POST. [...]

So you propose to keep the POST as is, thus requiring one to DELETE, then POST a permission if a flag has to be changed. AND support a PUT that will add the permission or update it with the good flag.

To this point I agree and think we are on the good track :).
However, do you plan to be backward compatible on the POST using only a name-encoded and support proper parameters on the PUT where we can specify access and scope?

Another idea. [...]

Here you refer to the GET /.../permissions
But the bit where the response dynamically switch based on the database content make me unconfortable. What is wrong to return the snippet you provide all the time?

we could even parse them during POST/PUT

Here you partially respond to my question above about how the flags are submitted to the POST/PUT. I understand that the POST could do that for backward compatibility, but for a new route I found it a little bit harsh not you?

[perm] by itself would never be directly saved in the database anymore.

And here I'm completely lost :). Do you mean that we would store the name-encoded flags only using a string as before, but with more options?

@fmigneault
Copy link
Collaborator Author

fmigneault commented Aug 31, 2020

sorry in advance for the wall of text :/

So you propose to keep the POST as is, thus requiring one to DELETE, then POST a permission if a flag has to be changed. AND support a PUT that will add the permission or update it with the good flag.

Yes -ish. Only change is to extend the current POST (not left 100% as is) so that it resolves all variants of [perm]-[allow|deny]-[match|recur] prior to creation or conflict response. At the moment, if [perm] exists and we POST non-existing [perm]-match, there is no (literal string equal) conflict. The pre-conversion would be to respect the requirement:

we should never have more than one permutation from the set [allow-match, allow-recur, deny-match, deny-recur]

We would indeed need to do explicit DELETE, then POST to replace [perm] by [perm]-match, or equivalently, just PUT [perm]-match directly.


But the bit where the response dynamically switch based on the database content make me unconfortable. What is wrong to return the snippet you provide all the time?

There is no dynamic response, both permission_names and permissions would be returned at all times. That was my original intention. You can just pick whichever variant suits your need (or like the best). They are just different representations of the same information, but permissions would be slightly more verbose on what is implied by string representation in permission_names.


However, do you plan to be backward compatible on the POST using only a name-encoded [...]

My intention is to support both str & object format on every method.
The general idea would be as follows:

incoming PUT/POST request

perm = get_perm(request)
if isinstance(perm, dict): 
   perm = perm_as_str(perm)  # convert + handle implicit flags
cur_perm = db.get_str_perm_for_[user/group/service/resource]()
if request is "PUT" and cur_perm:
   db.delete(cur_perm)
if request is "POST" and cur_perm:
  raise Conflict
db.create(perm)
return PUT: 200, POST: 201

for all GET requests:

cur_perm = db.get_str_perm_for_[user/group/service/resource]():
# extra inheritance resolution (as necessary according to request flags)
all_perm = set(cur_perm) + set(<other_applicable_perms>)
return {
   "permission_names": list(all_perm), 
   "permissions": [perm_as_obj(p) for p in all_perm]
}

[perm] by itself would never be directly saved in the database anymore.

And here I'm completely lost :). Do you mean that we would store the name-encoded flags only using a string as before, but with more options?

Yes.
To be backward compatible, I will "handle" the implicit conversion of [perm] to [perm]-allow-recur and [perm]-match to [perm]-allow-match during a POST/PUT, and it is that adjusted value that would be saved in the db.
Also to remain backward compatible, the GET [...]/permissions routes would return both the implicit [perm] and the explicit [perm]-allow-recur in (only) the string representation.
Therefore, if an external script was checking for "read", things won't start breaking suddenly (as it would if only "read-allow-recur" was now returned).
The new format as object will only return the explicit value {"name": "read", "access": "allow", "scope": "recur"} (implicit variant not included since they are of no use).


extra details

Probably better to understand why encoding as string is needed:

Because of ziggurat library running under the hood (as well as other pyramid's views implementation), every "permission" is a string. This means that when I call methods they provide to accomplish resource-tree resolution, I obtain only strings (cannot use the object-variant with extra access and scope flags).
Even worst is that if, let's way, I added 2 new columns in the db for permissions, I would still need to run many successive db queries after fetching the resource-tree, because I would only obtain a recursive representation of resource/permission_name.
The new access/scope flags in separate columns would be missing.
The best is then to encode the full [perm]-[access]-[scope] in the "permission_name" (with guarantee all 3 information are always there), and resolve tree-inheritance as usual by decoding them in-place. This way we avoid reimplementing the whole ziggurat methods.

@dbyrns
Copy link
Contributor

dbyrns commented Sep 1, 2020

The wall of text proved to be beneficial :)
Everything make sense and I approve all of that. This is a great solution, back compatible and compact.

@tlvu
Copy link
Contributor

tlvu commented Sep 16, 2020

I have just skimmed quickly all the previous discussions, I leave it to you for implementation details (ex: POST vs PUT, json payload, etc ...). I just have a few questions:

1 - Can this solution handle the situation in comment Ouranosinc/pavics-sdi#111 (comment) (restrict data access while granting anonymous access to metadata (and browsing)) and that whole issue in general?

2 - In the case of recursive deny, does it make sense to allow something under?

3 - If the parent have match deny but no recursive deny higher, meaning the child is accessible, how does the traversal handle that? Does it stop at the parent and never go down to the child?

@fmigneault
Copy link
Collaborator Author

  1. Theoretically yes, but we might need to implement specific resolution mechanism for thredds service. I don't remember exactly what is the difference between the access point of the metadata vs the actual netcdf file for an URL of the form {...}/thredds/dodsC/birdhouse/testdata/secure/<file>.nc. Is the metadata extracted from catalog instead of dodsC or something else? We might also consider adding more permissions values than only read/write (as currently), by adding something like getmetadata or whatever. The structure would be similar to following:

    thredds   [ (anonymous-group, getmetdata-allow-recursive),  (anonymous-group, read-allow-recursive) ]
      public    # all child resources of dir are read/metadata accessible by anyone (including unauthenticated)
        <some.nc>
      secure   [ (anonymous-group, read-deny-recursive) ]   # override read, metadata still allowed by top-level
        <private.nc>  # accessible ONLY if admin (see extra note below)
    

We basically need to tell was read, write and getmetadata will act on for the following:

Magpie/magpie/services.py

Lines 466 to 479 in 4c9d011

self.expand_acl(self.service, self.request.user)
elems = self.request.path.split("/")
if "fileServer" in elems:
first_idx = elems.index("fileServer")
elif "dodsC" in elems:
first_idx = elems.index("dodsC")
elems[-1] = elems[-1].replace(".html", "")
elif "catalog" in elems:
first_idx = elems.index("catalog")
elif elems[-1] == "catalog.html":
first_idx = elems.index(self.service.resource_name) - 1
else:
return self.acl

  1. The way permissions are parsed, if recursive deny is set, allow-permissions under it will not change anything (deny has priority). Permissions can be set at lower levels in case the deny clause is modified or lifted at later time, but they will remain ineffective while this deny is there.

  2. If deny-match, it will block only exactly that entry (not children under if any). A child would be accessible only if the appropriate match-permission is set on it OR one of its parent as the recursive-permission. It is important to remember that forbidden access remains the default behavior. Setting only deny permissions is redundant to default behavior. It should be used only to do things like "allow everything under some resource except that specific child".

extra note

Point 1 made me think about some special use-case:

If deny is added to group anonymous to block some resource, everybody gets blocked since everyone is a member of it (including admins). We cannot blindly deny in the case of admin. Some validation will need to be done to ensure admins still get pass-through/full-access regardless of this situation (might actually need to handle this special case manually).

If non-admin users (aka member of some other group) would need to have access to this resource that was denied access with anonymous group permission, I don't think it would be possible to employ this approach as groups are dynamically created (not feasible to hardcode some special handling as for admin use-case).

@tlvu
Copy link
Contributor

tlvu commented Sep 17, 2020

  1. I don't remember exactly what is the difference between the access point of the metadata vs the actual netcdf file for an URL of the form {...}/thredds/dodsC/birdhouse/testdata/secure/<file>.nc

Metadata: Just /twitcher/ows/proxy/thredds accessed by the browser.

Data: <service name="http" serviceType="HTTPServer" base="/twitcher/ows/proxy/thredds/fileServer/" />
Data: <service name="odap" serviceType="OpenDAP" base="/twitcher/ows/proxy/thredds/dodsC/" />
Metadata: <service name="ncml" serviceType="NCML" base="/twitcher/ows/proxy/thredds/ncml/"/>
Metadata: <service name="uddc" serviceType="UDDC" base="/twitcher/ows/proxy/thredds/uddc/"/>
Metadata: <service name="iso" serviceType="ISO" base="/twitcher/ows/proxy/thredds/iso/"/>
Not used currently but I think this should be classified as Data: <service name="wcs" serviceType="WCS" base="/twitcher/ows/proxy/thredds/wcs/" />
Not used currently but I think this should be classified as Data: <service name="wms" serviceType="WMS" base="/twitcher/ows/proxy/thredds/wms/" />

See https://github.com/bird-house/birdhouse-deploy/blob/a78ff4af531aa0357710cf5d1d87f6069586b6ac/birdhouse/config/thredds/catalog.xml.template#L6-L14

The actual subtree text (ex: dodsC or ncml or handle/future/sub-subtree) should probably be configurable and not hardcoded to avoid changing Magpie code if Thredds config changes.

  1. It is important to remember that forbidden access remains the default behavior.

So if no access control directive is found at the node and all higher ancestors, it is assumed to be deny-match for that node? This sounds reasonable.

If non-admin users (aka member of some other group) would need to have access to this resource that was denied access with anonymous group permission, I don't think it would be possible to employ this approach as groups are dynamically created (not feasible to hardcode some special handling as for admin use-case).

This case need to be handled somehow. Ex: the secure folder will have recursive deny for anonymous group but then we'll need to open it up for real users or real groups other than admin group.

This discussion brings me to wonder, how do you merge contradicting permissions? If a user belongs to all anonymous, users, projectA, projectB, external-collaborators groups, and the calculated effective permission on the exact same node is different for all of these groups, which one wins?

@dbyrns
Copy link
Contributor

dbyrns commented Sep 17, 2020

The use case we have to solve and express in Ouranosinc/pavics-sdi#111 (comment) is the following:
Lets say we have a tree like this : Root/PrivateDir/Child1
Anonymous group has a Root recursive-allow and PrivateDir recursive-deny permissions
GroupA has a read-match permission on Child1
As an anonymous user I should get access to all but PrivateDir ressources
As a member of GroupA I should get the read access on Child1

Based on the previously posted implementation :

bool authorized = False
if deny(file):
    return 401
if allow(file):
    authorized = True
# Parsing in bottom up order
parents = [dir, root]
for resource in parents:
    if deny-recursive(resource):
        return 401
    if allow-recursive(resource):
        authorized = True
if authorized:
    return 200

For admins, I think that anyway we can return authorized up front whatever the permissions (it is the case right now, no?)

If we now take into account the groups, it will means that the deny, allow deny-recursive and allow-recursive fcts are resolved using the inherited flag (taking into account all membership)? So, indeed being part of any group having a deny permission will block you from this ressource. Could it be possible to resolve the previous snippet for each of our group and return the authorization based on an OR basis of each group permission?

@dbyrns
Copy link
Contributor

dbyrns commented Sep 17, 2020

This discussion brings me to wonder, how do you merge contradicting permissions? If a user belongs to all anonymous, users, projectA, projectB, external-collaborators groups, and the calculated effective permission on the exact same node is different for all of these groups, which one wins?

To answer that specifically, up to now, both allow and deny permissions were ORed and deny had priority. So if you are allowed on any of the group you have a pending allowed permission, but if a deny permission applied on any of those groups the pending permission is revoked. I think that we should change the behavior as express in my previous comment.

@tlvu
Copy link
Contributor

tlvu commented Sep 17, 2020

Could it be possible to resolve the previous snippet for each of our group and return the authorization based on an OR basis of each group permission?

Agreed, this will allow setting very strict and aggressive recursive deny for anonymous group but will not block other groups with more forgiving permissions.

@tlvu
Copy link
Contributor

tlvu commented Sep 17, 2020

In Magpie web UI, I remember there is a button or checkbox "Show inherited permissions", does this button also show the effective combined group permissions for a user?

If not, it would be immensely useful to give the admin a way to visually see the effective permissions of a user taking in account all his groups.

@dbyrns
Copy link
Contributor

dbyrns commented Sep 17, 2020

In Magpie web UI, I remember there is a button or checkbox "Show inherited permissions", does this button also show the effective combined group permissions for a user?

In fact it's exclusively what it does. Right now, it combined the inherited permissions from groups (or not if unchecked) but does not resolve/show the recursive permissions.

If you look at the mock UI above, you'll see what we intend to do to better reflect the real effective permissions (what you see is what you get). A combobox will let you chosse the allow|deny/match|recurse flags and the read-only checkbox beside will let you know the resulting permission once all the allow/deny/recurse is resolved. The inherited permissions will still trigger if we include your group membership in the resolution or not.

@fmigneault
Copy link
Collaborator Author

fmigneault commented Sep 17, 2020

@tlvu

So if no access control directive is found at the node and all higher ancestors, it is assumed to be deny-match for that node? This sounds reasonable.

Yes. This is the current behavior and will remain in effect.

@dbyrns

For admins, I think that anyway we can return authorized up front whatever the permissions (it is the case right now, no?)

Yes. That is the intention. I will like to have a test that validates this though, just to ensure that admin-group has more importance than anonymous-group deny-permission when resolving the allow/deny.

Anonymous group has a Root recursive-allow and PrivateDir recursive-deny permissions
GroupA has a read-match permission on Child1
As a member of GroupA I should get the read access on Child1

No. That's where the issue comes in. Because everyone is a member of anonymous-group, if you set deny-recursive on PrivateDir, you effectively say "block everyone to access PrivateDir and its children" (final decision), and that rule has priority over "allow any GroupA user read on Child1". If denies don't have priority over allows, they will never work (eg: "Anonymous group has a Root recursive-allow" would override "PrivateDir recursive-deny") defeating the original purpose.

Keeping in mind that default is to block. So Deny are only for exceptions, and as such must override any previously allowed permission. The combination is something like:

acces = UserAllowPerm OR OneOf[UserGroupsAllowPerm] AND NOT Any[UserDenyPerm, UserGroupDenyPerm]
Allow if access else Deny

This is also how all pyramid permission scheme works:
https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html?highlight=deny#elements-of-an-acl

Could it be possible to resolve the previous snippet for each of our group and return the authorization based on an OR basis of each group permission?
Agreed, this will allow setting very strict and aggressive recursive deny for anonymous group but will not block other groups with more forgiving permissions.

There is no way to say GroupA is more important than GroupB, so it is not possible to say Allow-GroupA, but Deny-GroupB, and then have a User member of both GroupA and GroupB to be allowed.
The only "special" groups are admin and anonymous which we can provide special handling. Groups are already resolved using OR, and this is why OR(Allow-GroupA, Deny-GroupB) resolves to Deny. The deny is already very strict as it is the default.

I would argue that opening everything with recursive root permission, to then deny things one by one is only trying to work in the opposite manner Magpie works. Another set of permissions should be considered.

@dbyrns
Copy link
Contributor

dbyrns commented Sep 17, 2020

No. That's where the issue comes in

I know that's not how it works. I exposed the use case that should work.

If denies don't have priority over allows, they will never work

Yeah you're absolutely right! If I bring back my example, what's the difference between allow on Root, deny on PrivateDir : I want it to be denied. And deny on PrivateDir and allow on ChildA : I want it to be allowed.

In fact, the difference is the depth and since it's our own code that handle the recursive strategy and not pyramid, what could block us to handle it like this:

if deny(file):
    return 401
if allow(file):
    return 200
# Parsing in bottom up order
parents = [dir, root]
for resource in parents:
    if deny-recursive(resource):
        return 401
    if allow-recursive(resource):
        return 200

The difference is that the deepest permission always overide recursion, deny or allow. At the same level, deny still has priority. So if we got an allow permission on a file, I doesn't care that the whole branch has been forbidden for one of my group.

The drawback is that we would go against every other implementations... There must be a catch?

@dbyrns
Copy link
Contributor

dbyrns commented Sep 17, 2020

In fact, I think I just exposed how NTFS works : https://www.ntfs.com/ntfs-permissions-precedence.htm
So my main drawback is suddently less of a concern.

@fmigneault
Copy link
Collaborator Author

I agree NTFS' implementation makes a lot of sense and should handle all use cases for the recursive and inherited resolution (permissions closer to the actual item have more weight).

Just for completion, I would like to validate how to handle multiple group conflicting permissions on specific entries (no recursion involved and no direct user-permission). The NTFS doc does not explicitly details this properly. It only provides an example of additive different permissions (read+write) in no.4, and user priority > group inheritance in no.2, but not conflicting groups permissions at the same level. More specifically :

If groupA has deny-match, while groupB has allow-match, should user with both groupA and groupB memberships be allowed or denied, since they have equal priority. I can see the 2 variants having similarly strong arguments:

  1. When same level resource-groups have priority Allow > Deny, users that are intended to be denied via deny-groupA could mistakenly be allowed because of some other allow-groupB. On the other hand, if there was already defined deny-groupA, and an admin creates allow-groupB, the user wouldn't remain denied (otherwise, would be confusing since we just set the allow and "does not seem to work").
  2. When on the opposite we define Deny > Allow priority, groupA-deny would normally be set for an important reason (invert previous allow), so their users should not suddenly obtain more permissions via groupB-allow. Also, if only groupB-allow existed, but admin set groupA-deny, the resource would become now become restricted for these members as expected, instead of remaining allowed.

Because of case no.1of NTFS ""Deny" permissions generally take precedence over "allow" permissions", I would tend toward resolving to deny as higher priority, but both decisions make sense depending on context and can be justified.

@dbyrns
Copy link
Contributor

dbyrns commented Sep 18, 2020

I would, like you, definitely stick with deny has priority over allow. But in this case I guess that pyramid framework will do the resolution for you (at the same level). Hopefully we will not have to work against it.
Also, for your case #1, even if it can looks confusing, the ui designed with the effective permission should help to know what is going on. If the effective result is not understood one can still uncheck inherited or look into each group.

@dbyrns
Copy link
Contributor

dbyrns commented Sep 18, 2020

I would like to thank @tlvu also, since because of your intervention we will get in the end a more versatile design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api Something related to the API operations feature New feature to be developed ui Something related to the UI operations or display
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants