Handle Permissions on Properties correctly when searching

[tobiasschweizer]

At the moment, permissions are only checked on resources when searching.

However, when displaying a resources some of its properties may not be visible due to restrictions.

Example: Do a simple search for Berthold or an extended search for a book having Berthold as its creator. In the result, Berthold shows up as a creator but when you open the book, the creator property is not visible (only when you log in).

Permission checking on resources has to be added in the following SPARQL templates:

• searchFulltext.scala.txt
• searchExtended.scala.txt
• getResourceSearchResult.scala.txt

[benjamingeer]

I've been trying to implement this, and I'm not sure it's feasible with our current design.

There are two problems:

Counting results

First, we'd have to return the IRI, owner, project, and permissions on each matching value of each matching resource. But that means we'd get one row per matching value, and our whole design is based on the requirement that we get only one row per matching resource, so we can count the number of matching resources. For example, this is this query for return 18 books when we search for book titles that do not include the string 'Zeitglöcklein' (using a regular expression), modified to get value permissions:

prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix knora-base: <http://www.knora.org/ontology/knora-base#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

SELECT DISTINCT
    ?resource
    ?resourceLabel 
    ?resourceClass
    ?previewPath
    ?previewDimX
    ?previewDimY
    ?resourceOwner
    ?resourceProject
        ?valueObject0
        ?valueOwner0
        ?valueProject0
        ?targetResource0
        ?targetResourceOwner0
        ?targetResourceProject0
        (SAMPLE(?anyLiteral0) AS ?literal0)
        (GROUP_CONCAT(DISTINCT(?valuePermission0); separator=";") AS ?valuePermissions0)
        (GROUP_CONCAT(DISTINCT(?targetResourcePermission0); separator=";") AS ?targetResourcePermissions0)
    (GROUP_CONCAT(DISTINCT(?resourcePermission); separator=";") AS ?resourcePermissions)

    FROM <http://www.ontotext.com/explicit>

WHERE {
    BIND(STR("de") AS ?preferredLanguage)
    BIND(STR("en") AS ?fallbackLanguage)

            ?resource <http://www.knora.org/ontology/incunabula#title> ?valueObject0 ;
                knora-base:isDeleted false .
            ?valueObject0 knora-base:isDeleted false .

            ?valueObject0 knora-base:attachedToUser ?valueOwner0 .

            OPTIONAL {
                ?valueObject0 knora-base:attachedToProject ?valueProject0 .
            }

            ?valuePermissionProp0 rdfs:subPropertyOf+ knora-base:hasPermission .

            OPTIONAL {
                ?valueObject0 ?valuePermissionProp0 ?valuePermissionGroup0 .
                BIND(CONCAT(STR(?valuePermissionProp0), ",", STR(?valuePermissionGroup0)) as ?valuePermission0)
            }

        # Search criterion 0: operator !LIKE

                ?valueObject0 knora-base:valueHasString ?anyLiteral0 .

                FILTER (!regex(?anyLiteral0, 'Zeitglöcklein', "i"))

    ?resource knora-base:isDeleted false .
    ?resource rdfs:label ?resourceLabel .
    ?resource a ?resourceClass .

            # filter by restypeIri
            ?resource a <http://www.knora.org/ontology/incunabula#book> .

    OPTIONAL {
        ?resource knora-base:hasStillImageFileValue ?fileValue .
        ?fileValue a knora-base:StillImageFileValue .
        ?fileValue knora-base:isPreview true .
        ?fileValue knora-base:internalFilename ?previewPath .

        OPTIONAL {
            ?fileValue knora-base:dimX ?previewDimX .
            ?fileValue knora-base:dimY ?previewDimY .
        }
    }

    OPTIONAL {
        ?resourcePermissionProp rdfs:subPropertyOf+ knora-base:hasPermission .
        ?resource ?resourcePermissionProp ?resourcePermissionGroup .
        BIND(CONCAT(STR(?resourcePermissionProp), ",", STR(?resourcePermissionGroup)) as ?resourcePermission)
    }

    OPTIONAL {
        ?resource knora-base:attachedToUser ?resourceOwner .
    }

    OPTIONAL {
        ?resource knora-base:attachedToProject ?resourceProject .
    }
}

GROUP BY
    ?resource
    ?resourceLabel
    ?resourceClass
    ?previewPath
    ?previewDimX
    ?previewDimY
    ?resourceOwner
    ?resourceProject
        ?valueObject0
        ?valueOwner0
        ?valueProject0
        ?targetResource0
        ?targetResourceOwner0
        ?targetResourceProject0    
ORDER BY ?resource
        OFFSET 0
        LIMIT 25

There are 18 matching resources, but it returns 24 results, because there are some resources with multiple values that match.

Poor performance

Second, there seems to be a huge impact on performance. For example, the test return all the pages that are part of Zeitglöcklein des Lebens takes 665 milliseconds with the current code, but 12 seconds (i.e. 6 seconds for the COUNT and 6 to get the results) if it has to get permissions on all the link values and on the target resource, like this:

prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix knora-base: <http://www.knora.org/ontology/knora-base#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>


SELECT DISTINCT
    ?resource
    ?resourceLabel 
    ?resourceClass
    ?previewPath
    ?previewDimX
    ?previewDimY
    ?resourceOwner
    ?resourceProject
        ?valueObject0
        ?valueOwner0
        ?valueProject0
        ?targetResource0
        ?targetResourceOwner0
        ?targetResourceProject0
        (SAMPLE(?anyLiteral0) AS ?literal0)
        (GROUP_CONCAT(DISTINCT(?valuePermission0); separator=";") AS ?valuePermissions0)
        (GROUP_CONCAT(DISTINCT(?targetResourcePermission0); separator=";") AS ?targetResourcePermissions0)
    (GROUP_CONCAT(DISTINCT(?resourcePermission); separator=";") AS ?resourcePermissions)

    FROM <http://www.ontotext.com/explicit>

WHERE {
    BIND(STR("de") AS ?preferredLanguage)
    BIND(STR("en") AS ?fallbackLanguage)

        # Search criterion 0: operator EQ

                        BIND(IRI("http://data.knora.org/c5058f3a") AS ?targetResource0)

                        ?resource <http://www.knora.org/ontology/incunabula#partOf> <http://data.knora.org/c5058f3a> .

                        <http://data.knora.org/c5058f3a> rdfs:label ?anyLiteral0 ;
                            knora-base:isDeleted false ;
                            knora-base:attachedToUser ?targetResourceOwner0 ;
                            knora-base:attachedToProject ?targetResourceProject0 .

                        OPTIONAL {
                            ?targetResourcePermissionProp0 rdfs:subPropertyOf+ knora-base:hasPermission .
                            <http://data.knora.org/c5058f3a> ?targetResourcePermissionProp0 ?targetResourcePermissionGroup0 .
                            BIND(CONCAT(STR(?targetResourcePermissionProp0), ",", STR(?targetResourcePermissionGroup0)) as ?targetResourcePermission0)
                        }

                        ?resource <http://www.knora.org/ontology/incunabula#partOfValue> ?valueObject0 .

                        ?valueObject0 rdf:type knora-base:LinkValue ;
                            knora-base:isDeleted false ;
                            rdf:subject ?resource ;
                            rdf:predicate <http://www.knora.org/ontology/incunabula#partOf> ;
                            rdf:object <http://data.knora.org/c5058f3a> .

                        ?valueObject0 knora-base:attachedToUser ?valueOwner0 .

                        OPTIONAL {
                            ?valueObject0 knora-base:attachedToProject ?valueProject0 .
                        }

                        OPTIONAL {
                            ?valuePermissionProp0 rdfs:subPropertyOf+ knora-base:hasPermission .
                            ?valueObject0 ?valuePermissionProp0 ?valuePermissionGroup0 .
                            BIND(CONCAT(STR(?valuePermissionProp0), ",", STR(?valuePermissionGroup0)) as ?valuePermission0)
                        }

    ?resource knora-base:isDeleted false .
    ?resource rdfs:label ?resourceLabel .
    ?resource a ?resourceClass .

            # filter by restypeIri
            ?resource a <http://www.knora.org/ontology/incunabula#page> .

    OPTIONAL {
        ?resource knora-base:hasStillImageFileValue ?fileValue .
        ?fileValue a knora-base:StillImageFileValue .
        ?fileValue knora-base:isPreview true .
        ?fileValue knora-base:internalFilename ?previewPath .

        OPTIONAL {
            ?fileValue knora-base:dimX ?previewDimX .
            ?fileValue knora-base:dimY ?previewDimY .
        }
    }

    OPTIONAL {
        ?resourcePermissionProp rdfs:subPropertyOf+ knora-base:hasPermission .
        ?resource ?resourcePermissionProp ?resourcePermissionGroup .
        BIND(CONCAT(STR(?resourcePermissionProp), ",", STR(?resourcePermissionGroup)) as ?resourcePermission)
    }

    OPTIONAL {
        ?resource knora-base:attachedToUser ?resourceOwner .
    }

    OPTIONAL {
        ?resource knora-base:attachedToProject ?resourceProject .
    }
}

GROUP BY
    ?resource
    ?resourceLabel
    ?resourceClass
    ?previewPath
    ?previewDimX
    ?previewDimY
    ?resourceOwner
    ?resourceProject
        ?valueObject0
        ?valueOwner0
        ?valueProject0
        ?targetResource0
        ?targetResourceOwner0
        ?targetResourceProject0
ORDER BY ?resource
        OFFSET 0
        LIMIT 500

I don't know what we should do about this.

cc @lrosenth @subotic

[benjamingeer]

To deal with the counting problem, we could eliminate the COUNT queries altogether, and do the paging entirely in Scala. This would mean getting all the results each time a page is requested. Eliminating the COUNT query would also improve performance a lot.

We could also improve performance by automatically granting everyone view permission on all links, as long as you have permission to see the source and target resources. In the example above with return all the pages that are part of Zeitglöcklein des Lebens, this reduces the response time from 12 seconds to 5 seconds. This would be a reversal of the decision in #88.

We could also drastically limit the maximum number of results, e.g. to 100.

[benjamingeer]

Also, in the specific case where you're searching for a link to a particular resource using EQUALS, it would be much more efficient to do a separate query to get the permissions on that resource, rather than having them repeated in each row of the search results.

[benjamingeer]

I'm going to try this approach:

• Eliminate the COUNT queries as described above, and do the counting and paging entirely in Scala.
• Don't query permissions on target resources in the main search query. Instead, just collect their IRIs and do a separate query to get their permissions.

[subotic]

Great, sounds good.

[benjamingeer]

This seems to work OK. In SearchResponderV1Spec, all the tests are quite fast except for return all the pages that are part of Zeitglöcklein des Lebens and have a seqnum. For reasons I don't understand, it takes 6 seconds on GraphDB 6.6.4 and 4 seconds on Fuseki 2.3.0. But I think this is good enough for now.