<a> tag mapping to manage URL #fragment #1440

gfoo · 2019-09-17T09:01:20Z

resolves #1346

gfoo · 2019-09-18T07:03:37Z

Add URL #fragment <a> mapping
knora-ontologies/standoff-data.ttl generation (@loicjaouen @tobiasschweizer any docs somewhere?)
Update Knora doc
Add test
Upgrade script

gfoo · 2019-09-18T07:14:45Z

@loicjaouen @tobiasschweizer any docs somewhere?

I think I have to:

create the standard-mapping.json

{
    "knora-api:mappingHasName": "???",
    "knora-api:attachedToProject": {
        "@id": "???"
    },
    "rdfs:label": "???",
    "@context": {
        "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
        "knora-api": "http://api.knora.org/ontology/knora-api/v2#"
    }
}

call the endpoint to generate the new mapping:

$ curl -u [email protected]:test -X POST -F [email protected] -F [email protected]  http://localhost:3333/v2/mapping

generate the ttl:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
Construct {
    ?mapping ?p ?o .
    ?mapping knora-base:hasMappingElement ?mEle .
    ?mEle ?pp ?oo .
    ?oo ?ppp ?ooo .
}
WHERE {
    BIND(<http://rdfh.ch/standoff/mappings/StandardMapping> as ?mapping)
    ?mapping ?p ?o .

    OPTIONAL {
        ?mapping knora-base:hasMappingElement ?mEle .
    	OPTIONAL {
        	?mEle ?pp ?oo .
    		OPTIONAL {
        		?oo ?ppp ?ooo .
            }
        }
    }
}

To update this file knora-ontologies/standoff-data.ttl

gfoo · 2019-09-18T07:32:25Z

I'm a little afraid of updating the ttl using a sparql query because of IRIs... maybe we should update the ttl by hand? :|

And what about the upgrading of an existing Knora base?

tobiasschweizer · 2019-09-18T07:32:40Z

Yes, I think you have to add the new element to /Users/tobi/gitreps/Knora_github/webapi/_test_data/test_route/texts/mappingForStandardHTML.xml, create the new mapping in Knora and update /Users/tobi/gitreps/Knora_github/knora-ontologies/standoff-data.ttl.

Maybe you can do this incrementally, otherwise the IRIs will be all different and every TextValue with standoff would have to be updated (its reference to the mapping).

tobiasschweizer · 2019-09-18T07:34:55Z

I'm a little afraid of updating the ttl using a sparql query because of IRIs.

I am suffering from PTSD: posttraumatic standoff disorder myself ..

gfoo · 2019-09-18T09:18:23Z

@tobiasschweizer is there any tests to update?

@tobiasschweizer @benjamingeer As you can see in this commit 380121a I have just to add few triplets to an existing Knora base to upgrade it. Is that correct for you? Any guide lines or I just try to copy/paste the way you are doing your upgrade scripts in upgrade folder?

tobiasschweizer · 2019-09-18T09:24:55Z

is there any tests to update?

You added an element to the existing mapping, so everything is expected to run as before (so the tests should pass).

I think you should add a test that checks the new element.

gfoo · 2019-09-18T09:26:14Z

I think you should add a test that checks the new element.

in which test file? (sorry, never used your test framework...)

tobiasschweizer · 2019-09-18T09:27:37Z

in which test file?

that would be /Users/tobi/gitreps/Knora_github/webapi/src/test/scala/org/knora/webapi/e2e/v1/StandoffV1R2RSpec.scala

gfoo · 2019-09-18T13:21:12Z

Not very confortable with the CI failed, any clues?

tobiasschweizer · 2019-09-18T13:23:32Z

https://travis-ci.org/dhlab-basel/Knora/jobs/586477300#L6096

gfoo · 2019-09-18T15:38:00Z

@tobiasschweizer
I don't understand why the test perform a fulltext search for 'Dinge' fails, probably a side effect of my modification of standoff-data.ttl, but why?

It seems that an internal-link appears in the result (generating the diff error). If the test data come from the anything-data.ttl, there is no such data like that.

The cited value in the result is a TextValue containing a StandoffLinkTag but not an internal link...

tobiasschweizer · 2019-09-19T07:11:19Z

I don't understand why the test perform a fulltext search for 'Dinge' fails, probably a side effect of my modification of standoff-data.ttl, but why?

I think this is because the mapping is returned as well at it contains more elements than before.

tobiasschweizer · 2019-09-19T07:19:59Z

see https://github.com/dhlab-basel/Knora/blob/6555a8e7e4dfeb280c44c3bebc6fabc8ea8fcac3/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala#L1246-L1253

…4d2cff was correct!)

gfoo · 2019-09-19T09:45:15Z

I added a v1 test there using this data as input, but when I try to send the same data to the v2 API I get an error:

{
  "@type" : "anything:Thing",
  "anything:hasRichtext" : {
    "@type" : "knora-api:TextValue",
    "knora-api:textValueAsXml" : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<text><p>ref to note <a class=\"internal-link\" href=\"#_note1\" id=\"_ref-note1\">[1]</a></p><p><a class=\"internal-link\" href=\"#_ref-note1\" id=\"_note1\">[1]</a> note 1</p></text>",
    "knora-api:textValueHasMapping" : {
      "@id" : "http://rdfh.ch/standoff/mappings/StandardMapping"
    }
  },
  "knora-api:attachedToProject" : {
    "@id" : "http://rdfh.ch/projects/0001"
  },
  "rdfs:label" : "test thing",
  "@context" : {
    "rdf" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "knora-api" : "http://api.knora.org/ontology/knora-api/v2#",
    "rdfs" : "http://www.w3.org/2000/01/rdf-schema#",
    "xsd" : "http://www.w3.org/2001/XMLSchema#",
    "anything" : "http://0.0.0.0:3333/ontology/0001/anything/v2#"
  }
}

{
    "knora-api:error": "org.knora.webapi.AssertionException: Resource <http://rdfh.ch/0001/JdZmxDDiSvi_OqU7I5Ur0g> was saved, but one or more of its values are not correct",
    "@context": {
        "knora-api": "http://api.knora.org/ontology/knora-api/v2#"
    }
}

@benjamingeer it sounds like an encoding problem?

gfoo · 2019-09-19T09:56:35Z

I'm going to try this test there

gfoo · 2019-09-19T11:37:15Z

Error is ~~repoduced~~ there: https://travis-ci.org/dhlab-basel/Knora/jobs/586938555#L570

Update: not really reporduced because the errors are differents:

First error found using v2 API was saved, but one or more of its values are not correct
Error found by the test is not the same as the one that was submitted

gfoo · 2019-09-20T08:19:04Z

@benjamingeer my test above reveals an error described here #1221 (fixed here #1224)

Don't know if it is related to my first error: was saved, but one or more of its values are not correct ?

tobiasschweizer · 2019-09-25T14:43:31Z

@benjamingeer

That would be great, because I don't understand what you mean by standoff attribute conversions.

Please see 41475fd. Does this make sense to you?

tobiasschweizer · 2019-09-25T14:46:28Z

The SPARQL results for a StandoffInternalReferenceTag should include knora-base:targetHasOriginalXMLID. Try it and see.

I ran a v1 test to create the test data.

benjamingeer · 2019-09-25T14:47:10Z

I ran a v1 test to create the test data.

OK but your test is in v2. It would make sense to test both cases.

tobiasschweizer · 2019-09-25T14:53:42Z

OK but your test is in v2. It would make sense to test both cases.

StandoffUtilV2 is used both in v2 and v1. Yes, I will try to produce tests data for v2 too.

benjamingeer · 2019-09-25T14:55:08Z

webapi/src/main/scala/org/knora/webapi/util/standoff/StandoffTagUtilV2.scala

-                                    val originalId: String = standoffAssertions(value).getOrElse(OntologyConstants.KnoraBase.StandoffTagHasOriginalXMLID, throw InconsistentTriplestoreDataException(s"referred standoff $value node has no original XML id"))
+
+                                    // If a v2 SPARQL template was used, the XML ID is in this standoff tag.
+                                    val originalId: String = standoffTagAssertions.get(OntologyConstants.KnoraBase.TargetHasOriginalXMLID) match {


This line is used for v2.

benjamingeer · 2019-09-25T14:55:17Z

webapi/src/main/scala/org/knora/webapi/util/standoff/StandoffTagUtilV2.scala

+
+                                        case None =>
+                                            // If a v1 SPARQL template was used, we have to get the target node and to get its XML ID.
+                                            standoffAssertions(value).getOrElse(OntologyConstants.KnoraBase.StandoffTagHasOriginalXMLID, throw InconsistentTriplestoreDataException(s"referred standoff $value node has no original XML id"))


This line is used for v1.

tobiasschweizer · 2019-09-25T15:17:50Z

Ok I added tests for v1 and v2 in 40be041

Sorry, but I still don't understand the v2 case: Wouldn't createStandoffTagsV2FromSelectResults only be called after all pages of standoff have been gathered?

tobiasschweizer · 2019-09-25T15:20:26Z

Or are the pages of standoffAssertions converted to StandoffTagV2 in single steps and combined?

benjamingeer · 2019-09-25T15:23:05Z

The method createStandoffTagsV2FromSelectResults is called by createStandoffTagsV2FromConstructResults, which is called by ConstructResponseUtilV2.makeTextValueContentV2 for each page of standoff results. This is how the standoff route returns one page of standoff at a time to the client. I really recommend reading the comments in ConstructResponseUtilV2.makeTextValueContentV2.

tobiasschweizer · 2019-09-25T15:27:49Z

Ok, I understand! So v2 needs to read standoff in parts.

I really recommend reading the comments in ConstructResponseUtilV2.makeTextValueContentV2.

This the last day I can work on this before my holiday and I really try the best to be helpful.

benjamingeer · 2019-09-25T15:29:53Z

I really try the best to be helpful.

And I'm trying to be helpful by writing comments in the code so you can read them. :)

tobiasschweizer · 2019-09-25T15:36:49Z

And I'm trying to be helpful by writing comments in the code so you can read them. :)

Normally I would have. But today I just wanted to add that additional test ;-) You know how it is ...

…erties.

benjamingeer · 2019-09-25T15:39:41Z

OK I think that should fix it, let's see if all the tests pass.

tobiasschweizer · 2019-09-25T15:40:35Z

OK I think that should fix it, let's see if all the tests pass.

great, thx.

…LID.

benjamingeer · 2019-09-25T17:02:39Z

@tobiasschweizer Can we merge this now?

tobiasschweizer

@gfoo Thanks for this PR!

That was a quick and easy one ;-)

tobiasschweizer · 2019-09-25T17:19:17Z

@benjamingeer Yes, please go ahead.

tobiasschweizer · 2019-09-25T17:22:52Z

or as we say in Swiss German: "möööörtsch"

benjamingeer · 2019-09-25T18:14:47Z

@tobiasschweizer Thanks for your help, and have a good holiday!

gfoo · 2019-09-25T18:22:05Z

what a triplet team... ah ah ah :^)

feat: Added <a> tag mapping to manage URL #fragment

f6ead86

docs: updated

9339c11

gfoo added 2 commits September 18, 2019 10:22

doc: incremental update (by hand) of standoff-data.ttl

ec7d313

feat: updated due to new standard mapping

380121a

fix: indent

e2a5c87

gfoo added 3 commits September 18, 2019 11:52

test: HTML with internal anchor link

bd11ffc

fix: md typo

339c708

fix: md typo

e3513a1

gfoo added 2 commits September 18, 2019 16:19

fix: missing internal-link def in test data

a4d2cff

fix: undo last modifications (error comes from elsewhere)

6555a8e

fix: missing internal-link def in test data (finally fix in commit #a…

bfa2750

…4d2cff was correct!)

test: added internal-link on text value with standoff creation

62124f1

Benjamin Geer added 2 commits September 25, 2019 14:50

fix(standoff): Fix so v1 SPARQL queries also work.

bfe2f1d

feat(standoff): Add update script.

108b9f2

benjamingeer self-assigned this Sep 25, 2019

benjamingeer added the breaking/data label Sep 25, 2019

test (standoff): add tests for StandoffUtilV2

41475fd

benjamingeer reviewed Sep 25, 2019

View reviewed changes

test (standoff): add tests for StandoffUtilV2 for v1 and v2

40be041

fix(standoff): Include targetHasOriginalXMLID in standoff system prop…

5051772

…erties.

refactor(standoff): Remove redundant filtering of targetHasOriginalXM…

a35c8be

…LID.

tobiasschweizer approved these changes Sep 25, 2019

View reviewed changes

benjamingeer merged commit 08b4aa6 into develop Sep 25, 2019

benjamingeer deleted the wip/standoff-internal-anchor branch September 25, 2019 18:14

<a> tag mapping to manage URL #fragment #1440

<a> tag mapping to manage URL #fragment #1440

Conversation

gfoo commented Sep 17, 2019

gfoo commented Sep 18, 2019 • edited by benjamingeer Loading

gfoo commented Sep 18, 2019 • edited Loading

gfoo commented Sep 18, 2019

tobiasschweizer commented Sep 18, 2019 • edited Loading

tobiasschweizer commented Sep 18, 2019

gfoo commented Sep 18, 2019

tobiasschweizer commented Sep 18, 2019

gfoo commented Sep 18, 2019

tobiasschweizer commented Sep 18, 2019

gfoo commented Sep 18, 2019

tobiasschweizer commented Sep 18, 2019

gfoo commented Sep 18, 2019

tobiasschweizer commented Sep 19, 2019

tobiasschweizer commented Sep 19, 2019

gfoo commented Sep 19, 2019

gfoo commented Sep 19, 2019

gfoo commented Sep 19, 2019 • edited Loading

gfoo commented Sep 20, 2019

tobiasschweizer commented Sep 25, 2019 • edited Loading

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer Sep 25, 2019

Choose a reason for hiding this comment

benjamingeer Sep 25, 2019

Choose a reason for hiding this comment

tobiasschweizer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

tobiasschweizer left a comment

Choose a reason for hiding this comment

tobiasschweizer commented Sep 25, 2019

tobiasschweizer commented Sep 25, 2019

benjamingeer commented Sep 25, 2019

gfoo commented Sep 25, 2019

gfoo commented Sep 18, 2019 •

edited by benjamingeer

Loading

gfoo commented Sep 18, 2019 •

edited

Loading

tobiasschweizer commented Sep 18, 2019 •

edited

Loading

gfoo commented Sep 19, 2019 •

edited

Loading

tobiasschweizer commented Sep 25, 2019 •

edited

Loading