Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite date processing in API v2 #1006

Merged
merged 7 commits into from
Oct 12, 2018
Merged

Rewrite date processing in API v2 #1006

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6fe7b14
refactor (api-v2): Start refactoring date utility code (#928).
Oct 9, 2018
2bbba03
refactor (api-v2): Rewrite date processing in API v2 using ICU4J.
Oct 10, 2018
dcb087d
test (CalendarDateUtilV2): Add more era tests.
Oct 10, 2018
06c73e7
test (CalendarDateUtilV2): Add remaining tests from v1.
Oct 10, 2018
9f732de
style (api-v2): Clean up code a bit.
Oct 11, 2018
0a4bf0e
test (CalendarDateUtilV2): Clean up code.
Oct 11, 2018
c1977f8
feature (CalendarDateUtilV2): Check that start date is not later than…
Oct 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion webapi/WebapiBuild.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,8 @@ lazy val webApiLibs = Seq(
library.springSecurityCore,
library.swaggerAkkaHttp,
library.typesafeConfig,
library.xmlunitCore
library.xmlunitCore,
library.icu4j
)

lazy val library =
Expand Down Expand Up @@ -318,6 +319,8 @@ lazy val library =

// Java EE modules which are deprecated in Java SE 9, 10 and will be removed in Java SE 11
val jaxbApi = "javax.xml.bind" % "jaxb-api" % "2.2.12"

val icu4j = "com.ibm.icu" % "icu4j" % "62.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get rid of val jodd = "org.jodd" % "jodd" % "3.2.6"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you want to leave DateUtilV1 as it is? Could we make call v1 v2 methods?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DateUtilV1 uses jodd. I could refactor it to use the v2 code, but that would be more work. My idea was to avoid changing v1 if possible, because the priority is to finish v2.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could refactor it to use the v2 code, but that would be more work. My idea was to avoid changing v1 if possible, because the priority is to finish v2.

I understand. My concern is that both versions of the API have to behave identically when converting from and to JDN and that this might be more difficult to assure if we use two different libraries for v1 and v2.

But of course, there could be a version update of one of those libraries and then we would have the same issue if the new version behaves differently from the old one.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests for v1 and v2 are identical, so if that happens, I think we'll know. I'd be willing to change v1 to use v2, but I'd rather do it later, when v2 is done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, could you create a separate issue for that so we can do it later?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1007

}

lazy val javaRunOptions = Seq(
Expand Down
169 changes: 99 additions & 70 deletions ...src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,13 @@ import akka.pattern._
import akka.util.Timeout
import org.knora.webapi._
import org.knora.webapi.messages.admin.responder.usersmessages.UserADM
import org.knora.webapi.messages.v1.responder.valuemessages.{JulianDayNumberValueV1, KnoraCalendarV1, KnoraPrecisionV1}
import org.knora.webapi.messages.v2.responder._
import org.knora.webapi.messages.v2.responder.resourcemessages.ReadResourceV2
import org.knora.webapi.messages.v2.responder.standoffmessages.{GetMappingRequestV2, GetMappingResponseV2, MappingXMLtoStandoff, StandoffDataTypeClasses}
import org.knora.webapi.twirl.{StandoffTagAttributeV2, StandoffTagInternalReferenceAttributeV2, StandoffTagIriAttributeV2, StandoffTagV2}
import org.knora.webapi.util.DateUtilV2.{DateYearMonthDay, KnoraEraV2}
import org.knora.webapi.util.IriConversions._
import org.knora.webapi.util._
import org.knora.webapi.util.date._
import org.knora.webapi.util.jsonld._
import org.knora.webapi.util.standoff.StandoffTagUtilV2.TextWithStandoffTagsV2
import org.knora.webapi.util.standoff.{StandoffTagUtilV2, XMLUtil}
Expand Down Expand Up @@ -638,37 +637,38 @@ object ValueContentV2 extends ValueContentReaderV2[ValueContentV2] {
case class DateValueContentV2(ontologySchema: OntologySchema,
valueHasStartJDN: Int,
valueHasEndJDN: Int,
valueHasStartPrecision: KnoraPrecisionV1.Value,
valueHasEndPrecision: KnoraPrecisionV1.Value,
valueHasCalendar: KnoraCalendarV1.Value,
valueHasStartPrecision: DatePrecisionV2,
valueHasEndPrecision: DatePrecisionV2,
valueHasCalendar: CalendarNameV2,
comment: Option[String] = None) extends ValueContentV2 {
override def valueType: SmartIri = {
implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance
OntologyConstants.KnoraBase.DateValue.toSmartIri.toOntologySchema(ontologySchema)
}

// We compute valueHasString instead of taking it from the triplestore, because the
// string literal in the triplestore isn't in API v2 format.
override lazy val valueHasString: String = {
val startDate = DateUtilV2.jdnToDateYearMonthDay(
julianDayNumber = valueHasStartJDN,
private lazy val asCalendarDateRange: CalendarDateRangeV2 = {
val startCalendarDate = CalendarDateV2.fromJulianDayNumber(
julianDay = valueHasStartJDN,
precision = valueHasStartPrecision,
calendar = valueHasCalendar
calendarName = valueHasCalendar
)

val endDate = DateUtilV2.jdnToDateYearMonthDay(
julianDayNumber = valueHasEndJDN,
val endCalendarDate = CalendarDateV2.fromJulianDayNumber(
julianDay = valueHasEndJDN,
precision = valueHasEndPrecision,
calendar = valueHasCalendar
calendarName = valueHasCalendar
)

DateUtilV2.dateRangeToString(
startDate = startDate,
endDate = endDate,
calendar = valueHasCalendar
CalendarDateRangeV2(
startCalendarDate = startCalendarDate,
endCalendarDate = endCalendarDate
)
}

// We compute valueHasString instead of taking it from the triplestore, because the
// string literal in the triplestore isn't in API v2 format.
override lazy val valueHasString: String = asCalendarDateRange.toString

override def toOntologySchema(targetSchema: OntologySchema): ValueContentV2 = copy(ontologySchema = targetSchema)

override def toJsonLDValue(targetSchema: ApiV2Schema, settings: SettingsImpl): JsonLDValue = {
Expand All @@ -680,40 +680,24 @@ case class DateValueContentV2(ontologySchema: OntologySchema,
)

case ApiV2WithValueObjects =>
JsonLDObject(Map(
OntologyConstants.KnoraApiV2WithValueObjects.ValueAsString -> JsonLDString(valueHasString),
OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasCalendar -> JsonLDString(valueHasCalendar.toString)
) ++ toComplexDateValueAssertions)
}
}
val startCalendarDate: CalendarDateV2 = asCalendarDateRange.startCalendarDate
val endCalendarDate: CalendarDateV2 = asCalendarDateRange.endCalendarDate

/**
* Create knora-api assertions.
*
* @return a Map of [[ApiV2WithValueObjects]] value properties to numbers (year, month, day) representing the date value.
*/
def toComplexDateValueAssertions: Map[IRI, JsonLDValue] = {

val startDateConversion: DateYearMonthDay = DateUtilV2.jdnToDateYearMonthDay(valueHasStartJDN, valueHasStartPrecision, valueHasCalendar)

val startDateAssertions = startDateConversion.toStartDateAssertions.map {
case (k: IRI, v: Int) => (k, JsonLDInt(v))

} ++ startDateConversion.toStartEraAssertion.map {

case (k: IRI, v: String) => (k, JsonLDString(v))
}
val endDateConversion: DateYearMonthDay = DateUtilV2.jdnToDateYearMonthDay(valueHasEndJDN, valueHasEndPrecision, valueHasCalendar)
val startDateAssertions = Map(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartYear -> JsonLDInt(startCalendarDate.year)) ++
startCalendarDate.maybeMonth.map(month => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartMonth -> JsonLDInt(month)) ++
startCalendarDate.maybeDay.map(day => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartDay -> JsonLDInt(day)) ++
startCalendarDate.maybeEra.map(era => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartEra -> JsonLDString(era.toString))

val endDateAssertions = endDateConversion.toEndDateAssertions.map {
case (k: IRI, v: Int) => (k, JsonLDInt(v))
val endDateAssertions = Map(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndYear -> JsonLDInt(endCalendarDate.year)) ++
endCalendarDate.maybeMonth.map(month => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndMonth -> JsonLDInt(month)) ++
endCalendarDate.maybeDay.map(day => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndDay -> JsonLDInt(day)) ++
endCalendarDate.maybeEra.map(era => OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndEra -> JsonLDString(era.toString))

} ++ endDateConversion.toEndEraAssertion.map {

case (k: IRI, v: String) => (k, JsonLDString(v))
JsonLDObject(Map(
OntologyConstants.KnoraApiV2WithValueObjects.ValueAsString -> JsonLDString(valueHasString),
OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasCalendar -> JsonLDString(valueHasCalendar.toString)
) ++ startDateAssertions ++ endDateAssertions)
}

startDateAssertions ++ endDateAssertions
}

override def unescape: ValueContentV2 = {
Expand Down Expand Up @@ -752,6 +736,26 @@ case class DateValueContentV2(ontologySchema: OntologySchema,
* Constructs [[DateValueContentV2]] objects based on JSON-LD input.
*/
object DateValueContentV2 extends ValueContentReaderV2[DateValueContentV2] {
/**
* Parses a string representing a date range in API v2 simple format.
*
* @param dateStr the string to be parsed.
* @return a [[DateValueContentV2]] representing the date range.
*/
def parse(dateStr: String): DateValueContentV2 = {
val dateRange: CalendarDateRangeV2 = CalendarDateRangeV2.parse(dateStr)
val (startJDN: Int, endJDN: Int) = dateRange.toJulianDayRange

DateValueContentV2(
ontologySchema = ApiV2Simple,
valueHasStartJDN = startJDN,
valueHasEndJDN = endJDN,
valueHasStartPrecision = dateRange.startCalendarDate.precision,
valueHasEndPrecision = dateRange.endCalendarDate.precision,
valueHasCalendar = dateRange.startCalendarDate.calendarName
)
}

/**
* Converts a JSON-LD object to a [[DateValueContentV2]].
*
Expand All @@ -770,52 +774,77 @@ object DateValueContentV2 extends ValueContentReaderV2[DateValueContentV2] {
}

private def fromJsonLDObjectSync(jsonLDObject: JsonLDObject): DateValueContentV2 = {

implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance

val calendar: KnoraCalendarV1.Value = jsonLDObject.requireStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasCalendar, stringFormatter.validateCalendar)
// Get the values given in the the JSON-LD object.

val calendarName: CalendarNameV2 = jsonLDObject.requireStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasCalendar, CalendarNameV2.parse)

val dateValueHasStartYear: Int = jsonLDObject.requireInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartYear)
val maybeDateValueHasStartMonth: Option[Int] = jsonLDObject.maybeInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartMonth)
val maybeDateValueHasStartDay: Option[Int] = jsonLDObject.maybeInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartDay)
val maybeDateValueHasStartEra: Option[KnoraEraV2.Value] = jsonLDObject.maybeStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartEra, stringFormatter.validateEra)
val maybeDateValueHasStartEra: Option[DateEraV2] = jsonLDObject.maybeStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasStartEra, DateEraV2.parse)

val dateValueHasEndYear: Int = jsonLDObject.requireInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndYear)
val maybeDateValueHasEndMonth: Option[Int] = jsonLDObject.maybeInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndMonth)
val maybeDateValueHasEndDay: Option[Int] = jsonLDObject.maybeInt(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndDay)
val maybeDateValueHasEndEra: Option[KnoraEraV2.Value] = jsonLDObject.maybeStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndEra, stringFormatter.validateEra)
val maybeDateValueHasEndEra: Option[DateEraV2] = jsonLDObject.maybeStringWithValidation(OntologyConstants.KnoraApiV2WithValueObjects.DateValueHasEndEra, DateEraV2.parse)

// Check that the date precisions are valid.

val startDate = DateYearMonthDay(
if (maybeDateValueHasStartMonth.isEmpty && maybeDateValueHasStartDay.isDefined) {
throw AssertionException(s"Invalid date: $jsonLDObject")
}

if (maybeDateValueHasEndMonth.isEmpty && maybeDateValueHasEndDay.isDefined) {
throw AssertionException(s"Invalid date: $jsonLDObject")
}

// Check that the era is given if required.

calendarName match {
case _: CalendarNameGregorianOrJulian =>
if (maybeDateValueHasStartEra.isEmpty || maybeDateValueHasEndEra.isEmpty) {
throw AssertionException(s"Era is required in calendar $calendarName")
}

case _ => ()
}

// Construct a CalendarDateRangeV2 representing the start and end dates.

val startCalendarDate = CalendarDateV2(
calendarName = calendarName,
year = dateValueHasStartYear,
maybeMonth = maybeDateValueHasStartMonth,
maybeDay = maybeDateValueHasStartDay,
era = maybeDateValueHasStartEra.getOrElse(KnoraEraV2.CE)
maybeEra = maybeDateValueHasStartEra
)

val endDate = DateYearMonthDay(
val endCalendarDate = CalendarDateV2(
calendarName = calendarName,
year = dateValueHasEndYear,
maybeMonth = maybeDateValueHasEndMonth,
maybeDay = maybeDateValueHasEndDay,
era = maybeDateValueHasEndEra.getOrElse(KnoraEraV2.CE)
maybeEra = maybeDateValueHasEndEra
)

// TODO: convert the date range to start and end JDNs without first converting it to a string (#928).

val dateRangeStr = DateUtilV2.dateRangeToString(
startDate = startDate,
endDate = endDate,
calendar = calendar
val dateRange = CalendarDateRangeV2(
startCalendarDate = startCalendarDate,
endCalendarDate = endCalendarDate
)

val julianDateRange: JulianDayNumberValueV1 = DateUtilV1.createJDNValueV1FromDateString(dateRangeStr)
// Convert the CalendarDateRangeV2 to start and end Julian Day Numbers.

val (startJDN: Int, endJDN: Int) = dateRange.toJulianDayRange

DateValueContentV2(
ontologySchema = ApiV2WithValueObjects,
valueHasStartJDN = julianDateRange.dateval1,
valueHasEndJDN = julianDateRange.dateval2,
valueHasStartPrecision = startDate.getPrecision,
valueHasEndPrecision = endDate.getPrecision,
valueHasCalendar = calendar,
valueHasStartJDN = startJDN,
valueHasEndJDN = endJDN,
valueHasStartPrecision = startCalendarDate.precision,
valueHasEndPrecision = endCalendarDate.precision,
valueHasCalendar = calendarName,
comment = getComment(jsonLDObject)
)
}
Expand Down Expand Up @@ -976,11 +1005,11 @@ case class TextValueContentV2(ontologySchema: OntologySchema,
tagWithIndex => tagWithIndex.standoffNode.startIndex -> tagWithIndex.standoffTagInstanceIri
}.toMap

// resolve the original XML ids to standoff Iris every the `StandoffTagInternalReferenceAttributeV1`
// resolve the original XML ids to standoff Iris every the `StandoffTagInternalReferenceAttributeV2`
val standoffTagsWithNodeReferences: Seq[CreateStandoffTagV2InTriplestore] = standoffTagsWithOriginalXMLIDs.map {
standoffTag: CreateStandoffTagV2InTriplestore =>

// resolve original XML ids to standoff node Iris for `StandoffTagInternalReferenceAttributeV1`
// resolve original XML ids to standoff node Iris for `StandoffTagInternalReferenceAttributeV2`
val attributesWithStandoffNodeIriReferences: Seq[StandoffTagAttributeV2] = standoffTag.standoffNode.attributes.map {
attributeWithOriginalXMLID: StandoffTagAttributeV2 =>
attributeWithOriginalXMLID match {
Expand Down
4 changes: 2 additions & 2 deletions webapi/src/main/scala/org/knora/webapi/responders/v2/ResourcesResponderV2.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ import org.knora.webapi._
import org.knora.webapi.messages.admin.responder.permissionsmessages.{DefaultObjectAccessPermissionsStringForResourceClassGetADM, DefaultObjectAccessPermissionsStringResponseADM, ResourceCreateOperation}
import org.knora.webapi.messages.admin.responder.usersmessages.UserADM
import org.knora.webapi.messages.store.triplestoremessages.{SparqlConstructRequest, SparqlConstructResponse, SparqlUpdateRequest, SparqlUpdateResponse}
import org.knora.webapi.messages.v1.responder.valuemessages.KnoraCalendarV1
import org.knora.webapi.messages.v2.responder.ontologymessages._
import org.knora.webapi.messages.v2.responder.resourcemessages._
import org.knora.webapi.messages.v2.responder.searchmessages.GravsearchRequestV2
Expand All @@ -42,6 +41,7 @@ import org.knora.webapi.util.ActorUtil.{future2Message, handleUnexpectedMessage}
import org.knora.webapi.util.ConstructResponseUtilV2.{MappingAndXSLTransformation, ResourceWithValueRdfData}
import org.knora.webapi.util.IriConversions._
import org.knora.webapi.util._
import org.knora.webapi.util.date.CalendarNameGregorian
import org.knora.webapi.util.search.ConstructQuery
import org.knora.webapi.util.search.gravsearch.GravsearchParser

Expand Down Expand Up @@ -914,7 +914,7 @@ class ResourcesResponderV2 extends ResponderWithStandoffV2 {
valueObj.copy(
valueContent = dateContent.copy(
// act as if this was a Gregorian date
valueHasCalendar = KnoraCalendarV1.GREGORIAN
valueHasCalendar = CalendarNameGregorian
)
)

Expand Down
Loading