Divina profile and OPDS-PSE/ComicInfo.xml

[HadrienGardeur]

The Divina profile defines an additional set of features and constraints for BD, comics, manga and Webtoon: https://readium.org/webpub-manifest/profiles/divina

It's still a WIP with interest from a few organizations including De Marque (@mickael-menu, @chocolatkey, @qnga and myself), EDRLab (@llemeurfr) and Nota (@m-abs).
Recently, this group has mostly focused on guided navigation in Divina, bridging the gap between Media Overlay, panel by panel navigation and TTS.

OPDS-PSE was initially created by Ubooquity and implemented in Chunky Comic Reader. OPDS-PSE provides an easy way to stream pages for comics/manga through a URI template.
It is now being maintained under the Anansi-Project umbrella at: https://github.com/anansi-project/opds-pse/blob/master/v1.2.md

This updated version of OPDS-PSE is maintained by a group of developers working on servers and clients which includes @gotson, @majora2007, @ajslater and @fillito.

Initial discussion was started through the OPDS repo: opds-community/drafts#49 (comment)

A number of relevant discussions were also started in the OPDS-PSE repo, including a proposal for expressing OPDS-PSE using the RWPM model: anansi-project/opds-pse#10 (reply in thread)

A joint call between both groups will be organized in May.

[HadrienGardeur]

opds-community/drafts#49 (reply in thread)

@ajslater said:

Looking at the the Divina spec, the readingOrder section seems kind of heavyweight for most comic usage. It is interesting to be so explicit about the presentation of every page in a publication, and the panels schema seems useful, but I suspect for most comic book serving applications, a shorter, less expressive structure might be a simpler alternative.

Divina comes from a slightly different background than OPDS-PSE as it's meant to be potentially used in packaged publications as well.
In such publications, the path of each resource doesn't follow a convention and it's best to be explicit by indicating which resources are part of the readingOrder and in which order they should be displayed.

This has the benefit of giving us a place to provide additional metadata about each resource such as their media type, height, width or position in a spread.

With that use case in mind, Divina could potentially replace:

• the conventions of CBZ/CBR files where the reading order is inferred
• and ComicInfo.xml since most of the same metadata can be expressed as well

A Divina manifest could easily live side by side with ComicInfo.xml in a CBZ or with an OPF in an EPUB FXL to provide enhanced support in apps that support it.

In this context, the size of a compressed manifest barely matters at all compared to the size of a single image.

There's a tiny one time penalty on the server side to do the page analysis and then storage, but like, why do you want to send redundant information over the wire every time you sever a publication? 25 to hundreds of json fragments that all say "serve the next page exactly like the last one". gzip or brotli isn't going to compress that.

I think that's a fair point and this is why I initially reacted by saying that OPDS-PSE is a mix of simplicity and pragmatism:

OPDS-PSE is an interesting mix of things:

• it provides a way to stream comics/manga page by page (image by image)
• there's also a built-in support for requesting a resized version of each page (can be useful to get a lower resolution version on smaller devices for example)
• and it provides minimal info for restoring progression
  It's balancing out all these features with a single templated URI and a few attributes thanks to smart decisions and the linear nature of comics/manga.

Your proposal for an RWPM/OPDS 2.0 @ajslater is very similar to the one that I've made in the OPDS-PSE repo:

{
  "rel": "http://vaemendis.net/opds-pse/stream",
  "href": "https://example.com/opds-pse/xyz{?page,height,width}",
  "templated": true,
  "properties": {
    "maxHeight": 1600,
    "maxWidth": 1200,
    "supportedFormats": ["image/webp", "image/jpeg", "image/png", "image/gif"]
  }
}

Going over this proposal to make everything crystal clear:

• There's no need to repeat the number of pages for a given publication since this is already covered in metadata by numberOfPages.
• Based on the current OPDS-PSE specification, this example provides an URI template where a client can request a specific height/width with maximum values allowed by the server. @gotson mentioned that this is not currently implemented by anyone, if that's the case, you could drop it from this example.
• supportedFormats allows the client to know prior doing any request which media type will be supported by the client but it's very much optional.

If you boil it down to the most basic information, the only things left are:

• a rel value
• and a URI template with a page parameter

{
  "rel": "http://vaemendis.net/opds-pse/stream",
  "href": "https://example.com/opds-pse/xyz{?page}",
  "templated": true,
}

I would recommend minting a new rel value since it's no longer following the same URI template, for example: https://anansi-project.github.io/opds-pse

That said, I do believe there's value in also offering a streamable Divina manifest side by side with a packaged publication and a OPDS-PSE style URI template in an OPDS 2.0 catalog:

"links": [
  {
    "rel": "self",
    "type": "application/divina+json",
    "href": "manifest.json"
  },
  {
    "rel": "https://anansi-project.github.io/opds-pse",
    "href": "https://example.com/opds-pse/xyz{?page}",
    "templated": true,
  },
  {
    "rel": "http://opds-spec.org/acquisition",
    "type": "application/divina+zip",
    "href": "download.divina"
  }
]

• With this approach, while browsing an OPDS 2.0 catalog, clients will only receive publication metadata (which they need) but they won't receive the readingOrder (it doesn't matter anymore if the publication has a large number of items in there).
• If a client wants the full power of Divina to stream a publication, they could easily follow the self link, fetch the manifest and cache it temporarily.
• If a client prefers to keep it simple, they could follow the OPDS-PSE rel and use the URI template instead.
• And if a client wants to download a publication to read it offline, it can follow the OPDS acquisition link.

[HadrienGardeur]

Going back to this topic, based on our first call it feels that the option of dropping OPDS-PSE entirely is being considered.

How would it look like without it? Pretty similar to the example that I provided above:

"links": [
  {
    "rel": "self",
    "type": "application/divina+json",
    "href": "manifest.json"
  },
  {
    "rel": "http://opds-spec.org/acquisition",
    "type": "application/divina+zip",
    "href": "download.divina"
  }
]

Fetching the self link would return a Divina with the usual reading order:

"readingOrder": [
  {
    "href": "cover.jpg",
    "type": "image/jpeg"
  },
  {
    "href": "page1.jpg",
    "type": "image/jpeg",
    "properties": {
      "page": "left"
    }
  },
  {
    "href": "page2.jpg",
    "type": "image/jpeg",
    "properties": {
      "page": "right"
    }
  }
]

For clients, it would be fairly easy to remain compatible with OPDS-PSE.

Let's say that you receive the following OPDS-PSE URI template:

<link rel="http://vaemendis.net/opds-pse/stream"
      type="image/jpeg"
      href="/opds-comics/stream/1217?page={pageNumber}"
      pse:count="3" />

Clients could easily create a compatibility layer that converts this back to a Divina:

"readingOrder": [
  {
    "href": "/opds-comics/stream/1217?page=0",
    "type": "image/jpeg"
  },
  {
    "href": "/opds-comics/stream/1217?page=1",
    "type": "image/jpeg"
  },
  {
    "href": "/opds-comics/stream/1217?page=2",
    "type": "image/jpeg"
  }
]

In various Readium toolkits, we already abstract away the difference between OPDS 1.2 and 2.0 since they're parsed into the same in-memory model where clients do not need to know about OPDS versions at all. We could do the same with OPDS-PSE and handle that on behalf of clients.

Any thoughts about this approach? How do people feel about dropping the URI template approach entirely?

[gotson]

That's really a question for clients more than servers. Opds providers will still need to provide both versions for a foreseeable future.

By the way, is there a good way to provide both opds versions in a way that is easily discoverable from client point of view?

[HadrienGardeur]

You can always include an alternate link in your feed that would point to another OPDS variant, but this has to be supported by clients of course.

In Atom, it would look like this:

<link rel="alternate"
      type="application/opds+json"
      href="feed.json" />

[HadrienGardeur]

Based on our latest call with @majora2007 and @therobbiedavis plus a meeting with @fillito in Madrid, it feels like everyone agrees with the general direction of using Divina as a replacement for OPDS-PSE rather than keeping them both side by side.

[HadrienGardeur]

Here's a first proposal of topics to discuss between the Readium and Anansi Project communities:

• metadata, more specifically comparing ComicInfo.xml with RWPM/Divina to build a mapping and identify potential gaps
• URI template vs manifest approach for streaming (see discussion above)
• handling spreads in both contexts

[HadrienGardeur]

Here's my early take on comparing ComicInfo.xml with RWPM/Divina. I'm excluding from the scope of this comparaison all elements that are related to spreads or info that would typically be included in readingOrder or resources in RWPM/Divina.

Title

There's not much to say, but RWPM/Divina goes beyond ComicInfo.xml with the ability to indicate a title in multiple languages or a separate string for sorting.

Contributors

Both of them provide extensive support for various roles, with a lot of roles in common:

• available in both ComicInfo.xml and RWPM/Divina: writer, penciler, inker, colorist, letterer, editor, translator and publisher
• only available in ComicInfo.xml: cover artist

Overall RWPM/Divina provides a longer list of roles (with built-in extensibility through MARC relators) and goes beyond strings for expressing contributors with support for expressions in other languages, identifiers (using an ISNI is highly recommended) or a sort string.

That said, there's really no equivalent for cover artist in RWPM or in MARC relators. The closest thing supported in RWPM/Divina would be artist while MARC contains the cov code to indicate a "Cover designer":

A person or organization responsible for the graphic design of a book cover, album cover, slipcase, box, container, etc. For a person or organization responsible for the graphic design of an entire book, use Book designer; for book jackets, use Bookjacket designer

Given the importance of covers for comics, we could explore if RWPM/Divina needs to go beyond what's currently supported.

Series

ComicInfo.xml has the concept of series and alternate series while RWPM/Divina has a different approach, where series is repeatable without any concept of main/alternate series.

While RWPM/Divina offer the usual enhancements over strings (identifier and sort string), it lacks the concept of a "count" available in ComicInfo.xml.

Considering the open nature of many series, this feels mostly useful once a series is completed (with the risk or prequels/sequels being published afterwards) and could be problematic if various publications provide inconsistent info about the series count.

It would be interesting to get some feedback from the Anansi Projet community about the usefulness and risks of this property.

Reading progression

ComicInfo.xml has a rather specialized approach for reading direction. Instead of simply indicating "ltr" or "rtl" like in RWPM/Divina, there's a specialized Manga property which supports four values: "Unknown", "Yes", "No" and "YesAndRightToLeft".

I'm not really convinced by this approach as it takes two different concepts and mash them together:

• the type of publication and/or genre that it belong to is better expressed through a specialized vocabulary, such as THEMA. For example, there's a code for seinen (https://ns.editeur.org/thema/en/XAML) that can be used with subject in RWPM/Divina.
• the reading direction is better expressed as a separate value, which is the case with readingProgression in RWPM/Divina

That said, we could clearly benefit in having more guidelines on how to use subject in RWPM/Divina. We could even recommend using specific codes from THEMA or BISAC to do so.

Age rating and audience

RWPM/Divina lack the native ability to express the audience of a publication. Although the built-in schema.org extensibility could be used for that, we could benefit from having something better in our default context.

I've been working on this topic for some time, but audience is a complicated topic which covers:

• general audience info, which doesn't always cover a specific age range (most publications end up under "General/adult" which doesn't say much)
• ratings, with many variations per region and that can cover a mix of age recommandations ("16+") or subjects ("Violence", "Drug use"…)
• educational-level schemes that indicate if a publication targets a specific education level or exam
• reading-level schemes that indicate the level of vocabulary necessary to properly understand a publication

ComicInfo.xml is stuck in between general audience info ("Early Childhood", "Kids to Adults", "Teen", "Everyone") and ratings ("PG", "MA15+").

It would be interesting to get some feedback on how these values are really used in practice, which would help us with the addition of an audience property in RWPM/Divina.

Other properties

The following properties present in ComicInfo.xml also have a direct equivalent in RWPM/Divina:

• Summary (description)
• PageCount (numberOfPages)
• Genre (subject)
• GTIN (identifier)
• Year, Month and Day (published)
• LanguageISO (language)

There are also a number of other properties that would be either supported only through schema.org or with new additions:

• Tags
• Web
• BlackAndWhite
• Characters
• Teams
• Locations
• ScanInformation
• StoryArc
• StoryArcNumber
• SeriesGroup
• CommunityRating
• MainCharacterOrTeam
• Review

It's not exactly clear to me how each of these properties are used or how useful they are in practice, but it would be interesting to cover them as part of our discussion.

[ajslater]

One small point of clarity here about the different purposes of Divina & ComicInfo.xml:

• Divina is a new, well thought out, comprehensive protocol meant to be sent down the wire to a client.
• ComicInfo.xml is a legacy archive metadata format meant to support a defunct program that has been de-facto adopted as a comic standard because of ComicRack's (former?) popularity. It's proper analog in the book world is not Divina, but something like Open Container Format's (e.g. EPub's) META-INF directory or metadata.xml.

ComicInfo.xml however, does represents a superset of valuable metadata to comic book apps.

Contributors
Cover Artists are an important for comics, but I feel this might be a good extension that relates to all books. Like how to represent in that Boris Vallejo did the cover of this Conan novel, but someone else did the interior interstitial illustrations.
Either a new tag or a 'cov' attribute would be fine for this.

Series
The concept of Alternate Series does exist in comic world, but practically I never see it used. I might be out of the loop here. Maybe some of the people who have been working on Comic Reading Lists standards recently have a take on this.

Reading progression
Divina's approach is superior here. There's an Anansi project thread looking to create a ComicInfo.xml tag with four directions and also a "continuous" modifier. The overloaded Manga tag is a mismash of concepts. I think it's bad and should be superseded. Your points about this are all correct.
There's a bunch of good information that's possible to express in the Readium Presentation extension too. I wonder a bit about what attributes belong in this Presentation extension vs Divina, but that's not really a part of this discussion.

Age rating and audience
In practice these are rarely used or unused. The schema is a strange set of values partially mirroring American Movie ratings. There is an argument that the concept is useful, but if it were adopted I would think about it from first principles and not use ComicInfo.xml's implementation as much of a guide. This would not be an obstruction for the comic world's existing usage, because I think the ComicInfo.xml implementation is largely unused.

Other properties

• Tags - this is practically unused, it feels sort of useful to me as a generic tag, but no one uses it, so maybe not.
• Web - This is very useful. It is used a link to more metadata on the web or the source of metadata. I think there are rough equivalents to this in OPDS 2.0, and maybe they could be used for this. Otherwise maybe it's a link element with a new rel type.
• BlackAndWhite - Unused. Idk it's sort of interesting to know which comics or books are monochromatic or not, but not especially.
• Characters, Teams, Locations - Widely used, actually. In OPDS 1.0 I wedge these into the old 'categories' tag. in OPDS 2.0 I put them into 'subject'. It would be nice to have another equivalent.
• ScanInformation - This is almost a kind of credit, the equivalent for books would be who did the scan or OCR if that was necessary. e.g. GoogleBooks.
• StoryArc & StoryArcNumber - The order of reading comics across story arcs is very important to people and these tags are inadequate to the task. I hope someone from the recent Comic Book Reading List efforts is part of this discussion. There's a lot of activity on this topic and I haven't been paying attention. I wish the information would be encoded into individual comics with extensions to tags like these but afaik, the real effort has been towards centralized Reading Lists that are external to the comics and then rely on matching books to those mentioned in the reading list. The Anansi project's recent proposed addition of a GUID (ISBN etc.) for comics could help enormously in any approach to this unsolved problem.
• SeriesGroup - Unused to my knowledge.
• CommunityRating - Unused but occasionally a feature request from users. Difficult to make this global but i admit it has merit.
• MainCharacterOrTeam - Unused. If the concept is useful, it should probably be an attribute of Character or Team tags.
• Review - ??? this is new to me.

Many of my opinions are expressed above, my experience is limited and I obviously don't speak for the entire comic book community. I'm sure I've made mistakes, feedback is very welcome.

[majora2007]

I want to jump in and provide some clarifications.

• Age Rating. I do agree that the ComicInfo.xml's implementation is pretty poor, but I can say they are used in practice. Many servers use Age Ratings as a way to build child friendly views, so having something akin to this would be ideal and that can be mapped back to an appropriate rating in ComicInfo would be ideal for software that would support both ComicInfo and Divina.
• Web is important but also lacking in ComicInfo as there is only one field and misses an opportunity to store multiple links, like links to different sites that have metadata or some relationships. I believe Divina can support multiples.
• BlackAndWhite I have heard is used to know what can be used on eInk readers, but I do agree it is likely largely unused by many people.
• SeriesGroup is used for including series in collections. Not sure how many people actually use this feature though, but I do know support exists in software to read these tags, so they must be used. Would be interesting to see if we need them going forward or if we can use something like belongsTo to replace it.

@DieselTech might be able to provide some insights on StoryArc/StoryArcNumber as he works in reading list orders.

[gotson]

I think @ajslater's analysis is spot on, and @HadrienGardeur your understanding and analysis is very good.

Keep in mind that ComicInfo.xml was never meant to be an interchange format, it was used by ComicRack to store information you would input in the GUI, and fetch it later. The use of ComicInfo.xml as an interchange metadata format is a by product of that, and the fact that it's so badly design shows.

The Anansi Project tried to document the fields, from the usage observed in the wild. Some fields have been twisted from their original purpose too.

Here are some thoughts or comments:

• Count attribute should not be part of the metadata, because it will always be wrong. Applications consuming ComicInfo.xml would only keep the maximum value of all the values present in a Series. I personally don't think it makes sense to be in Divina.
• Age Rating: i agree with Hadrien, it bears different information, and the one in ComicInfo.xml is useless. If we need something we should start from scratch.
• Tags may be too generic. Some websites use Themes, or Genres, or Categories. I think it makes sense to be able to characterize a publication via a simple bag of words.
• AlternateSeries: this is very much for cross-overs, or reading orders. It seems the belongsTo field can contain 1 series and 1 collection. This seem to be roughly equivalent.
• StoryArc: This one has been twisted from its original purpose. It was originally designed to store which Story Arc within the Series the book was part of. For example for Undiscovered Country the first 6 books are part of the story arc called "Destiny". Books 7-12 are part of "Unity". However the community started using this field to represent the comics Reading Orders, where a reading order would contain comics from different series, in a specific order. That's why we ended up adding StoryArcNumber to help with the ordering.

One more thing that is not covered in ComicInfo.xml is the handling of Comic one-shots. There are usually some specials that are not part of the main series, but would need to be read within that series. For example Undiscovered Country: Destiny Man #1 is a special, that is part of a series called "Undiscovered Country: Destiny Man", but should be read between Undiscovered Country 18 and 19.

I think the belongsTo field is probably flexible enough to accomodate the series, any alternate series or collection, reading orders, and also specials that need to be read within a series. The field may need to cater for more than just series and collection though.

[HadrienGardeur]

I've created a separate discussion about the use of THEMA for subjects: #98

I feel that this is a better approach for handling them than tags, although they can't cover the same range of topics (which might be a good thing here).