normative references clause missing from ToC; possibly due to incorrect kanji in subclause heading

[ReesePlews]

in checking a generated plateau document 001-v4 from 2024-10-24, clause 1.4, normative references, is not listed in the table of contents of the documents. shown in this image below from a generate of 001-v4, from 2024-10-24.

the clause is shown in the document. (shown below)

so i checked the heading and i am noticing a possible difference in the kanji used.

the current heading is: 引⽤規格等 of which the encoding is:

the 2nd character is slightly different, instead it should be 引用規格等:

perhaps there are multiple uses in the flavor template? i am not sure if this is the reason why the subclause is not in the ToC but it is a discrepancy.

thank you.

[ronaldtse]

I believe we are not properly encoding the bibliography in the documents 001 and 002. I have to check.

[ReesePlews]

i was not sure, i just noticed the kanji. thanks for checking,

[opoudjis]

In the latest version I have pulled, only the bibliography is marked up as a Metanorma clause. The Normative References are not. 001-v3 contains a table with no separate clause and no bibliographic markup, commented as

// TODO: These are normative references for standards

[[table1]]
.標準製品仕様書が引用する規格等
[cols="2",options="header"]
|===
| 文書名 | URL

If there is no Normative References clause in the Asciidoc, there isn't going to be one in the Table of Contents either. That is an editorial issue.

The problem with the Kanji is that we do not speak Japanese, and therefore I have been obligated to cut and paste from JIS documents. Something or someone is doing with titles what we had identified recently as misusing radical symbols instead of the correct Kanji; I suspect this is a PDF issue, given what we recently found.

My preference is to get you to proofread all those files, @ReesePlews, because you read Japanese and I don't. But if you cannot eyeball these issues readily, I will try to write a script to identify where the radicals are in the internationalisation files.

The files to check are:

https://github.com/relaton/relaton-render/blob/main/lib/isodoc-yaml/i18n-ja.yaml
https://github.com/metanorma/isodoc/blob/main/lib/isodoc-yaml/i18n-ja.yaml
https://github.com/metanorma/metanorma-iso/blob/main/lib/isodoc/iso/i18n-ja.yaml
https://github.com/metanorma/metanorma-jis/blob/main/lib/isodoc/jis/i18n-ja.yaml
https://github.com/metanorma/metanorma-plateau/blob/main/lib/isodoc/plateau/i18n-ja.yaml

[ronaldtse]

@opoudjis 001-v3 is purely for legacy purposes so it is not applicable.

I believe the references sections in 001-v4 and 002-v4 may not be fully encoded due to missing rendering support. I'll check.

[ReesePlews]

thank you @opoudjis i will have a look at those yaml files and see if i can identify any incorrect characters.
i will try and do that today or tomorrow.

[ReesePlews]

hello @opoudjis i extracted the lines with kanji chars from the files you listed. i could not find characters designated as "radicals"; perhaps that was the only one. surprisingly though i could not find 引用規格等 in the files you listed. perhaps there are more character mapping files? if you would like the text files, please let me know. i just used regex to remove the left part of the lines then removed blank lines and non-kanji-lines. i pasted the remaining text into the website used above and searched the output for "radical"; not the best process.

i dont have anything more to add here regarding this kanji issue.

i am not sure about the references and bibliography. @ronaldtse and the other developers i believe were handling those things.

if that is not related here, i will let you or Ron close this issue. thank you.

[opoudjis]

I've just seared for 引用規格等 on github/metanorma, and the only place it turns up is in the documents themselves:

https://raw.githubusercontent.com/metanorma/mn-samples-plateau/3674bd6314b410063cee352c109f2763bdc94c00/sources/001-v4/sections/section-01/section-04-normrefs.adoc

[heading="Normative references"]
[bibliography]
=== 引用規格等

This means that this is a separate issue: the Table of Contents generator not expecting to find a normative references clause treated as a subclause.

So, the markup is correct, the Kanji used in the source text are incorrect (you tell me why), and the bug is instead in the existing HTML table of contents generator, which I will fix. (@ronaldtse This is not a "leave it for Firelight" issue, because Firelight is not going to be adopted universally next week; in any case, this should be a one-line fix.)

[ReesePlews]

hello @opoudjis thanks for locating the issue. i also checked the .adoc file and those character codes are correct. the issue is happening in the PDF (the code of the 2nd character in the PDF is incorrect). i confirmed the "document.html" file (from a generate on 20241024) is also correct, so i think only the PDF has the problem.

if you need me to show this, i can paste them again here, however the correct character string is listed above, so that can be used to fix the PDF.

thank you.

[ReesePlews]

i should have checked the adoc file before. sorry about that. it would have pinpointed the problem better from the creation of this issue. sorry about that!

[opoudjis]

@Intelligent2013 this is a bug report on PDF TOC for 001-v4: it is not being found, presumably because your TOC builder, like mine, wasn't expecting to find /sections/clause/references.

@ronaldtse also indicates it is possible that the docker version of mn2pdf may not contain the fixes that dealt with radical characters.

[Intelligent2013]

@ReesePlews I don't see the images in the first post:

Do you see them? If this isn't GitHub issue, could you attach them again, please?

[ReesePlews]

hello @Intelligent2013 thank you for checking these things. i have re-loaded the images in the original post (above) and clarified the document from which they originated. if they are still not showing, please let me know. thank you

[Intelligent2013]

@ReesePlews thank you! It's ok now!

So, there are two issues:

• missing 1.4 item in the ToC
  I'm working on this.
  Hmm, in the bookmarks there is the item 1.4:
  

• wrong kangi encoding
  This issue was fixed earlier (in kanji codeset differences between adoc and pdf mn-samples-plateau#168, mentioned the mn2pdf version 1.99, but actually 2.00). But mn2pdf after the version 1.99 can't generate correct PDF ((URGENT) PDF output on GitHub Actions are broken due to old version of mn2pdf #121). The latest version 2.08 (released today) allows to generate PDF properly.
  I'll try to correct Gemfile in mn-samples-plateau to retrieve the latest mn2pdf gem and use it in the docker.

[Intelligent2013]

@ReesePlews sorry for a long answer. The issues fixed in the last week Metanorma release.
The generated PDF's available in the artifacts in https://github.com/metanorma/mn-samples-plateau/actions/runs/11957118049.

[ReesePlews]

thank you @Intelligent2013 we will have a look at this and get back to you.