IATI XML in multiple languages loads incorrectly

[zzgvh]

When the API loads IATI data the lang attribute is ignored leading to the "last" attribute being used rather than the one specified as default.

Example: in the Cordaid July dataset a number of activites have two title tags, one in English and one in French. The French tag comes after the English one in the XML and is used in the import.

[adriancollier]

@zzgvh is this an issue we want to fix, or is this something we should include in our "advice" to partners when submitting a file?

[zzgvh]

@kasper I think you've misunderstood the problem here. The issue is that the XML may have two tags with the same information in different languages. Example:

    <iati-activity xml:lang="en">
    ...
        <title>Integration of disabled people in the society</title>
        <title xml:lang="fr">L'introduction de PBF et RBC au CFHMB</title>
    ...

Currently the XSLT doesn't look at the xml:lang attribute at all, the end result being that the last tag is the one that's used, if there are several that only differ by language. If I'm reading the commit right you're just deleting the xml:lang attribute, making no difference to the end result.

We might simply delete the whole tag, if it has an xml:lang attribute differing from the default, but that is problematic if there is no data in the default language and we can't be sure that's not the case.

So what's needed is way to get the tag that matches the default if it exists, but still returns something if there is no data in the default lang.

[KasperBrandt]

Ah, completely misunderstood it indeed.. Will have another go at this one.

[zzgvh]

There's a problem when the xml:lang tag is found deep in the tree. I used this file to test on:

<?xml version="1.0" encoding="utf-8"?>
<iati-activities version="1.03" generated-datetime="2013-10-29T15:54:35+00:00" xmlns:akvo="http://www.akvo.org">
  <iati-activity last-updated-datetime="2012-01-23T16:36:05+00:00" xml:lang="en" default-currency="EUR" akvo:business-unit-id="K6020" akvo:image-caption="Girls from Chennai " akvo:publish="true" akvo:internal-project-id="104691" akvo:photo-id="31535">
    <iati-identifier>NL-KVK-41160054-104691</iati-identifier>
    <title>Foster Care for Children in Slums</title>
    <description type="1" akvo:type="4">Foster care for destitute and exploited street children </description>
    <description type="1" akvo:type="5">This is a pilot project to find out how alternative care for street children can work in Chennai. Don Bosco Anbu Illam will indentify and train foster families to take care of 75 former street children.  Local communities and Child Welfare Committees will be sensitized and trained on this issue. </description>
    <description type="2" akvo:type="8">To provide alternative care and protection for street children, with support of local foster families </description>
    <description type="3" akvo:type="3">Many children living on the streets in Chennai do not have a chance to return to their families. DonBosco Anbu Illam used to take care of these children in homes but has started some years ago a program to find alternative care , especially with families. Since it is proven that foster care is much better for children than in homes. With the support of communities and local authorities, foster families are being recruited in Chennai and approx  75 children will get a new home. </description>
    <description type="1" akvo:type="6">Many street children in India do not have a chance to return to their families. In the past, these children would stay in homes where they would live in a group with many more other children. Research has proven that growing up in families (or in smaller units) is much better for children.
The past years we have been promoting alternative care for children and we are member of the Better Care Network Netherlands in which a number of organizations working with children are promoting alternative care and sharing their experiences  </description>
    <description type="1" akvo:type="7">The main activities are:
1. Identification of foster care families and linking with the children
2. Preparation of the families in their role as foster family
3. Awareness building in the communities and local duty bearers like the police, on streetchildren, care and follow-up
4. Training of the Child Welfare Committees on alternative care </description>
    <description type="1" akvo:type="10">Foster Care is relatively new for the Don Bosco organizations and this pilot project will involve local Child Welfare Committees to share their experiences and to be able to get support for the foster care families, also from government side.  Awareness building and  training of interested  families, are important to make the foster care a success. It is new for many families to adopt a child outside of their direct family but it has been proven that it is possible with sufficient support and guidance.
Don Bosco Anbu Illam is also active in local fundraising from the general public to support its work.</description>
    <description type="1" akvo:type="9">This is the final year of support to Don Bosco Anbu Illam, since the children's program of Cordaid will be phased out in India. DonBoco Anbu Illam has been joining in trainings on (local) fundraising that were organised by Don Bosco National Forum on Youth at Risk  for all the Don Bosco organizations,  working with street and other vulnerable children. Besides this small programme, the organizations is also implementing a street and working children project.</description>
    <activity-status code="2">Implementation</activity-status>
    <activity-date type="start-planned" iso-date="2010-12-01"></activity-date>
    <activity-date type="end-planned" iso-date="2013-11-30"></activity-date>
    <activity-date type="start-actual" iso-date="2010-12-01"></activity-date>
    <activity-date type="end-actual" iso-date="2013-11-30"></activity-date>
    <reporting-org ref="NL-KVK-41160054" type="21">Stichting Cordaid</reporting-org>
    <participating-org role="Accountable" ref="NL-KVK-41160054" type="21" akvo:internal-org-ref="27239">Cordaid</participating-org>
    <participating-org role="Implementing" type="23" akvo:internal-org-ref="18105">Don Bosco Anbu Illam Chennai</participating-org>
    <participating-org role="Funding" type="21" akvo:internal-org-ref="27591">Cordaid Kinderstem</participating-org>
    <recipient-country percentage="100" code="IN">INDIA</recipient-country>
    <location percentage="100">
      <location-type code="PPLA">Seat of a first-order administrative division</location-type>
      <name>Chennai </name>
      <administrative country="IN">INDIA</administrative>
      <coordinates latitude="13.060422000000000000" longitude="80.495830000000000000" precision="1"></coordinates>
    </location>
    <default-finance-type code="110">Aid grant excluding debt reorganisation</default-finance-type>
    <default-tied-status code="Untied">Untied</default-tied-status>
    <budget type="1" akvo:budget-from="Cordaid">
      <period-start iso-date="2010-12-01"></period-start>
      <period-end iso-date="2013-11-30"></period-end>
      <value value-date="2010-12-01">53722</value>
    </budget>
    <budget type="1" akvo:budget-from="Others">
      <period-start iso-date="2013-12-01"></period-start>
      <period-end iso-date="2013-11-30"></period-end>
      <value value-date="2013-12-01">9480</value>
    </budget>
    <sector vocabulary="DAC" code="15150" percentage="60">Strengthening civil society</sector>
    <sector vocabulary="DAC" code="16010" percentage="40">Social/ welfare services</sector>
    <result type="2">
      <title>Better care, education and support for exploited children</title>
      <indicator measure="1" akvo:indicator-id="289">
        <title>Community organizations with strengthened management and social audit capacity</title>
        <title xml:lang="fr">Les organismes communautaires avec une gestion renforcée et la capacité d'audit social</title>
        <baseline year="2010" value="0.00" />
        <period>
          <period-start iso-date="2010-12-01"></period-start>
          <period-end iso-date="2013-11-30"></period-end>
          <target value="10.00" />
        </period>
      </indicator>
      <indicator measure="1" akvo:indicator-id="294">
        <title>Vulnerable or exploited children getting care and protection</title>
        <baseline year="2010" value="0.00" />
        <period>
          <period-start iso-date="2010-12-01"></period-start>
          <period-end iso-date="2013-11-30"></period-end>
          <target value="75.00">
            <comment>children receiving foster care </comment>
          </target>
        </period>
      </indicator>
    </result>
  </iati-activity>
</iati-activities>

As you can see I've added an extra <title> tag with an xml:lang="fr" attr in the first result/indicator and that results in an error "Element is not a child of this node.". Haven't analyzed the cause of this.

[zzgvh]

And a small style comment: since you're using '{http://www.w3.org/XML/1998/namespace}lang' in so many places I'd place it in a "constant", i.e. XML_LANG_NS = '{http://www.w3.org/XML/1998/namespace}lang' or something. Makes for somewhat shorter lines 😉

[KasperBrandt]

Merged in #532

[KasperBrandt]

Test plan

GIVEN the Cordaid import script
WHEN two similar nodes have a different xml:lang attribute
THEN the node similar to the xml:lang of the top (project) node should be saved, or the node without xml:lang attribute if no match is found