Add 'Recipient Country' field to the API (estimate: 16)

[nadiagorchakova]

ICCO has reported that their project country data does not get through the API anymore. The 'missing' country field seems to be a legacy issue as it was removed from Editor some time ago, while the API was not updated.

We should add 'Recipient country' from section 7 in the Editor to the API code. It should also be marked that the old country field must not be used.

(It might be that the original country field was removed as part of this issue #2277)

[punchagan]

The API endpoint under discussion is end-points like rest/v1/project_map_location/.

This issue seems to have the discussion on why the Country field was made a legacy field. Looks like, the attempt was to switch to using the IATI country lists alone, instead of that along with our own Country lists. (#2405 is a newer issue for this). But, there are few issues with how this was done, which we'll need to figure out how to resolve/fix.

• BaseLocation (which is subclassed by ProjectLocation) has city, state, etc., but doesn't have a country.
• A project can have multiple ProjectLocations and multiple RecepientCountrys. How do we associate a country with a location, is not very clear, in this case.

[Geerts]

A project can have multiple ProjectLocations and multiple RecepientCountrys. How do we associate a country with a location, is not very clear, in this case.

We handle that showing multiple locations with a percentage, this is something that the consumer of the API should handle as well. Imo it shouldn't be a problem giving several countries back for one project.

[punchagan]

We handle that showing multiple locations with a percentage, this is something that the consumer of the API should handle as well. Imo it shouldn't be a problem giving several countries back for one project.

Which API end-points are we talking about here, exactly? If we are talking about rest/v1/project_map_location/?location_target=<project_id> and rest/v1/project_location/?location_target=<project_id>, then we would expect each latitude and longitude returned in the results to be associated with a country. This association is happening from a project location to a recipient country via a project. If a project has multiple such locations, and multiple such countries, it's not straight forward to map them.

[Geerts]

Sounds good to me. Only question left then is if we remove the legacy field from the API to avoid confusion. Now that I think about it again I think that would be good practise to avoid confusion in the future.

[GeertSoet]

Created via Reamaze:

Link: https://akvoo.reamaze.com/admin/conversations/akvorsr-support-request-number-33810-from-lissy-at-akvo-dot-org

[nadiagorchakova]

We'll have to communicate to partners about the changes to API anyway, so we can mention as well that the legacy field will be deleted.

[punchagan]

Let me try again, looks like I was unable to communicate clearly.

   +-----------+                                            +---------+
   |           |                                            |         |
   | Recipient |                                            |         |
   |  Country  |                                           /+  Proj.  |
   |           +-                                      /--- |   Loc.  |
   |           | \--           +-----------+        /--     |         |
   |           |    \---       |           |     /--        |         |
   +-----------+        \--    |           | /---           |         |
                           \-- |           +-               +---------+
                              \+  Project  |                           
  +---------------+           /+           +--              +---------+
  |               |        /-- |           |  \-----        |         |
  |   Recipient   |    /---    |           \        \-----  |  Proj.  |
  |    Country    | /--        |           |\-            \-+   Loc.  |
  |               +-           +-----------+  \-            |         |
  |               |                             \           |         |
  |               |                              \-         |         |
  +---------------+                                \-       +---------+
                                                     \-                
                                                       \    +---------+
                                                        \-  |         |
                                                          \-|  Proj.  |
                                                            \   Loc.  |
                                                            |         |
                                                            |         |
                                                            |         |
                                                            +---------+

A ProjectLocation has the associated Latitude, Longitude, Street, City,
etc. But, no country (ignore the legacy field).

A RecipientCountry needs to be associated with a ProjectLocation to be able
to respond with a "country" in API responses like those
here.

It is not yet clear (to me) how we can do that, when we have multiple
RecipientCountry and ProjectLocation associated with a Project.

There can also be projects with ProjectLocation(s), but no RecipientCountry
in which case the country will be "null" in the response. It doesn't make
sense to me to have a Location model with all the fields except Country. May
be @zzgvh will have comments on this.

[Geerts]

A RecipientCountry needs to be associated with a ProjectLocation to be able to respond with a "country" in API responses like those here.

Imo we shouldn't aim for that, merging two dynamic fields seems to be impossible. If we add recipient country separate, users can request both separately and also process them in the way they need it.

[Geerts]

We have a new case of a partner dealing with this. Via water is trying to show the city and country of a project together on their website. This is possible by doing two different API requests namely:
http://rsr.akvo.org/rest/v1/recipient_country/?format=json&project_id=4342
http://rsr.akvo.org/rest/v1/project/?format=json&project_id=4342

However, at this moment it is not to fill in multiple city's hence it is also not possible to connect and match those two fields.

[GeertSoet]

Created via Reamaze:

Link: https://akvoo.reamaze.com/admin/conversations/feature-request-multiple-photos-in-update

[zzgvh]

I don't know the reasoning behind the removal of country from ProjectLocation (or any location object for that matter...) but couldn't we make the country field in ProjectLocation a foreign key to a RecipientCountry object for the project? We should be able to migrate this mostly automatically by first checking if the legacy country field matches one of the recipients, and second using a geo lookup API like http://ws.geonames.org/countryCodeJSON?lat=59&lng=18&username=demo for cases where there is no country.

It seems that this would work in most cases. You could even add recipient countries that have a commitment percentage of zero if a location is not in one of the "real" recipients.

I have not investigated how this impacts the IATI import.

[Geerts]

I don't know the reasoning behind the removal of country from ProjectLocation (or any location object for that matter...)

This came from the recipient country field. Guess it felt like we had two places to fill in country.

[nadiagorchakova]

Are there any action points from the above discussion? @punchagan @zzgvh

[punchagan]

The action items seem to be what @zzgvh has listed in this comment

1. make the country field in ProjectLocation a foreign key to a RecipientCountry object for the project
2. check if the legacy country field matches one of the recipients
3. using a geo lookup API for cases where there is no recipient country, probably add a recipient country that has a commitment percentage of zero
4. OrganisationLocation will also need to be fixed similarly.
5. Does it make sense that ProjectUpdateLocation (or BaseLocation) has everything else in the address, but country?
6. How would the UI for adding a new project location change?
   • If the project has a RecipientCountry?
   • If it doesn't?

@Geerts @zzgvh @nadiagorchakova comments?

[punchagan]

Stats from a DB about a week old

Description	Count	Query/Code
Total ProjectLocations	8106	ProjectLocation.objects.count()
ProjectLocations with no legacy country	2401	ProjectLocation.objects.filter(country=None).count()
No legacy country & No recipient country on Project	133	ProjectLocation.objects.filter(country=None).annotate(country_count=Count('location_target__recipient_countries')).filter(country_count=0).count()
Have legacy country; but no recipient country	29	ProjectLocation.objects.exclude(country=None).annotate(country_count=Count('location_target__recipient_countries')).filter(country_count=0).count()
Legacy country not a recipient country	31	sum([1 for loc in ProjectLocation.objects.exclude(country=None).select_related('location_target') if not loc.location_target.recipient_countries.filter(country__iexact=loc.country.iso_code).exists()])

[Geerts]

Does it make sense that ProjectUpdateLocation (or BaseLocation) has everything else in the address, but country?

—> No this doesn’t make sense. We got into this situation because of mixing rsr and iati fields and treating them as separate things. I think from the design perspective we should go back to the question:

• what does the user want to fill in here
• what does the user needs to fill in to comply with IATI

A possible solution could that we have one (recipient) country field where depending on if you report in IATI or not it shows you the % field.

[zzgvh]

• make the country field in ProjectLocation a foreign key to a RecipientCountry object for the project
• check if the legacy country field matches one of the recipients
• using a geo lookup API for cases where there is no recipient country, probably add a recipient country that has a commitment percentage of zero

These three operations make sense together. But the geo-lookup might be hard/take time. If we skip that we have a number of records that need manual fixing.

• OrganisationLocation will also need to be fixed similarly.

OrgLoc has iati_country that has data for each record so it seems we can just migrate to using that?

• Does it make sense that ProjectUpdateLocation (or BaseLocation) has everything else in the address, but country?

No. Again geo-lookup is ideal but maybe "expensive".

• How would the UI for adding a new project location change?
  • If the project has a RecipientCountry?
  • If it doesn't?

I think we need to decide if it's mandatory for a Project to have at least one associated Country. If so we can force the use of the exisiting countries from associated RecipientCountry objects when creating locations. That in turn means that RecipientCountry objects have to be added before you can create locations. Kinda awkward, I know...

Otherwise I think we need to implement automatic lookup of country given a coordinate pair. If we do that, we can find the country, and either use an existing RecipientCountry object or create one with 0%. Again cost/benefit decision.

Seems to me that experimenting with geo-lookup for a day or so might be a good idea to gauge what realistic options we have.