Shapefile Dataset (with its ISO 19115/19139 metadata) not found after uploading it using API

[eblondel]

Expected Behavior

A shapefile + its ISO 19115/19139 XML metadata uploaded through the web API, and made visible over GeoNode.

Actual Behavior

The API returns a success (status code 200) with status "finished" and an URL pointing to the catalogue dataset; BUT

• the dataset is not visible in the GeoNode web-interface.
• Trying to get the resource/dataset through the API also leads to a "Not found" error
• Additional tests, that seem to indicate the upload actually works:
  • Trying to re-upload it with the same ISO XML metadata, and same UUID as identifier; leads to an error saying the UUID is already in the system
  • Perform a DELETE operation with the API also works (which allows to re-upload it with same metadata UUID after)

Live examples:

• when pushing the shapefile WITHOUT the metadata: https://stable.demo.geonode.org/catalogue/#/dataset/6160 (visible)
• when pushing the shapefile WITH metadata: https://stable.demo.geonode.org/catalogue/#/dataset/6161 (not visible, although it seems to be there according to above diagnosis)

It seems to be an issue finding the dataset through the search

( Are there prequisites of metadata content for the XML file to be uploaded together with the shapefile, and that could be the cause of dataset "invisibility"? )

Steps to Reproduce the Problem

1. Get this shapefile (and associated ISO 19115 XML file): https://github.com/eblondel/geonode4R/tree/main/inst/extdata/samples
2. Try to upload it with the API

The integration test here (R language) reproduces the full test: https://github.com/eblondel/geonode4R/blob/main/tests/testthat/test_uploads.R

Specifications

https://stable.demo.geonode.org/

[kikislater]

If we tried to upload a zip without xml then upload xml it returns :

The UUID identifier from the XML Metadata, is different from the one saved]

Edit: Even with one download from another geonode then upload to new one : same error.

[kikislater]

So I made some tests on stable Geonode :

• Creation of dataset + metadata from GUI then exporting shape + metadata iso format
• Delete all data in Geonode
• upload same shapefile and same xml.... not possible : invalid Geonode report invalid xml
• upload shapefile ... then import ISO Metadata in dataset : same error as above => The UUID identifier from the XML Metadata, is different from the one saved]
• Checking dataset uuid from api and replace in proper xml in this block :

   <gmd:fileIdentifier>
     <gco:CharacterString>4eb36cec-0763-47f1-ade7-9ae9bbe0aadc</gco:CharacterString>
   </gmd:fileIdentifier>

• Reimport modified xml : it works !
• checking metadata : some are modified from xml but some disappears like :
  • Attribution
  • Restrictions
  • Edition
  • DOI

Please note as well that some Geonode api have PUT and others POST methods to upload dataset

[eblondel]

thanks @kikislater it clarifies some part, but what is not clear is why Geonode provides a method to upload dataset (eg. shapefile) - with method endpoint uploads/upload - which includes support for ISO 19115 xml file; if Geonode expects to have this XML file with a fileIdentifier that corresponds to internal UUID assigned by Geonode for the dataset. Through a single method call, i push an XML metadata that is has an pre-defined identifier (uuid or other semantic id), that should not be a blocker for this, and if Geonode would expect to modify the metadata fileIdentifier, this should be post-processed. Am i missing something?

I see there is another method that allows to upload an ISO 19115 separately (method endpoint datasets/{dataset_id}/metadata), but it seems new and supported only since recent versions of Geonode

[kikislater]

Agree and you could test it through api interface: https://master.demo.geonode.org/api/v2/uploads/upload

• First load of page logged in =>

• Second attempt after trying to POST a shapefile =>
  

... different form but should not affect api if using proper base_file, dbf_file among others

[eblondel]

Actually the api answered with a success when I add the metadata, but then the dataset is not visible and discovered by geonode Le mer. 22 mars 2023 à 08:29, Sylvain POULAIN ***@***.***> a écrit :

…

Agree and you could test it through api interface: https://master.demo.geonode.org/api/v2/uploads/upload - First load of page logged in => [image: image] <https://user-images.githubusercontent.com/13238064/226830397-685e9df3-3c32-491c-969a-d9d8cae32057.png> - Second attempt after trying to POST a shapefile => [image: image] <https://user-images.githubusercontent.com/13238064/226830769-275c3593-fbc1-48d7-bc2c-b330e27d5736.png> ... different form but should not affect api if using proper base_file, dbf_file among others — Reply to this email directly, view it on GitHub <#10762 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKDK3HRIA3CRN3ISD6VVW3W5KS5FANCNFSM6AAAAAAVWKP5CY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kikislater]

If you have access to django administration interface it should be in dataset with untick "approved" and "is published" :

May be through this point on api : /api/v2/datasets

But this issue let me remind me that it's not possible : #8850

[kikislater]

Here you can post metadata :

https://docs.geonode.org/en/master/devel/api/usage/index.html?highlight=token#dataset-update-metadata

import requests

url = ROOT + "api/v2/datasets/" + DATASET_ID
auth = (LOGIN_NAME, LOGIN_PASSWORD)

data = {
    "title": "a new title",
    "license": 4,
}
response = requests.patch(url, auth=auth, json=data)

https://docs.geonode.org/en/master/devel/api/V2/index.html#put--api-v2-datasets-id-metadata-

[kikislater]

Affects only stable version, master and dev publish metadata

[kikislater]

153736b

[giohappy]

@eblondel I wasn't able to reproduce the issue. I've tested with Postman and the upload with the XML file worked fine. I could see the resource and the attributes from the file were assigned correctly.
Since I see you're using R this is a script that is working for me:

library(httr)

headers = c(
  'Authorization' = 'Basic <you base64 credentials>',
)

body = list(
  'base_file' = upload_file('./shapefile1.shp'),
  'dbf_file' = upload_file('./shapefile1.dbf'),
  'shx_file' = upload_file('./shapefile1.shx'),
  'prj_file' = upload_file('./shapefile1.prj'),
  'xml_file' = upload_file('./shapefile1.xml')
)

res <- VERB("POST", url = "http://localhost:8000/api/v2/uploads/upload", body = body, add_headers(headers), encode = 'multipart')

cat(content(res, 'text'))

Same test with cURL

curl --location 'http://localhost:8000/api/v2/uploads/upload' \
--header 'Authorization: Basic <you base64 credentials>' \
--form 'base_file=@"./shapefile1.shp"' \
--form 'dbf_file=@"./shapefile1.dbf"' \
--form 'shx_file=@"./shapefile1.shx"' \
--form 'prj_file=@"./shapefile1.prj"' \
--form 'xml_file=@"./shapefile1.xml"'

[eblondel]

@giohappy Thanks for your tests, what is your localhost geonode? All my tests were done on https://stable.demo.geonode.org @kikislater 's test sound to indicate the same, it doesn't work on the stable geonode.

[eblondel]

I've tested it over https://development.demo.geonode.org it works, but not on the stable release, so I understand that something was fixed over development. do you confirm?

[kikislater]

153736b

This is here

[eblondel]

we can close this, as it works as expected. Thanks for your support