Shapefile upload with not ASCII field names fails

[capooti]

Observed in 2.7.x. Error log:

[29/May/2018 21:05:00] "GET /upload/progress HTTP/1.1" 200 17
uploaded_files: [<InMemoryUploadedFile: ming_female_1.zip (application/zip)>]
shapefile components: [u'shp', 'dbf', 'shx', 'prj']
valid_extensions: ('zip',)
relevant_files: [<InMemoryUploadedFile: ming_female_1.zip (application/zip)>]
Name: ming_female_1, ext: .zip
base_file: /tmp/tmp7WMjHJ/ming_female_1.zip
spatial_files: [<SpatialFile base_file=/tmp/tmp7WMjHJ/ming_female_1.shp file_type=<FileType shp> aux=[u'/tmp/tmp7WMjHJ/ming_female_1.dbf', u'/tmp/tmp7WMjHJ/ming_female_1.shx', u'/tmp/tmp7WMjHJ/ming_female_1.prj'] sld=[] xml=[]>]
Uploading layer: ming_female_1, files [<SpatialFile base_file=/tmp/tmp7WMjHJ/ming_female_1.shp file_type=<FileType shp> aux=[u'/tmp/tmp7WMjHJ/ming_female_1.dbf', u'/tmp/tmp7WMjHJ/ming_female_1.shx', u'/tmp/tmp7WMjHJ/ming_female_1.prj'] sld=[] xml=[]>]
Using name as requested
Name for layer: 'ming_female_1'
GET http://localhost:8080/geoserver/rest/workspaces.xml
GET http://localhost:8080/geoserver/rest/workspaces/geonode/datastores.xml
GET http://localhost:8080/geoserver/rest/workspaces/geonode/coveragestores.xml
GET http://localhost:8080/geoserver/rest/workspaces/geonode/wmsstores.xml
files_to_upload: [u'/tmp/tmp7WMjHJ/ming_female_1.shp', u'/tmp/tmp7WMjHJ/ming_female_1.dbf', u'/tmp/tmp7WMjHJ/ming_female_1.shx', u'/tmp/tmp7WMjHJ/ming_female_1.prj']
Uploading featureType
Finished upload of [ming_female_1] to GeoServer without errors.
provided sld is None
saved session : <geonode.upload.upload.UploaderSession object at 0x7f7a2180a350>
[29/May/2018 21:05:01] "POST /upload/ HTTP/1.1" 200 109
Unexpected error in upload step
Traceback (most recent call last):
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/views.py", line 671, in view
    Upload.objects.update_from_session(upload_session)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/models.py", line 42, in update_from_session
    upload_session)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/models.py", line 107, in update_from_session
    self.save()
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 827, in _save_table
    forced_update)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 877, in _do_update
    return filtered._update(values) > 0
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 580, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1062, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 83, in execute
    sql = self.db.ops.last_executed_query(self.cursor, sql, params)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/operations.py", line 218, in last_executed_query
    return cursor.query.decode('utf-8')
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfd in position 1443: invalid start byte
Traceback (most recent call last):
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/views.py", line 671, in view
    Upload.objects.update_from_session(upload_session)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/models.py", line 42, in update_from_session
    upload_session)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/geonode/upload/models.py", line 107, in update_from_session
    self.save()
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 827, in _save_table
    forced_update)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 877, in _do_update
    return filtered._update(values) > 0
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 580, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1062, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 83, in execute
    sql = self.db.ops.last_executed_query(self.cursor, sql, params)
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/operations.py", line 218, in last_executed_query
    return cursor.query.decode('utf-8')
  File "/home/ubuntu/worldmap_28/worldmap-geonode-2.7.x/env/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfd in position 1443: invalid start byte

[capooti]

This is happening because of a problem with GeoServer REST API.

When checking gs_resource.native_bbox in Step 6 of geoserver/upload.py the error is raised. This is the endpoint which is checked:

http://localhost:8080/geoserver/rest/workspaces/geonode/datastores/wmdata/featuretypes/ming_female_1.xml

and this is the exception:

*** Exception: ('GeoServer gave non-XML response for [GET http://localhost:8080/geoserver/rest/workspaces/geonode/datastores/wmdata/featuretypes/ming_female_1.xml]: \n ming_female_1\n ming_female_1\n \n geonode\n <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="alternate" href="http://localhost:8080/geoserver/rest/namespaces/geonode.xml" type="application/xml"/>\n \n <title>ming_female_1</title>\n ....

@afabiani any chance this can be fixed in GeoServer?

In case you want to replicate I can send you the shapefile.

[capooti]

@afabiani is this a know problem? should I report this in the GeoServer mailing list?

[afabiani]

Hi @capooti we should open a JIRA on GeoServer for this one I guess.

[capooti]

done! thanks
https://osgeo-org.atlassian.net/projects/GEOS/issues/GEOS-8890?filter=allissues

[capooti]

Update: Andrea Aime suggests (look at the comments in the ticket) to prevent field names containing characters like "@", "/" etc... when uploading shapefiles and add this to our fields sanitizer. I can look into this soon and come with a fix.

[capooti]

This was fixed now with PR #4026