Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression with uploading a shapefile with no ascii characters #4025

Closed
capooti opened this issue Oct 25, 2018 · 7 comments
Closed

Regression with uploading a shapefile with no ascii characters #4025

capooti opened this issue Oct 25, 2018 · 7 comments
Assignees
@afabiani
Labels
blocker Critical issue blocking next major release and/or (People are badly blocked with no workaround) regression Issues related to regressions.
Milestone
2.10

Comments

@capooti
Copy link
Member

capooti commented Oct 25, 2018 •
edited
Loading

Using master I cannot upload this shapefile, which a few months could be uploaded:

ming_female_1.zip

The error is:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfd in position 1437: invalid start byte

This is happening in the upload.models.update_from_session() when the session is saved (self.save()).

I believe this can happen to any shapefile with sends some non-ascii character in the unicode_session and it can't be encoded/decoded. @afabiani any hint on this?
@capooti capooti added the blocker Critical issue blocking next major release and/or (People are badly blocked with no workaround) label Oct 25, 2018
@afabiani afabiani added the regression Issues related to regressions. label Oct 25, 2018
@afabiani afabiani self-assigned this Oct 25, 2018
@afabiani
Copy link
Member

taking a look

@capooti
Copy link
Member Author

capooti commented Oct 25, 2018 •
edited
Loading

thanks @afabiani
Using the geoserver importer (from the GeoServer admin) the shapefile is correctly imported. The error happens in GeoNode

@capooti
Copy link
Member Author

capooti commented Oct 25, 2018

I realized it is not a regression, I tried the method as it was one year ago before your PR with mosaics and there is the same error. It could be that when I uploaded it correctly it was using geonode.rest vs geonode.importer (which I am currently using)

@afabiani
Copy link
Member

hi @capooti indeed there is an issue, which should be handled somehow.

However, in your specific use case I'm getting an error from GeoServer Importer, not being able to ingest the layer

java.io.IOException: Error occurred creating table
	at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:795)
	at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:150)
	at org.geoserver.importer.Importer.loadIntoDataStore(Importer.java:1334)
	at org.geoserver.importer.Importer.doIndirectImport(Importer.java:1105)
	at org.geoserver.importer.Importer.run(Importer.java:928)
	at org.geoserver.importer.Importer.run(Importer.java:891)
	at org.geoserver.importer.Importer$5.callInternal(Importer.java:962)
	at org.geoserver.importer.Importer$5.callInternal(Importer.java:1)
	at org.geoserver.importer.Importer$SecurityContextCopyingJob.call(Importer.java:993)
	at org.geoserver.importer.job.Job.call(Job.java:16)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERRORE: la colonna "__" è stata specificata più di una volta
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2476)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2189)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:260)
	at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
	at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
	at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:787)
	... 13 more

@capooti
Copy link
Member Author

capooti commented Oct 25, 2018 •
edited
Loading

Which GeoServer version are you using? I am with 2.14.x, and this is what I see in log:

2018-10-25 17:35:48,864 TRACE [importer.format] - Error reading file as json
java.lang.RuntimeException: Unexpected character (G) at position 0.
	at org.geotools.geojson.feature.FeatureJSON$FeatureCollectionIterator.readNext(FeatureJSON.java:725)
	at org.geotools.geojson.feature.FeatureJSON$FeatureCollectionIterator.hasNext(FeatureJSON.java:710)
	at org.geoserver.importer.format.GeoJSONFormat.sniff(GeoJSONFormat.java:116)
	at org.geoserver.importer.format.GeoJSONFormat.canRead(GeoJSONFormat.java:106)
	at org.geoserver.importer.DataFormat.lookup(DataFormat.java:50)
	at org.geoserver.importer.Directory.prepare(Directory.java:192)
	at org.geoserver.importer.Importer.addTasks(Importer.java:472)
	at org.geoserver.importer.Importer.update(Importer.java:457)
	at org.geoserver.importer.rest.ImportTaskController.acceptData(ImportTaskController.java:277)
	at org.geoserver.importer.rest.ImportTaskController.taskPost(ImportTaskController.java:178)

Furthermore I can successfully import the shapefile with the GeoServer importer from GeoServer admin site. I believe it is definitely a GeoNode error. Thanks!

@capooti
Copy link
Member Author

capooti commented Oct 25, 2018

I definitely think is a regression. I have downloaded a few shapefiles from our GeoNode instance - which were successfully uploaded with GeoNode 2.7.x and which have Chinese field names and now they all fail to upload.

@capooti capooti mentioned this issue Oct 25, 2018
Closed
afabiani pushed a commit that referenced this issue Oct 26, 2018
[Fixes #4025] Regression with uploading a shapefile with no ascii cha…
5a08be8 
…racters
@afabiani
Copy link
Member

@capooti can you give a try to this patch here?

#4026

afabiani pushed a commit to geosolutions-it/geonode that referenced this issue Oct 29, 2018
[Fixes GeoNode#4025] Regression with uploading a shapefile with no as…
5afb5ad 
…cii characters
afabiani pushed a commit to geosolutions-it/geonode that referenced this issue Oct 29, 2018
[Fixes GeoNode#4025] Regression with uploading a shapefile with no as…
ac64c41 
…cii characters
afabiani pushed a commit that referenced this issue Oct 29, 2018
[Fixes #4025] Regression with uploading a shapefile with no ascii cha…
b31cbf7 
…racters
afabiani pushed a commit to geosolutions-it/geonode that referenced this issue Oct 29, 2018
[Fixes GeoNode#4025] Regression with uploading a shapefile with no as…
bd3bba5 
…cii characters
afabiani pushed a commit to geosolutions-it/geonode that referenced this issue Oct 30, 2018
[Fixes GeoNode#4025] Regression with uploading a shapefile with no as…
ffad0fc 
…cii characters
frafra pushed a commit to frafra/geonode that referenced this issue Dec 19, 2018
@frafra
[Fixes GeoNode#4025] Regression with uploading a shapefile with no as…
0e0d1bf 
…cii characters (GeoNode#4026)

* [Fixes GeoNode#4025] Regression with uploading a shapefile with no ascii characters

* [Fixes GeoNode#4025] Regression with uploading a shapefile with no ascii characters

*  - Rename columns of non-UTF-8 shapefiles attributes before ingesting

 - Fix test cases
@frafra frafra mentioned this issue Dec 19, 2018
Merged
@afabiani afabiani added this to the 2.10 milestone Jun 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@afabiani afabiani

Labels
blocker Critical issue blocking next major release and/or (People are badly blocked with no workaround) regression Issues related to regressions.
Projects
None yet
Milestone
2.10
Development

No branches or pull requests
2 participants
@capooti @afabiani