Encoding problem in Metadata

[dvictori]

I have GeoNode 2.8 installed from package in Ubuntu.

I uploaded several shapefiles along with their metadata in a xml file using geonode importlayers command. Everything worked fine and I can see the metadata using the Metadata detail button, including the accented characters in the title and keywords field.

But when I try to edit the metadata (Edit layers \ metadata wizard) I get an error. Looking at the apache error log, it mentions a problem with encoding.

--- Apache error log ---

[Wed Sep 12 06:55:17.755914 2018] [wsgi:error] [pid 30108:tid 139760428979968] Internal Server Error: /layers/geonode:go1_a01_2014_lasboundary/metadata
[Wed Sep 12 06:55:17.755940 2018] [wsgi:error] [pid 30108:tid 139760428979968] Traceback (most recent call last):
[Wed Sep 12 06:55:17.755945 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
[Wed Sep 12 06:55:17.755955 2018] [wsgi:error] [pid 30108:tid 139760428979968] response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Wed Sep 12 06:55:17.755959 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
[Wed Sep 12 06:55:17.755962 2018] [wsgi:error] [pid 30108:tid 139760428979968] return view_func(request, *args, **kwargs)
[Wed Sep 12 06:55:17.755965 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/local/lib/python2.7/dist-packages/geonode/layers/views.py", line 1093, in layer_metadata
[Wed Sep 12 06:55:17.755969 2018] [wsgi:error] [pid 30108:tid 139760428979968] getattr(settings, 'GROUP_MANDATORY_RESOURCES', False),
[Wed Sep 12 06:55:17.755972 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/shortcuts.py", line 67, in render
[Wed Sep 12 06:55:17.755975 2018] [wsgi:error] [pid 30108:tid 139760428979968] template_name, context, request=request, using=using)
[Wed Sep 12 06:55:17.755978 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader.py", line 99, in render_to_string
[Wed Sep 12 06:55:17.755981 2018] [wsgi:error] [pid 30108:tid 139760428979968] return template.render(context, request)
[Wed Sep 12 06:55:17.755984 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/backends/django.py", line 74, in render
[Wed Sep 12 06:55:17.755988 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.template.render(context)
[Wed Sep 12 06:55:17.755991 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 210, in render
[Wed Sep 12 06:55:17.755994 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self._render(context)
[Wed Sep 12 06:55:17.755997 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render
[Wed Sep 12 06:55:17.756000 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.nodelist.render(context)
[Wed Sep 12 06:55:17.756003 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756006 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756009 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756012 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756015 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 135, in render
[Wed Sep 12 06:55:17.756019 2018] [wsgi:error] [pid 30108:tid 139760428979968] return compiled_parent._render(context)
[Wed Sep 12 06:55:17.756022 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render
[Wed Sep 12 06:55:17.756025 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.nodelist.render(context)
[Wed Sep 12 06:55:17.756028 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756031 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756034 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756037 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756040 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 135, in render
[Wed Sep 12 06:55:17.756046 2018] [wsgi:error] [pid 30108:tid 139760428979968] return compiled_parent._render(context)
[Wed Sep 12 06:55:17.756050 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render
[Wed Sep 12 06:55:17.756053 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.nodelist.render(context)
[Wed Sep 12 06:55:17.756056 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756059 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756062 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756065 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756068 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 135, in render
[Wed Sep 12 06:55:17.756071 2018] [wsgi:error] [pid 30108:tid 139760428979968] return compiled_parent._render(context)
[Wed Sep 12 06:55:17.756075 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render
[Wed Sep 12 06:55:17.756078 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.nodelist.render(context)
[Wed Sep 12 06:55:17.756081 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756084 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756087 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756090 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756093 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 65, in render
[Wed Sep 12 06:55:17.756096 2018] [wsgi:error] [pid 30108:tid 139760428979968] result = block.nodelist.render(context)
[Wed Sep 12 06:55:17.756099 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756102 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756105 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756109 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756112 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 65, in render
[Wed Sep 12 06:55:17.756115 2018] [wsgi:error] [pid 30108:tid 139760428979968] result = block.nodelist.render(context)
[Wed Sep 12 06:55:17.756118 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756121 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756124 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756127 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756130 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/floppyforms/templatetags/floppyforms.py", line 500, in render
[Wed Sep 12 06:55:17.756136 2018] [wsgi:error] [pid 30108:tid 139760428979968] output = nodelist.render(context)
[Wed Sep 12 06:55:17.756139 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 212, in render
[Wed Sep 12 06:55:17.756142 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self._render(context)
[Wed Sep 12 06:55:17.756145 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render
[Wed Sep 12 06:55:17.756148 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.nodelist.render(context)
[Wed Sep 12 06:55:17.756151 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756154 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756158 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756161 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756164 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 56, in render
[Wed Sep 12 06:55:17.756167 2018] [wsgi:error] [pid 30108:tid 139760428979968] result = self.nodelist.render(context)
[Wed Sep 12 06:55:17.756170 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 905, in render
[Wed Sep 12 06:55:17.756173 2018] [wsgi:error] [pid 30108:tid 139760428979968] bit = self.render_node(node, context)
[Wed Sep 12 06:55:17.756176 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 79, in render_node
[Wed Sep 12 06:55:17.756179 2018] [wsgi:error] [pid 30108:tid 139760428979968] return node.render(context)
[Wed Sep 12 06:55:17.756182 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/template/debug.py", line 92, in render
[Wed Sep 12 06:55:17.756186 2018] [wsgi:error] [pid 30108:tid 139760428979968] output = force_text(output)
[Wed Sep 12 06:55:17.756189 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/utils/encoding.py", line 92, in force_text
[Wed Sep 12 06:55:17.756192 2018] [wsgi:error] [pid 30108:tid 139760428979968] s = six.text_type(s)
[Wed Sep 12 06:55:17.756195 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/utils/html.py", line 404, in
[Wed Sep 12 06:55:17.756198 2018] [wsgi:error] [pid 30108:tid 139760428979968] klass.unicode = lambda self: mark_safe(klass_unicode(self))
[Wed Sep 12 06:55:17.756201 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/forms/forms.py", line 537, in str
[Wed Sep 12 06:55:17.756204 2018] [wsgi:error] [pid 30108:tid 139760428979968] return self.as_widget()
[Wed Sep 12 06:55:17.756207 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/lib/python2.7/dist-packages/django/forms/forms.py", line 593, in as_widget
[Wed Sep 12 06:55:17.756211 2018] [wsgi:error] [pid 30108:tid 139760428979968] return force_text(widget.render(name, self.value(), attrs=attrs))
[Wed Sep 12 06:55:17.756214 2018] [wsgi:error] [pid 30108:tid 139760428979968] File "/usr/local/lib/python2.7/dist-packages/geonode/base/forms.py", line 115, in render
[Wed Sep 12 06:55:17.756217 2018] [wsgi:error] [pid 30108:tid 139760428979968] vals = ','.join([str(i.tag.name) for i in values])
[Wed Sep 12 06:55:17.756220 2018] [wsgi:error] [pid 30108:tid 139760428979968] UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 3: ordinal not in range(128)

[dvictori]

This was first discussed in the GeoNode users mailing list on Sep 12, 2018.

@afabiani mentioned this issue was already fixed in 2.7.x branch and I should compare the code base. He also suggested I should convert the email tread to an issue.

He also commented that

so the issue happens in this method, probably in the context preparation

https://github.com/GeoNode/geonode/blob/master/geonode/layers/views.py#L798

WARNING: your code will be different

From the stack trace basically there should be a non ASCII character somewhere on the metadata fields which cause issue to the Django template. Probably a keyword? You should be able to identify the strange characters from the metadata XML.

However, basically, once you identified which attribute is causing the issue, you should be able to go to the Layers Models or Base ResourceBase Models and add a line of code like this one

<whatever_attribute_causing_issue>.encode("utf-8")

[dvictori]

I believe it's a problem with the keyword being accented, since accents only in the Title do not cause an error.

So I assume I must place keywords.encode("utf-8") somewhere inside the ResourceBase class in base/model.py. So where should I place this? At the end of the class? Or do I need to change something else?

[afabiani]

@dvictori

this PR #4026 contains changes which should address your issue.

[frafra]

Would it be possible to have it merged on 2.8.x too?
Update: done.