UnicodeDecodeError via client/base_request.py", line 92, in SendRequest

[blueyed]

I am experimenting with Vim mappings to Alt/Meta keys in terminal Vim (resulting in strange characters being used), and when I had opened the output from :map via redir in a new buffer, it resulted in the following YCM error.

It is really nasty, because it keeps popping up after having closed the buffer.

Is there a way to disable YCM, without having to restart Vim? :YcmRestart did not help.

Error detected while processing function <SNR>242_OnBufferVisit:
line   12:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "…/YouCompleteMe/autoload/../python/ycm/youcompleteme.py", line 247, in OnBufferVisit
    SendEventNotificationAsync( 'BufferVisit', extra_data )
  File "…/autoload/../python/ycm/client/event_notification.py", line 71, in SendEventNotificationAsync
    event.Start()
  File "…/autoload/../python/ycm/client/event_notification.py", line 41, in Start
    'event_notification' )
  File "…/autoload/../python/ycm/client/base_request.py", line 77, in PostDataToHandlerAsync
    return BaseRequest._TalkToHandlerAsync( data, handler, 'POST', timeout )
  File "…/autoload/../python/ycm/client/base_request.py", line 113, in _TalkToHandlerAsync
    return SendRequest( data, handler, method, timeout )
  File "…/autoload/../python/ycm/client/base_request.py", line 92, in SendRequest
    data = json.dumps( data ),
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xdb in position 5409: invalid continuation byte

[Valloric]

There's a bug somewhere here, but it's hard to track down. I think Vim is giving YCM some badly encoded string data, but I can't find where.

If you could produce a test case for this, I'd appreciate it. This is likely related to having non-ASCII chars in your buffer or path to the buffer (neither of which would normally be a problem, and yet we have this issue).

[blueyed]

Like I've said, I can reproduce it when redirecting the output from :map.

I have tracked it down to a single char/byte: Hex 00db.

However, when saving and reading the buffer which contains it, it changes from being displayed as <db> to Û, and the error disappears.

It can be reproduced by defining the following map, and then redirecting the output from :map to a buffer, where YCM is active:

:nmap \e[ <A-[>

:redir => message
:  silent map
:redir END
:put=message

[blueyed]

Besides fixing the bug, it would be very nice, if errors like that wouldn't be that noisy.

[xiaket]

hi, there.

I bump into a similar problem like this one. I was given some files zipped under Windows and when I unzip it, its file name is not right:

~/fm/Fairy Master %C0%A6+%B1%E3%B8+%F5%CD+%A6%B3/GameServer

And after I closed files opened under that directory, YCM throws an error:

Error detected while processing function <SNR>56_OnBufferUnload:
line    5:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/youcompleteme.py", line 232, in
OnBufferUnload
    { 'unloaded_buffer': deleted_buffer_file } )
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/client/event_notification.py", l
ine 71, in SendEventNotificationAsync
    event.Start()
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/client/event_notification.py", l
ine 41, in Start
    'event_notification' )
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 77
, in PostDataToHandlerAsync
    return BaseRequest._TalkToHandlerAsync( data, handler, 'POST', timeout )
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 11
3, in _TalkToHandlerAsync
    return SendRequest( data, handler, method, timeout )
  File "/Users/xiaket/.xiaket/share/Dropbox/git/github/etc/vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 92
, in SendRequest
    data = json.dumps( data ),
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 40: invalid start byte

I printed the data and found that the data dictionary contains non-UTF8 characters, so the dumps function in json module is not working properly. I added an 'ensure_ascii=False' option to the dumps function, and the problem was solved.

I suspect @blueyed bumped into a similar problem and json dumps just failed.

I'm not sure whether it is safe to add this 'ensure_ascii' option to every instance of dumps function call in YCM, so I did not prepare a pull request. I really hope @Valloric would look into the problem.

Sorry again that I cannot provide a valid pull request.

[Valloric]

@xiaket Thanks for the heads-up on the ensure_ascii param. I've now set that to false and made the code explicitly encode the data as utf8.

I still can't repro this issue so please confirm that it's fixed.

[Remiol]

It is not fixed. I have the same error.

[blueyed]

@Remiol
Can you provide a test case? Do you get the same traceback?

My test case from above appears to be fixed.

[Remiol]

@blueyed
Sorry, I can't. I stopped trying to compile ycm for windows 64 bits. I uninstalled visual studio, mingw, python 64 bits, vim 64 bits, because I had too much troubles. After 4 days of trying I just wasn't enable to create a good working ycm_core.pyd file.
I reinstalled python 32 bits and vim 32 bits.

[xiaket]

@Remiol If I may ask, how can you be sure that you bumped into this error instead of an installation issue?

[Remiol]

@xiaket maybe it was an installation issue, I'm not sure.

[Oblomov]

For what it's worth, I'm still experiencing this issue, while editing ikiwiki file (basically markdown). The files I edit are UTF-8 encoded, enc=utf-8 as well, and they contain some UTF-8 characters (mostly em-dash U+2014, ellipsis U+2026, plus some smart quotes) and some less-used 7-bit control characters such as no-break space U+000A and form feed U+000C.

Now the interesting part is that the error only seems to trigger on long documents, and then again only when the form feeds are there (I use them to separate longer documents into sections since vim section navigation is what it is). In the files where I can reproduce it, simply removing the form feeds seems to be sufficient to not trigger the error. I'm not sure how this may be related though.

I can share one such file if needed.

[puremourning]

If you have a reproducible test case, that would be great!

Could you raise a new issue and follow the instructions in the CONTRIBUTING.md and/or the new issue template?

[Oblomov]

Sure, I'll give it a go. Let me first check if I can reduce the test case to a more manageable size

[Oblomov]

There we go. The test case isn't as minimal as I could hope, so I'm hosting it on a website of mine.

[puremourning]

Thanks. Any test case is better than no test case! :)