fix: previewers & entry maker file encoding

[jamestrew]

Description

Previewers and some entry_makers currently has poor support for non UTF8 encoded files.
This PR looks to add a new option file_encoding to pickers with for previewers file_previewer & grep_previewer, qflist_previewer.
This new option is consumed by buffer_previewer.file_maker to do some string processing with vim.iconv. A similar thing also happens in make_entry.gen_from_vimgrep.

Fixes #2415

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list relevant details about your configuration

This change impacts the following pickers:
live_grep
grep_string
find_files
git_files
treesitter
current_buffer_fuzzy_find
oldfiles
buffers
marks
lsp_dynamic_workspace_symbols
lsp_workspace_symbols
lsp_document_symbols
lsp_implementations
lsp_type_definitions
lsp_definitions
lsp_outgoing_calls
lsp_incoming_calls
lsp_references

• For each picker above, confirm previewer and entries look good for non-UTF8 encoded files

1. create file with acentuação
2. write file with latin1 encoding :write ++enc=latin1
3. test pickers with :Telescope <picker> file_encoding=latin1 and verify previewer/entry

• Regression test with standard utf8 file and no file_encoding option

Configuration:

• Neovim version (nvim --version): NVIM v0.9.0-dev-1070+g2630341db
• Operating system and version: Linux archlinux 6.2.5-arch1-1

Checklist:

• My code follows the style guidelines of this project (stylua)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (lua annotations)

[tjdevries]

One other note would be since:

        Converted string if conversion succeeds, `nil` otherwise.

Maybe we should default back to using the input text if we get nil back from the iconv just in case? I'm not super familiar with file encoding stuff though.

[jamestrew]

Maybe we should default back to using the input text if we get nil back from the iconv just in case? I'm not super familiar with file encoding stuff though.

I'm not super familiar either but probably a good idea to handle this case. Updated the code.

[tjdevries]

i prefer something like line = line or c and then do the insert.

[jamestrew]

Actually, going back to the handling nil thing, if vim.iconv returns nil wouldn't the above "ternary operator" handle it and just return c.
We should be able to simply table.insert(t, line).

[tjdevries]

sorry, one little nit and then we can merge. Thanks :)

[Conni2461]

thanks :)

[Conni2461]

and sorry for my really late response. i just rebased and fixed the conflicts

[brneor]

Just tested this change. Even though I can now see the preview correctly I still can't search for this characters.

Fox example, in a file with the content acentuação, using Telescope live_grep file_encoding=latin1 a search for ç gives me no results.

[Conni2461]

can you open a new issue, for that? also it would be good to know which sorter you use. fzf-native, fzy-native, .... so yeah basically we need information to your configuration, so a new issue would be great

[jamestrew]

I can take a look at this. I think we simply need to pass the file_encoding=latin1 option to ripgrep.

Edit: #2570
@brneor should work when using this PR.