Dealing with character encoding issues and CSV files

[brianrandell]

I'm trying to use Slacker against the old Microsoft Northwind sample database. I'm guessing the issue is more of a Ruby / OS config issue than a Slacker problem. However, I'm just learning so I thought I'd ask.

I have a simple test:

# Method "describe" opens up an example group.
describe 'customers' do
    it 'customer cactu has results' do
        result = sproc('CustOrderHist', :CustomerID => 'CACTU')

        expect(result).to match('cactuorder_hist.csv')
    end
end

The cactuorder_hist.csv has the records from the database.

When I run the test, I get the following error:

Encoding::CompatibilityError:
incompatible character encodings: Windows-1252 and ASCII-8BIT

I'm running on Windows 10 1709 x64 against a local SQL Server 2017 instance. I've tried both ODBC and the tinyTDS option.

I've tried saving the file using the various encoding options (UTF-8, Unicode, etc.) in Notepad but I get other errors. Currently the file is saved with ANSI encoding and shows the above error.

Here's the resultset:
ProductName,Total
Geitost,10
Gorgonzola Telino,1
Ipoh Coffee,7
Jack's New England Clam Chowder,10
Laughing Lumberjack Lager,20
Rhönbräu Klosterbier,20
Röd Kaviar,10
Rössle Sauerkraut,8
Sasquatch Ale,20
Scottish Longbreads,6
Steeleye Stout,3

[vassilvk]

@brianrandell, the issue you are experiencing is twofold.

1. The odbc driver used by Slacker encodes data in ASCII.
2. Slacker expects CSV files to be encoded using Windows-1252 encoding as that matches the default encoding of the odbc driver.

I released a new version of Slacker - version 1.0.21, which properly encodes CSV contents using UTF-8 encoding.

This solves the second issue, but the first one stays open.
The good news is that tiny_tds has proper support for UTF-8.

I tested your scenario using slacker 1.0.21 and tiny_tds - worked without a hitch.

Please follow these steps to test on your end:

• Grab the new version of slacker:

  gem install slacker
  gem cleanup slacker

• Switch to tiny_tds using these instructions.
• Run your tests.

I am reopening the issue.
Please test on your end and close if successful.

[brianrandell]

Thanks! That did it.