setup_bundler's rails_app? incompatible with non-ascii characters

[D-system]

Description

Reproduction steps

Having the config/application.rb with Japanese characters (eg: あ)

In https://github.com/Shopify/ruby-lsp/blob/88b85b810061aafd7d8d7078df6b5aa713979e54/test/fixtures/rails_application.rb add あ

module MyApp
  class Application < Rails::Application
    # あ
  end
end

Details

From this change #2218 ruby-lsp error out on

ruby-lsp/lib/ruby_lsp/setup_bundler.rb

Line 292 in 88b85b8

/class .* < (::)?Rails::Application/.match?(application_contents)

Ruby-lsp: 0.17.14
Ruby: 3.3.4
Rails: 6.1.7.8

Error: setup_bundler.rb:295:in 'match?': invalid byte sequence in US-ASCII (ArgumentError)
Full output:

2024-08-20 17:32:33.321 [info] (app) Checking if chruby is available on the path with command: /bin/zsh -i -c 'chruby --version'
2024-08-20 17:32:35.982 [info] (app) Checking if rbenv is available on the path with command: /bin/zsh -i -c 'rbenv --version'
2024-08-20 17:32:36.935 [info] (app) Discovered version manager rbenv
2024-08-20 17:32:36.935 [info] (app) Running command: `rbenv exec ruby -W0 -rjson -e 'STDERR.print({env: ENV.to_h,yjit:!!defined?(RubyVM::YJIT),version:RUBY_VERSION}.to_json)'` in /Users/thomas/projects/app using shell: /bin/zsh
2024-08-20 17:32:38.382 [info] (app) /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:295:in `match?': invalid byte sequence in US-ASCII (ArgumentError)

      /class .* < (::)?Rails::Application/.match?(application_contents)
                                                  ^^^^^^^^^^^^^^^^^^^^
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:295:in `rails_app?'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `bind_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `validate_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:53:in `initialize'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `bind_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `validate_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/exe/ruby-lsp:67:in `new'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/exe/ruby-lsp:67:in `<top (required)>'
	from /Users/thomas/.rbenv/versions/3.3.4/bin/ruby-lsp:25:in `load'
	from /Users/thomas/.rbenv/versions/3.3.4/bin/ruby-lsp:25:in `<main>'

2024-08-20 17:32:38.390 [info] (app) [Error - 5:32:38 PM] Server process exited with code 1.
2024-08-20 17:32:38.391 [info] (app) [Error - 5:32:38 PM] Server initialization failed.

Using application_contents.force_encoding("UTF-8") solve the problem.

[andyw8]

@D-system what does the following return in your app?

file -mime config/application.rb

[tekin]

I've also started seeing this error. My application.rb file has a single ’ non-ASCII character that has been there for some time:

# Override Rails’ default public exception handling
config.exceptions_app = self.routes

$ file -mime config/application.rb
config/application.rb: Ruby script text, Unicode text, UTF-8 text

[Earlopain]

Can you show the output of Encoding.default_external and Encoding.default_internal in irb?

[vinistock]

This problem doesn't seem to be isolated to multibyte characters. There's some encoding issue in play here. Adding Japanese or other multibyte characters to the fixture still passes the test because the file contents are read as UTF-8.

Even forcing the file encoding to be US-ASCII passes the test.

[vinistock]

I created a revert #2466 while we're trying to understand what's going on, since this blocks people from using the LSP.

[Earlopain]

If you run RUBYOPT="--encoding US-ASCII" bundle exec ruby -Itest ./test/setup_bundler_test.rb --name SetupBundlerTest#test_ruby_lsp_rails_is_automatically_included_in_rails_apps, the test will fail (if it contains non-ascii characters). Though, I'm not sure if forcing the encoding to UTF8 when reading the file would be the correct solution.

[andyw8]

It also fails with LANG=anything bundle exec ruby -Itest ./test/setup_bundler_test.rb --name SetupBundlerTest#test_ruby_lsp_rails_is_automatically_included_in_rails_apps, so I'm wondering if something in the environment is setting that. 🤔

[andyw8]

#2469 should solve this but I'd still like to understand why if/why LANG being changed.

[tekin]

$ echo $LANG                                                                               
en_US.UTF-8

[D-system]

@andyw8

what does the following return in your app?

❯ file -mime config/application.rb
config/application.rb: Ruby script text, Unicode text, UTF-8 text

[D-system]

#2469 should solve this but I'd still like to understand why if/why LANG being changed.

I tried it with 0.17.4 and yes, it solved the issue.
And the 0.17.4 was able to update to 0.17.5 and runs great. Thanks for the reactivity.
Closing the ticket.

So the issue depends on the var env LANG which Rails solved the problem by forcing the encoding to UTF-8 https://github.com/rails/rails/blob/v7.1.3.3/railties/lib/rails.rb#L21-L25

[D-system]

Can you show the output of Encoding.default_external and Encoding.default_internal in irb?

irb(main):001> Encoding.default_external
=> #<Encoding:UTF-8>
irb(main):002> Encoding.default_internal
=> nil

[D-system]

FYI: maintenance task had the same issue from ActiveStorage that reads CSV files in ASCII-8bit/binary even that file is text with UTF-8 encoding at upload time.
Shopify/maintenance_tasks#1025