Failed to load WithCLexer, using pure Ruby lexer

[hmdne]

I have Opal installed with distribution provided gem and bundler commands (without rvm). I wasn't able to enable c_lexer, because an error happened all the time:

Failed to load WithCLexer, using pure Ruby lexer

I tried to inspect the issue further. I tried doing "require 'c_lexer'" manually, but

LoadError: cannot load such file -- /usr/share/gems/gems/c_lexer-2.6.4.1.0/lib/lexer

Hmm I guess you meant lexer.so, but it's not in lib, it's in ext/lexer actually. I replaced this line in lib/c_lexer.rb from:

require_relative 'lexer'

To:

require_relative '../ext/lexer/lexer'

And it finally worked. Would you consider changing it upstream or somehow work towards improving the build tools? Or should I consider upgrading some tool? I use distribution provided ruby-2.6.3 and bundler-1.17.2.

[iliabylich]

I can't reproduce this issue locally. By having this code in the Gemfile:

source 'https://rubygems.org'
ruby '2.6.4'

gem 'parser', '= 2.6.4.1'
gem 'c_lexer', '= 2.6.4.1.0'

I get 0 errors:

$ ruby -rbundler/setup -rparser -rc_lexer -e 'puts Parser::CLexer'
Parser::CLexer

because c_lexer.so (well, c_lexer.bundle on my system) is in the lib directory:

$ tree /Users/ilya/.rvm/gems/ruby-2.6.4@c_lexer-crash-test/gems/c_lexer-2.6.4.1.0/lib
/Users/ilya/.rvm/gems/ruby-2.6.4@c_lexer-crash-test/gems/c_lexer-2.6.4.1.0/lib
├── c_lexer
│   └── version.rb
├── c_lexer.rb
└── lexer.bundle

1 directory, 3 files

lexer.so should've been created in the lib directory, we have a test suite on Travis that runs all original tests from whitequark/parser and we don't include ext directory into the load path, and it works fine

[hmdne]

Yes, the difference is that I don't use rvm. I also tested using rvm and the .so file ended up in a lib/ directory.

The issue is broader, I encountered it first when trying to install sassc-ruby. There's an issue in their tracker: sass/sassc-ruby#146 (some posts refer to a different issue) but I can say it at least spreads across 4 distributions: CentOS, openSUSE, Fedora, maybe Ubuntu as well. For the record, sassc-ruby 2.1.0 works, because the installed gem is a binary distribution and 2.2.0 isn't.

By the way, I also see different gems with .so files in ext/ directory working on my machine, like eventmachine. They use a require 'rubyeventmachine' construct which works for loading rubyeventmachine.so from both lib/ and ext/ (and it's located in ext/, but it's located both in ext/ and lib/ when I build the library with rvm).

In short, I think that even if it's a bug in those distributions, it could be a good idea to try to load the .so file from both directories (here it's in ext/lexer/ actually, not just in ext/ like in eventmachine) to avoid the users a bug that's quite hard to debug.

[iliabylich]

well, then we should fix it by adding ext directory to the load path. Want to send a PR? If not I'll do it myself a bit later today

[hmdne]

That won't do it. There's require_relative being used and it's not ext but ext/lexer. I would suggest something like:

begin
  require_relative 'lexer'
rescue LoadError
  require_relative '../ext/lexer/lexer'
end

I can do a pull request.