Rails extremely slow with carmen-rails gem

[ghost]

When I run 'rails r' or 'rails s' on my Windows 7 64bit machine, it takes 3 minutes before something happens. This has only begun recently. 'gem install' and 'bundle install' still run at normal speed.

In order to find out what caused the problem I upgraded to the latest Rails version (3.2.13) and Ruby Installer 1.9.3-p392. That didn't help. Then I got a tip to look at my gemfile. So I started removing gems from my gemfile. Once I commented out gem 'carmen-rails' startup times where back to normal.

This is only recently. It has worked just fine, as I am using carmen-rails in production. I tried going back to an older version with gem 'carmen-rails', '1.0.0.beta2' but that didn't help either.

I also had an issue with "Too many files open", that also pointed to carmen-rails. Any idea what might be going on? Any recent changes that might explain this behavior? Thanks.

[ghost]

Hey Nick,

Sorry to hear. We did recently issue a pull request yesterday but I will do some speed tests tonight. And get back to you. I will get this figured out ASAP.

Thanks for the heads up.

[ghost]

http://cl.ly/image/0U3f0F3j2V34

As you can see it is taking a bit longer to fire. But that shouldn't slow the rails console or server down that much. I am doing it now on a fresh app to test and seeing no lag at all. But I can see from the testing there that the load/boot time is wayy too long. So I will take a look and see if we can speed that up.

Also will add the gem to a bigger project of mine and see if it slows things down.

[ghost]

Hi Dain,

thanks for your help. How did you generate that screenshot? Maybe I can run it on my setup and see what it produces.

[ghost]

Someone created a nice ruby script for benchmarking Gem times, that I rely on for a lot of production stuff as well.

Just cd into the dir of your app, and run this
curl -fsSL https://gist.github.com/raw/5022636/benchmark.rb | ruby

It will give you a nice printout. If you could attach a screenshot here, so we can see what the discrepancies are (if any) between a new rails app and the one you're running.

[ghost]

I made two screenshots from the project one with carmen-rails enabled:

and the other with carmen-rails disabled.

[ghost]

Woah. That's crazy, yeah I've never seen a gem take that long. I will spend some considerable time today trying to replicate that using a few large apps. Can you upload your exact gem file I so I can test it using what you are.

[ghost]

It is indeed crazy. Uploaded the gemfile as a jpg, otherwise github wouldn't accept it. Remove the jpg part and I hope it works. If not, let me know and I'll upload it to my server.

[ghost]

I have also managed to get the "Too many open files" error. Attached a screenshot of the error. Maybe this is related to the slow startup. This error does not occur always.

I also have a carmen.rb file in config/initializers (see attachment). This has nothing to do with the slow startup, but maybe with the too many files open error. When I remove this file, I get errors stating that world.yml can not be found in config/locales/carmen/

If I need to test something or you need further information, please let me know.

[ghost]

I can't replicate this. Very strange. I will keep trying. I have used your exact gemfile and ruby/rails versions. I feel like maybe I have a gem version that is different than yours perhaps and that version discrepancy has code changes that are causing a clash for some reason and causing this.

Give me a few more days to figure this out. Going to dive deep into this over the weekend.

[ghost]

Hi Dain,

thanks for the effort. I have installed ruby-prof in an effort to try to find the problem area. I have uploaded a profile file to http://www.n-consulting.be/profile.html . Too me it appears Module::UnicodeUtils::Impl#read_names is the culprit. But it is the first time I am running this tool, so maybe it gives you some hints as to what may be the problem.

This profile was generated by running - ruby-prof -p graph_html C:\Ruby\Ruby193\bin\rake environment > profile.html

I am now running one for rails s, but it has already been running for 2 hours and still no results. So I do not know if it will ever finish, if it does I will also post the results.

[ghost]

Hi Dain,

I might have found the problem area. I have two projects that use carmen-rails, one small one and one big one. In the big project carmen-rails is slow, in the small project it is still fast.

I checked everything and finally looked in the Gemfile.lock file. In the slow one I have:

carmen (1.0.0)
|---> unicode_utils (> 1.4.0)
carmen-rails (1.0.0)
|---> carmen (> 1.0.0.beta2)
|---> rails

While in the fast one I have

carmen (1.0.0.beta2)
carmen-rails (1.0.0.beta3)
|---> carmen (~> 1.0.0.beta2)
|---> rails

Notice the dependency of carmen 1.0.0 on unicode_utils which also used the most time in the profiler. I think this is causing all the problems. |---> is to show indents, as github removes spaces.

[ghost]

Hey Nick,

Will be meeting with Jim about this today or tomorrow. I think we have a good handle on the problem. Thanks for your input. Expect a fix soon.

[ghost]

Found the cause of the problem. Working on a fix in a few different areas. Should be speedier soon. Will let you know when we get it out (hopefully by this weekend or mid-week next week). Thanks.

[ghost]

Great to hear that you found the cause of the problem. Looking forward to the fix.

[ghost]

Any idea when the fix will be ready? Thanks.

[rreynier]

I am having the same issue. With this gem enabled my rake and rails commands take 10-20 times longer. I should mention I tried this on multiple machines all running windows 7 64 bit with the same result.

[kjeremy]

Any update on this?

[aidanmorgan]

I am having this same issue on Windows8, even running with the gem pointing to github. I've checked my Gemfile.lock and I have no other libraries depending on carmen/carmen-rails.

[nashbridges]

one possible cause is that https://github.com/jim/carmen-rails/blob/master/lib/carmen-rails.rb#L9 adds all carmen locale files to the I18n.load_path. Simple I18n backend (which is the default backend in Rails) eager loads all of them, that is about 400 files to be read from a drive

[gkulasik]

I am also having this issue. Have been hoping for a fix for the last few months. When I use carmen-rails start up time for rails s, rails c, migration generation goes up significantly. Using windows 8, rails 3.2.13, ruby 2. Just wondering if any update will be coming out soon.

[G0dwin]

Also encountering this issue, running Win8.1, Rails 4.0, Ruby 2.0, carmen-rails 1.01.

I'm not sure that this issue is caused by I18n, I'm using i18n-active_record, I have an SSD, and I've also tried removing all yml translations from the gem dir and it still takes roughly 1m to get a response after executing "rails server" (less than 5s without carmen-rails).

I'd happily help by providing more details or digging into the code to try to find the issue but this really is a major issue, all rails, rake, bundle, etc commands take a minute to run making development (and especially testing) a very painful process.

[G0dwin]

I have determined the bulk of the problem is actually in the unicode_utils gem which is being referenced by the carmen gem (see lang/unicode_utils#6). My suggestion for the authors would be to leave out the unicode_utils gem altogether since only one method actually gets used from the gem: char_name.

[eikes]

Carmen uses unicode_utils only once: UnicodeUtils.casefold in lib/carmen/querying but I think that can be replaced with something like this

name.mb_chars.downcase.normalize === region.name.mb_chars.downcase.normalize

[eikes]

I just wanted to follow up, that I was successful in speeding up carmen on windows by replacing unicode_utils with activesupport.

The fork is here: https://github.com/eikes/carmen/tree/active_support_not_unicode_utils

[ahukkanen]

I'd really love to see @eikes suggestion or something similar that improves the performance on Windows machines integrated into this gem. It's really taking forever to even load the gem.