Overall reactivity, database updating, and usability issues. Possible npm issues

[JoeyLyman]

Expected behavior

When doing a fresh reaction init, I don't expect 'npm list' to show a whole bunch of missing packages. Also expect general usability (though, I am not sure if I have a local problem, and would be stoked if you could spot anything that I may be missing in the screencast video I made).

Actual behavior

Video of a fresh install, and attempted usage of RC: https://www.youtube.com/watch?v=NEX9f72bJqE&t=53s

When I run reaction, there is a large lack of reactivity; sometimes the database updates, and sometimes it doesn't. This is for admin settings, new products, and new accounts. Also, sometimes data from one product will be superimposed on another. Also, sometimes when I am not logged in (for example in an incognito window), there won't be a sign in button at all on the navbar. Also, when not logged in, sometimes when I click on a product, it will take me to a page that says I don't have permission to view that product. Also, when clicking on the marketplace dropdown in the menu, sometimes the options show up below it, and sometimes they don't (i.e. enable users to opt to be sellers). Also, images don't always upload - sometimes new products that I tried to put images on will just show up to be white on the product grid, and sometimes they will have the ReactionCommerce watermark; sometimes when images show up on the product grid, they still won't show up on the product detail page.

Steps to reproduce the behavior

Please see video. I did a fresh install and did everything in the Requirements and Installation docs. Then ran reaction, and tried to do general things, without much luck.

Thanks lots,

Joey

Versions

Node: 8.7.0
NPM: 5.4.2
Meteor Node: 4.8.4
Meteor NPM: 4.6.1
Reaction CLI: 0.18.1
Reaction: 1.5.1
Reaction branch: master

[brent-hoover]

I get this if I try to view the video

[JoeyLyman]

sorry, it defaulted to private I just set it to public
https://youtu.be/NEX9f72bJqE

[spencern]

To give some more context:

Here's a screenshot of the machine specs from this video:

@JoeyLyman will you run ulimit -a on your machine and post the output.

A few notes:

bcrypt

the npm install --save bcrypt warning is due to us intentionally leaving that out. It causes certain builds to break, that warning is not coming directly from Reaction, but is passed through.
You can read up on the bcrypt issues for more info: #2333, #1946 and the PR where we removed it #2571

We're hoping to be able to reintroduce node bcrypt once it's compatible with meteor builds again.

In the meantime, we highly recommend not installing the node version of bcrypt and ignoring the warning. The mention of 3x slower is only when running bcrypt, this happens only when a password needs to be checked, (login, account creation, etc) and so it's not making the whole app run 3x slower. Running the pure javascript version of bcrypt is not the reason for the slowness you are experiencing.

Testing

• The current version of reaction should have 299 tests and they should all be passing. I see in your reaction -v output that you're running the latest version 1.5.1 but not sure why we have different numbers of tests, any ideas on that? I've just run reaction init and reaction test to create a fresh install and am seeing 299 Passing when running tests.
• Watching the video, it's clear that the tests are taking longer than they should.
• I clocked a full test from a freshly reaction inited install and completed tests in 6 minutes and 0 seconds on my machine just now (with two other servers running simultaneously on different ports). I see that on your machine it took ~ 12 minutes and 14 seconds, a huge disparity, I don't have a good explanation for that right now, but it seems also that we're running different versions somehow.
• When running tests after the first run, I was seeing times below 4 minutes for the whole test suite to load and run.

Running the app

• I'm seeing times of ~ 1 minute 45 seconds for the initial startup of the app the first time (after running tests, like in your video). This is from the time I hit enter to run reaction until I see `=> App running at: http://localhost:3000/
• In the video, it's taking ~ 3 minutes 45 seconds to get up and running. This is another large disparity and I do question if it's simply my computer being faster, an issue with memory, or an issue with the way Reaction is running on your machine.

I'd be curious if you look at your memory usage and swap usage while Reaction is running to see if perhaps it's just hammering your swap.

App UI Responsiveness

• We've seen issues with people being logged out, and were tracking a meteor bug that led to that, but it appeared to be fixed in the 1.5.2.2 release of Meteor.
• It seems that we need to disable buttons (such as the create product button) once they've been hit until the action they trigger is finished. I think some of the duplicate empty products you're seeing may be the result of your system running sluggishly.
• I've not been able to reproduce the issues you're able to show with the product editing panel where you tab through product fields and the values you enter disappear after moving on to the next input.

When running Reaction in development mode, every file is watched, and all javascript is loaded as separate files, this can be very taxing on a computer and I can imagine that a computer that has to swap memory into the hard drive could produce a lot of these results.

My primary dev machine is about 3 years newer and has 16GB memory, but I've got a 2012 machine that also doesn't exhibit this level of slowness. 8GB of memory should be plenty for development, but my older machine also runs 16GB, so I'm unable to test that specifically.

If anyone else is experiencing these issues, I think we may need some more data points to pin down exactly what the problems are.

[JoeyLyman]

Here's my output for 'ulimit -a'

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited

And cool, thanks for the the explanation for bcrypt. The requirements doc at https://docs.reactioncommerce.com/reaction-docs/master/requirements does still seem to recommend installing the node version: "Installing a local native compiled version of bcrypt is optional, but may yield some performance benefits and is recommended for production environments."

Testing

Hmm, after uninstalling node bcrypt, 299 tests ran, 298 passed and 1 failed (before first reaction launch), then after first launch all 299 passed. Still a couple errors though, not sure if it's just in the lack of configuration of my version? Here's the log:

Also, yes I have an SSD in here that is only 25% full.

Here's an activity snapshot of when I am trying to log in (after being logged out when I refreshed the page after a freeze), where it is stuck again trying to log me in. The activity was pretty similar to this on startup, and when things were going smoothly during the first few interactions (adding a product and a title to the product).

Also, is it normal to have all those npm ERR! missing: ... packages?

Also, when I did reaction pull tonight, it says it is up to date, but reaction -v still says v1.5.1 though when I run reaction it suggests I update to 1.5.2.

Thanks again for the help and the long answer, let me know what else I can show you i.e. video of usage, more in depth / consolidated with some of the potential bugs, etc.

[spencern]

@JoeyLyman Thanks for spending time pulling this together.

Regarding the reaction -v issue, we had an issue where we released 1.5.2 yesterday but the package.json version did not get bumped. This will be corrected when we release 1.5.3.

Sounds like you've got the correct version and the correct amount of tests. Not sure what I was seeing in the last video.

I don't see any issues with your ulimit configuration.

One of the next focus points for our core team will be app performance, I'd expect that we'll address many of the underlying issues that may be causing the sluggishness you're experiencing as well.

[JoeyLyman]

Cool, so no issue with npm thinking there are all these missing dependencies, or with the errors in the test?

[spencern]

We've got one or two tests that seem to fail occasionally, so it seems like one of those got you.
The missing dependencies shouldn't be a problem, and I think those are just warnings, as long as the meteor npm install went through correctly, I wouldn't worry about that.

[cantuket]

Hey @spencern,

I'm having some really slow dev build times, which sound out of line from everyone else's experiences. I don't mind the initial build time, but the restart time is definitely prohibitive to getting much done...

Initial Server Start: ($ reaction)
5min 36sec

Restart on file save: ( in imports > plugins > custom > some-JSX-file.js)
5min 10sec

I'm running on an embarrassingly old system Mac Book Pro (Mid 2010), but have 16GB of RAM and have never has an issue with any other modern Node environments. I'm running the latest release (Reaction 1.6.3) and below are the complete specs on my machine....

MacBook Pro (13-inch, Mid 2010)
OS: High Sierra 10.13.2
Processor: 2.4 GHz Intel Core 2 Duo
Memory: 16 GB 1067 MHz DDR3
Graphics: NVIDIA GeForce 320M 256 MB

Available RAM (@ initialization): 10.4 GB
Available Storage (@ initialization): 62.9 GB (SSD)

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1418
virtual memory (kbytes, -v) unlimited

I'm getting a new machine soon, but would love to be able to work on a project in the mean time. If you're aware of a previous version I can roll back to that may help, or if I'm missing something from the development process then I'd appreciate any help.

Thanks!

[jshimko]

@cantuket Well, I suspect it's because of where you're putting the file. You can't just throw custom files in /imports/plugins/custom/. The plugin loader looks for very specific index files to import custom code within each plugin (see any existing plugin for an example). You first need to create a plugin folder for your custom code like...

/imports/plugins/custom/my-plugin/

Then within that add one or both of...

./your-plugin/client/index.js
./your-plugin/server/index.js

You then use those index files to import or export any of your custom plugin code.

You could also automatically generate the above by running...

reaction plugins create --name my-plugin

That generates the following...

/imports/plugins/custom/my-plugin/
/imports/plugins/custom/my-plugin/package.json
/imports/plugins/custom/my-plugin/register.js
/imports/plugins/custom/my-plugin/client/index.js
/imports/plugins/custom/my-plugin/server/index.js

The four generated files above are the only files reaction-cli looks for within a plugin directory. Whatever customizations you want to load then need to be imported from one of those two index files. Without doing that, reaction-cli and Meteor have no idea your custom code exists and it will not be added to the app or trigger a reload when the file changes.

See the docs for more detail on customizing Reaction.

https://docs.reactioncommerce.com

[cantuket]

Thanks for the detailed response @jshimko,

I should have been more detailed with my question because I was actually working on imported files in the client directory of a registered plugin, as you specified, just left that critical piece of info out. Sorry!

I still appreciate the explanation, which gives me a much better understanding of how the compilation is working. Unfortunately the above mentioned build times still apply, but waiting for a new machine to be delivered in the next few day so I assume that will improve my situation significantly.

[spencern]

Closing this issue for now as we don't have solid reproduction steps and don't have anything that we can work on specifically on this issue. We're starting a big performance push this week, if you're having similar issues in a couple weeks once we've made some performance improvements or if you have specific reproduction steps please open another issue.