Offer unpacked code from package.json

[cwillisf]

Proposed Changes

This change declares ./src/index.js as the "main" script in package.json, and stops building dist/node/* with webpack. This change also adjusts the webpack configuration for compatibility with similarly unpacked audio, render, and storage modules.

Reason for Changes

This sets us up to build scratch-gui in a single step, allowing webpack to better optimize our files and reducing the duplication of code within our output bundles.

Test Coverage

Test coverage has not changed: the tests were already using the unpacked source.

[cwillisf]

I dug a bit further into resolve: { symlinks: false } and it looks like we have a decision to make.

Background info: when Node.js requires a module across a symlink, for example when using npm link, the symlinks get resolved before further processing. That is, if you have /src/scratch-vm and /src/scratch-audio, and /src/scratch-vm/node_modules/scratch-audio is a symbolic link to /src/scratch-audio, then require('scratch-audio') will result in Node loading /src/scratch-audio/src/index.js -- not /src/scratch-vm/node_modules/scratch-audio/src/index.js. For most purposes this has no effect -- it's the same file, after all -- but a few operations are affected by this change. For more info about this, see: nodejs/node#3402

In webpack, this behavior can be configured using the resolve.symlinks setting. The default is equivalent to resolve: { symlinks: true } and acts like Node; setting the value to false makes webpack act more-or-less as if the symlink were a regular directory. Specifically, in the situation described above:

• resolve: { symlinks: true } will result in webpack loading /src/scratch-audio/src/index.js
• resolve: { symlinks: false } will result in webpack loading /src/scratch-vm/node_modules/scratch-audio/src/index.js

This difference matters if, for example, we use require.resolve in the test clause of any webpack rules -- for example, a loader rule specifying to use Babel. Since require.resolve is a Node function it will always resolve symlinks, so the test will only pass if we configure webpack to resolve file names in the same way as Node -- that is, if we set resolve.symlinks to true or leave it default.

Unfortunately, things go wrong when webpack tries to load a plugin or preset. It appears that webpack looks for its plugins and presets only in parent directories of the file currently being processed. That is, if we're currently processing /src/scratch-audio/src/index.js then webpack will look in /src/scratch-audio/node_modules for presets and plugins. Notably, in this situation webpack will not look in /src/scratch-vm/node_modules even if /src/scratch-audio/src/index.js is being loaded by a require in the scratch-vm source (with resolve.symlinks enabled).

Because of this, enabling resolve.symlinks means webpack will fail to find a plugin or preset which the parent project (scratch-vm or scratch-gui) uses unless it is currently installed in every npm linked module used by that parent project.

I scoured the web for solutions more elegant than what I describe below; I didn't find any. Some people recommend adding path.resolve(__dirname, "node_modules") to resolve.modules but that didn't help: webpack doesn't appear to use this setting when searching for plugins and presets.

There are two solutions I've found; they're described below and each is represented as a commit in this PR. Please don't let the order of the commits affect your opinion: the order reflects when I figured out each technique, not necessarily a preference. I do have a preference but I'd like your unbiased input. :)

Option 1: Use resolve: { symlinks: false }

• Implemented in 4f095b7
  • When looking at that commit, disregard the base.module.rules.concat changes on lines 54, 59, 87, and 116: that change was necessary regardless of the value of resolve.symlinks.
• The behavior of webpack's require emulation during build will not match that of require and require.resolve commands executed by Node during webpack configuration.
  • Because of this, we won't be able to use require.resolve in the test clause of any webpack config items, such as loader rules. Instead, use path.resolve, which will normalize the path but will not resolve symlinks.
• Any path-sensitive code in the webpack bundle will behave differently in the bundle as compared to Node.js, though I'm not aware of any such code currently.

Option 2: Use absolute paths to load webpack plugins & presets (let resolve.symlinks default true)

• Implemented in a4a0d60
• The behavior of webpack's require emulation during build will match that of require and require.resolve commands executed by Node during webpack configuration.
• Requires post-processing various paths in the webpack config in order to resolve the paths to their build-time values, since we don't know until build time whether a given module is on the other side of a symlink. Basically this comes down to calling fs.realpathSync. I chose to implement this in a function, though it feels a bit "clever" to me (in a negative way).

Thoughts? Preferences?

[cwillisf]

Closing this in favor of #529 for now. There are a few other changes, like proper version numbers, which should happen before I try this kind of change again.