assets:precompile dosnt create packs dir - shakapacker 7.0.0.rc.0

[lpincu]

Hello
i working on replacing Webpacker with shakapacker
Ive started with Webpacker 5.2.2 and replaced it with shakapacker using this guide

Webpacker 5.2.2 -> shakapacker v6.0.0.rc.6 -> shakapacker 6.5.6 -> shakapacker 6.6.0 -> shakapacker 7.0.0.rc.0

The reason im upgrading from V6 to V7 is because of this issue

V6 worked great for me (in dev env) and all the react pages works fine and render without any errors

After upgrading to V7RC assets:precompile is not compiling anything, i didnt move any of my component's entry points.
all of the entry points are located in frontend/app/javascript/packs/

while trying to reach to my app im getting the following error

Shakapacker can't find application.js in /MY_APP_PATH/public/packs/manifest.json. Possible causes:
1. You forgot to install node packages (try `yarn install`) or are running an incompatible version of Node
2. Your app has code with a non-standard extension (like a `.jsx` file) but the extension is not in the `extensions` config in `config/shakapacker.yml`
3. You have set compile: false (see `config/shakapacker.yml`) for this environment
   (unless you are using the `bin/shakapacker -w` or the `bin/shakapacker-dev-server`, in which case maybe you aren't running the dev server in the background?)
4. webpack has not yet FINISHED running to reflect updates.
5. You have misconfigured Shakapacker's `config/shakapacker.yml` file.
6. Your webpack configuration is not creating a manifest.

Your manifest contains:
{
}

when trying to run RAILS_ENV=dev bundle exec rake assets:precompile nothing is happening and packs dir is not exists anywhere
(we are using RAILS_ENV=dev and not development didnt have any issues with it in V6)
cd public/packs -bash: cd: public/packs: No such file or directory

RAILS_ENV=dev bundle exec rake assets:precompile
DEPRECATION WARNING: Initialization autoloaded the constants ApplicationRecord, Provider, and UtcArgumentsSerializer.

Being able to do this is deprecated. Autoloading during initialization is going
to be an error condition in future versions of Rails.

Reloading does not reboot the application, and therefore code executed during
initialization does not run again. So, if you reload ApplicationRecord, for example,
the expected changes won't be reflected in that stale Class object.

`config.autoloader` is set to `classic`. These autoloaded constants would have been unloaded if `config.autoloader` had been set to `:zeitwerk`.

Please, check the "Autoloading and Reloading Constants" guide for solutions.
 (called from <top (required)> at MY_PATH/config/environment.rb:5)
yarn install v1.22.19
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.93s.

(Note: the warnings are not new and we have them in webpacker 5.2.2 too)

when im using ./bin/shakapacker its seems like its works fine and packs dir along with all the expected files are created
and the app working perfectly fine

there are any breaking changes related to assets:precompile in this version?

frontend/config/shakapacker.yml

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript/packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/webpacker
  shakapacker_compile_output: true

  # You can have a subdirectory of the source_path, like 'packs' (recommended).
  # Alternatively, you can use '/' to use the whole source_path directory.
  source_entry_path: /

  # Additional paths webpack should look up modules
  # ['app/assets', 'engine/foo/app/assets']
  additional_paths: ['app/javascript']

  # If nested_entries is true, then we'll pick up subdirectories within the source_entry_path.
  # You cannot set this option to true if you set source_entry_path to '/'
  nested_entries: false

  #  While using a File-System-based automated bundle generation feature, miscellaneous warnings suggesting css order
  #  conflicts may arise due to the mini-css-extract-plugin. For projects where css ordering has been mitigated through
  #  consistent use of scoping or naming conventions, the css order warnings can be disabled by setting
  #  css_extract_ignore_order_warnings to true
  # css_extract_ignore_order_warnings: true

  # See https://github.com/shakacode/shakapacker#deployment
  shakapacker_precompile: true

  # Location for manifest.json, defaults to {public_output_path}/manifest.json if unset
  manifest_path: public/packs/manifest.json

  # Select loader to use, available options are 'babel' (default), 'swc' or 'esbuild'
  webpack_loader: 'babel'

  # Set to true to enable check for matching versions of shakapacker gem and NPM package - will raise an error if there is a mismatch or wildcard versioning is used
  ensure_consistent_versioning: false

  # Select whether the compiler will use SHA digest ('digest' option) or most most recent modified timestamp ('mtime') to determine freshness
  compiler_strategy: digest

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: false

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .gif
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .jsx
    - .mjs
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

e2e:
  <<: *default
  compile: true

dev:
  <<: *default
  compile: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    # Hot Module Replacement updates modules while the application is running without a full reload
    hmr: false
    # Defaults to the inverse of hmr. Uncomment to manually set this.
    # live_reload: true
    client:
      # Should we show a full-screen overlay in the browser when there are compiler errors or warnings?
      overlay: true
      # May also be a string
      # webSocketURL:
      #  hostname: "0.0.0.0"
      #  pathname: "/ws"
      #  port: 8080
    # Should we use gzip compression?
    compress: true
    # Note that apps that do not check the host are vulnerable to DNS rebinding attacks
    allowed_hosts: "all"
    pretty: true
    headers:
      'Access-Control-Allow-Origin': '*'
    static:
      watch:
        ignored: '**/node_modules/**'

test:
  <<: *default
  compile: true

  # Compile test packs to a separate directory
  public_output_path: packs-test

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Extract and emit a css file
  extract_css: true

  # Cache manifest.json for performance
  cache_manifest: true

frontend/config/webpack/webpack.config.js

const { env, generateWebpackConfig, merge } = require('shakapacker')
const webpackConfig = generateWebpackConfig()
const { existsSync } = require('fs')
const { resolve } = require('path')

const envSpecificConfig = () => {
    const path = resolve(__dirname, `${env.nodeEnv}.js`)
    console.log(`path ${path}`)
    if (existsSync(path)) {
        console.log(`Loading ENV specific webpack configuration file ${path}`)
        return require(path)
    } else {
        // Probably an error if the file for the NODE_ENV does not exist
        throw new Error(`Got Error with NODE_ENV = ${env.nodeEnv}`);
    }
}

const options = {
    resolve: {
        extensions: ['.css', '.js', '.jsx'],
        fallback: {
            "os": require.resolve("os-browserify/browser"),
            "path": require.resolve("path-browserify"),
            "fs": false
        }
    }
}

module.exports = merge({}, webpackConfig, options, envSpecificConfig())

frontend/config/webpack/development.js

process.env.NODE_ENV = process.env.NODE_ENV || 'development'

const webpackConfig = require('./webpack.config')

module.exports = webpackConfig

Expected behavior:

assets:precompile should create packs dir along with manifest.json and other dir and files
important note i put console.log in webpack.config.js and the log is printed while im using ./bin/shakapacker and not when using assets:precompile

Actual behavior:

nothing happen
no errors in console

Setup environment:

Rails 6.0.6.1
ruby 2.6.6p146
Node v14.17.1

package.json
"shakapacker": "7.0.0-rc.0",
Gemfile
gem 'shakapacker', '7.0.0.rc.0'

[justin808]

@lpincu webpack has an error and thus there's no manifest.json. Run /bin/shakapacker and post the output. You can wrap in <details> tags so that the area expands if long.

[lpincu]

@justin808

./bin/shakapacker
Warning: 'development' environment not found in the configuration. Using 'production' configuration as a fallback.
path APP_PATH/config/webpack/development.js
Loading ENV specific webpack configuration file 
APP_PATH/config/webpack/development.js
assets by status 162 KiB [cached] 1 asset
assets by path static/ 1.25 MiB
  assets by path static/app/assets/images/ 136 KiB 53 assets
  assets by path static/node_modules/ 1.11 MiB
    assets by path static/node_modules/font-awesome/fonts/ 929 KiB 5 assets
    assets by path static/node_modules/bootstrap/dist/fonts/ 211 KiB 5 assets
assets by path js/*.js 9.08 MiB
  assets by chunk 8.21 MiB (id hint: vendors)
    asset js/vendors-node_modules_mui_icons-material_esm_Visibility_js-node_modules_mui_icons-material_esm-ea85f7.js 1.8 MiB [emitted] (id hint: vendors) 1 related asset
    asset js/vendors-node_modules-common-ui_dist_index_modern_js.js 1.32 MiB [emitted] (id hint: vendors) 1 related asset
    asset js/vendors-node_modules-data-grid_lib_index_js.js 1.2 MiB [emitted] (id hint: vendors) 1 related asset
    + 15 assets
  + 16 assets
asset css/vendors-node_modules-common-ui_dist_index_modern_js.css 150 KiB [emitted] (id hint: vendors) 1 related asset
asset manifest.json 30.7 KiB [emitted]
orphan modules 9.02 MiB (javascript) 1.27 MiB (asset) 1.08 KiB (runtime) [orphan] 11829 modules
runtime modules 4.98 KiB 16 modules
cacheable modules 7.26 MiB (javascript) 149 KiB (css/mini-extract) 136 KiB (asset)
  modules by path ./node_modules/ 6.8 MiB (javascript) 149 KiB (css/mini-extract)
    javascript modules 6.8 MiB 902 modules
    css modules 149 KiB
      css ./node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[2].use[1]!./node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[2].use[2]!./node_modules/bootstrap/dist/css/bootstrap.min.css 113 KiB [built] [code generated]
      css ./node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[2].use[1]!./node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[2].use[2]!./node_modules/font-awesome/css/font-awesome.css 36.4 KiB [built] [code generated]
    ./node_modules/axios/package.json 1.92 KiB [built] [code generated]
  modules by path ./app/ 471 KiB (javascript) 136 KiB (asset)
    javascript modules 469 KiB 182 modules
    asset modules 2.17 KiB (javascript) 136 KiB (asset) 53 modules
  ./node.js (ignored) 15 bytes [built] [code generated]
webpack 5.80.0 compiled successfully in 24208 ms

[ahangarha]

@justin808
I confirm we have an issue with invoking shakapacker:compile when assets:precompile is invoked. This might cause this issue.

[ahangarha]

@lpincu
Are you sure you have development entry in your config/shakapacker.yml or config/webpacker.yml?

[lpincu]

@ahangarha
We are using dev as our RAILS_ENV and not development
i any case i also tried to add development to my config/shakapacker.yml and use RAILS_ENV=development
both ways (dev\ development) didnt work for me

RAILS_ENV=dev bundle exec assets:precompile worked fine in shakapacker 6.6.0

[ahangarha]

@lpincu
May you please use fix-check-binstubs-rake-task branch for Shakapacker and verify resolution and share the result?

In case you need information. please update your Gemfile with the following line:

gem "shakapacker", github: "shakacode/shakapacker", branch: "fix-check-binstubs-rake-task"

[ahangarha]

We are using dev as our RAILS_ENV and not development
i any case i also tried to add development to my config/shakapacker.yml and use RAILS_ENV=development
both ways (dev\ development) didnt work for me

It would be hard for me to be able to reproduce this issue. May you please create a simple repo demonstrating this issue and share it with us so we can work on the issue?

[lpincu]

@lpincu May you please use fix-check-binstubs-rake-task branch for Shakapacker and verify resolution and share the result?

In case you need information. please update your Gemfile with the following line:

gem "shakapacker", github: "shakacode/shakapacker", branch: "fix-check-binstubs-rake-task"

@ahangarha
Its seems like its working now!
Thanks!!

[ahangarha]

Thanks for the update
And thanks for testing this RC release. Happy to get it fixed before the release.

Is the whole issue resolved?

[benj3]

Any view on getting this in to a rc?

[ahangarha]

We might make the final release. Meanwhile, use gem "shakapacker", github: "shakacode/shakapacker" to get the latest changes from master branch to test.

[benj3]

I can confirm using the latest changes on the master branch fixes this issue for me.

[ahangarha]

@benj3 Thanks for testing and confirming the solution.