[Bug] SyntaxError when running a "bin" file that isn't authored in JS

[cameronhunter]

• I'd be willing to implement a fix

Describe the bug

Yarn 2.0 fails with a SyntaxError when running a bin file that isn't authored in JS. For example, the bin file may be an executable file with a shebang #!/usr/bin/env sh.

Example bin file:

#!/usr/bin/env sh
echo 'Hello world!'

To Reproduce

See sample package (@cameronhunter/berry-bin-bug) that defines a bin that is an executable file with a shebang.

const { promises: { writeFile, chmod } } = require('fs');

await writeFile('./hello-world', '#!/usr/bin/env sh\necho "Hello world!"');
await chmod('./hello-world', 0o765);

await packageJsonAndInstall({
  bin: { 'hello-world': './hello-world' }
});

await expect(yarn('hello-world')).resolves.toContain('Hello world!');

Example of failure:

$ yarn berry-bin-bug
/Users/chunter/workspace/github/cameronhunter/berry-bin-bug/bin/berry-bin-bug:3
echo "Hello world!"
     ^^^^^^^^^^^^^^

SyntaxError: Unexpected string
    at Module._compile (internal/modules/cjs/loader.js:895:18)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
    at Module.load (internal/modules/cjs/loader.js:815:32)
    at Function.module_1.Module._load (/Users/chunter/workspace/github/cameronhunter/berry-bin-bug/.pnp.js:13519:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
    at internal/main/run_main_module.js:17:11

Environment if relevant (please complete the following information):

• OS: MacOS Mojave (10.14.5)
• Node version: 12.14.0
• Yarn version: 2.0.0-rc.28

[yarnbot]

This issue reproduces on master:

Error: expect(received).resolves.toContain()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js hello-world
/tmp/tmp-2722xAfw0xGBGh/hello-world:2
echo "Hello world!"
     ^^^^^^^^^^^^^^

SyntaxError: Unexpected string
    at Module._compile (internal/modules/cjs/loader.js:895:18)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
    at Module.load (internal/modules/cjs/loader.js:815:32)
    at Function.module_1.Module._load (/tmp/tmp-2722xAfw0xGBGh/.pnp.js:13576:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
    at internal/main/run_main_module.js:17:11

]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-2.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:11:7)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-2.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-2.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:16)
    at async Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-2.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-2.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-2.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-2.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-2.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

[arcanis]

That would be a wontfix as third-party binaries aren't portable and cannot be run from zip archives. At worst, you can make a JS script that uses execFileSync.

[cameronhunter]

Thanks for your response, @arcanis. Cross-platform isn't a goal for many of our (internal) tools so it's a shame that we have to spin up node to execute a shebang executable. I can certainly work around this using your suggestion though.

Perhaps this ticket could be changed to focus on improving the error message so that other users don't fall into the same confusion.

Side-question: Is cross-platform portability a goal of berry or is it more the implementation detail of not being able to run them from zip archives?

[arcanis]

Side-question: Is cross-platform portability a goal of berry or is it more the implementation detail of not being able to run them from zip archives?

A bit of both - I'd say the main reason is that we'd like to encourage applications to avoid using shell scripts as it puts the burden on users (installs take more time, and zero installs don't work). So if you want to absolutely do it, better do it explicitly.

[ti1024]

Some NPM packages (e.g., https://www.npmjs.com/package/elm) are meant to provide a native binary executable for various platforms and expose it so that it can be found via PATH when run from a script specified in package.json (often indirectly, e.g., via a specific loader for webpack). These packages cannot be used with Yarn Berry, even with yarn unplug (or even without Plug'n'Play altogether).

Can Yarn provide a configuration option to skip the shell script shim for a particular executable?

[ysfaran]

Is there away to run TypeScript bin files?

I used

#!npx esr --cache

as shebang with

"name": "@comp/scripts",
"bin": {
    "my-script": "./src/cli/my-script.ts"
}

in the package.json. With yarn 1.0 workspace setup I could then simply execute yarn my-script and it worked just fine. I wanted to migrate to yarn 3.0.2, but there it's not working anymore..

Surely I could compile the script to JS first, but it was a quite nice developer experience to just execute the TS file without having a build step always in between..

[milesj]

I agree with @ti1024. More and more packages are providing native binaries (Rust, Go), and not having these work via Yarn is a huge problem IMO.

What if yarn inspected the os, cpus, etc fields in package.json and inferred compatibility through that? This seems to be top of mind since Yarn 3.2 adds support for libc field. https://dev.to/arcanis/yarn-32-libc-yarn-explain-next-major--o22

[devongovett]

btw, this worked in yarn v1 too

[rtrembecky]

Hi, I encountered a similar problem now. I installed the ngrok npm package via yarn, and while yarn ngrok worked in previous versions, it crashes with PnP:

richard@MacBook-Pro coffeebot % yarn ngrok 
/Users/richard/vl/bots/coffeebot/.yarn/unplugged/ngrok-npm-4.3.1-f30055099e/node_modules/ngrok/bin/ngrok:1
����


SyntaxError: Invalid or unexpected token
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.external_module_.Module._load (/Users/richard/vl/bots/coffeebot/.pnp.cjs:13342:14)
    at Function.moduleExports.runMain (/Users/richard/vl/bots/coffeebot/.pnp.cjs:13543:31)
    at node:internal/main/run_main_module:17:47

Node.js v17.4.0

I also tried yarn run ngrok or yarn exec ngrok.

However, I found a workaround here: evanw/esbuild#237 (comment)

Running $(yarn bin ngrok) works. So I created a script in package.json:

  "ngrok": "$(yarn bin ngrok)",

Now the yarn ngrok works correctly 🙂 Hope it helps somebody.

[Guara92]

I'm facing the same issue with some elm tools (elm-test & elm-review), the proposed workaround seems doesn't work in this case

"elm:test": "$(yarn bin elm-test)",

node@project:/code$ yarn elm:test
Internal Error: spawn ENOTDIR
    at ChildProcess.spawn (node:internal/child_process:413:11)
    at Object.spawn (node:child_process:700:9)
    at fK (/home/node/.node/corepack/yarn/3.2.1/yarn.js:4:7429)
    at Ud.implementation (/home/node/.node/corepack/yarn/3.2.1/yarn.js:397:18472)
    at Ud.exec (/home/node/.node/corepack/yarn/3.2.1/yarn.js:400:1582)
    at Ud.run (/home/node/.node/corepack/yarn/3.2.1/yarn.js:400:1753)
    at C_ (/home/node/.node/corepack/yarn/3.2.1/yarn.js:406:6555)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async UDe (/home/node/.node/corepack/yarn/3.2.1/yarn.js:408:16)
    at async o (/home/node/.node/corepack/yarn/3.2.1/yarn.js:408:146)

any news on this?

[larixer]

In my opinion native binaries should be supported, at least on unplugged packages. Someone should step in, implement and maintain this feature - yes, of course. At least it should not be viewed as wont fix.

[AndreKR]

The package getcomposer seems to be affected by this as well?

Its package.json contains:

"bin": {
  "composer": "./composer.phar"
}

composer.phar starts with:

#!/usr/bin/env php
<?php

It seems that the shebang line is actually parsed when running yarn install with nodeLinker: node-modules? Because node_modules/.bin/composer.cmd looks like this:

@IF EXIST "%~dp0\php.exe" (
  "%~dp0\php.exe"  "%~dp0\..\getcomposer\composer.phar" %*
) ELSE (
  calc
  @SETLOCAL
  @SET PATHEXT=%PATHEXT:;.JS;=;%
  php  "%~dp0\..\getcomposer\composer.phar" %*
)

But when I define a script in my project like this:

"scripts": {
  "foo": "composer --version"
}

node_modules/.bin/composer.cmd is not even executed, instead it tries to execute node composer.phar which of course fails.

[AndreKR]

I found a workaround though. If I define my script like this, it will execute correctly:

"scripts": {
  "foo": "node_modules/.bin/composer --version"
}

[arcanis]

Hello! This thread has been discussing two separate issues.

• bin entries should be able to point to native binaries; we finally fixed this today in:
  Adds support for running native binaries without JS intermediaries #5508 🎉

• bin entries should be able to point to non-JS shebang scripts; this is still not possible at the moment.

It's a little difficult to follow the thread with those two separate issues kept together, so I think it's best to keep it archived. However, since the second use case is not resolved yet (a bin entry pointing to a non-JS shebang script), if you would still benefit from it, please feel free to open a new thread and reference this one.

Another reason why I'd like another thread to be opened is to assess how important is this use case in practice. While supporting native binaries were clearly high value (I'm aware of a couple of packages who had to make workarounds for Yarn), I'm not quite convinced the same is true for non-JS shebang scripts. We had some internal debates, and settling them would benefit from having clear example of semi-popular packages using such patterns - so far, we don't have any in mind, hence why we'd need your help.

[dec0dOS]

Hello! Trunk is affected by this as well.

$(yarn bin trunk) works fine on macOS. Is that an acceptable workaround?
Are there any pitfalls?
Thanks!

[evisong]

Hi, as indicated in mklement0/ttab#52, ttab package is affected as well, as it's based on a #!/usr/bin/env bash shebang (https://github.com/mklement0/ttab/blob/master/bin/ttab#L1-L5).

@rtrembecky's workaround $(yarn bin ttab) works well on my Mac, thanks!

FYI, I even run yarn 4 workspaces command for my mono-repo like this:

TTAB=$(yarn bin ttab) yarn workspaces foreach -A exec '$TTAB -G exec yarn start'