Illegal instruction

[JosephBotros]

Hi,
I am developing for Nodejs, on local machine everything is running smoothly. But when I pushed the project on a VPS and try to run the project I get "Illegal instruction" and that's it. I traced the problem and found that this error occurs when I write either
const faceapi = require('@vladmandic/face-api')
or
const tensor = require('@tensorflow/tfjs-node')
I figured that tensorflow is not compatible with the system, although the project runs without issues on Heroku server, but the client won't be using Heroku.
After much struggling I managed to install tensorflow on the server but I still get the error. I even tried to build tensorflow on the server but the machine runs out of memory before the build is done. I'm lost and any help is appreciated.
My VPS environment:
Ubuntu Linux 20.04
QEMU Virtual CPU version 2.5+, 2 cores
2 GB RAM
Kernel and CPU: Linux 5.4.0-81-generic on x86_64

[vladmandic]

• can you post actual backtrace that shows "illegal instruction"?
• what is the version of nodejs and tfjs-node package?
• can you double-check there are no warning messages during tfjs-node package installation?
  installation process builds napi binary bindings towards tensorflow binaries
  (btw, there is no need to install tensorflow as tfjs-node comes with tensorflow binaries as part of the package)

but...having said all that, i believe the issue is that tensorflow that comes with tfjs-node is built using advanced CPU instructions that are not supported by QEMU (most likely culprits are AVX2 and FMA).

and if that's the case, the only solution is to custom build tfjs-node (and included tensorflow) from sources while changing configuration not to use those instructions - and that is pretty painful thing to do.

btw, you don't need to build on that VPS server, you can build locally - important is to have correct configuration.
and regarding OOM, it can be tuned how much memory bazel tool uses for parallel builds (by default it's extremely hungry)

before going down that route, you can try old tfjs-node v1.2
(i believe that is the latest version that was built without using AVX instruction set)

[JosephBotros]

• can you post actual backtrace that shows "illegal instruction"?
  There is no error or anything else, just Illegal instruction. I put a console.log() before every require and the one before tensor or faceapi is the last console that is printed, right after that Illegal instruction shows up. I made an empty script that has only
  var faceapi = require('@vladmandic/face-api');' and showed Illegal instruction`

• what is the version of nodejs and tfjs-node package?
  Node.js: v14.17.6.
  @tensorflow/tfjs-node: ^3.8.0

• can you double-check there are no warning messages during tfjs-node package installation?
  Here's the log:

npm install @tensorflow/tfjs-node
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142

added 634 packages, and audited 635 packages in 60s

34 packages are looking for funding
  run `npm fund` for details

6 vulnerabilities (5 low, 1 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.

[JosephBotros]

I will try the old version you mentioned now

[JosephBotros]

Still Illegal instrcution

[JosephBotros]

I thought I need to build tensor on the VPS, I'll try building it locally

[vladmandic]

I thought I need to build tensor on the VPS, I'll try building it locally

no, its cross-platform compile, you can even build for ARM while on x86 - it all depends on configuration

[vladmandic]

just occured to me, why not use wasm backend in nodejs? that is actually possible with nodejs v14 and higher
it's about 30% slower than native tfjs-node, but still far better than using cpu backend

only gotcha is that wasm binaries must be loaded from https target, it doesn't support local fs i/o handler
but you can use a cdn for that
(i have enhancement request open with tfjs team for that)
btw, models can be locally hosted, it's only wasm loader that is not linked vs https handler

while this is not ideal it may not be a bad start