Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error when building firebase-admin #317

Closed
markozxuu opened this issue Mar 19, 2019 · 12 comments
Closed

error when building firebase-admin #317

markozxuu opened this issue Mar 19, 2019 · 12 comments
Assignees
Milestone

Comments

@markozxuu
Copy link

markozxuu commented Mar 19, 2019

ncc version: 0.16.1

Steps to reproduce

git clone https://github.com/MAPESO/ncc-bug-firebase
cd ncc-bug-firebase
yarn install  # dependency installation
ncc build index.js   # build in single file
node dist/index.js  # note the error output

logs of build

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/write-file-atomic" attempted to require "worker _threads" but could not be resolved, assuming external

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/hash-stream-validation" attempted to require "fast-crc32c" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/retry-request" attempted to require "request" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/write-file-atomic" attempted to require "worker _threads" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/debug/src" attempted to require "supports color" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.

ncc: Module directory "/Users/mark/ncc-bug-firebase/node_modules/bytebuffer/dist" attempted to require "memcpy" but could not be resolved, assuming external.

(node:41366) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. (/Users/mark/ncc-bu
g-firebase/google/api)
    at EventEmitter.emit (events.js:171:17)
    at EventEmitter.<anonymous> (/Users/mark/ncc-bug-firebase/dist/index.js:187929:15)
    at Object.onceWrapper (events.js:273:13)
    at EventEmitter.emit (events.js:182:13)
    at fn (/Users/mark/ncc-bug-firebase/dist/index.js:187764:17)
    at statter (/Users/mark/ncc-bug-firebase/dist/index.js:187802:7)
    at walkdir (/Users/mark/ncc-bug-firebase/dist/index.js:187933:3)
    at Function.module.exports.walkdir.sync (/Users/mark/ncc-bug-firebase/dist/index.js:187679:10)
    at module.exports.COMMON_PROTO_DIRS.map.dir (/Users/mark/ncc-bug-firebase/dist/index.js:51861:17)
    at Array.map (<anonymous>)
(node:41366) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by thro
wing inside of an async function without a catch block, or by rejecting a promise which was not handled with .ca
tch(). (rejection id: 1)
(node:41366) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise r
ejections that are not handled will terminate the Node.js process with a non-zero exit code.

we can extract as something interesting is this phrase "but could not be resolved, assuming external". ncc does not offer much detail but I feel that these modules are causing the problem in some way

this is curious because that route exists but not in that order

(node:41366) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. (/Users/mark/ncc-bug-firebase/google/api)

the correct order would be Users/mark/ncc-bug-firebase/dist/protos/google/api

result of index.js 51861

return walk.sync(dir)

result of index.js 187679

  return walkdir(path,options,eventHandler);

result of index.js 187764

emitter.emit('fail',path,err);

result of index.js 187802

statAction(ex,stat);

result of index.js 187929

emitter.emit('error',path,err);

result of index.js 187933

statter(path,1);
@guybedford
Copy link
Contributor

The underlying error here is in google-gax, in this file - https://unpkg.com/[email protected]/build/src/grpc.js.

The code that is causing the problem is:

// INCLUDE_DIRS is passed to @grpc/proto-loader
const INCLUDE_DIRS = [];
INCLUDE_DIRS.push(googleProtoFilesDir);
// COMMON_PROTO_FILES logic is here for protobufjs loads (see
// GoogleProtoFilesRoot below)
const COMMON_PROTO_DIRS = [
    // This list of directories is defined here:
    // https://github.com/googleapis/googleapis/blob/master/gapic/packaging/common_protos.yaml
    'api',
    path.join('iam', 'v1'),
    path.join('logging', 'type'),
    path.join('monitoring', 'v3'),
    'longrunning',
    'protobuf',
    'rpc',
    'type',
].map(dir => path.join(googleProtoFilesDir, 'google', dir));
INCLUDE_DIRS.push(...COMMON_PROTO_DIRS);
const COMMON_PROTO_FILES = COMMON_PROTO_DIRS
    .map(dir => {
    return walk.sync(dir)
        .filter(f => path.extname(f) === '.proto')
        .map(f => path.normalize(f).substring(googleProtoFilesDir.length + 1));
})
    .reduce((a, c) => a.concat(c), []);

where walk.sync is trying to do a dynamic directory walk on a dynamically known directory.

There is no general analysis fix unfortunately here, so like the express templates case, we need to work out what is a specific fix we can inline for this exact package to make things work out ok.

If I run this on my machine, I get COMMON_PROTO_FILES containing:

[ 'google/api/annotations.proto',
  'google/api/auth.proto',
  'google/api/backend.proto',
  'google/api/billing.proto',
  'google/api/client.proto',
  'google/api/config_change.proto',
  'google/api/consumer.proto',
  'google/api/context.proto',
  'google/api/control.proto',
  'google/api/distribution.proto',
  'google/api/documentation.proto',
  'google/api/endpoint.proto',
  'google/api/experimental/authorization_config.proto',
  'google/api/experimental/experimental.proto',
  'google/api/expr/v1alpha1/cel_service.proto',
  'google/api/expr/v1alpha1/checked.proto',
  'google/api/expr/v1alpha1/conformance_service.proto',
  'google/api/expr/v1alpha1/eval.proto',
  'google/api/expr/v1alpha1/explain.proto',
  'google/api/expr/v1alpha1/syntax.proto',
  'google/api/expr/v1alpha1/value.proto',
  'google/api/expr/v1beta1/decl.proto',
  'google/api/expr/v1beta1/eval.proto',
  'google/api/expr/v1beta1/expr.proto',
  'google/api/expr/v1beta1/source.proto',
  'google/api/expr/v1beta1/value.proto',
  'google/api/field_behavior.proto',
  'google/api/http.proto',
  'google/api/httpbody.proto',
  'google/api/label.proto',
  'google/api/launch_stage.proto',
  'google/api/log.proto',
  'google/api/logging.proto',
  'google/api/metric.proto',
  'google/api/monitored_resource.proto',
  'google/api/monitoring.proto',
  'google/api/quota.proto',
  'google/api/resource.proto',
  'google/api/service.proto',
  'google/api/servicecontrol/v1/check_error.proto',
  'google/api/servicecontrol/v1/distribution.proto',
  'google/api/servicecontrol/v1/log_entry.proto',
  'google/api/servicecontrol/v1/metric_value.proto',
  'google/api/servicecontrol/v1/operation.proto',
  'google/api/servicecontrol/v1/quota_controller.proto',
  'google/api/servicecontrol/v1/service_controller.proto',
  'google/api/servicemanagement/v1/resources.proto',
  'google/api/servicemanagement/v1/servicemanager.proto',
  'google/api/source_info.proto',
  'google/api/system_parameter.proto',
  'google/api/usage.proto',
  'google/iam/v1/iam_policy.proto',
  'google/iam/v1/logging/audit_data.proto',
  'google/iam/v1/policy.proto',
  'google/logging/type/http_request.proto',
  'google/logging/type/log_severity.proto',
  'google/monitoring/v3/alert.proto',
  'google/monitoring/v3/alert_service.proto',
  'google/monitoring/v3/common.proto',
  'google/monitoring/v3/dropped_labels.proto',
  'google/monitoring/v3/group.proto',
  'google/monitoring/v3/group_service.proto',
  'google/monitoring/v3/metric.proto',
  'google/monitoring/v3/metric_service.proto',
  'google/monitoring/v3/mutation_record.proto',
  'google/monitoring/v3/notification.proto',
  'google/monitoring/v3/notification_service.proto',
  'google/monitoring/v3/span_context.proto',
  'google/monitoring/v3/uptime.proto',
  'google/monitoring/v3/uptime_service.proto',
  'google/longrunning/operations.proto',
  'google/protobuf/any.proto',
  'google/protobuf/api.proto',
  'google/protobuf/descriptor.proto',
  'google/protobuf/duration.proto',
  'google/protobuf/empty.proto',
  'google/protobuf/field_mask.proto',
  'google/protobuf/source_context.proto',
  'google/protobuf/struct.proto',
  'google/protobuf/timestamp.proto',
  'google/protobuf/type.proto',
  'google/protobuf/util/json_format.proto',
  'google/protobuf/util/json_format_proto3.proto',
  'google/protobuf/wrappers.proto',
  'google/rpc/code.proto',
  'google/rpc/error_details.proto',
  'google/rpc/status.proto',
  'google/type/calendar_period.proto',
  'google/type/color.proto',
  'google/type/date.proto',
  'google/type/dayofweek.proto',
  'google/type/expr.proto',
  'google/type/fraction.proto',
  'google/type/latlng.proto',
  'google/type/money.proto',
  'google/type/postal_address.proto',
  'google/type/quaternion.proto',
  'google/type/timeofday.proto' ]

Reading https://github.com/googleapis/googleapis/blob/master/gapic/packaging/common_protos.yaml, it seems the list above is actually fixed by package version of the google-proto-files package, so that one fix could simply be to do the following transformation:

  • When processing google-gax/build/src/grpc.js, replace the definition of COMMON_PROTO_FILES to simply be the inlined list above, provided the version of google-gax matches our current major.
  • We then need to make sure we keep the above list in sync with what is defined in google-proto-files. One option might be to actually build this list in the ncc build itself with this package as a dependency to make sure it stays up-to-date.

I just tested out the above code replacement manually but unfortunately that then gives me a new error:

Error: no such type: Document
    at Service.lookupType (/node_modules/protobufjs/src/namespace.js:382:1)
    at Method.resolve (/node_modules/protobufjs/src/method.js:148:1)
    at Service.resolveAll (/node_modules/protobufjs/src/service.js:111:1)
    at getProtobufServiceAttrs (/node_modules/grpc/src/protobuf_js_6_common.js:106:1)
    at loadObject (/node_modules/grpc/src/protobuf_js_6_common.js:134:1)
    at values.forEach.nested (/node_modules/grpc/src/protobuf_js_6_common.js:143:1)
    at Array.forEach (<anonymous>)
    at loadObject (/node_modules/grpc/src/protobuf_js_6_common.js:142:1)
    at values.forEach.nested (/node_modules/grpc/src/protobuf_js_6_common.js:143:1)
    at Array.forEach (<anonymous>)

which unfortunately doesn't give a good stack trace but seems like it would be a dynamic require issue in dynamically requesting a .proto file, since the standard dynamic require analysis issues would apply there.

Unfortunately we still haven't made much progress on dynamic require approaches though, and would need to start with a much simpler case to work up to this sort of a problem.

So for now, unfortunately, solving this firebase-admin case remains elusive.

@styfle styfle added package issue and removed bug Something isn't working labels Mar 22, 2019
@markozxuu
Copy link
Author

markozxuu commented Mar 23, 2019

@guybedford I really appreciate the research you just did, thank you very much. I'll have to resort to mongoDB Atlas as serverless db.

@markozxuu
Copy link
Author

markozxuu commented Mar 23, 2019

@rauchg That now and now-dev are not compatible with firestore will be a problem. Users will not have many options when using DB serverless :|

@guybedford
Copy link
Contributor

I finally managed to find a fix for this in vercel/webpack-asset-relocator-loader#12.

styfle pushed a commit that referenced this issue Mar 25, 2019
This includes the updates for correct parser errors, the express templates support, as well as firebase admin support.

We have tests for all the above in the asset relocator loader itself, although the firebase-admin case is not a full integration test as that would involve storing credentials.

Fixes #317
Fixes #323
@markozxuu
Copy link
Author

@guybedford firebase-admin works perfectly usingncc but using now-dev I get the following error Error: package.json does not exist at /private

this error is related to this issue #306

54338076-7b097700-45f6-11e9-871e-ee668ea7a1e6

@styfle
Copy link
Member

styfle commented Mar 26, 2019

If it works with ncc, then this is not the appropriate place for the issue.

Please discuss with TooTallNate.

@Yuripetusko
Copy link

I am using ncc to build my ts api project which i later upload to AWS serverless lambda. I do use firebase-admin and custom node10 runtime on lambda. After executing my function I get following error:

"errorMessage": "/var/task/lib/src/node/extension_binary/node-v64-darwin-x64-unknown/grpc_node1.node: invalid ELF header",

is it related to ncc? Let me know if I need to create a separate issue

@styfle
Copy link
Member

styfle commented May 6, 2019

@Yuripetusko This error looks like you compiled on Mac and then ran on Linux. Perhaps you should try deploying to Now so that your build environment matches your run-time environment.

@Yuripetusko
Copy link

Thanks @styfle

@diogoviannaaraujo
Copy link

diogoviannaaraujo commented Aug 19, 2019

I still get this error when deploying on now, when using now-cli everything goes ok, but on now i get:

ERROR	Unhandled rejection: { Error: ENOENT: no such file or directory, open 'google/protobuf/api.proto'
    at Object.openSync (fs.js:443:3)
    at Object.readFileSync (fs.js:343:35)
    at fetch (/var/task/page.js:32228:34)
    at Root.load (/var/task/page.js:32262:13)
    at Root.loadSync (/var/task/page.js:32303:17)
    at Object.loadSync (/var/task/page.js:46295:17)
    at Object.8ZNE (/var/task/page.js:34417:37)
    at __webpack_require__ (/var/task/page.js:23:31)
    at Object.<anonymous> (/var/task/page.js:128795:25)
    at Object.b1/w (/var/task/page.js:129054:30)
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: 'google/protobuf/api.proto' }

@styfle
Copy link
Member

styfle commented Aug 19, 2019

Hi @diogoviannaaraujo

The @now/node builder no longer uses ncc.

Can you make sure you using the latest @now/node in your now.json and update the CLI to the latest version by running npm i -g now@latest?

@diogoviannaaraujo
Copy link

diogoviannaaraujo commented Aug 19, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants