Package task aws lambda not able to find module and package name month is wrong

[tamoanxx]

Aws is unable to locate node modules but when I zip it up manually it is able to locate them.

Also the package name date is one month behind.

[Tim-B]

Hi tamtamoanxx,
Have you put your dependencies in the bundledDependencies section? Also have you tried looking in the generated package to see if any of the dependencies are there?

[tamoanxx]

Yes

The contents of the package are correct visually. The only thing that I could find is that I’m coding all this in a windows environment that could be part of the issue.

If I zip all the contents manually it works just fine.

On Mar 20, 2015, at 2:44 AM, Tim [email protected] wrote:

Hi tamtamoanxx,
Have you put your dependencies in the bundledDependencies section? Also have you tried looking in the generated package to see if any of the dependencies are there?

—
Reply to this email directly or view it on GitHub #6 (comment).

[Tim-B]

Are you able to create a simple projects that exhibits this behaviour which I can try and run?

What about if you upload the zipped file manually, does it work then?

[tamoanxx]

I will generate the project and send it over to you shortly.

It is the same result if I upload manually.

Thanks,
Dustin

On Mar 20, 2015, at 3:54 AM, Tim [email protected] wrote:

Are you able to create a simple projects that exhibits this behaviour which I can try and run?

What about if you upload the zipped file manually, does it work then?

—
Reply to this email directly or view it on GitHub.

[x3mka]

Same here. Reproducible at Windows 8.
Requiring npm modules (from ./node_modules) or custom files from nested folders (like ./nested/my.js) fails after grunt deploy.
Manual zip and upload works.

It looks like a permission issue on files in nested folders.

To reproduce:

1. add some dep ('async', for example) in package.json dependencies and bundle dependencies.
2. add var async = require('async'); and some empty handler code in index.js.
3. npm install, package, deploy, invoke in console

[jagedn]

Same here with Windows 8.

I have deployed correctly after doing a little patch to archiver\lib\core.js changing the data.mode to 777

near line 179:

if (typeof data.mode === 'number') {
data.mode &= 0777;
} else if (data.stats && data.mode === null) {
//jorge, removed data.mode = data.stats.mode & 0777;
data.mode = 0777;
//jorge
} else if (data.mode === null) {
data.mode = isDir ? 0755 : 0644;
}

[Tim-B]

I bumped the adm-zip version to the latest one which I hope will resolve this issue. I also tested it in a Windows 7 VM and it all worked fine.

[jagedn]

Hi Tim-B

I just tested the new version 0.4.0 (Windows 8.1) but deps are not detected . If I change core.js it works fine.
it must be something about how aws unzips the file and the internal permissions

[Tim-B]

I realised I should have bumped node-archiver not adm-zip, but node-archiver is already the latest version.

It might be worth reporting this upstream: https://github.com/archiverjs/node-archiver

I can't rule out that my implementation isn't to blame, but the options say "When using the bulk or file methods, fs stat data is used as the default value." so perhaps it's not correctly reading the stat data on Windows.

I might look into replacing the whole zip library with a different one.

[jagedn]

as far I can see your implementation is correct. I tried to unzip the package with windows and with unix and they seem ok, so I think it's something about the unzip process of lambda.
I was trying to use other zip-systems but adm-zip seems so easy ...

p.d.
your implementation saved me the week and improved my deploy process at 200% so thank you so much

[Tim-B]

I have a suspicion of what it is, it looks like zips don't support unix file permissions explicitly - but some zip implementations specify them via a custom field in the ZIP header: http://stackoverflow.com/questions/13633488/can-i-store-unix-permissions-in-a-zip-file-built-with-apache-ant

If I run a zip generated on my machine (Ubuntu) through zipinfo most of the file permissions are -rw-rw-r--, but I suspect it's reading and therefore setting incorrect permissions on Windows. When lambda unzips the package on linux it abides by the faulty file permissions in the zip.

Would you be able to somehow share a zip generated on your machine and I'll run it through zipinfo to see if the permissions differ.

[jagedn]

sure,
try it at
https://s3-eu-west-1.amazonaws.com/puravida-share-public/orangemovil_1-0-0_2015-4-23-13-5-52.zip

[Tim-B]

So on my machine your zip produces:

Archive:  orangemovil_1-0-0_2015-4-23-13-5-52.zip
Zip file size: 29117 bytes, number of entries: 14
-rw-rw-rw-  4.5 unx      554 bl defN 15-May-23 11:05 index.js
drw-rw-rw-  4.5 unx        0 b- stor 15-May-23 11:05 node_modules/
drw-rw-rw-  4.5 unx        0 b- stor 15-May-23 11:05 node_modules/async/
-rw-rw-rw-  4.5 unx      674 bl defN 15-May-23 11:05 node_modules/async/bower.json
-rw-rw-rw-  4.5 unx      297 bl defN 15-May-23 11:05 node_modules/async/component.json
drw-rw-rw-  4.5 unx        0 b- stor 15-May-23 11:05 node_modules/async/lib/
-rw-rw-rw-  4.5 unx     1063 bl defN 15-May-23 11:05 node_modules/async/LICENSE
-rw-rw-rw-  4.5 unx     1846 bl defN 15-May-23 11:05 node_modules/async/package.json
-rw-rw-rw-  4.5 unx    52082 bl defN 15-May-23 11:05 node_modules/async/README.md
-rw-rw-rw-  4.5 unx    34652 bl defN 15-May-23 11:05 node_modules/async/lib/async.js
drw-rw-rw-  4.5 unx        0 b- stor 15-May-23 11:05 node_modules/async/support/
-rw-rw-rw-  4.5 unx     1553 bl defN 15-May-23 11:05 node_modules/async/support/sync-package-managers.js
-rw-rw-rw-  4.5 unx     1874 bl defN 15-May-23 11:05 orangemovil.js
-rw-rw-rw-  4.5 unx      512 bl defN 15-May-23 11:05 package.json
14 files, 95107 bytes uncompressed, 27181 bytes compressed:  71.4%

But when I generate the same zip I get:

Archive:  orangemovil_1-0-0_2015-4-23-21-21-10.zip
Zip file size: 29486 bytes, number of entries: 15
-rw-rw-r--  4.5 unx      508 bl defN 15-May-23 09:38 Gruntfile.js
-rw-rw-r--  4.5 unx      554 bl defN 15-May-23 01:05 index.js
drwxrwxr-x  4.5 unx        0 b- stor 15-May-23 11:20 node_modules/
drwxrwxr-x  4.5 unx        0 b- stor 15-May-23 11:21 node_modules/async/
-rw-rw-r--  4.5 unx     1063 bl defN 15-May-19 08:44 node_modules/async/LICENSE
-rw-rw-r--  4.5 unx    52082 bl defN 15-May-19 08:44 node_modules/async/README.md
-rw-rw-r--  4.5 unx      674 bl defN 15-May-19 08:44 node_modules/async/bower.json
-rw-rw-r--  4.5 unx      297 bl defN 15-May-19 08:44 node_modules/async/component.json
drwxrwxr-x  4.5 unx        0 b- stor 15-May-19 08:44 node_modules/async/lib/
-rw-rw-r--  4.5 unx    34652 bl defN 15-May-19 08:44 node_modules/async/lib/async.js
-rw-rw-r--  4.5 unx     1859 bl defN 15-May-23 11:20 node_modules/async/package.json
drwxrwxr-x  4.5 unx        0 b- stor 15-May-19 08:44 node_modules/async/support/
-rwxrwxr-x  4.5 unx     1553 bl defN 15-May-19 08:44 node_modules/async/support/sync-package-managers.js
-rw-rw-r--  4.5 unx     1874 bl defN 15-May-23 01:05 orangemovil.js
-rw-rw-r--  4.5 unx      491 bl defN 15-May-23 11:21 package.json
15 files, 95607 bytes uncompressed, 27434 bytes compressed:  71.3%

I think the problem could be that the execute bit is missing on directories. I'll take a look and see if I can either fix that, or strip the permission information automatically.

[Tim-B]

I did a bit of a monkey patch which I think will fix this issue by making all files and folders have 777 permissions. https://github.com/Tim-B/grunt-aws-lambda/blob/master/tasks/lambda_package.js#L76

Hopefully that will fix the issue, it would be great if someone who uses Windows could give the latest version a try and let me know how they go.

[x3mka]

I ended up making my own aws lambda app template with gulp tasks. Gulp-zip doesn't have any issues. Besides, I needed package files and node modules lists config per function.

https://github.com/x3mka/aws-lambda-template