Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vertx caching folder cannot be created with Amazon Lambda extension #4254

Closed
oztimpower opened this issue Sep 28, 2019 · 30 comments
Closed

vertx caching folder cannot be created with Amazon Lambda extension #4254

oztimpower opened this issue Sep 28, 2019 · 30 comments
Labels
area/vertx kind/bug Something isn't working
Milestone
0.25.0

Comments

@oztimpower
Copy link
Contributor

oztimpower commented Sep 28, 2019 •
edited
Loading

Amazon Lambda extension requires an immutable file system.

Upon execution on AWS, vertx attempts to create the vertx cache folder. This causes the AWS Lambda execution to fail with the stack trace below. When run locally using sam-local (docker) it works ok.

The vertx configuration in io.quarkus.vertx.core.runtime.config.VertxConfiguration.caching is set to true by default. For the Amazon Lambda this should be set to false.

Execution on AWS fails:

java.lang.IllegalStateException: Failed to create cache dir
 at io.vertx.core.file.impl.FileResolver.setupCacheDir(FileResolver.java:332)
 at io.vertx.core.file.impl.FileResolver.<init>(FileResolver.java:87)
 at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:165)
 at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
 at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
 at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
 at io.vertx.core.Vertx.vertx(Vertx.java:85)
 at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:100)
 at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:82)
 at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb104.deploy_0(VertxCoreProcessor$buildWeb104.zig:71)
 at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb104.deploy(VertxCoreProcessor$buildWeb104.zig:96)
 at io.quarkus.runner.ApplicationImpl5.doStart(ApplicationImpl5.zig:226)
 at io.quarkus.runtime.Application.start(Application.java:93)
 at io.quarkus.runtime.Application.run(Application.java:213)
 at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:34)

Code (Kotlin):

package org.tjpower

import com.amazonaws.services.lambda.runtime.Context
import com.amazonaws.services.lambda.runtime.RequestHandler
import javax.enterprise.context.ApplicationScoped
import javax.inject.Inject


class HelloRequest() {
    var firstName: String? = null
    var lastName: String? = null
}

@ApplicationScoped
open class HelloGreeter() {
    open fun greet(firstName: String?, lastName: String?): String {
        return "$firstName, $lastName"
    }
}

class HelloLambda : RequestHandler<HelloRequest, String> {

//    @Inject
//    lateinit var greeter: HelloGreeter

    val greeter = HelloGreeter()

    override fun handleRequest(request: HelloRequest, context: Context): String {
        return greeter.greet(request.firstName, request.lastName)
    }
}
@oztimpower oztimpower added the kind/bug Something isn't working label Sep 28, 2019
@geoand
Copy link
Contributor

geoand commented Sep 28, 2019

@cescoffier what are the implications of disabling caching?

@cescoffier
Copy link
Member

@geoand the main impact would be when serving static files, as they are unpacked into the cache dir and read from there. In a lambda environment, serving static file is probably not used (@patriot1burke can you confirm?), so we could disable that.

Vert.x use the system tmp directory to create the cache directory, do it means that on AWS there is no tmp?

@oztimpower
Copy link
Contributor Author

I did lots of digging today to drill into the problem. With a pure java implementation, it works. However with the native-image version it doesn't. The difference is that the native image uses "/var/tmp" as the temp folder.

I tested it manually, modifying the bootstrap shell script, creating the directories in both /tmp and /var/tmp. I can create /tmp/vertx-cache, but I cannot create /var/tmp/vertx-cache, which is what native-image sees.

mkdir: cannot create directory ‘/var/tmp/vertx-cache’: Read-only file system

package io.quarkus.vertx.core.runtime;
@Recorder
public class VertxCoreRecorder {

...
136:

        String fileCacheDir = System.getProperty(CACHE_DIR_BASE_PROP_NAME,
                System.getProperty("java.io.tmpdir", ".") + File.separator + "vertx-cache");

@oztimpower
Copy link
Contributor Author

oztimpower commented Sep 30, 2019 via email

@cescoffier
Copy link
Member

cescoffier commented Sep 30, 2019 via email

@oztimpower
Copy link
Contributor Author

oztimpower commented Sep 30, 2019 via email •
edited
Loading

oztimpower added a commit to oztimpower/quarkus that referenced this issue Oct 2, 2019
@oztimpower
Issue quarkusio#4254 Force java.io.tmpdir = /tmp for native executabl…
b5274fb 
…e to address vertx-cache folder creation problem
cescoffier added a commit that referenced this issue Oct 3, 2019
@cescoffier
Merge pull request #4327 from oztimpower/master
edc9cea 
Issue #4254 Force java.io.tmpdir = /tmp for native executable
@gsmet
Copy link
Member

gsmet commented Oct 12, 2019

However next problem, -H:EnableURLProtocols=http needs to be passed to the native-image command.

@oztimpower , I'm curious about this part: how did you create your project? IIRC we enable the handler when in the Maven project template.

@oztimpower
Copy link
Contributor Author

oztimpower commented Oct 12, 2019 via email

@gsmet
Copy link
Member

gsmet commented Oct 12, 2019

Yeah, it's probably something we should push via the extension mechanism.

Could you create a specific issue for that?

I'm leaning towards closing this one as you fixed the initial issue. It will be clearer if we have a specific one.

Thanks!

@cescoffier cescoffier added this to the 0.25.0 milestone Oct 15, 2019
@johnoliver
Copy link
Contributor

for future reference for anyone who may stumble across this issue when deploying to openshift, I had to set -Dvertx.cacheDirBase=/tmp/vertx by creating an environment variable called JAVA_OPTS set to that value. Setting -Djava.io.tmpdir=/tmp alone did not seem to be sufficient.

@johnoliver johnoliver mentioned this issue Nov 7, 2019
Closed
@machi1990
Copy link
Member

for future reference for anyone who may stumble across this issue when deploying to openshift, I had to set -Dvertx.cacheDirBase=/tmp/vertx by creating an environment variable called JAVA_OPTS set to that value. Setting -Djava.io.tmpdir=/tmp alone did not seem to be sufficient.

Thanks for the tip. Maybe we can add it in the doc, WDYT? @gsmet @cescoffier

@geoand
Copy link
Contributor

geoand commented Nov 7, 2019

That would seem like a valuable addition to the vertx guide

@machi1990
Copy link
Member

@johnoliver would you like to contribute the tip here https://github.com/quarkusio/quarkus/blob/9eb5164f24a2c3bc6b4015bc4e399a7ef955704c/docs/src/main/asciidoc/vertx.adoc ?

@johnoliver
Copy link
Contributor

@machi1990 #5287

@mihapolak mihapolak mentioned this issue Nov 29, 2019
Closed
@sherl0cks
Copy link
Contributor

@oztimpower was there a resolution to this on lambda? The suggestions for OpenShift do not work on lambda for me.

@oztimpower
Copy link
Contributor Author

Hi @sherl0cks yes the issue was resolved for Amazon Lambda. I recall seeing a similar question about OpenShift that forcing the /tmp folder did not work. Recommend you search the issues or create a new issue to track it.

@sherl0cks
Copy link
Contributor

sherl0cks commented Dec 9, 2019 •
edited
Loading

@oztimpower @cescoffier @gsmet can you share the resolution for lambda? I see #4327, but that file is no longer present in master (moved in 0.27) and it is unclear to me how I can set this command line switch in the lambda custom runtime. I've tried all sorts of ways to disable vertx caching to no effect.

To be clear, I'm not interest in OpenShift, only aws lambda native. I'm using 1.0.1.final of quarkus.

@cescoffier
Copy link
Member

@sherl0cks setting -Dvertx.cacheDirBase=/somewher/in/the/cloud/that/is/writable should still work.

@oztimpower
Copy link
Contributor Author

oztimpower commented Dec 10, 2019 •
edited
Loading

@sherl0cks have you tried using the SAM CLI as per the latest Quarkus documentation?

Amazon lambda works for me, the Maven deployment automated the bundling and the /tmp folder issue was migrated to code, and thus dropped from the script.

Possible to post your error message, particularly with SAM local?

I use Quarkus with native lambda in Prod, works well.

@sherl0cks
Copy link
Contributor

sherl0cks commented Dec 10, 2019 •
edited
Loading

Just got this working. Below are the steps I took. @cescoffier would you be interested in a PR to the maven archetype for step 7 and info in the docs to explain how this works for future users?

  1. unzip function.zip
  2. rename bootstrap to something like runner
  3. create a new shell script called bootstrap with something like ./runner -Djava.io.tmpdir=/tmp. @oztimpower I was unable to achieve the same effect via setting a system property in code. Do you have example of what you did?
  4. make sure bootstrap is executable
  5. zip bootstrap & runner into a new function.zip
  6. run update-native.sh (assuming you came from quarkus-aws-hello-world example) and then test with invoke-native.sh
  7. automate steps 1-6 by adding the files to src and updating the zip.xml config for the maven assembly plugin that creates the .zip file

I'm not using SAM cli as I generally find it to be more trouble than its worth. Reading the maven archetype and the docs, I do not see anything in that guide that would address this concern. If you can link to something specific that I missed, please do.

For any future readers, my code calls an https endpoint and thus I need to update the zip.xml to package in libsunec.so and cacerts per the native ssl guide.

@patriot1burke
Copy link
Contributor

Did you increase your cold start latency?

@sherl0cks
Copy link
Contributor

sherl0cks commented Dec 11, 2019 •
edited
Loading

I can’t say until I find a way to change vertx temp without this extra script. That said, I’m getting ~250ms init on a cold start right now, which is basically flat from 128mb to to max 3008mb. Duration starts at about ~250ms and then levels out at ~25ms at 512mb.

@ctron-te
Copy link

ctron-te commented Feb 11, 2020 •
edited
Loading

Strangely when I try the sample project with Java everything works fine. Then I converted it to Kotlin, which works fine with java runtime but when I move to provided runtime I get java.lang.IllegalStateException: Failed to create cache dir.

This is using Quarkus 1.2.0.Final

@patriot1burke
Copy link
Contributor

Any reason to have quarkus set the java.io.tmpdir and vertx cache dir automatically? Pretty sure /tmp is guaranteed to be available for lambda.

@patriot1burke
Copy link
Contributor

My issue is I can't reproduce the problem.

@patriot1burke
Copy link
Contributor

BTW all, we want to avoid requiring a bootstrap shell script around the runner so as to reduce cold start latency even more.

@oztimpower
Copy link
Contributor Author

I'm experiencing the same problem as @ctron-te with head (999-SNAPSHOT).

Kotlin AWS Lambda project that can reproduce it. Project is a Kotlin version of the Java AWS Lambda template, i.e. InputObject, OutputObject, ProcessingService.

  1. git clone https://github.com/oztimpower/kotlin-lambda.git
  2. mvn package -Dnative -Dquarkus.native.container-build=true
  3. ./manage.sh native create
  4. ./manage.sh native invoke

START RequestId: 555e0118-5147-4830-86ec-d73469fd046e Version: $LATEST
2020-03-04 14:09:01,667 ERROR [io.qua.application] (main) Failed to start application: java.lang.IllegalStateException: Failed to create cache dir: /var/tmp//vertx-cache/file-cache-14c1dacd-5b2b-4f42-bd36-1e706fa769b2
at io.vertx.core.file.impl.FileResolver.setupCacheDir(FileResolver.java:332)
at io.vertx.core.file.impl.FileResolver.(FileResolver.java:87)
at io.vertx.core.impl.VertxImpl.(VertxImpl.java:167)
at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
at io.vertx.core.Vertx.vertx(Vertx.java:85)

@oztimpower
Copy link
Contributor Author

I should add it works fine with Java source Native executable. Issue is with Kotlin.

@oztimpower oztimpower mentioned this issue Mar 5, 2020
Closed
@patriot1burke
Copy link
Contributor

This PR might help. #7619

@oztimpower
Copy link
Contributor Author

I confirm that #7619, resolves the issue with my Kotlin Amazon Lambda project. i.e., forcing the tmp dirs in AmazonLambdaProcessor.java works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/vertx kind/bug Something isn't working
Projects
None yet
Milestone
0.25.0
Development

No branches or pull requests
9 participants
@cescoffier @patriot1burke @gsmet @sherl0cks @johnoliver @geoand @machi1990 @oztimpower @ctron-te