`-Djavax.net.ssl.trustStore` certs not properly passed when downloading deps in BSP #2963

jiramares · 2024-06-14T08:41:57Z

Versions
Scala CLI version: 1.3.2
Scala version (default): 3.4.2

IntelliJ IDEA 2024.1.3 (Ultimate Edition)
Build #IU-241.17890.1, built on June 4, 2024

Description
We used company artifactory to resolve/cache artefacts. Therefor I have to set up environment varianble COURSIER_REPOSITORIES to handle it right. Because we are using self signed certificate on server there is problem with resolving artefacts directly using coursier (upgrades scala-cli, scala or java).

I realized that there is config repositories.default that seems to me to fulfil the same thing but only for scala-cli (not system wide). So I started using it. It works well but breaks the integration with IDE.

After starting using repositories.default instead of COURSIER_REPOSITORIES I can't build in IDEA via BSP. In BSP window there has been no tree under the root of project. The problem is with our artifactory server with self signed certificate because using maven central causes everything works fine.

To Reproduce

scala-cli config repositories.default ivy2local https://our-server/with-self-signed-certificate/repo --power
create directory test
in it create file test.sc with content

//> using scala 3.3

println(sum(2, 3))

def sum(a: Int, b: Int) = a + b

in the directory test run scala-cli setup-ide test.sc
open IDEA and File / New / Project from Existing Sources - choose folder test and in following dialog choose model BSP

In BSP window there is no tree and println is marked as uknown symbol.

Expected behaviour
There should be dependencies in BSP window and println shoul be known.

The text was updated successfully, but these errors were encountered:

jiramares · 2024-06-14T14:22:05Z

And now it is not working with artifact proxy with self-signed certificate via https at all. Neither COURSIER_REPOSITORIES environment variable nor repositories.default scala-cli config. I don't know what to do :-(

Gedochao · 2024-06-18T08:09:40Z

I don't know what to do :-(

Hey, I'll try to tackle this soon (likely won't be this week, however).
I'd suspect something is wrong with the config sub-command's repositories.* settings, but COURSIER_REPOSITORIES is known and proven to be reliable... 😕
Can't tell you much until I take a closer look.

jiramares · 2024-06-19T14:23:53Z

Hi @Gedochao,
I have been a little more investigating and it looks that problem is with https protocol with artifact proxy (not with its setting via COURSIER_REPOSITORIES or repositories.default). I have to restart computer between tests because I can't reset the environment well (killing all java processes have not been enough).
scala-cli works great with http and https too (for https I have to use -Djavax.net.ssl.trustStore). But the IDEA is not able to resolve using https at all (not possible to validate server certificate - I have imported it into IDEA's store - it looks like idea is using coursier under the hood too but I can't specify -Djavax.net.ssl.trustStore). To resolve dependencies is job of IDEA or bloop?

jiramares · 2024-06-20T09:47:45Z

From my point of view it directs to coursier that is run under the hood to resolve artifacts. But when run via scala-cli I can add -Djavax.net.ssl.trustStore but when run in IDEA it is not possible and therfore not working.

Gedochao · 2024-06-24T07:30:46Z

To resolve dependencies is job of IDEA or bloop?

Definitely Scala CLI/Bloop, depending on which dependencies. Both use Coursier under the hood.
However, I can't be sure what IDEA does in the background.

From my point of view it directs to coursier that is run under the hood to resolve artifacts. But when run via scala-cli I can add -Djavax.net.ssl.trustStore but when run in IDEA it is not possible and therfore not working.

@jiramares Can you raise a corresponding ticket on the JetBrains issue tracker? (https://youtrack.jetbrains.com/issues)
So that we can establish this works as expected on the IDEA side of things.

jiramares · 2024-06-24T07:39:31Z

I have already raised one an the result is - it is not IDEA problem (https://youtrack.jetbrains.com/issue/SCL-22728/scala-cli-project-not-working)](https://youtrack.jetbrains.com/issue/SCL-22728/scala-cli-project-not-working)

jiramares · 2024-06-24T07:41:26Z

But .. who starts the bloop server, from mine point of view it is started by scala-cli and IDEA in calling it via BSP? Therefore if the bloop is responsible for resolving why it is run okay through scala-cli and not running via IDEA? I am confused....

Gedochao · 2024-06-24T08:55:15Z

One extra question - have you tried running it in Metals?
I just want to confirm this isn't IDEA specific, since from what I understand it works when run outside of IDEA.
I'd theorise maybe some coursier instance ran inside of IntelliJ is using a JVM without the cert flag.

I hope we won't end up bouncing this back and forth...

jiramares · 2024-06-24T09:12:44Z

No, I haven't ... be honest don't know how to make it.
Just install Metals into VSC and run scala-cli setup-id . and open it in VSC? Or it is necessary to do something more?

Gedochao · 2024-06-24T10:03:59Z

No, I haven't ... be honest don't know how to make it. Just install Metals into VSC and run scala-cli setup-id . and open it in VSC? Or it is necessary to do something more?

Exactly this.
plus start the Metals server and check if everything works.

jiramares · 2024-06-24T10:45:21Z

I try to make it but I am sorry not to test it now ... hope to make it this week. j

jiramares · 2024-06-28T13:40:12Z

I have installed metals into VSC and have same result as in IDEA:

024.06.28 15:33:28 INFO  BSP server: [error]   download error: Caught javax.net.ssl.SSLHandshakeException (PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) while downloading https://ids/artifactory/repo/org/typelevel/cats-core_3/2.12.0/cats-core_3-2.12.0.pom

Looks like that the -Djavax.net.ssl.trustStore have to be somehow passed to bloop too...

tgodzik · 2024-07-01T09:35:48Z

@jiramares do you use cats within your project?

jiramares · 2024-07-01T18:42:56Z

Yes, because of dylemma.xml-spac

tgodzik · 2024-07-02T08:22:26Z

My guess is that we are not adding the certs when downloading sources somehow, but it seems unexpected 🤔

Gedochao · 2024-10-03T10:58:15Z

@jiramares are you still encountering this issue with Scala CLI v1.5.1?
I suspect we may have fixed it in #3169.

jiramares · 2024-10-08T10:10:16Z

It is still not working. If I repeat the steps in issue description I am getting following stacktrace:

java.lang.Exception: Error getting Bloop class path
  bloop.rifle.BloopRifle$.startServer(BloopRifle.scala:51)
  bloop.rifle.BloopServer$.startBloop$1(BloopServer.scala:77)
  bloop.rifle.BloopServer$.ensureBloopRunning(BloopServer.scala:108)
  bloop.rifle.BloopServer$.bsp(BloopServer.scala:156)
  bloop.rifle.BloopServer$.buildServer(BloopServer.scala:186)
  scala.build.bsp.BspImpl.$anonfun$10(BspImpl.scala:424)
  scala.build.compiler.BloopCompiler.<init>(BloopCompiler.scala:15)
  scala.build.bsp.BspImpl.newBloopSession(BspImpl.scala:429)
  scala.build.bsp.BspImpl.run(BspImpl.scala:465)
  scala.cli.commands.bsp.Bsp$.runCommand$$anonfun$1(Bsp.scala:185)
  scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
  scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
  scala.build.bsp.BspThreads$.withThreads(BspThreads.scala:23)
  scala.cli.commands.bsp.Bsp$.runCommand(Bsp.scala:174)
  scala.cli.commands.bsp.Bsp$.runCommand(Bsp.scala:76)
  scala.cli.commands.ScalaCommand.run(ScalaCommand.scala:390)
  scala.cli.commands.ScalaCommand.run(ScalaCommand.scala:371)
  caseapp.core.app.CaseApp.main(CaseApp.scala:166)
  scala.cli.commands.ScalaCommand.main(ScalaCommand.scala:356)
  caseapp.core.app.CommandsEntryPoint.main(CommandsEntryPoint.scala:370)
  scala.cli.ScalaCliCommands.main(ScalaCliCommands.scala:125)
  scala.cli.ScalaCli$.main0(ScalaCli.scala:320)
  scala.cli.ScalaCli$.main(ScalaCli.scala:124)
  scala.cli.ScalaCli.main(ScalaCli.scala)
scala.build.errors.FetchingDependenciesError: Error downloading ch.epfl.scala:bloop-frontend_2.12:2.0.2
  not found: /home/jmares/.ivy2/local/ch.epfl.scala/bloop-frontend_2.12/2.0.2/ivys/ivy.xml
  download error: Caught javax.net.ssl.SSLHandshakeException (PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) while downloading https://our-server/with-self-signed-certificate/repo/ch/epfl/scala/bloop-frontend_2.12/2.0.2/bloop-frontend_2.12-2.0.2.pom
  scala.build.Artifacts$.toFetchingDependenciesError(Artifacts.scala:723)
  scala.build.Artifacts$.fetchCsDependencies$$anonfun$1$$anonfun$2(Artifacts.scala:697)
  scala.util.Either$LeftProjection.map(Either.scala:622)
  scala.build.Artifacts$.fetchCsDependencies$$anonfun$1(Artifacts.scala:691)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Artifacts$.fetchCsDependencies(Artifacts.scala:671)
  scala.build.Artifacts$.fetchAnyDependenciesWithResult$$anonfun$1(Artifacts.scala:590)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Artifacts$.fetchAnyDependenciesWithResult(Artifacts.scala:569)
  scala.build.Artifacts$.fetchAnyDependencies$$anonfun$1(Artifacts.scala:555)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Artifacts$.fetchAnyDependencies(Artifacts.scala:546)
  scala.build.Artifacts$.artifacts$$anonfun$1(Artifacts.scala:486)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Artifacts$.artifacts(Artifacts.scala:484)
  scala.build.Bloop$.bloopClassPath$$anonfun$1(Bloop.scala:74)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Bloop$.bloopClassPath(Bloop.scala:72)
  scala.build.Bloop$.bloopClassPath$$anonfun$2(Bloop.scala:105)
  scala.build.EitherCps$Helper.apply(EitherCps.scala:19)
  scala.build.Bloop$.bloopClassPath(Bloop.scala:95)
  scala.cli.commands.shared.SharedCompilationServerOptions.$anonfun$13(SharedCompilationServerOptions.scala:265)
  bloop.rifle.BloopRifle$.startServer(BloopRifle.scala:50)
  bloop.rifle.BloopServer$.startBloop$1(BloopServer.scala:77)
  bloop.rifle.BloopServer$.ensureBloopRunning(BloopServer.scala:108)
  bloop.rifle.BloopServer$.bsp(BloopServer.scala:156)
  bloop.rifle.BloopServer$.buildServer(BloopServer.scala:186)
  scala.build.bsp.BspImpl.$anonfun$10(BspImpl.scala:424)
  scala.build.compiler.BloopCompiler.<init>(BloopCompiler.scala:15)
  scala.build.bsp.BspImpl.newBloopSession(BspImpl.scala:429)
  scala.build.bsp.BspImpl.run(BspImpl.scala:465)
  scala.cli.commands.bsp.Bsp$.runCommand$$anonfun$1(Bsp.scala:185)
  scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
  scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
  scala.build.bsp.BspThreads$.withThreads(BspThreads.scala:23)
  scala.cli.commands.bsp.Bsp$.runCommand(Bsp.scala:174)
  scala.cli.commands.bsp.Bsp$.runCommand(Bsp.scala:76)
  scala.cli.commands.ScalaCommand.run(ScalaCommand.scala:390)
  scala.cli.commands.ScalaCommand.run(ScalaCommand.scala:371)
  caseapp.core.app.CaseApp.main(CaseApp.scala:166)
  scala.cli.commands.ScalaCommand.main(ScalaCommand.scala:356)
  caseapp.core.app.CommandsEntryPoint.main(CommandsEntryPoint.scala:370)
  scala.cli.ScalaCliCommands.main(ScalaCliCommands.scala:125)
  scala.cli.ScalaCli$.main0(ScalaCli.scala:320)
  scala.cli.ScalaCli$.main(ScalaCli.scala:124)
  scala.cli.ScalaCli.main(ScalaCli.scala)
coursier.error.ResolutionError$CantDownloadModule: Error downloading ch.epfl.scala:bloop-frontend_2.12:2.0.2
  not found: /home/jmares/.ivy2/local/ch.epfl.scala/bloop-frontend_2.12/2.0.2/ivys/ivy.xml
  download error: Caught javax.net.ssl.SSLHandshakeException (PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) while downloading https://our-server/with-self-signed-certificate/repo/ch/epfl/scala/bloop-frontend_2.12/2.0.2/bloop-frontend_2.12-2.0.2.pom
  coursier.Resolve$.$anonfun$validate$1(Resolve.scala:403)
  scala.collection.immutable.List.map(List.scala:247)
  scala.collection.immutable.List.map(List.scala:79)
  coursier.Resolve$.validate(Resolve.scala:401)
  coursier.Resolve.validate0$1(Resolve.scala:160)
  coursier.Resolve.$anonfun$ioWithConflicts0$4(Resolve.scala:210)
  coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
  coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
  coursier.util.Task$.wrap(Task.scala:82)
  coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
  scala.concurrent.impl.Promise$Transformation.run(Promise.scala:470)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)
  com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
  com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)

it looks like the problem is with coursier that is running its own JDK and we have to somewhere specify -Djavax.net.ssl.trustStore=/opt/jdk/lib/security/cacerts' but I don't know where :-(

PS. The JAVA_HOME jdk and IntelliJ IDEA JDK cacerts contains imported certificate to verify self-signed server certificate.

PS2. I am a little bit confused because the file .scala-build/ide-envs.json contains value:
"COURSIER_REPOSITORIES":"ivy2local|http://our-server/with-self-signed-certificate/repo"
that is specified using ENV variable COURSIER_REPOSITORIES but coursier loads artefacts from repository specified in scala-cli config.

PS3. If I removed this environment variable the value COURSIER_REPOSITORIES has been missing in .scala-build/ide-envs.json file.

tgodzik · 2024-10-08T14:15:00Z

Did you do scala-cli setup-ide -Djavax.net.ssl.trustStore=/opt/jdk/lib/security/cacerts . ? It should then use that property while downloading.

jiramares · 2024-10-09T09:11:27Z

@tgodzik That's it ... it works. Great .... I am going to try to finalize solution based on it.

jiramares added the bug Something isn't working label Jun 14, 2024

Gedochao added IDE bsp Issues tied to the implementation of BSP (Build Server Protocol) config Issues tied to the config sub-command. labels Jun 14, 2024

Gedochao added the intellij Issues tied with IDEA IntelliJ interoperability label Jun 24, 2024

Gedochao changed the title ~~Setting "repositories.default" breaks the IntelliJ Idea integration~~ -Djavax.net.ssl.trustStore not properly passed when downloading deps in BSP Jul 2, 2024

Gedochao changed the title ~~-Djavax.net.ssl.trustStore not properly passed when downloading deps in BSP~~ -Djavax.net.ssl.trustStore certs not properly passed when downloading deps in BSP Jul 2, 2024

Gedochao removed intellij Issues tied with IDEA IntelliJ interoperability config Issues tied to the config sub-command. labels Jul 2, 2024

Gedochao added this to Scala CLI Board Jul 8, 2024

github-project-automation bot moved this to To do in Scala CLI Board Jul 8, 2024

Gedochao mentioned this issue Sep 12, 2024

Ensure that passing Java props into Scala CLI as launcher args would also pass it into BSP configuration #3169

Merged

jiramares closed this as completed Oct 9, 2024

github-project-automation bot moved this from To do to Done in Scala CLI Board Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`-Djavax.net.ssl.trustStore` certs not properly passed when downloading deps in BSP #2963

`-Djavax.net.ssl.trustStore` certs not properly passed when downloading deps in BSP #2963

jiramares commented Jun 14, 2024

jiramares commented Jun 14, 2024

Gedochao commented Jun 18, 2024

jiramares commented Jun 19, 2024 •

edited

Loading

jiramares commented Jun 20, 2024

Gedochao commented Jun 24, 2024

jiramares commented Jun 24, 2024

jiramares commented Jun 24, 2024

Gedochao commented Jun 24, 2024

jiramares commented Jun 24, 2024

Gedochao commented Jun 24, 2024 •

edited

Loading

jiramares commented Jun 24, 2024

jiramares commented Jun 28, 2024 •

edited

Loading

tgodzik commented Jul 1, 2024

jiramares commented Jul 1, 2024

tgodzik commented Jul 2, 2024

Gedochao commented Oct 3, 2024

jiramares commented Oct 8, 2024

tgodzik commented Oct 8, 2024

jiramares commented Oct 9, 2024

-Djavax.net.ssl.trustStore certs not properly passed when downloading deps in BSP #2963

-Djavax.net.ssl.trustStore certs not properly passed when downloading deps in BSP #2963

Comments

jiramares commented Jun 14, 2024

jiramares commented Jun 14, 2024

Gedochao commented Jun 18, 2024

jiramares commented Jun 19, 2024 • edited Loading

jiramares commented Jun 20, 2024

Gedochao commented Jun 24, 2024

jiramares commented Jun 24, 2024

jiramares commented Jun 24, 2024

Gedochao commented Jun 24, 2024

jiramares commented Jun 24, 2024

Gedochao commented Jun 24, 2024 • edited Loading

jiramares commented Jun 24, 2024

jiramares commented Jun 28, 2024 • edited Loading

tgodzik commented Jul 1, 2024

jiramares commented Jul 1, 2024

tgodzik commented Jul 2, 2024

Gedochao commented Oct 3, 2024

jiramares commented Oct 8, 2024

tgodzik commented Oct 8, 2024

jiramares commented Oct 9, 2024

`-Djavax.net.ssl.trustStore` certs not properly passed when downloading deps in BSP #2963

`-Djavax.net.ssl.trustStore` certs not properly passed when downloading deps in BSP #2963

jiramares commented Jun 19, 2024 •

edited

Loading

Gedochao commented Jun 24, 2024 •

edited

Loading

jiramares commented Jun 28, 2024 •

edited

Loading