SSL error while starting Client

[aznwarmonkey]

Hello,

I am new to skein and dask in general. I am trying to create a hello world dask-yarn application and running into ssl certificate errors. Does anyone know how to get around it?

Here's the code i am using:

 import skein
 c = skein.Client(log_level='debug', java_options="-Djava.io.tmpdir=/path/to/some/dir -Dcom.sun.net.ssl.checkRevocation=false")
 

and here's the error message:

19/07/08 09:25:06 DEBUG skein.Driver: Starting Skein version 0.7.4
19/07/08 09:25:06 DEBUG skein.Driver: Logging in using ticket cache
19/07/08 09:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/08 09:25:07 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
19/07/08 09:25:08 INFO client.AHSProxy: Connecting to Application History server at cap-compute02/172.27.57.54:10200
19/07/08 09:25:08 INFO skein.Driver: Driver started, listening on 37170
19/07/08 09:25:08 DEBUG skein.Driver: Reporting gRPC server port back to the launching process
E0708 09:25:08.618234364  161031 ssl_transport_security.cc:1238] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
19/07/08 09:25:08 DEBUG skein.Driver: Starting process disconnected, shutting down
19/07/08 09:25:08 DEBUG skein.Driver: Driver shut down
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/software/virtual_env/aesop_20190707_env/lib/python3.7/site-packages/skein/core.py", line 366, in __init__
    self._call('ping', proto.Empty())
  File "/software/virtual_env/aesop_20190707_env/lib/python3.7/site-packages/skein/core.py", line 289, in _call
    raise ConnectionError("Unable to connect to %s" % self._server_name)
skein.exceptions.ConnectionError: Unable to connect to driver
 

[jcrist]

Apologies for the delay here, I've been out-of-office (vacation, then SciPy).

Why are you passing in java_options? Most users don't need to use these (the defaults are usually sufficient).

For your specific error, it looks like the connection between python and the driver subprocess is failing, resulting in the driver shutting down. To prevent lingering processes, when creating a non-global driver the java subprocess watches stdin, and upon close (usually due to parent process termination) shuts itself down. The specific lines are:

skein/java/src/main/java/com/anaconda/skein/Driver.java

Lines 298 to 299 in 3baa06c

	while (System.in.read() != -1) {}
	LOG.debug("Starting process disconnected, shutting down");

It's not clear to me why you're seeing this issue. A few questions:

• What OS are you running?
• What version of Java?
• What version of Hadoop?
• What version of openssl?

What happens if you start a global driver separately, then connect to it? This avoids this section of the code, and may help diagnose if there's another issue. The easiest way to test this is using the CLI (this is also available in the Python api with skein.Driver.start_global_driver/skein.Driver.from_global_driver).

$ skein driver start --log-level debug --log mylog.txt
$ skein application ls
$ skein driver stop

After running that, it would be good to know if it succeeded, and the output of mylog.txt and the console logs.