Debugger invoked when run fails

[amitlan]

I installed pgloader from the latest 3.2.0 rpm. Then the following:

$ pgloader --help

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1006DB3133}>:
  Error opening shared object "libsybdb.so":
  libsybdb.so: cannot open shared object file: No such file or directory.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE       ] Skip this shared object and continue.
  1: [RETRY          ] Retry loading this shared object.
  2: [CHANGE-PATHNAME] Specify a different pathname to load the shared object from.

(SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libsybdb.so" :NAMESTRING "libsybdb.so" :HANDLE NIL :DONT-SAVE NIL))
0] 

It seems (sbcl) debugger was invoked upon not finding libsybdb.so or something like that. It remains in interactive mode until I enter (sb-ext:quit), which is something I found on Googling. I also saw where I got that little trick is that sbcl can be invoked in --non-interactive mode. Maybe that should be the default as this's possibly surprising behavior. Not sure if it applies here though.

Thoughts?

[dimitri]

It's not supposed to be happening that way, but I see that I might be missing some error handling in fact. Could you please type in back at the debugger's prompt before quitting, and paste here the backtrace?

[amitlan]

Here you go:

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1006DB2FD3}>
0: (SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libsybdb.so" :NAMESTRING "libsybdb.so" :HANDLE NIL :DONT-SAVE NIL))
1: (SB-ALIEN::TRY-REOPEN-SHARED-OBJECT #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libsybdb.so" :NAMESTRING "libsybdb.so" :HANDLE NIL :DONT-SAVE NIL))
2: (SB-SYS:REOPEN-SHARED-OBJECTS)
3: (SB-IMPL::FOREIGN-REINIT)
4: (SB-IMPL::REINIT)
5: ((FLET #:WITHOUT-INTERRUPTS-BODY-54 :IN SAVE-LISP-AND-DIE))
6: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

[dimitri]

Ok it seems to be quite a simple error I made when refactoring the image hooks. Can you build from source to confirm that the bug is fixed? I think it warrant a 3.2.1 release already, so I will schedule that for this week when I have your confirmation that the bug is fixed.

[amitlan]

Thanks Dimitri for the fix.

I just cloned the repo and tried make; but haven't had success although this time it was the build itself (sorry I'm not quite familiar with the build environment):

$ make
mkdir -p build
curl -o build/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56097  100 56097    0     0   700k      0 --:--:-- --:--:-- --:--:--  913k
sbcl --no-sysinit --no-userinit --load build/quicklisp.lisp                         \
             --eval '(quicklisp-quickstart:install :path "build/quicklisp")' \
             --eval '(quit)'
This is SBCL 1.0.38-3.el6, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

  ==== quicklisp quickstart 2014-02-01 loaded ====

    To continue with installation, evaluate: (quicklisp-quickstart:install)

    For installation options, evaluate: (quicklisp-quickstart:help)

WARNING: Making quicklisp part of the install pathname directory
(quicklisp-quickstart:install)

debugger invoked on a SB-BSD-SOCKETS:OPERATION-TIMEOUT-ERROR in thread #<THREAD
                                                                         "initial thread" RUNNING
                                                                         {1002C27731}>:
  Socket error in "connect": ETIMEDOUT (Connection timed out)

Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Ignore runtime option --eval "(quicklisp-quickstart:install :path \"build/quicklisp\")".
  1: [ABORT   ] Skip rest of --eval and --load options.
  2:            Skip to toplevel READ/EVAL/PRINT loop.
  3: [QUIT    ] Quit SBCL (calling #'QUIT, killing the process).

(SB-BSD-SOCKETS:SOCKET-ERROR "connect")
0] CONTINUE
sbcl --no-sysinit --no-userinit --load build/quicklisp/setup.lisp                 \
             --eval '(push "/home/amit/postgres/pgxn/pgloader/" asdf:*central-registry*)'      \
             --eval '(ql:quickload "pgloader")'                     \
             --eval '(quit)'
This is SBCL 1.0.38-3.el6, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD
                                                           "initial thread" RUNNING
                                                           {1002C27991}>:
  Couldn't load #P"build/quicklisp/setup.lisp": file does not exist.

Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Ignore runtime option --load "build/quicklisp/setup.lisp".
  1: [ABORT   ] Skip rest of --eval and --load options.
  2:            Skip to toplevel READ/EVAL/PRINT loop.
  3: [QUIT    ] Quit SBCL (calling #'QUIT, killing the process).

(LOAD #P"build/quicklisp/setup.lisp")[:EXTERNAL]
0] CONTINUE  

debugger invoked on a SB-INT:SIMPLE-READER-PACKAGE-ERROR in thread #<THREAD
                                                                     "initial thread" RUNNING
                                                                     {1002C27991}>:
  SB-INT:SIMPLE-READER-PACKAGE-ERROR at 13 (line 1, column 13) on #<SB-IMPL::STRING-INPUT-STREAM

                                                                    {1002C50A91}>:
    package "QL" not found

Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
  1: [ABORT   ] Skip rest of --eval and --load options.
  2:            Skip to toplevel READ/EVAL/PRINT loop.
  3: [QUIT    ] Quit SBCL (calling #'QUIT, killing the process).

(SB-IMPL::READ-TOKEN #<SB-IMPL::STRING-INPUT-STREAM {1002C50A91}> #\q)
0] QUIT
make: *** [build/manifest.ql] Error 1

Is this invocation of the debugger normal?

[dimitri]

In the middle of the build output we see the connection failure, that makes it all fail. Try again with network attached?

  Socket error in "connect": ETIMEDOUT (Connection timed out)

[amitlan]

The virtual machine DOES have network attached.

Is it possible that:

Socket error in "connect": ETIMEDOUT (Connection timed out)

may be caused by a proxy sitting between my box and whatever it is trying to connect to?

[dimitri]

To use a proxy, edit the Makefile as shown in the example you found... Other than that, you can use a build machine with direct connection then sync the binary over, it's self-contained.

[amitlan]

Ok, making some change as you suggested in the Makefile, things seem to proceed only to stop and invoke debugger with stack looking like:

0: (CFFI::FL-ERROR "Unable to load foreign library (~A).~%  ~A")[:EXTERNAL]
1: (CFFI::LOAD-FOREIGN-LIBRARY-PATH SYBDB "libsybdb.so" NIL)
2: ((FLET CFFI::%DO-LOAD) #<FOREIGN-LIBRARY SYBDB> SYBDB "libsybdb.so")
3: (LOAD-FOREIGN-LIBRARY SYBDB)[:EXTERNAL]
4: ((SB-C::TOP-LEVEL-FORM
     (BLOCK #:BLOCK0
       (LET ((#:G1 NIL))
         (TAGBODY
           (RESTART-BIND ???
             ???)
          #:TAG2
           (RETURN-FROM #:BLOCK0 ???))))))[:TOPLEVEL]
5: (SB-FASL::LOAD-FASL-GROUP
    #<SB-SYS:FD-STREAM
      for "file /home/amit/.cache/common-lisp/sbcl-1.0.38-3.el6-linux-x64/home/amit/postgres/pgxn/pgloader/build/quicklisp/dists/quicklisp/software/cl-mssql-20131003-git/src/mssql.fasl"
      {1004643001}>)
6: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK))
7: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-LOCK]324))
8: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK
    #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK) {7FFFEFF03AE9}>
    #S(SB-THREAD:MUTEX
       :NAME "World Lock"
       :%OWNER #<SB-THREAD:THREAD "initial thread" RUNNING {1002C27991}>
       :STATE 1))

<snip>

The same libsybdb.so related failure.

debugger invoked on a LOAD-FOREIGN-LIBRARY-ERROR in thread #<THREAD
                                                             "initial thread" RUNNING
                                                             {1002C27991}>:
  Unable to load foreign library (SYBDB).
  Error opening shared object "libsybdb.so":
  libsybdb.so: cannot open shared object file: No such file or directory.

Does the previous commit miss something?

By the way, I did following to the Makefile:

--- a/Makefile
+++ b/Makefile
@@ -84,7 +84,7 @@ $(QLDIR)/setup.lisp:
        mkdir -p $(BUILDDIR)
        curl -o $(BUILDDIR)/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp
        $(CL) $(CL_OPTS) --load $(BUILDDIR)/quicklisp.lisp                         \
-             --eval '(quicklisp-quickstart:install :path "$(BUILDDIR)/quicklisp")' \
+             --eval '(quicklisp-quickstart:install :path "$(BUILDDIR)/quicklisp" :proxy "$(http_proxy)")' \
              --eval '(quit)'

[dimitri]

Yeah freetds-devel is an hard dependency for building pgloader, sorry I should have mentioned that. It's only optional when using the artefact binary.

Your http_proxy change unfortunately breaks building pgloader when http_proxy is not defined in the environment... I will see about that with the Quicklisp project directly, maybe they could use the env directly.

[amitlan]

And then I got hit by the SBCL 1.0.38 related issue with:

Symbol "DELETE-DIRECTORY" not found in the SB-EXT package.

I'm on CentOS 6, :(

[dimitri]

SBCL 1.0.38 is so old that no SBCL maintainer I've heard of would consider working on it. Please install a more recent one on the build machine, consider a local static binary as from http://prdownloads.sourceforge.net/sbcl/sbcl-1.2.7-x86-64-linux-binary.tar.bz2 (from http://sbcl.org/platform-table.html).

[amitlan]

Sweet, deploying SBCL 1.2.7 binary made it work for me.

By the way, I had to turn off the compress option:

--- a/Makefile
+++ b/Makefile
-COMPRESS_CORE ?= yes
+COMPRESS_CORE ?= no

[dimitri]

Yeah, or recompile sbcl from source as described in the INSTALL.md file... do you also confirm the bugfix is effective?

[amitlan]

Yes, all the problems I reported in this issue have been fixed thanks to your commits up to the point that I can now run pgloader, 👍

[dimitri]

Awesome, thanks for confirming!

[hron84]

I installed pgloader from PGDG repository, and this annoying debugger appears. It says libssl.so.1.0.1e is missing but after i say CONT in the debugger, it still works. How on the earth can I just disable the debugger and ask pgloder to act as other linux commands - do its job or coredump if cannot?

[dimitri]

@hron84 which version are you using? Normally it's fixed now, you might need to use the current tree version for that tho. I'm thinking of cutting a release “soon”, but I'm not there yet.

[ksyz]

Hit the same error today, using postgresql96 from pgdg repository. system in question is fully updated 64b scientific linux 7.4 with pgloader-3.4.1-1.rhel7.src.rpm and openssl-1.0.2k-8.el7.src.rpm. pgloader continues uppon hitting cont. Anyway, I think it's different issue than original topic. So I'm here just to provide more details to comment that I found to be the same issue. Feel free to split/ignore.

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10068F63A3}>:
  Error opening shared object "libssl.so.1.0.1e":
  libssl.so.1.0.1e: cannot open shared object file: No such file or directory.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE       ] Skip this shared object and continue.
  1: [RETRY          ] Retry loading this shared object.
  2: [CHANGE-PATHNAME] Specify a different pathname to load the shared object from.

(SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libssl.so.1.0.1e" :NAMESTRING "libssl.so.1.0.1e" :HANDLE NIL :DONT-SAVE NIL))
0] BACK

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10068F63A3}>
0: (SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libssl.so.1.0.1e" :NAMESTRING "libssl.so.1.0.1e" :HANDLE NIL :DONT-SAVE NIL))
1: (SB-ALIEN::TRY-REOPEN-SHARED-OBJECT #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libssl.so.1.0.1e" :NAMESTRING "libssl.so.1.0.1e" :HANDLE NIL :DONT-SAVE NIL))
2: (SB-SYS:REOPEN-SHARED-OBJECTS)
3: (SB-IMPL::FOREIGN-REINIT)
4: (SB-IMPL::REINIT)
5: ((FLET #:WITHOUT-INTERRUPTS-BODY-86 :IN SAVE-LISP-AND-DIE))
6: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

[hron84]

@dimitri you can check yourself which pgloader is contained in PGDG repository, but IIRC it was https://apt.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7.6-x86_64/pgloader-3.4.1-1.rhel7.x86_64.rpm (I already changed job so I have no longer access to the affected system).

This version is may be outdated. The problem is I often use restricted environments where discouraged or disallowed to install stuffs from a targz binary, especially in production environment. Mostly the only thing I can and would do is install an RPM/DEB package because these stuffs keep managed by the platform's own package manager, however, there is enough task to not have enough time to make an RPM/DEB package from every single stuff.

I recommend you to check https://packagecloud.io/ that supports both popular platforms, and provides a repository for them. The only thing you need is make a proper RPM Spec / DEB packaging scripts and build a package from your stuff. Also be sure you do not include unnecessary dependencies.