Undefined symbols for architecture x86_64 on OSX Big Sur

[phumke]

Hi tree-sitter-python,

I was able to develop using tree-sitter-python without any issues until I upgraded my computer to OSX Big Sur. Now I'm running into the following error:

Python 3.7.7 (default, Mar 10 2020, 16:11:21) 
[Clang 11.0.0 (clang-1100.0.33.12)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tree_sitter import Language
>>> Language.build_library('languages.so', ["tree-sitter-python"])
Undefined symbols for architecture x86_64:
  "std::__1::__vector_base_common<true>::__throw_length_error() const", referenced from:
      std::__1::vector<unsigned short, std::__1::allocator<unsigned short> >::__recommend(unsigned long) const in scanner.o
      std::__1::vector<(anonymous namespace)::Delimiter, std::__1::allocator<(anonymous namespace)::Delimiter> >::__recommend(unsigned long) const in scanner.o
  "std::logic_error::logic_error(char const*)", referenced from:
      std::length_error::length_error(char const*) in scanner.o
  "std::length_error::~length_error()", referenced from:
      std::__1::__throw_length_error(char const*) in scanner.o
  "std::terminate()", referenced from:
      ___clang_call_terminate in scanner.o
  "typeinfo for std::length_error", referenced from:
      std::__1::__throw_length_error(char const*) in scanner.o
  "vtable for std::length_error", referenced from:
      std::length_error::length_error(char const*) in scanner.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "operator delete(void*)", referenced from:
      _tree_sitter_python_external_scanner_create in scanner.o
      _tree_sitter_python_external_scanner_destroy in scanner.o
      std::__1::_DeallocateCaller::__do_call(void*) in scanner.o
  "operator new(unsigned long)", referenced from:
      _tree_sitter_python_external_scanner_create in scanner.o
      std::__1::__libcpp_allocate(unsigned long, unsigned long) in scanner.o
  "___cxa_allocate_exception", referenced from:
      std::__1::__throw_length_error(char const*) in scanner.o
  "___cxa_begin_catch", referenced from:
      ___clang_call_terminate in scanner.o
  "___cxa_call_unexpected", referenced from:
      std::__1::__vector_base<(anonymous namespace)::Delimiter, std::__1::allocator<(anonymous namespace)::Delimiter> >::__destruct_at_end((anonymous namespace)::Delimiter*) in scanner.o
      std::__1::allocator<(anonymous namespace)::Delimiter>::deallocate((anonymous namespace)::Delimiter*, unsigned long) in scanner.o
      std::__1::__vector_base<unsigned short, std::__1::allocator<unsigned short> >::__destruct_at_end(unsigned short*) in scanner.o
      std::__1::allocator<unsigned short>::deallocate(unsigned short*, unsigned long) in scanner.o
      std::__1::vector<(anonymous namespace)::Delimiter, std::__1::allocator<(anonymous namespace)::Delimiter> >::__destruct_at_end((anonymous namespace)::Delimiter*) in scanner.o
      std::__1::vector<unsigned short, std::__1::allocator<unsigned short> >::max_size() const in scanner.o
      std::__1::__split_buffer<unsigned short, std::__1::allocator<unsigned short>&>::__destruct_at_end(unsigned short*, std::__1::integral_constant<bool, false>) in scanner.o
      ...
  "___cxa_free_exception", referenced from:
      std::__1::__throw_length_error(char const*) in scanner.o
  "___cxa_throw", referenced from:
      std::__1::__throw_length_error(char const*) in scanner.o
  "___gxx_personality_v0", referenced from:
      _tree_sitter_python_external_scanner_create in scanner.o
      _tree_sitter_python_external_scanner_destroy in scanner.o
      (anonymous namespace)::Scanner::Scanner() in scanner.o
      std::__1::__vector_base<(anonymous namespace)::Delimiter, std::__1::allocator<(anonymous namespace)::Delimiter> >::__destruct_at_end((anonymous namespace)::Delimiter*) in scanner.o
      std::__1::allocator<(anonymous namespace)::Delimiter>::deallocate((anonymous namespace)::Delimiter*, unsigned long) in scanner.o
      std::__1::__vector_base<unsigned short, std::__1::allocator<unsigned short> >::__destruct_at_end(unsigned short*) in scanner.o
      std::__1::allocator<unsigned short>::deallocate(unsigned short*, unsigned long) in scanner.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/unixccompiler.py", line 204, in link
    self.spawn(linker + ld_args)
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python3.7/site-packages/tree_sitter/__init__.py", line 72, in build_library
    compiler.link_shared_object(object_paths, output_path)
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/ccompiler.py", line 717, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/System/Volumes/Data/export/apps/python/3.7.7/lib/python3.7/distutils/unixccompiler.py", line 206, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'cc' failed with exit status 1

To add to this mystery I've been able to build tree-sitter-java, tree-sitter-javascript, and tree-sitter-scala successfully, however also run into the error when attempting tree-sitter-cpp.

I've played around with different versions of python 3 (to see if clang was the issue) to no avail. Ex on python 3.6:

Python 3.6.10 (default, Jan 16 2020, 13:37:48) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tree_sitter import Language
>>> Language.build_library('languages.so', ["tree-sitter-python"])
Undefined symbols for architecture x86_64:
  "std::__1::__vector_base_common<true>::__throw_length_error() const", referenced from:
      std::__1::vector<unsigned short, std::__1::allocator<unsigned short> >::__recommend(unsigned long) const in scanner.o
      std::__1::vector<(anonymous namespace)::Delimiter, std::__1::allocator<(anonymous namespace)::Delimiter> >::__recommend(unsigned long) const in scanner.o
...

I've also tried different versions of tree-sitter and tree-sitter-python and seen the error consistently in every variation: tree-sitter 0.19.0 and tree-sitter-python 0.19.0, as well as tree-sitter 0.2.0 and the versions of tree-sitter-python 0.13.6 - 0.17.1.

I also have a linux machine and don't have any problems over there.

Do you know where this issue is coming from and how to get around it?

Thank you!

[phumke]

I wanted to ping again because I am still limping along using my linux machine as a workaround. Ideally I would like to use this on the mac since most of my fellow devs at work all use macs. I'm not sure if this is a problem within a language-binding issue or tree-sitter core specifically. @maxbrunsfeld do you happen to know where this problem is coming from?

Based on some research it sounds like switching from cc to cc++ for compilation may fix this issue, but I didn't want to muck around in compiler settings if there is a better option.

[maxbrunsfeld]

I think we should close this and you should subscribe to this issue instead: tree-sitter/py-tree-sitter#41. I would say that this is an issue with py-tree-sitter, and not this grammar in particular, because py-tree-sitter (specifically the build_library function that you mentioned) has logic for compiling and linking the tree-sitter parsers.

The symptom you're seeing is a bit different (in that on macOS, build_library itself is giving the error), so it may be a different problem, but they're both related to linking in the C++ standard library. You're having problems using parsers that require the C++ standard library.

[phumke]

In case anyone else runs across this issue, here is the resolution:

The issue boiled down to the recent OSX breaking how the ctypes.util.find_library command works in python, always returning None (example error, issue 1 in cpython, issue 2 in cypthon). The find_library command is used within py-tree-sitter in order to see if the c++ or stdc++ libraries are available and loading one of them (see discussion). Since None is always returned, the necessary libraries are never loaded. However, the actual load_library works. I did not have access to the patched versions of python so monkey patching around the load_library to just return anything is enough to get this system back up and running.

Here is my monkey patch, which is a variation of the patch listed here:

import sys
import ctypes.util

if sys.platform == 'darwin':
    real_find_library = ctypes.util.find_library

    def find_library(name):
        if name in {"c++", "stdc++"}:
            return f"/usr/lib/lib{name}.dylib"
        return real_find_library(name)
    ctypes.util.find_library = find_library

Language.build_library('languages.so', languages)

[maxbrunsfeld]

Wow, thanks for sharing your findings!

[phumke]

My pleasure! This looks like an issue that will eventually resolve itself (at least in python 3.8/3.9), so this is hopefully just a temporary workaround in the meantime.