Allow installation of embedPy alongside PyQ

[abalkin]

Currently, both PyQ and embedPy install p.so in QHOME and this name conflict means that users cannot use both in the same kdb+ instance.

Why would users want to have both PyQ and embedPy? While the two projects provide some overlapping functionality, PyQ is primarily focused on accessing kdb+ data from Python while embedPy has more features for q programers to call Python functions.

Implementation plan

• rename p.so in PyQ to pyq.so
• provide means of (optionally?) installing embedPy components into a PyQ environment, e.g, pip install pyq[embedpy]

[abalkin]

@jhanna-kx, @effbiae - what do you think of this plan? Do you think having emdedPy pip installable independently from PyQ is something of interest?

[abalkin]

Can pip compile (c)source modules?

Yes, but it relies on distutils/setuptools and the logic in the setup.py file to achieve that. Try

pip install -v --no-binary pyq pyq

to watch how p.so is compiled in pyq.

can pip give us more config info about the python environment?

No. Pip is a package manager, not a build tool. In Python 3, distutils gives you full access to everything in the python's own (autoconf-generated) Makefile, which is likely more than you would ever want to know.

On the other hand, I am currently experimenting with conda (the package manager for the Anaconda Python distribution) and I think it deserves a serious consideration. Apart from being very popular among data scientists, conda has the following advantages over pip:

1. Support for non-python packages. For example, it installs its own ffi and blas libs.
2. Binary relocation support. Conda knows how to patch RPATH on Linux and how to do equivalent name tool magic on macOS.
3. I did not investigate this aspect, but conda is likely to be more friendly to commercial software than pip. AFAIK, it has the concept of "channels" that allow publishers control what different classes of users can install.

/cc: @effbiae

[abalkin]

pip install pyq installs embedPy if its requirements are met: Python 3 with numpy.

[TKTheTechie]

Hi, I'm trying to run pyq alongside embedPy but I get the following error:

'/opt/miniconda3/envs/kdb/q/l64/./p.so: undefined symbol: init
  [2]  /opt/miniconda3/envs/kdb/q/p.q:6:
 `L`M`H set'@[system"python3 ",;c;{system"python ",c}];if[count M;if[k~key k:`$":",M;L::M]];
 .p:(`:./p 2:(`init;2))[L;H]]
           ^
  [0]  (<load>)

  )^C'stop
  [3]  (.Q.dr)

I'm not clear if this issue has been resolved. If has, could you provide steps on how to get both these items to be working together?

[sashkab]

@thomas-kunnumpurath please open new issue.