Opened 15 months ago

Closed 3 months ago

Last modified 3 months ago

#30534 closed enhancement (invalid)

Repackage pynac as a pip-installable package

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: packages: standard Keywords: pynac, sd110, sd111
Cc: rws, slelievre, gh-tobiasdiez, was, fbissey, tscrim, gh-kliem, vdelecroix, embray Merged in:
Authors: Reviewers: François Bissey
Report Upstream: Reported upstream. No feedback yet. Work issues:
Branch: Commit:
Dependencies: #30446, #31064 Stopgaps:

Status badges

Description (last modified by mkoeppe)

Pynac has a compile-time dependency on the Python library but is not installed using Python package infrastructure. This is problematic because Python users cannot install it using standard Python tools - for example for testing different Python versions.

It would make sense for src/sage/libs/pynac/pynac.pxd and pynac_wrap.h to be shipped with this Python package, introducing a dependency on Cython. (The only thing blocking this is from sage.libs.gmp.types cimport mpz_t, mpq_t, mpz_ptr, mpq_ptr - but that can be fixed by some minimal cut&paste.)

(The other parts of src/sage/libs/pynac, constant.pxd, constant.pyx, pynac.pyx use various imports from sage and must remain in sage.libs.pynac)

Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org: https://pypi.org/project/Pynac

Attachments (1)

py-ginac-0.7.27.tar.gz (576.6 KB) - added by mkoeppe 9 months ago.

Download all attachments as: .zip

Change History (77)

comment:1 Changed 15 months ago by slelievre

  • Cc rws slelievre added
  • Description modified (diff)
  • Keywords pynac added

Cc-ing Ralf Stephan who maintains Pynac.

comment:2 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:3 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 15 months ago by slelievre

Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org:

https://pypi.org/project/Pynac

Collision between

  • Pynac = "Py-Dynac" = Python bindings for the Dynac particle tracking code
  • Pynac = "Py-GiNaC" = Python derivative of the C++ library GiNaC

Possible solutions:

  • rename these projects to Py-Dynac and Py-GiNaC, or Pydynac and Pyginac
  • no renaming but use the name pyginac on PyPI for "Py-GiNaC"

Note:

  • "Py-Dynac" got the pynac name on PyPI
  • "Py-GiNaC" got the pynac.org domain name

Regardless of how we handle getting "Py-GiNaC" on PyPI, a disambiguation note in both "Pynac" projects would be nice, with links to each other.

I will propose changes via pull requests or other to add such a note in relevant places.

comment:5 Changed 15 months ago by slelievre

Or use pynac-ginac as the PyPI name (more discoverable?).

comment:6 Changed 14 months ago by mkoeppe

  • Cc gh-tobiasdiez added

comment:7 Changed 13 months ago by mkoeppe

  • Cc was added
  • Keywords sd110 added

We briefly discussed it at sd110.

From William Stein (CoCalc) to Everyone: (1:37 PM)

py-ginac is pretty good.

comment:8 Changed 13 months ago by fbissey

  • Cc fbissey added

comment:9 follow-up: Changed 13 months ago by chapoton

There is already some py_ginac somewhere :

http://moebinv.sourceforge.net/pyGiNaC.html

comment:10 in reply to: ↑ 9 Changed 13 months ago by mkoeppe

Replying to chapoton:

There is already some py_ginac somewhere :

http://moebinv.sourceforge.net/pyGiNaC.html

Thanks for the pointer. This appears to be an active project. From https://sourceforge.net/projects/pyginac.moebinv.p/:

"This is a refresh and maintained version of the previous stalled project https://sourceforge.net/projects/pyginac/ It works with the current version of GiNaC."

The source (git) is at https://sourceforge.net/p/moebinv/pyginac/code/ci/master/tree/

It provides the Python packages cginac and ginac.

comment:11 Changed 13 months ago by mkoeppe

It calls itself "pyGiNaC" but it does not define a distribution of this name. It must be built using scons and is not pip-installable - so it also does not claim a project name on PyPI.

comment:12 Changed 13 months ago by mkoeppe

  • Report Upstream changed from N/A to Reported upstream. No feedback yet.

comment:13 Changed 12 months ago by mkoeppe

  • Keywords sd111 added

comment:14 Changed 10 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:16 Changed 9 months ago by mkoeppe

  • Dependencies set to #30446

comment:17 Changed 9 months ago by mkoeppe

  • Branch set to u/mkoeppe/repackage_pynac_as_a_pip_installable_package

comment:18 Changed 9 months ago by git

  • Commit set to 44e4dd88ade4ef49f9dde054b3d75f4b57a4956a

Branch pushed to git repo; I updated commit sha1. New commits:

44e4dd8Fixup

comment:19 Changed 9 months ago by mkoeppe

  • Authors set to Matthias Koeppe, ...

The new package installs correctly. sage.libs.pynac needs more work.

comment:20 Changed 9 months ago by mkoeppe

  • Cc tscrim added

comment:21 Changed 9 months ago by mkoeppe

  • Cc gh-kliem vdelecroix added

I think I'll need some help here from people who know about import and cimport in Cython...

comment:22 Changed 9 months ago by tscrim

With what aspects? Just when to use one versus the other?

comment:23 Changed 9 months ago by mkoeppe

I am getting runtime linker errors like the following on this branch:

[dochtml]     import sage.libs.pynac.pynac # initialize pynac before .ring
[dochtml] ImportError: dlopen(/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so, 2): Symbol not found: __ZN5GiNaC12library_initD1Ev
[dochtml]   Referenced from: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so
[dochtml]   Expected in: flat namespace
[dochtml]  in /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so

comment:24 Changed 9 months ago by tscrim

  • Cc embray added

That is beyond my knowledge as it has something to do with how the packages are linked. Perhaps looking at how p_group_cohomology does its setup might have a clue?

Erik, do you know about these things or know a good person to ask?

comment:25 Changed 9 months ago by fbissey

The double leading underscore is very strange. If you remove one it becomes simply

$ c++filt -n _ZN5GiNaC12library_initD1Ev
GiNaC::library_init::~library_init()

but with two, it is nothing. Some naming or interface is not set right.

comment:26 follow-up: Changed 9 months ago by gh-kliem

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

comment:27 Changed 9 months ago by gh-kliem

And something isn't right about pynac/setup.py:

(sage-sh) kliem@cofio:sage$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ginac.pynac import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ginac.pynac'
>>> from ginac.libpynac import *
Segmentation fault

comment:28 Changed 9 months ago by gh-kliem

I even get segementation fault in ipython when it tries to load the modules of ginac.

comment:29 follow-up: Changed 9 months ago by gh-kliem

This part in numeric.cpp leads to a segmentation fault when importing the module:

3321 #ifdef PYNAC_HAVE_LIBGIAC
3322 giac::gen* numeric::to_giacgen(giac::context* cptr) const
3323 {
3324         if (t == LONG)
3325                 return new giac::gen(v._long);
3326 +----  7 lines: if (t == MPZ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3333 +---- 10 lines: if (t == MPQ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3343         else
3344                 return nullptr;
3345 }
3346 #endif

giac might not be linked correctly, but I don't know. Also ./bootstrap is missing from setup.py. Just running ./configure isn't working.

comment:30 follow-up: Changed 9 months ago by gh-kliem

Also I would guess that you need to include the header files as depends=....

What is definitely missing is gmptypes. Also it seems weird that you name the module ginac.libpynac but then you cimport from ginac.pynac import * in pynac.pyx. This isn't defined.

Anyway, if I know just ./bootstrap; make pynac and then include the folder ginac, then I'm back to your orignal problem.

First I got:

ImportError: /home/mi/kliem/Applications/pynac/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZTVN5GiNaC9containerISt6vectorEE

This I made go away by declaring things static in pynac_wrap.h. Then I get the original error about library_init.

The class library_init is defined in ex.h. However, the methods are then defined in utils.cpp.

But even if I change this, the error doesn't go ayay. By now, I don't understand anything anymore.

comment:31 Changed 9 months ago by fbissey

I noticed while building the package that giac is automagically detected and used. Current sage package explicitly disable it. I remember chatting with Ralph when he made it available and that it was currently broken. So, first thing will be to disable it.

comment:32 Changed 9 months ago by mkoeppe

See https://github.com/pynac/pynac/pull/372 "configure: Change default to --with-giac=no"

comment:33 Changed 9 months ago by fbissey

I pulled on your python_package branch, I guess I should have used the attached tarball.

comment:34 Changed 9 months ago by mkoeppe

I've just the branch into python_package and will also prepare a new tarball

comment:35 in reply to: ↑ 30 Changed 9 months ago by mkoeppe

Replying to gh-kliem:

Also it seems weird that you name the module ginac.libpynac but then you cimport from ginac.pynac import * in pynac.pyx. This isn't defined.

Thanks, renamed now

comment:36 Changed 9 months ago by git

  • Commit changed from 44e4dd88ade4ef49f9dde054b3d75f4b57a4956a to 72da71ac5f262afae58655ceccc2da30954e2175

Branch pushed to git repo; I updated commit sha1. New commits:

0a16129WIP
72da71apynac 0.7.27.p16

comment:37 Changed 9 months ago by fbissey

OK, so pulling from your branch and building with my system packages and tools on gentoo with python setup.py install --prefix=$someprefix I cannot see anything obviously wrong with what is installed. ldd -r pynac.cpython-39-x86_64-linux-gnu.so just report what I am expecting to be missing, that is symbols from python itself.

I am not currently setup for compiling on OS X, which may have trickier issue at linking.

comment:38 in reply to: ↑ 29 Changed 9 months ago by mkoeppe

Replying to gh-kliem:

Also ./bootstrap is missing from setup.py. Just running ./configure isn't working.

I've now added some instructions to the README

comment:39 Changed 9 months ago by gh-kliem

https://github.com/kliem/pynac/tree/python_package

I did some experiments. Not all changes in this branch are necessary to get things to work. In particular, I did many experiments trying to avoid this segmentation fault, but I couldn't find something. It's ipython specific. Sage just complains about linkage.

As reported above I get a weird segmentation fault, whenever I import ginac.pynac.

It goes away magically, if I import ginac.gmptypes beforehand. I don't know why.

Getting sage to start still does not work, because

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

I don't know why it would be undefined. It is sage specific. In an Ipython session it is fine even if I add something that uses it to `ginac/pynac.pyx'

+cdef py_gcd(n, k):
+    return n*k
+
+
+def init_function_table():
+    py_funcs.py_gcd = &py_gcd
+

If I pip install this, I can run this function just fine in ipython:

(sage-sh) kliem@cofio:pynac$ ipython
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ginac import gmptypes                                                                                                                                                                                                                                             

In [2]: from ginac.pynac import *                                                                                                                                                                                                                                              

In [3]: init_function_table()  

What is even more strange about this segementation fault is that in sage on develop branch I can run import os beforehand to fix this. This does not work in ipython.

comment:40 follow-up: Changed 9 months ago by fbissey

Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  # (at your option) any later version.
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from cpython cimport *
  from ginac.pynac cimport *
  ^
  ------------------------------------------------------------

  ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      defined by GiNaC. This allows us to prevent collisions with C++ level
      special functions when a user asks to construct a symbolic function
      with the same name.
      """
      global GINAC_FN_SERIAL
      GINAC_FN_SERIAL = g_registered_functions().size()
                       ^
  ------------------------------------------------------------

  ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions
  Compiling ginac/pynac.pyx because it changed.
  [1/1] Cythonizing ginac/pynac.pyx

comment:41 Changed 9 months ago by git

  • Commit changed from 72da71ac5f262afae58655ceccc2da30954e2175 to f24d6466798713a884877b0fa4afb1c6b20d29fb

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

cea4cd5ci-cygwin-standard.yml: More stages, continue-on-error: true
cf31b79Fixup after cherry-pick
073124cMerge branch 't/31084/makefile__add__ptest__targets_that_do_not_depend_on_the_docbuild' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
8769bd6.github/workflows/ci-cygwin-*.yml: Separate docbuild and ptest
3826222Fix up stage iv
598b0c2local-cygwin-choco: Do not pass through PATH, use full pathname of choco instead
d65299cMerge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
a5e4051Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
42f4458Merge tag '9.3.beta7' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
f24d646Merge #31064

comment:42 Changed 9 months ago by mkoeppe

  • Dependencies changed from #30446 to #30446, #31064

comment:43 Changed 9 months ago by mkoeppe

I have pushed a few more changes to the pynac branch and have set up testing against this ticket - running at https://github.com/mkoeppe/pynac/runs/2010476345

comment:44 in reply to: ↑ 40 ; follow-up: Changed 9 months ago by gh-kliem

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

After adding it, the build runs fine. Sage doens't start with the same stupid error:

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

The segmentation fault in ipython went away for an ImportError:

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4giac25i_lex_is_strictly_greaterERKNS_7index_mES2_

Replying to fbissey:

Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  # (at your option) any later version.
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from cpython cimport *
  from ginac.pynac cimport *
  ^
  ------------------------------------------------------------

  ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      defined by GiNaC. This allows us to prevent collisions with C++ level
      special functions when a user asks to construct a symbolic function
      with the same name.
      """
      global GINAC_FN_SERIAL
      GINAC_FN_SERIAL = g_registered_functions().size()
                       ^
  ------------------------------------------------------------

  ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions
  Compiling ginac/pynac.pyx because it changed.
  [1/1] Cythonizing ginac/pynac.pyx

comment:45 in reply to: ↑ 26 ; follow-up: Changed 9 months ago by gh-kliem

Replying to gh-kliem:

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

Has anyone experienced this? This seems to be a bug in our build system?

comment:46 in reply to: ↑ 44 ; follow-up: Changed 9 months ago by mkoeppe

Replying to gh-kliem:

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

pynac.pxd is generated by setup.py from pynac.pxd.in via the configure script. Installation of the new package can be seen to succeed in https://github.com/mkoeppe/pynac/runs/2010476345

comment:47 in reply to: ↑ 45 Changed 9 months ago by mkoeppe

Replying to gh-kliem:

Replying to gh-kliem:

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

Has anyone experienced this? This seems to be a bug in our build system?

Yes, quite possible that the install cleaner only handles Python modules and extensions but not package data.

comment:48 Changed 9 months ago by git

  • Commit changed from f24d6466798713a884877b0fa4afb1c6b20d29fb to c0805f54143cab58d56e351c19cb5b28944f0148

Branch pushed to git repo; I updated commit sha1. New commits:

c0805f5build/pkgs/pynac: Use 9c89d8bbc723456b2d94efb78ca1df1820694ff0

comment:49 Changed 9 months ago by mkoeppe

Just packaged up current HEAD of my pynac python_package branch.

Changed 9 months ago by mkoeppe

comment:50 Changed 9 months ago by git

  • Commit changed from c0805f54143cab58d56e351c19cb5b28944f0148 to 93fe11f753cff24f7d81f00b91ec393bcbfc1721

Branch pushed to git repo; I updated commit sha1. New commits:

93fe11fbuild/pkgs/pynac: Use 5701911df8f00c99bc7c7ed676360729200fb726

comment:51 Changed 9 months ago by mkoeppe

(and another time.)

comment:52 in reply to: ↑ 46 Changed 9 months ago by gh-kliem

This is fixed with the new attachment. Before there was a version where ginac/pynac.pxd was missing from configure for some reason.

Replying to mkoeppe:

Replying to gh-kliem:

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

pynac.pxd is generated by setup.py from pynac.pxd.in via the configure script. Installation of the new package can be seen to succeed in https://github.com/mkoeppe/pynac/runs/2010476345

comment:53 Changed 9 months ago by gh-kliem

With the new branch I can now do from ginac.pynac import * without an error raising in ipython. The ImportError when loading sage itself still persists.

comment:54 Changed 9 months ago by gh-kliem

So maybe this module needs to include something on runtime and this doesn't work. So from ginac.pynac import * works fine, but the cimport doesn't.

In my experiments, the os.path.dirname(__file__) in the setup file didn't do anything. Maybe the include_dirs are set up wrong?

comment:55 Changed 9 months ago by fbissey

OK, tried with all the new stuff. Stopped at the documentation again but different symbol than in comment:23

[dochtml] ImportError: /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

which is

$ c++filt -n _ZN5GiNaC8py_funcsE
GiNaC::py_funcs

comment:56 Changed 9 months ago by fbissey

Running ldd, removing python calls and going through c++filt -n the following symbols are undefined and should be replaced by calls since you cannot link to the python module

ldd -r /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so | grep _Z | c++filt -n
undefined symbol: GiNaC::py_funcs       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::container<std::vector>      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::numeric   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::symbol    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::~library_init()  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::container<std::vector>::tinfo_static   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::constant    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::I      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::_num0_bp       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::pseries   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::basic     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&)   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::library_init()   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Float(_object*)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, std::vector<GiNaC::ex, std::allocator<GiNaC::ex> > const&, bool)      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_I(_object*)      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: std::vector<GiNaC::ex, std::allocator<GiNaC::ex> >::~vector() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::decide() const     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&, GiNaC::ex const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::sorted_op(unsigned long) const     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_basic(GiNaC::basic const&)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::pseries::is_terminating() const        (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int)       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::registered_functions()       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::compare(GiNaC::ex const&) const    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::numeric::to_pyobject() const   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::the_operator() const       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&)     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_int(int)    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::constant::constant()   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::hold() const    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Integer(_object*)        (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::operator=(GiNaC::basic const&)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)

comment:57 follow-up: Changed 9 months ago by mkoeppe

Looks like trying to pull in everything to sage.libs.pynac by from ginac.pynac cimport * is a mistake and instead all modules that need pynac should cimport more specifically from ginac.pynac

comment:58 in reply to: ↑ 57 Changed 9 months ago by gh-kliem

Replying to mkoeppe:

Looks like trying to pull in everything to sage.libs.pynac by from ginac.pynac cimport * is a mistake and instead all modules that need pynac should cimport more specifically from ginac.pynac

But why does

sage: cython(''' 
....: from sage.libs.pynac.pynac cimport * 
....: ''')    

works just fine on develop? I would say that things should work just fine, once everything is set up correctly. However, I don't know how to resolve the problems as I'm currently not even able to cimport from my own little two files project.

comment:59 Changed 9 months ago by gh-kliem

I might have found the problem:

In setup.py:

-       package_data = {'ginac': ['*.pxd', '*.h', '*.hpp']},
+       package_data = {'ginac': ['*.pxd', '*.h', '*.hpp', '*.cpp']},

This will place the `.cpp` files in the correct directory. We need them, because when we cimport we recompile some of them. When the `.cpp` files are not there, we get the missing symbols.

After doing this, I get a new error:

{{{
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/constant.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/constant.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/ring.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/ring.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/function.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/function.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/pynac.o -L/srv/public/kliem/sage/local/lib -lgsl -lflint -lgmp -lm -lopenblas -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/expression.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/expression.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources -lpari
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] error: command 'g++' failed with exit status 1
}}}

But I think, this might be easier to resolve.
Version 0, edited 9 months ago by gh-kliem (next)

comment:60 follow-up: Changed 9 months ago by mkoeppe

This can't be the right fix. The .cpp files should not be installed.

comment:61 Changed 9 months ago by mkoeppe

Where's that -lginac coming from??

comment:62 follow-up: Changed 9 months ago by git

  • Commit changed from 93fe11f753cff24f7d81f00b91ec393bcbfc1721 to b4da02f2732e16a646c097c67c06c80686ba391f

Branch pushed to git repo; I updated commit sha1. New commits:

b4da02fcimport directly from ginac.pynac

comment:63 in reply to: ↑ 62 Changed 9 months ago by gh-kliem

Same problem. Same undefined symbol.

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

b4da02fcimport directly from ginac.pynac

comment:64 in reply to: ↑ 60 ; follow-up: Changed 9 months ago by gh-kliem

Replying to mkoeppe:

This can't be the right fix. The .cpp files should not be installed.

When you cimport you recompile .pxd file. And by this it is trying to recompile everything that was included there. If the corresponding .cpp files aren't found, there is a problem with that.

If we don't want to recompile everything, then we should put the cdef extern from into pynac.pyx and only do the type definitions in pynax.pxd.

comment:65 in reply to: ↑ 64 Changed 9 months ago by mkoeppe

Replying to gh-kliem:

we should put the cdef extern from into pynac.pyx and only do the type definitions in pynax.pxd.

I don't know about this mechanism, but it sounds like what I am looking for. Could you elaborate on this or push a change to the branch?

comment:66 Changed 9 months ago by gh-kliem

I'll check, if that fixes things.

comment:67 Changed 9 months ago by gh-kliem

I don't know, if that will work. I get many errors and by no means do I know how to fix them.

E.g.

[pynac-0.7.28.p18]   gcc -DNDEBUG -g -fwrapv -O2 -Wall -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iginac -I./ginac -I. -I. -I/srv/public/kliem/sage/local/lib/python3.7/site-packages/pip/_vendor/pep517 -I/srv/public/kliem/sage/local -I/usr/lib/python37.zip -I/usr/lib/python3.7 -I/usr/lib/python3.7/lib-dynload -I/srv/public/kliem/sage/local/lib/python3.7/site-packages -I/srv/public/kliem/sage/local/include -I/usr/include/python3.7m -c ginac/pynac.cpp -o build/temp.linux-x86_64-3.7/ginac/pynac.o -std=c++11 -DSING_NDEBUG -DOM_NDEBUG -DSING_NDEBUG -DOM_NDEBUG -I/srv/public/kliem/sage/local/include
[pynac-0.7.28.p18]   ginac/pynac.cpp:1875:50: error: no declaration matches '__pyx_t_5ginac_5pynac_GFunctionOptVector& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18]    static __pyx_t_5ginac_5pynac_GFunctionOptVector &GiNaC::function::registered_functions(void); /*proto*/
[pynac-0.7.28.p18]                                                     ^~~~~
[pynac-0.7.28.p18]   In file included from ginac/ginac.h:54,
[pynac-0.7.28.p18]                    from ginac/pynac_wrap.h:13,
[pynac-0.7.28.p18]                    from ginac/pynac.cpp:786:
[pynac-0.7.28.p18]   ginac/function.h:409:41: note: candidate is: 'static std::vector<GiNaC::function_options>& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18]     static std::vector<function_options> & registered_functions();
[pynac-0.7.28.p18]                                            ^~~~~~~~~~~~~~~~~~~~
[pynac-0.7.28.p18]   ginac/function.h:340:7: note: 'class GiNaC::function' defined here
[pynac-0.7.28.p18]    class function : public exprseq

I don't even know if it is possible to import a cppclass in the .pyx part and to define in in the header then.

This option is definitely not mentioned here: https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html

The problem with this entire setup is that ginac was never properly compiled and linked as a C++ module. I have the feeling that you are trying to achieve something that cython doesn't want you to do.

  • If you have a C++ library you can create a cython header for this and directly expose the cpp classes via that header (e.g. you can create a pip installable GMP header that uses an existing GMP installation).
  • If you have some C++ files and you want to expose them directly by a cython wrapper, but the header of this wrapper will recompile your files and thus every module cimporting from this wrapper needs to be able to recompile this as well.
  • You can create a proper wrapper class of a cpp class. This class can be used by other modules without recompiling the actual cpp class.

This is what the picture looks like to me.

comment:68 Changed 5 months ago by gh-kliem

I digged into a bit. Not that I know a solution now, but at least I understand somehow what is going on.

The question is really what we want. cdef extern from in a .pxd-file assumes that every file cimporting this is linked against the same library or has access to the same sources. I don't think we can change this. So the way this is we must either compile ginac as a library and link against it or ship the sources (the later being really awful, because we have plenty of duplications and it takes forever to compile).

You can use cdef extern in the pyx however and expose the definitions in the header. This seems a very clean way of doing it, but there are tons of obstructions along the way. E.g. the functions must be declared static: https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#implementing-functions-in-c

comment:69 Changed 5 months ago by gh-kliem

And apparently with a cppclass it works even in the header? (Different project, but there it seemed to work.)

I'm confused. I will see, if I can find out, what is causing the problem.

comment:70 Changed 5 months ago by mkoeppe

Thanks for looking into this!

comment:71 Changed 4 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:72 Changed 3 months ago by mkoeppe

  • Authors Matthias Koeppe, ... deleted
  • Milestone changed from sage-9.5 to sage-duplicate/invalid/wontfix

New approach: #32386 (Merge pynac as src/sage/symbolics/ginac)

comment:73 Changed 3 months ago by mkoeppe

  • Status changed from new to needs_review

comment:74 Changed 3 months ago by fbissey

  • Reviewers set to François Bissey
  • Status changed from needs_review to positive_review

Sure, let's try the other way.

comment:75 Changed 3 months ago by mkoeppe

  • Resolution set to invalid
  • Status changed from positive_review to closed

comment:76 Changed 3 months ago by gh-kliem

  • Branch u/mkoeppe/repackage_pynac_as_a_pip_installable_package deleted
  • Commit b4da02f2732e16a646c097c67c06c80686ba391f deleted

Removing the branch to avoid confusion in the future.

Note: See TracTickets for help on using tickets.