Repackage pynac as a pip-installable package

[mkoeppe]

Pynac has a compile-time dependency on the Python library but is not installed using Python package infrastructure. This is problematic because Python users cannot install it using standard Python tools - for example for testing different Python versions.

• ​Pynac home
• ​Pynac repo

It would make sense for src/sage/libs/pynac/pynac.pxd and pynac_wrap.h to be shipped with this Python package, introducing a dependency on Cython. (The only thing blocking this is from sage.libs.gmp.types cimport mpz_t, mpq_t, mpz_ptr, mpq_ptr - but that can be fixed by some minimal cut&paste.)

(The other parts of src/sage/libs/pynac, constant.pxd, constant.pyx, pynac.pyx use various imports from sage and must remain in sage.libs.pynac)

Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org: ​https://pypi.org/project/Pynac

[slelievre]

Cc-ing Ralf Stephan who maintains Pynac.

[slelievre]

Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org:

​​https://pypi.org/project/Pynac

Collision between

• Pynac = "Py-Dynac" = Python bindings for the Dynac particle tracking code
• Pynac = "Py-GiNaC" = Python derivative of the C++ library GiNaC

Possible solutions:

• rename these projects to Py-Dynac and Py-GiNaC, or Pydynac and Pyginac
• no renaming but use the name pyginac on PyPI for "Py-GiNaC"

Note:

• "Py-Dynac" got the pynac name on PyPI
• "Py-GiNaC" got the pynac.org domain name

Regardless of how we handle getting "Py-GiNaC" on PyPI, a disambiguation note in both "Pynac" projects would be nice, with links to each other.

I will propose changes via pull requests or other to add such a note in relevant places.

[slelievre]

Or use pynac-ginac as the PyPI name (more discoverable?).

[mkoeppe]

We briefly discussed it at sd110.

From William Stein (CoCalc) to Everyone: (1:37 PM)

py-ginac is pretty good.

[chapoton]

There is already some py_ginac somewhere :

​http://moebinv.sourceforge.net/pyGiNaC.html

[mkoeppe]

Replying to chapoton:

There is already some py_ginac somewhere :

​http://moebinv.sourceforge.net/pyGiNaC.html

Thanks for the pointer. This appears to be an active project. From ​https://sourceforge.net/projects/pyginac.moebinv.p/:

"This is a refresh and maintained version of the previous stalled project ​https://sourceforge.net/projects/pyginac/ It works with the current version of GiNaC."

The source (git) is at ​https://sourceforge.net/p/moebinv/pyginac/code/ci/master/tree/

It provides the Python packages cginac and ginac.

[mkoeppe]

It calls itself "pyGiNaC" but it does not define a distribution of this name. It must be built using scons and is not pip-installable - so it also does not claim a project name on PyPI.

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

​https://github.com/pynac/pynac/pull/371

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​44e4dd8

Fixup

[mkoeppe]

The new package installs correctly. sage.libs.pynac needs more work.

[mkoeppe]

I think I'll need some help here from people who know about import and cimport in Cython...

[tscrim]

With what aspects? Just when to use one versus the other?

[mkoeppe]

I am getting runtime linker errors like the following on this branch:

[dochtml]     import sage.libs.pynac.pynac # initialize pynac before .ring
[dochtml] ImportError: dlopen(/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so, 2): Symbol not found: __ZN5GiNaC12library_initD1Ev
[dochtml]   Referenced from: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so
[dochtml]   Expected in: flat namespace
[dochtml]  in /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so

[tscrim]

That is beyond my knowledge as it has something to do with how the packages are linked. Perhaps looking at how ​p_group_cohomology does its setup might have a clue?

Erik, do you know about these things or know a good person to ask?

[fbissey]

The double leading underscore is very strange. If you remove one it becomes simply

$ c++filt -n _ZN5GiNaC12library_initD1Ev
GiNaC::library_init::~library_init()

but with two, it is nothing. Some naming or interface is not set right.

[gh-kliem]

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

[gh-kliem]

And something isn't right about pynac/setup.py:

(sage-sh) kliem@cofio:sage$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ginac.pynac import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ginac.pynac'
>>> from ginac.libpynac import *
Segmentation fault

[gh-kliem]

I even get segementation fault in ipython when it tries to load the modules of ginac.

[gh-kliem]

This part in numeric.cpp leads to a segmentation fault when importing the module:

3321 #ifdef PYNAC_HAVE_LIBGIAC
3322 giac::gen* numeric::to_giacgen(giac::context* cptr) const
3323 {
3324         if (t == LONG)
3325                 return new giac::gen(v._long);
3326 +----  7 lines: if (t == MPZ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3333 +---- 10 lines: if (t == MPQ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3343         else
3344                 return nullptr;
3345 }
3346 #endif

giac might not be linked correctly, but I don't know. Also ./bootstrap is missing from setup.py. Just running ./configure isn't working.

[gh-kliem]

Also I would guess that you need to include the header files as depends=....

What is definitely missing is gmptypes. Also it seems weird that you name the module ginac.libpynac but then you cimport from ginac.pynac import * in pynac.pyx. This isn't defined.

Anyway, if I know just ./bootstrap; make pynac and then include the folder ginac, then I'm back to your orignal problem.

First I got:

ImportError: /home/mi/kliem/Applications/pynac/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZTVN5GiNaC9containerISt6vectorEE

This I made go away by declaring things static in pynac_wrap.h. Then I get the original error about library_init.

The class library_init is defined in ex.h. However, the methods are then defined in utils.cpp.

But even if I change this, the error doesn't go ayay. By now, I don't understand anything anymore.

[fbissey]

I noticed while building the package that giac is automagically detected and used. Current sage package explicitly disable it. I remember chatting with Ralph when he made it available and that it was currently broken. So, first thing will be to disable it.

[mkoeppe]

See ​https://github.com/pynac/pynac/pull/372 "configure: Change default to --with-giac=no"

[fbissey]

I pulled on your python_package branch, I guess I should have used the attached tarball.

[mkoeppe]

I've just the branch into python_package and will also prepare a new tarball

[mkoeppe]

Replying to gh-kliem:

Also it seems weird that you name the module ginac.libpynac but then you cimport from ginac.pynac import * in pynac.pyx. This isn't defined.

Thanks, renamed now

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0a16129	WIP
​72da71a	pynac 0.7.27.p16

[fbissey]

OK, so pulling from your branch and building with my system packages and tools on gentoo with python setup.py install --prefix=$someprefix I cannot see anything obviously wrong with what is installed. ldd -r pynac.cpython-39-x86_64-linux-gnu.so just report what I am expecting to be missing, that is symbols from python itself.

I am not currently setup for compiling on OS X, which may have trickier issue at linking.

[mkoeppe]

Replying to gh-kliem:

Also ./bootstrap is missing from setup.py. Just running ./configure isn't working.

I've now added some instructions to the README

[gh-kliem]

​https://github.com/kliem/pynac/tree/python_package

I did some experiments. Not all changes in this branch are necessary to get things to work. In particular, I did many experiments trying to avoid this segmentation fault, but I couldn't find something. It's ipython specific. Sage just complains about linkage.

As reported above I get a weird segmentation fault, whenever I import ginac.pynac.

It goes away magically, if I import ginac.gmptypes beforehand. I don't know why.

Getting sage to start still does not work, because

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

I don't know why it would be undefined. It is sage specific. In an Ipython session it is fine even if I add something that uses it to `ginac/pynac.pyx'

+cdef py_gcd(n, k):
+    return n*k
+
+
+def init_function_table():
+    py_funcs.py_gcd = &py_gcd
+

If I pip install this, I can run this function just fine in ipython:

(sage-sh) kliem@cofio:pynac$ ipython
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ginac import gmptypes                                                                                                                                                                                                                                             

In [2]: from ginac.pynac import *                                                                                                                                                                                                                                              

In [3]: init_function_table()  

What is even more strange about this segementation fault is that in sage on develop branch I can run import os beforehand to fix this. This does not work in ipython.

[fbissey]

Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  # (at your option) any later version.
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from cpython cimport *
  from ginac.pynac cimport *
  ^
  ------------------------------------------------------------

  ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      defined by GiNaC. This allows us to prevent collisions with C++ level
      special functions when a user asks to construct a symbolic function
      with the same name.
      """
      global GINAC_FN_SERIAL
      GINAC_FN_SERIAL = g_registered_functions().size()
                       ^
  ------------------------------------------------------------

  ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions
  Compiling ginac/pynac.pyx because it changed.
  [1/1] Cythonizing ginac/pynac.pyx

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​cea4cd5	ci-cygwin-standard.yml: More stages, continue-on-error: true
​cf31b79	Fixup after cherry-pick
​073124c	Merge branch 't/31084/makefile__add__ptest__targets_that_do_not_depend_on_the_docbuild' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​8769bd6	.github/workflows/ci-cygwin-*.yml: Separate docbuild and ptest
​3826222	Fix up stage iv
​598b0c2	local-cygwin-choco: Do not pass through PATH, use full pathname of choco instead
​d65299c	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​a5e4051	Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​42f4458	Merge tag '9.3.beta7' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​f24d646	Merge #31064

[mkoeppe]

I have pushed a few more changes to the pynac branch and have set up testing against this ticket - running at ​https://github.com/mkoeppe/pynac/runs/2010476345

[gh-kliem]

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

After adding it, the build runs fine. Sage doens't start with the same stupid error:

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

The segmentation fault in ipython went away for an ImportError:

ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4giac25i_lex_is_strictly_greaterERKNS_7index_mES2_

Replying to fbissey:

Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  # (at your option) any later version.
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from cpython cimport *
  from ginac.pynac cimport *
  ^
  ------------------------------------------------------------

  ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      defined by GiNaC. This allows us to prevent collisions with C++ level
      special functions when a user asks to construct a symbolic function
      with the same name.
      """
      global GINAC_FN_SERIAL
      GINAC_FN_SERIAL = g_registered_functions().size()
                       ^
  ------------------------------------------------------------

  ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions
  Compiling ginac/pynac.pyx because it changed.
  [1/1] Cythonizing ginac/pynac.pyx

[gh-kliem]

Replying to gh-kliem:

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

Has anyone experienced this? This seems to be a bug in our build system?

[mkoeppe]

Replying to gh-kliem:

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

pynac.pxd is generated by setup.py from pynac.pxd.in via the configure script. Installation of the new package can be seen to succeed in ​https://github.com/mkoeppe/pynac/runs/2010476345

[mkoeppe]

Replying to gh-kliem:

Replying to gh-kliem:

First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.

Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.

Has anyone experienced this? This seems to be a bug in our build system?

Yes, quite possible that the install cleaner only handles Python modules and extensions but not package data.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c0805f5

build/pkgs/pynac: Use 9c89d8bbc723456b2d94efb78ca1df1820694ff0

[mkoeppe]

Just packaged up current HEAD of my pynac python_package branch.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​93fe11f

build/pkgs/pynac: Use 5701911df8f00c99bc7c7ed676360729200fb726

[mkoeppe]

(and another time.)

[gh-kliem]

This is fixed with the new attachment. Before there was a version where ginac/pynac.pxd was missing from configure for some reason.

Replying to mkoeppe:

Replying to gh-kliem:

Yes, same error here. ginac/pynax.pdx is missing from the configure file now.

pynac.pxd is generated by setup.py from pynac.pxd.in via the configure script. Installation of the new package can be seen to succeed in ​https://github.com/mkoeppe/pynac/runs/2010476345

[gh-kliem]

With the new branch I can now do from ginac.pynac import * without an error raising in ipython. The ImportError when loading sage itself still persists.

[gh-kliem]

So maybe this module needs to include something on runtime and this doesn't work. So from ginac.pynac import * works fine, but the cimport doesn't.

In my experiments, the os.path.dirname(__file__) in the setup file didn't do anything. Maybe the include_dirs are set up wrong?

[fbissey]

OK, tried with all the new stuff. Stopped at the documentation again but different symbol than in comment:23

[dochtml] ImportError: /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE

which is

$ c++filt -n _ZN5GiNaC8py_funcsE
GiNaC::py_funcs

[fbissey]

Running ldd, removing python calls and going through c++filt -n the following symbols are undefined and should be replaced by calls since you cannot link to the python module

ldd -r /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so | grep _Z | c++filt -n
undefined symbol: GiNaC::py_funcs       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::container<std::vector>      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::numeric   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::symbol    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::~library_init()  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::container<std::vector>::tinfo_static   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::constant    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::I      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::_num0_bp       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::pseries   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::basic     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&)   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::library_init()   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Float(_object*)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, std::vector<GiNaC::ex, std::allocator<GiNaC::ex> > const&, bool)      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_I(_object*)      (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: std::vector<GiNaC::ex, std::allocator<GiNaC::ex> >::~vector() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::decide() const     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&, GiNaC::ex const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::sorted_op(unsigned long) const     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_basic(GiNaC::basic const&)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::pseries::is_terminating() const        (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int)       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::registered_functions()       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::compare(GiNaC::ex const&) const    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::numeric::to_pyobject() const   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::the_operator() const       (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&)     (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_int(int)    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::constant::constant()   (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::hold() const    (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Integer(_object*)        (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::operator=(GiNaC::basic const&)  (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)

[mkoeppe]

Looks like trying to pull in everything to sage.libs.pynac by from ginac.pynac cimport * is a mistake and instead all modules that need pynac should cimport more specifically from ginac.pynac

[gh-kliem]

Replying to mkoeppe:

Looks like trying to pull in everything to sage.libs.pynac by from ginac.pynac cimport * is a mistake and instead all modules that need pynac should cimport more specifically from ginac.pynac

But why does

sage: cython(''' 
....: from sage.libs.pynac.pynac cimport * 
....: ''')    

works just fine on develop? I would say that things should work just fine, once everything is set up correctly. However, I don't know how to resolve the problems as I'm currently not even able to cimport from my own little two files project.

[gh-kliem]

I might have found the problem:

In setup.py:

-       package_data = {'ginac': ['*.pxd', '*.h', '*.hpp']},
+       package_data = {'ginac': ['*.pxd', '*.h', '*.hpp', '*.cpp']},

This will place the `.cpp` files in the correct directory. We need them, because when we cimport we recompile some of them. When the `.cpp` files are not there, we get the missing symbols.

After doing this, I get a new error:

{{{
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/constant.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/constant.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/ring.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/ring.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/function.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/function.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/pynac.o -L/srv/public/kliem/sage/local/lib -lgsl -lflint -lgmp -lm -lopenblas -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/expression.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/expression.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources -lpari
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] error: command 'g++' failed with exit status 1
}}}

But I think, this might be easier to resolve.

[mkoeppe]

This can't be the right fix. The .cpp files should not be installed.

[mkoeppe]

Where's that -lginac coming from??

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b4da02f

cimport directly from ginac.pynac

[gh-kliem]

Same problem. Same undefined symbol.

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​b4da02f

cimport directly from ginac.pynac

[gh-kliem]

Replying to mkoeppe:

This can't be the right fix. The .cpp files should not be installed.

When you cimport you recompile .pxd file. And by this it is trying to recompile everything that was included there. If the corresponding .cpp files aren't found, there is a problem with that.

If we don't want to recompile everything, then we should put the cdef extern from into pynac.pyx and only do the type definitions in pynax.pxd.

[mkoeppe]

Replying to gh-kliem:

we should put the cdef extern from into pynac.pyx and only do the type definitions in pynax.pxd.

I don't know about this mechanism, but it sounds like what I am looking for. Could you elaborate on this or push a change to the branch?

[gh-kliem]

I'll check, if that fixes things.

[gh-kliem]

I don't know, if that will work. I get many errors and by no means do I know how to fix them.

E.g.

[pynac-0.7.28.p18]   gcc -DNDEBUG -g -fwrapv -O2 -Wall -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iginac -I./ginac -I. -I. -I/srv/public/kliem/sage/local/lib/python3.7/site-packages/pip/_vendor/pep517 -I/srv/public/kliem/sage/local -I/usr/lib/python37.zip -I/usr/lib/python3.7 -I/usr/lib/python3.7/lib-dynload -I/srv/public/kliem/sage/local/lib/python3.7/site-packages -I/srv/public/kliem/sage/local/include -I/usr/include/python3.7m -c ginac/pynac.cpp -o build/temp.linux-x86_64-3.7/ginac/pynac.o -std=c++11 -DSING_NDEBUG -DOM_NDEBUG -DSING_NDEBUG -DOM_NDEBUG -I/srv/public/kliem/sage/local/include
[pynac-0.7.28.p18]   ginac/pynac.cpp:1875:50: error: no declaration matches '__pyx_t_5ginac_5pynac_GFunctionOptVector& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18]    static __pyx_t_5ginac_5pynac_GFunctionOptVector &GiNaC::function::registered_functions(void); /*proto*/
[pynac-0.7.28.p18]                                                     ^~~~~
[pynac-0.7.28.p18]   In file included from ginac/ginac.h:54,
[pynac-0.7.28.p18]                    from ginac/pynac_wrap.h:13,
[pynac-0.7.28.p18]                    from ginac/pynac.cpp:786:
[pynac-0.7.28.p18]   ginac/function.h:409:41: note: candidate is: 'static std::vector<GiNaC::function_options>& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18]     static std::vector<function_options> & registered_functions();
[pynac-0.7.28.p18]                                            ^~~~~~~~~~~~~~~~~~~~
[pynac-0.7.28.p18]   ginac/function.h:340:7: note: 'class GiNaC::function' defined here
[pynac-0.7.28.p18]    class function : public exprseq

I don't even know if it is possible to import a cppclass in the .pyx part and to define in in the header then.

This option is definitely not mentioned here: ​https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html

The problem with this entire setup is that ginac was never properly compiled and linked as a C++ module. I have the feeling that you are trying to achieve something that cython doesn't want you to do.

• If you have a C++ library you can create a cython header for this and directly expose the cpp classes via that header (e.g. you can create a pip installable GMP header that uses an existing GMP installation).
• If you have some C++ files and you want to expose them directly by a cython wrapper, but the header of this wrapper will recompile your files and thus every module cimporting from this wrapper needs to be able to recompile this as well.
• You can create a proper wrapper class of a cpp class. This class can be used by other modules without recompiling the actual cpp class.

This is what the picture looks like to me.

[gh-kliem]

I digged into a bit. Not that I know a solution now, but at least I understand somehow what is going on.

The question is really what we want. cdef extern from in a .pxd-file assumes that every file cimporting this is linked against the same library or has access to the same sources. I don't think we can change this. So the way this is we must either compile ginac as a library and link against it or ship the sources (the later being really awful, because we have plenty of duplications and it takes forever to compile).

You can use cdef extern in the pyx however and expose the definitions in the header. This seems a very clean way of doing it, but there are tons of obstructions along the way. E.g. the functions must be declared static: ​https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#implementing-functions-in-c

[gh-kliem]

And apparently with a cppclass it works even in the header? (Different project, but there it seemed to work.)

I'm confused. I will see, if I can find out, what is causing the problem.

[mkoeppe]

Thanks for looking into this!

[mkoeppe]

New approach: #32386 (Merge pynac as src/sage/symbolics/ginac)

[fbissey]

Sure, let's try the other way.

[gh-kliem]

Removing the branch to avoid confusion in the future.