Opened 4 years ago

Closed 4 years ago

#17605 closed enhancement (fixed)

"How to call C code from Sage" thematic tutorial

Reported by: ncohen Owned by:
Priority: major Milestone: sage-6.5
Component: documentation Keywords:
Cc: azi, jmantysalo, vbraun, kcrisman, aschilling Merged in:
Authors: Nathann Cohen Reviewers: Karl-Dieter Crisman, David Coudert
Report Upstream: N/A Work issues:
Branch: 2f37b6f (Commits) Commit: 2f37b6fc906262f0cda839734eec8ab6f9cbf389
Dependencies: Stopgaps:

Description (last modified by ncohen)

This branch adds to Sage's thematic tutorial an explanation of how to interface a C code (or a compiled library) with Sage. It does not replace the developer's manual section entitled "Packing third-party code" as it is only a way to make it work "on your own compputer", without the complications that are only cause by Sage's infrastructure.

I have been asked the same instructions several times, and with this we will have a link to provide instead of the same explanation and the example files.

Nathann

Change History (40)

comment:1 Changed 4 years ago by ncohen

  • Branch set to public/17605
  • Status changed from new to needs_review

comment:2 Changed 4 years ago by git

  • Commit set to 462b6e2ef174a80b33c06a18e999ec2a459d4700

Branch pushed to git repo; I updated commit sha1. New commits:

462b6e2trac #17605: "How to call C code from Sage" thematic tutorial

comment:3 follow-up: Changed 4 years ago by kcrisman

I don't know enough about this to say whether it is all correct, but it looks very helpful (at least, to me!). Did you stop early, though?

+Calling a function defined in a library
+---------------------------------------
+
+  ~/a$ gcc -c -fPIC double_vector.c -o libdoublevector.o

I guess you have compiled the library file, but where is the calling part?

comment:4 Changed 4 years ago by ncohen

I am an idiot. I made many attempts to link this .pyx file with a compiled library and failed, and this is just the beginning of the section that I do not know how to write. I will update my patch in a second to remove that.

I would love to write it, but I just don't know how. There are quite some things that I know how to do by modifying sage, but not without that :-P

Nathann

comment:5 Changed 4 years ago by git

  • Commit changed from 462b6e2ef174a80b33c06a18e999ec2a459d4700 to a84624002490417b18adb6534a1c73843822306e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a846240trac #17605: "How to call C code from Sage" thematic tutorial

comment:6 in reply to: ↑ 3 Changed 4 years ago by ncohen

I don't know enough about this to say whether it is all correct, but it looks very helpful (at least, to me!).

Well, the point is actually that with this you can give it a try and witness that it does work ;-)

Volker? Do you know how to link to a compiled library while outside of Sage? Can you teach me? And I'll write the doc!

Nathann

comment:7 Changed 4 years ago by ncohen

  • Cc aschilling added

comment:8 follow-up: Changed 4 years ago by vbraun

Note: An object file (.o) is not a library (.a / .so).

You can link compiled C/C++ code with a stand-alone Cython project like in https://github.com/vbraun/lattice_phi4. Then just import as module in Sage.

comment:9 in reply to: ↑ 8 Changed 4 years ago by ncohen

Hello Volker,

You can link compiled C/C++ code with a stand-alone Cython project like in https://github.com/vbraun/lattice_phi4. Then just import as module in Sage.

I tried it but I was not able to compile it. It seems that I am missing some headers.

I kept on whipping my test files and go to something new, though: I was able to make it work provided that I copied libdoublevector.so to SAGE_ROOT/local/lib/.

Now, I am trying to make it all run from my working directory but I meet the following problem:

  • When calling "%runfile double_vector_sage.pyx" there are two calls to GCC
  • The one which contains -ldoublevector is not the one that contains -L/home/ncohen/a

As a result the second call always fails, because the library is not found.

sage: %runfile double_vector_sage.pyx
Compiling ./double_vector_sage.pyx...
...
RuntimeError: Error compiling ./double_vector_sage.pyx:
running build
running build_ext
building '_home_ncohen_a_double_vector_sage_pyx_5' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
 -I/home/ncohen/.Sage/local/include/csage -I/home/ncohen/.Sage/local/include
 -I/home/ncohen/.Sage/local/include/python2.7 
 -I/home/ncohen/.Sage/local/lib/python/site-packages/numpy/core/include 
 -I/home/ncohen/.Sage/src/sage/ext -I/home/ncohen/.Sage/src 
 -I/home/ncohen/.Sage/src/sage/gsl -I. -I/home/ncohen/.Sage/local/include/python2.7 
 -c _home_ncohen_a_double_vector_sage_pyx_5.c 
 -o build/temp.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.o 
 -w -O2 -L/home/ncohen/a/
gcc -pthread -shared -L/home/ncohen/.Sage/local/lib build/temp.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.o 
 -L/home/ncohen/.Sage/local/lib/ -L/home/ncohen/.Sage/local/lib -ldoublevector 
 -lmpfr -lgmp -lgmpxx -lstdc++ -lpari -lm -lec -lgsl -lgslcblas -latlas -lntl 
 -lcsage -lpython2.7 -o build/lib.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.so 
 -L/home/ncohen/.Sage/local/lib

The header of my .pyx file is:

#clib doublevector
#cargs -L/home/ncohen/a/

Thanks to you if you can give me some hint. In the meantime I'll be trying again to make it work.

Nathann

comment:10 Changed 4 years ago by git

  • Commit changed from a84624002490417b18adb6534a1c73843822306e to 2ad9b510df07b6f97741f934e9830b3b405401e8

Branch pushed to git repo; I updated commit sha1. New commits:

2ad9b51trac #17605: Make it work for libraries

comment:11 Changed 4 years ago by ncohen

Okay found a way ;-)

Needs review again !

Nathann

comment:12 Changed 4 years ago by ncohen

  • Description modified (diff)

comment:13 follow-up: Changed 4 years ago by vbraun

There is probably some alternate reality where the new description is useful for others to read/search, I just lack the imagination for it.

Once you install the shared library into Sage you might just as well go through module_list.py as well.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 4 years ago by ncohen

  • Description modified (diff)

There is probably some alternate reality where the new description is useful for others to read/search, I just lack the imagination for it.

There is probably one in which comments like that don't make people uselessly angry. Are we in the reality in which you also review the branch ?

Once you install the shared library into Sage you might just as well go through module_list.py as well.

I do not understand this comment.

Nathann

comment:15 in reply to: ↑ 14 ; follow-up: Changed 4 years ago by vbraun

Replying to ncohen:

Once you install the shared library into Sage you might just as well go through module_list.py as well.

I do not understand this comment.

Presumably you want to build a stand-alone C/C++ and Cython project without having to install stuff into Sage. By definition, this should exclude steps that modify Sage directly. I mentioned the code in my github repo, this is the only way to do this currently: Write Makefile that calls gcc and cython directly.

Also, you shouldn't build shared libraries the way you describe. Its very platform dependent and your steps won't work under OSX, say.

comment:16 in reply to: ↑ 15 Changed 4 years ago by ncohen

Presumably you want to build a stand-alone C/C++ and Cython project without having to install stuff into Sage. By definition, this should exclude steps that modify Sage directly.

I do not understand. In the tutorial no instruction advises the user to change anything inside of Sage. What did you see that I missed?

I mentioned the code in my github repo, this is the only way to do this currently: Write Makefile that calls gcc and cython directly.

I want to make it as simple as possible, and the current instructions avoid that.

Also, you shouldn't build shared libraries the way you describe. Its very platform dependent and your steps won't work under OSX, say.

Oh. Could you tell me how to build this library under OSX ? I will update the tutorial to explain both situations.

Nathann

comment:17 follow-up: Changed 4 years ago by vbraun

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book. E.g. on a global installation (like SMC) you just don't have permissions to do that.

comment:18 in reply to: ↑ 17 Changed 4 years ago by ncohen

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book. E.g. on a global installation (like SMC) you just don't have permissions to do that.

The tutorial does not do that.

Nathann

comment:19 follow-up: Changed 4 years ago by kcrisman

Well, the point is actually that with this you can give it a try and witness that it does work

Sadly, not today :( though it would be more fun than what I have to do. Though I would recommend removing the ncohen references to something more generic, no doubt an oversight.

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book.

The tutorial does not do that.

I think Volker is referring to the earlier comment

I kept on whipping my test files and go to something new, though: I was able to make it work provided that I copied libdoublevector.so to SAGE_ROOT/local/lib/.

which however I think you are right is not in the tutorial, you seem to have taken a different approach.

As for the OS X, I think the .so files are called .dylib (at least sometimes) and on Cygwin .dll... but I am not an expert in such things. Good luck! This would indeed be a good tutorial to have.

comment:20 in reply to: ↑ 19 Changed 4 years ago by ncohen

Hello,

Sadly, not today :( though it would be more fun than what I have to do. Though I would recommend removing the ncohen references to something more generic, no doubt an oversight.

Okay I can do that... I saw Jeroen's home path in several documents of the developer's manual, so I thought it did not matter.

I think Volker is referring to the earlier comment

Yes, probably.

As for the OS X, I think the .so files are called .dylib (at least sometimes) and on Cygwin .dll... but I am not an expert in such things. Good luck! This would indeed be a good tutorial to have.

HMmm.. Yep, but I need help to write that. I never used those architectures.

Nathann

comment:21 follow-up: Changed 4 years ago by kcrisman

I saw Jeroen's home path in several documents of the developer's manual, so I thought it did not matter.

Probably not that much, but might as well make it more generic if you have the opportunity.

comment:22 in reply to: ↑ 21 Changed 4 years ago by ncohen

Probably not that much, but might as well make it more generic if you have the opportunity.

http://www.sagemath.org/doc/developer/doctesting.html

I'll fix them all...

Nathann

comment:23 follow-up: Changed 4 years ago by kcrisman

I'll fix them all...

Well, I'd make that a separate ticket, to be on the safe side. Because it could be that several other people, including yourself, disagree with me about this, and I don't feel strongly enough about it to complain that hard - should have the opportunity to shoot me down without making this ticket's review process too long. My personal feeling is that usually it's best to say something early on, as we do in some tutorials, like "Here we assume that your home directory and username are username/ or something analogous to that.

comment:24 Changed 4 years ago by git

  • Commit changed from 2ad9b510df07b6f97741f934e9830b3b405401e8 to 96177d8ed62e5c88325c95f168406b27249b2ea1

Branch pushed to git repo; I updated commit sha1. New commits:

96177d8trac #17605: no trace that Sage is written by jews

comment:25 in reply to: ↑ 23 Changed 4 years ago by ncohen

Replying to kcrisman:

I'll fix them all...

Well, I'd make that a separate ticket

I changed those introduced by this branch.

Because it could be that several other people, including yourself, disagree with me about this

For this kind of stuff, let's say that I prefer to make the changes and get my ticket reviewed.

My personal feeling is that usually it's best to say something early on, as we do in some tutorials, like "Here we assume that your home directory and username are username/ or something analogous to that.

I hope that just changing the prompt will make it (more) obvious in this case.

Nathann

comment:26 Changed 4 years ago by kcrisman

I hope that just changing the prompt will make it (more) obvious in this case.

Yes, this does a good job.

comment:27 follow-up: Changed 4 years ago by kcrisman

  • Reviewers set to Karl-Dieter Crisman
  • Status changed from needs_review to needs_work

Issues I found.

  • My compiler gives this.
    $ gcc hello.c -o hello; ./hello
    hello.c: In function ‘main’:
    hello.c:7: warning: return type of ‘main’ is not ‘int’
    Hello World
    
  • Inconsistent names here:
    +  def my_interface_function():
    +      hello_world() # This is the C function from hello.c
    ...
    +  sage: my_bridge_function()
    +  Hello World
    
  • Compiler error:
    sage: %runfile hello_sage.pyx
    Compiling ./hello_sage.pyx...
    Error compiling cython file:
    Error converting ./hello_sage.pyx to C:
    
    
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    include "interrupt.pxi"  # ctrl-c interrupt block support
    include "stdsage.pxi"  # ctrl-c interrupt block support
    
    include "cdefs.pxi"
    cdef extern from "hello.c":
    void hello_world()
    ^
    ------------------------------------------------------------
    
    _Users_..._hello_sage_pyx_0.pyx:7:0: Expected an increase in indentation level
    
    
  • Even after fixing this:
    sage: %runfile hello_sage.pyx
    Compiling ./hello_sage.pyx...
    Error compiling cython file:
    Error compiling ./hello_sage.pyx:
    running build
    ...
    _Users_..._hello_sage_pyx_0.c:349:19: fatal error: hello.c: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    Maybe my file needs to be in my PATH? But that didn't seem to help.
  • And since presumably I needed to compile double_vector.c first, I tried that, but...
    $ gcc double_vector.c -o double_vector
    double_vector.c: In function ‘sum_of_two_vectors’:
    double_vector.c:4: warning: incompatible implicit declaration of built-in function ‘malloc’
    Undefined symbols for architecture x86_64:
      "_main", referenced from:
          start in crt1.10.6.o
    ld: symbol(s) not found for architecture x86_64
    collect2: ld returned 1 exit status
    
    SO tells me there are two errors - need to include stdio.h and then compile with -c or provide a main function. Still get the weird error about it not existing when I try to %runfile, though.
  • Similar problems with the final library example.

comment:28 in reply to: ↑ 27 Changed 4 years ago by ncohen

Yo ! Thank you for looking at this.

  • My compiler gives this.
    $ gcc hello.c -o hello; ./hello
    hello.c: In function ‘main’:
    hello.c:7: warning: return type of ‘main’ is not ‘int’
    Hello World
    

My compiler does not say a word. I changed the return type of main to int and added a 'return 0'. I would prefer to remove this return 0 (my compiler does not complain either if I do) for that is irrelevant information to what I want to illustrate. Can you tell me what it gives for you ?

  • Inconsistent names here:

Fixed.

  • Compiler error:

I fixed the indentation level.

  • Even after fixing this:
    sage: %runfile hello_sage.pyx
    Compiling ./hello_sage.pyx...
    Error compiling cython file:
    Error compiling ./hello_sage.pyx:
    running build
    ...
    _Users_..._hello_sage_pyx_0.c:349:19: fatal error: hello.c: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    

This I do not understand. Is hello_sage is in the same folder as hello.c ? O_o I really do not understand that.

Maybe my file needs to be in my PATH? But that didn't seem to help.

No no, you should not have to. And I would prefer to avoid asking the user to change the PATH anyway O_o

  • And since presumably I needed to compile double_vector.c first, I tried that, but...

Nono, you do not need to compile it by itself (and when you do it complains because no 'main' function is defined, which is correct). It gets compiled when you call "runfile". Actually, you do not need to compile 'hello.c' at first. It is just to give an example. If it gets complicated we can remove this compilation of "hello world", actually. It is not necessary.

  • Similar problems with the final library example.

When you say "similar problem", is it also about 'main' ? O_o

Nathann

comment:29 Changed 4 years ago by git

  • Commit changed from 96177d8ed62e5c88325c95f168406b27249b2ea1 to bd0dae3b8577fe2e339fca82934a388639e95668

Branch pushed to git repo; I updated commit sha1. New commits:

bd0dae3trac #17605: Errors...

comment:30 Changed 4 years ago by git

  • Commit changed from bd0dae3b8577fe2e339fca82934a388639e95668 to 23139b5ae0c7a8e5baa6fe6afd2efd52777dfaff

Branch pushed to git repo; I updated commit sha1. New commits:

23139b5trac #17605: Remove useless (and problematic) compilation (+indentation fixes)

comment:31 Changed 4 years ago by git

  • Commit changed from 23139b5ae0c7a8e5baa6fe6afd2efd52777dfaff to 5f20f0a3872d2f3f12b3c716c9a4e445530969bc

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5f20f0atrac #17605: Remove useless (and problematic) compilation (+indentation fixes)

comment:32 Changed 4 years ago by ncohen

  • Status changed from needs_work to needs_info

comment:33 Changed 4 years ago by git

  • Commit changed from 5f20f0a3872d2f3f12b3c716c9a4e445530969bc to 2f37b6fc906262f0cda839734eec8ab6f9cbf389

Branch pushed to git repo; I updated commit sha1. New commits:

2f37b6ftrac #17605: Syntax highlighting

comment:34 Changed 4 years ago by ncohen

  • Status changed from needs_info to needs_review

comment:35 follow-up: Changed 4 years ago by dcoudert

  • Reviewers changed from Karl-Dieter Crisman to Karl-Dieter Crisman, David Coudert

Hello,

I tried with my MBA/OSX Yosemite. The documentation is clear and useful. I'm able to reproduce all examples without error/warning.

Unless others have difficulties on other systems, I vote for positive review. Let me know if I can change the status.

David.

comment:36 in reply to: ↑ 35 Changed 4 years ago by ncohen

Yo,

I tried with my MBA/OSX Yosemite. The documentation is clear and useful. I'm able to reproduce all examples without error/warning.

Oh, so it works on Mac too ? Cool !

Unless others have difficulties on other systems, I vote for positive review. Let me know if I can change the status.

Well I'd vote for that.. And if there is something wrong with some distribution, we can add a note later. And hopefully in the meantime the guy will figure out how to make it work.

Nathann

comment:37 Changed 4 years ago by dcoudert

  • Status changed from needs_review to positive_review

So let's go ;)

comment:38 follow-up: Changed 4 years ago by kcrisman

This also works for me now, though I have no idea why. Maybe because we removed the original compilation and Sage compiled it wherever Sage compiles things.

comment:39 in reply to: ↑ 38 Changed 4 years ago by ncohen

This also works for me now, though I have no idea why. Maybe because we removed the original compilation and Sage compiled it wherever Sage compiles things.

Ahahah. Well, remember? "Theory is when you know everything but nothing works, practice is when everything works but nobody knows why" :-P

Nathann

comment:40 Changed 4 years ago by vbraun

  • Branch changed from public/17605 to 2f37b6fc906262f0cda839734eec8ab6f9cbf389
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.