Opened 21 months ago

Closed 20 months ago

Last modified 20 months ago

#23518 closed enhancement (fixed)

Upgrade cypari2 package

Reported by: robharron Owned by:
Priority: major Milestone: sage-8.1
Component: packages: standard Keywords: cypari2, sd87
Cc: jdemeyer, defeo, vdelecroix, fbissey Merged in:
Authors: Jeroen Demeyer Reviewers: Frédéric Chapoton
Report Upstream: N/A Work issues:
Branch: 749c534 (Commits) Commit:
Dependencies: Stopgaps:

Change History (40)

comment:1 Changed 21 months ago by chapoton

any release coming soon ?

comment:2 Changed 21 months ago by chapoton

  • Cc defeo added

comment:3 Changed 20 months ago by chapoton

  • Cc vdelecroix added

ping ? this is the current blocking point in my experimental python3 build.

comment:4 Changed 20 months ago by chapoton

  • Status changed from new to needs_info

comment:5 Changed 20 months ago by robharron

I don't really know about what kind of automated stuff happens on trac anymore, so let me ask this perhaps naive question: why is this a block for anything? I only created this ticket because of #15829 which itself is still under development (has never even been marked as "needs review").

comment:6 Changed 20 months ago by chapoton

Sorry, I have used your ticket for my own purposes. I need (for something having to do with making sage works with python3) to have a new release of cypari2. So I would just embark in this ticket, which is also asking for a release of cypari2.

comment:7 Changed 20 months ago by robharron

Aha! I understand now. It may be worth updating the ticket description: it might encourage people to work on this ticket!

comment:8 Changed 20 months ago by fbissey

  • Cc fbissey added

comment:9 Changed 20 months ago by vdelecroix

  • Description modified (diff)
  • Summary changed from Update cypari2 with version including idealmoddivisor to Updgrade cypari2 package

comment:10 Changed 20 months ago by chapoton

  • Description modified (diff)

comment:11 Changed 20 months ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Summary changed from Updgrade cypari2 package to Upgrade cypari2 package

comment:12 Changed 20 months ago by jdemeyer

  • Description modified (diff)

comment:13 Changed 20 months ago by jdemeyer

  • Branch set to u/jdemeyer/upgrade_cypari2_package

comment:14 Changed 20 months ago by jdemeyer

  • Commit set to 75cd924e9b264c49598d7909b84a37c6bb050511
  • Status changed from needs_info to needs_work

Something very strange happens while testing the package:

[cypari-1.1.0] Testing cypari2.pari_instance
[cypari-1.1.0] Traceback (most recent call last):
[cypari-1.1.0]   File "tests/rundoctest.py", line 24, in <module>
[cypari-1.1.0]     test = doctest.testmod(mod, optionflags=doctest.ELLIPSIS|doctest.REPORT_NDIFF|doctest.IGNORE_EXCEPTION_DETAIL)
[cypari-1.1.0]   File "/usr/local/src/sage-config/local/lib/python2.7/doctest.py", line 1911, in testmod
[cypari-1.1.0]     runner.run(test)
[cypari-1.1.0]   File "/usr/local/src/sage-config/local/lib/python2.7/doctest.py", line 1454, in run
[cypari-1.1.0]     return self.__run(test, compileflags, out)
[cypari-1.1.0]   File "/usr/local/src/sage-config/local/lib/python2.7/doctest.py", line 1368, in __run
[cypari-1.1.0]     exc_info)
[cypari-1.1.0]   File "/usr/local/src/sage-config/local/lib/python2.7/doctest.py", line 1235, in report_unexpected_exception
[cypari-1.1.0]     'Exception raised:\n' + _indent(_exception_traceback(exc_info)))
[cypari-1.1.0]   File "/usr/local/src/sage-config/local/lib/python2.7/doctest.py", line 1251, in _failure_header
[cypari-1.1.0]     return '\n'.join(out)
[cypari-1.1.0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 16: ordinal not in range(128)

This happens only when testing the Sage package. Any other way of running the testsuite works fine.

I have absolutely no idea why and I hope that somebody else can fix it.


New commits:

75cd924Upgrade cypari2 to version 1.1.0

comment:15 Changed 20 months ago by chapoton

How can I run this testsuite ?

comment:16 Changed 20 months ago by jdemeyer

The failing testsuite is ./sage -i -c cypari

But like I said, the strange thing is that manually running spkg-check works.

comment:17 Changed 20 months ago by chapoton

I can confirm the failure of ./sage -i -c cypari, just the same as what you got.

How can I run manually "spkg-check" ?

comment:18 Changed 20 months ago by chapoton

wiht python3-sage, I get something else

[cypari-1.1.0] ****************************************************
[cypari-1.1.0] Uninstalling cypari2-1.0.0:
[cypari-1.1.0]   Successfully uninstalled cypari2-1.0.0
[cypari-1.1.0] Installing package cypari2 using pip3
[cypari-1.1.0] Ignoring indexes: https://pypi.python.org/simple
[cypari-1.1.0] Processing /home/chapoton/sage3/local/var/tmp/sage/build/cypari-1.1.0/src
[cypari-1.1.0]   Running setup.py (path:/tmp/pip-yihktdba-build/setup.py) egg_info for package from file:///home/chapoton/sage3/local/var/tmp/sage/build/cypari-1.1.0/src
[cypari-1.1.0]     Running command python setup.py egg_info
[cypari-1.1.0]     running egg_info
[cypari-1.1.0]     creating pip-egg-info/cypari2.egg-info
[cypari-1.1.0]     writing pip-egg-info/cypari2.egg-info/PKG-INFO
[cypari-1.1.0]     writing dependency_links to pip-egg-info/cypari2.egg-info/dependency_links.txt
[cypari-1.1.0]     writing top-level names to pip-egg-info/cypari2.egg-info/top_level.txt
[cypari-1.1.0]     writing manifest file 'pip-egg-info/cypari2.egg-info/SOURCES.txt'
[cypari-1.1.0]     Generating PARI functions: (!_) (#_) (%) (%#) (+_) (-_) Catalan Col Colrev (DEBUGLEVEL) Euler I List Map MatTraceback (most recent call last):
[cypari-1.1.0]       File "<string>", line 1, in <module>
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/setup.py", line 84, in <module>
[cypari-1.1.0]         cmdclass=dict(build_ext=build_ext, bdist_egg=no_egg)
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/distutils/core.py", line 148, in setup
[cypari-1.1.0]         dist.run_commands()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
[cypari-1.1.0]         self.run_command(cmd)
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/distutils/dist.py", line 974, in run_command
[cypari-1.1.0]         cmd_obj.run()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 279, in run
[cypari-1.1.0]         self.find_sources()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 294, in find_sources
[cypari-1.1.0]         mm.run()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 521, in run
[cypari-1.1.0]         self.add_defaults()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 557, in add_defaults
[cypari-1.1.0]         sdist.add_defaults(self)
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
[cypari-1.1.0]         self._add_defaults_ext()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
[cypari-1.1.0]         build_ext = self.get_finalized_command('build_ext')
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
[cypari-1.1.0]         cmd_obj.ensure_finalized()
[cypari-1.1.0]       File "/home/chapoton/sage3/local/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
[cypari-1.1.0]         self.finalize_options()
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/setup.py", line 50, in finalize_options
[cypari-1.1.0]         rebuild()
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/autogen/__init__.py", line 26, in rebuild
[cypari-1.1.0]         G()
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/autogen/generator.py", line 336, in __call__
[cypari-1.1.0]         self.handle_pari_function(**v)
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/autogen/generator.py", line 239, in handle_pari_function
[cypari-1.1.0]         self.gen_file, doc, obsolete)
[cypari-1.1.0]       File "/tmp/pip-yihktdba-build/autogen/generator.py", line 313, in write_method
[cypari-1.1.0]         print(s, file=file)
[cypari-1.1.0]     UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 314: ordinal not in range(128)
[cypari-1.1.0] Cleaning up...
[cypari-1.1.0] Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-yihktdba-build/
[cypari-1.1.0] Exception information:
[cypari-1.1.0] Traceback (most recent call last):
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
[cypari-1.1.0]     status = self.run(options, args)
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/commands/install.py", line 324, in run
[cypari-1.1.0]     requirement_set.prepare_files(finder)
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
[cypari-1.1.0]     ignore_dependencies=self.ignore_dependencies))
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/req/req_set.py", line 634, in _prepare_file
[cypari-1.1.0]     abstract_dist.prep_for_dist()
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
[cypari-1.1.0]     self.req_to_install.run_egg_info()
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/req/req_install.py", line 439, in run_egg_info
[cypari-1.1.0]     command_desc='python setup.py egg_info')
[cypari-1.1.0]   File "/home/chapoton/sage3/local/lib/python3.6/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
[cypari-1.1.0]     % (command_desc, proc.returncode, cwd))
[cypari-1.1.0] pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-yihktdba-build/
[cypari-1.1.0] Error: installing with pip3 failed
[cypari-1.1.0] 
[cypari-1.1.0] real	0m4.142s
[cypari-1.1.0] user	0m2.116s
[cypari-1.1.0] sys	0m0.224s
[cypari-1.1.0] ************************************************************************
[cypari-1.1.0] Error installing package cypari-1.1.0
[cypari-1.1.0] ************************************************************************

comment:19 Changed 20 months ago by jdemeyer

Can I please have

from __past__ import bytes_literals

comment:20 Changed 20 months ago by chapoton

Sorry, I do not understand..

EDIT: oh, some kind of joke ?

Last edited 20 months ago by chapoton (previous) (diff)

comment:21 Changed 20 months ago by jdemeyer

Yes, a joke of course.

Let me just say that I'm not a fan of the way how Python deals with unicode encoding issues...

comment:22 Changed 20 months ago by jdemeyer

OK, the issues are due to LC_ALL=C. This is something that will need to be fixed upstream.

comment:23 follow-up: Changed 20 months ago by embray

I just saw your e-mail about this. Did you figure it out? You mentioned something about LC_ALL=C. And by "fixed upstream" do you mean in cypari? Should I take a look at how cypari is handling encodings?

comment:24 in reply to: ↑ 23 Changed 20 months ago by jdemeyer

  • Description modified (diff)

Replying to embray:

I just saw your e-mail about this. Did you figure it out?

Yes. The issue is that Python's open() and print() functions depend on the locale:

$ LC_ALL=en_US.UTF-8 python3 -c 'print(open("/dev/null"))'
<_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>
$ LC_ALL=C python3 -c 'print(open("/dev/null"))'
<_io.TextIOWrapper name='/dev/null' mode='r' encoding='ANSI_X3.4-1968'>

It turns out that cypari2 was assuming that files were opened in UTF-8 mode. I now made this explicit with open(...., encoding="utf-8") in Python 3.

comment:25 follow-up: Changed 20 months ago by embray

By the way, this is a useful guide handling encodings when reading text files:

http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html#text-file-processing

I would suggest also adding errors='surrogateescape'. This way if any file happens to contain non-ASCII characters that were written using any encoding other than UTF-8, it will at least handle the ASCII parts correctly and not blow up with a UnicodeDecodeError.

comment:26 Changed 20 months ago by git

  • Commit changed from 75cd924e9b264c49598d7909b84a37c6bb050511 to 749c534e8fd56f3ec32c29d2b8422292cefff5f7

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

749c534Upgrade cypari2 to version 1.1.1

comment:27 in reply to: ↑ 25 Changed 20 months ago by jdemeyer

Replying to embray:

I would suggest also adding errors='surrogateescape'. This way if any file happens to contain non-ASCII characters that were written using any encoding other than UTF-8, it will at least handle the ASCII parts correctly and not blow up with a UnicodeDecodeError.

This is not applicable here. In this case, we know that all the files we deal with are UTF-8. The input files come from the PARI documentation which is written in UTF-8. And the output files contain auto-generated Cython code which has a # -*- coding: utf-8 -*- header.

comment:28 Changed 20 months ago by jdemeyer

  • Status changed from needs_work to needs_review

This version builds and passes its testsuite. I did not yet run Sage doctests.

comment:29 Changed 20 months ago by embray

That's good, that makes it easy then.

comment:30 Changed 20 months ago by jdemeyer

Side comment: the # -*- coding: utf-8 -*- header doesn't actually matter since Cython supports PEP 3120.

comment:31 Changed 20 months ago by chapoton

not yet smooth for python3:

[cypari-1.1.1]     Running setup.py install for cypari2: finished with status 'done'
[cypari-1.1.1]   Removing source in /tmp/pip-qsn9psry-build
[cypari-1.1.1] Successfully installed cypari2-1.1.1
[cypari-1.1.1] Cleaning up...
[cypari-1.1.1] 
[cypari-1.1.1] real	3m51.177s
[cypari-1.1.1] user	3m42.104s
[cypari-1.1.1] sys	0m5.896s
[cypari-1.1.1] Successfully installed cypari-1.1.1
[cypari-1.1.1] Running the test suite for cypari-1.1.1...
[cypari-1.1.1] make[2]: Entering directory '/home/chapoton/sage3/local/var/tmp/sage/build/cypari-1.1.1/src'
[cypari-1.1.1] python tests/rundoctest.py
[cypari-1.1.1] Traceback (most recent call last):
[cypari-1.1.1]   File "tests/rundoctest.py", line 15, in <module>
[cypari-1.1.1]     cypari2.string_utils.encoding = "utf-8"
[cypari-1.1.1] AttributeError: 'module' object has no attribute 'string_utils'
[cypari-1.1.1] Makefile:16: recipe for target 'check' failed
[cypari-1.1.1] make[2]: *** [check] Error 1
[cypari-1.1.1] make[2]: Leaving directory '/home/chapoton/sage3/local/var/tmp/sage/build/cypari-1.1.1/src'
[cypari-1.1.1] 
[cypari-1.1.1] real	0m0.346s
[cypari-1.1.1] user	0m0.120s
[cypari-1.1.1] sys	0m0.012s
[cypari-1.1.1] ************************************************************************

comment:32 Changed 20 months ago by jdemeyer

Sigh... but Python 3 works fine outside of Sage in Travis CI, running the same commands (pip install . and make check).

comment:33 Changed 20 months ago by chapoton

Well, I am using "export SAGE_PYTHON3=yes", pull the branch here and "sage -c -f cypari" (in a dedicated sage install for py3 experimental work)

Maybe it is because the python3 general setup is not firmly in place..

comment:34 Changed 20 months ago by jdemeyer

Doctests pass (on Python 2).

I don't understand why this fails with SAGE_PYTHON3=yes, especially given that the error seems to be not specific to Python 3 and given that it works with Python 3 outside of Sage. Maybe there is simply a problem with the particular setup of chapoton.

In any case, I suggest that this should not be a blocker for merging this ticket. It doesn't make the Python 3 situation any worse than before and it helps people who need to update cypari2 for other reasons.

comment:35 Changed 20 months ago by chapoton

ok, ok. But still, if somebody could try to reproduce that error that I find..

EDIT: I do not have a particular setup, I think.

Last edited 20 months ago by chapoton (previous) (diff)

comment:36 Changed 20 months ago by embray

I'll have a look at the Python 3 issue. I agree with Jeroen it needn't hold up this issue but I can try to investigate ASAP (I need to do a SAGE_PYTHON3=yes build anyways to help with other issues on that).

comment:37 Changed 20 months ago by chapoton

  • Reviewers set to Frédéric Chapoton
  • Status changed from needs_review to positive_review

ok, let this go in. We will see later what to do for tests to pass in python3.

comment:38 Changed 20 months ago by vbraun

  • Branch changed from u/jdemeyer/upgrade_cypari2_package to 749c534e8fd56f3ec32c29d2b8422292cefff5f7
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:39 Changed 20 months ago by jdemeyer

  • Commit 749c534e8fd56f3ec32c29d2b8422292cefff5f7 deleted

Any news from the PYTHON3 front?

comment:40 Changed 20 months ago by chapoton

Not an answer on the cypari2 relation to python3..

Well, this ticket fixes the issue that I had. One of the first blocking point for PYTHON3 is now the division problem in #22875. If you feel like handling this one, it would be great, because I do not understand exactly what you ask for there.

And #23785 would be helpful too.

Note: See TracTickets for help on using tickets.