Opened 9 years ago

Closed 7 years ago

#9921 closed enhancement (fixed)

nose testing suite as an optional spkg

Reported by: jason Owned by: tbd
Priority: major Milestone: sage-5.7
Component: packages: optional Keywords: sd41
Cc: drkirkby, jhpalmieri Merged in: sage-5.7.beta2
Authors: John Palmieri Reviewers: Karl-Dieter Crisman, Martin Raum
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

Several projects we depend on use nose to do self-tests. In order to test these packages, we'd have to have nose installed.

This is fairly simple without an spkg:

wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz # or the latest version at http://pypi.python.org/pypi/nose/
tar xzvf nose-1.1.2.tar.gz
cd nose-1.1.2
sage -python setup.py install  

However, it might make sense to have nose be an optional spkg.

Matplotlib relies on nose to do tests (http://matplotlib.sourceforge.net/devel/coding_guide.html#testing), as well as numpy/scipy (http://projects.scipy.org/numpy/wiki/TestingGuidelines).


New spkg: version 1.1.2.

Apply trac_9921-nose-scripts.patch to the scripts repo.

Attachments (3)

trac_9921-nose-scripts.patch (367 bytes) - added by jhpalmieri 7 years ago.
scripts repo
trac_9921-nose-spkg.patch (3.3 KB) - added by jhpalmieri 7 years ago.
patch for spkg, for review only
trac_9921-nose-spkg.git.patch (3.3 KB) - added by jhpalmieri 7 years ago.
patch for spkg, for review only (version 1.1.3)

Download all attachments as: .zip

Change History (35)

comment:1 Changed 9 years ago by jason

The current website for nose is: http://code.google.com/p/python-nose/

comment:2 Changed 9 years ago by jason

  • Description modified (diff)

comment:3 Changed 9 years ago by drkirkby

IMHO it would be sensible to have nose as a standard package.

I suggested that once, and it got a luke warm reception. Having to install an optional item to test a standard package seems a bad idea to me.

Perhaps if a list of packages which make use of 'nose' could be produced, it might be possible to argue for it to be standard.

Dave

comment:4 follow-up: Changed 9 years ago by jason

Yes, I'm thinking (according to guidelines) optional for now, standard after a probationary period.

comment:5 in reply to: ↑ 4 Changed 9 years ago by drkirkby

Replying to jason:

Yes, I'm thinking (according to guidelines) optional for now, standard after a probationary period.

Yes, according to guidelines it should be optional for a while. But there have been exceptions to that. Since it does not actually link to anything in Sage, and would only be called when running spkg-check, I think one could argue the risk is minimal. In contrast it would allow a number of packages to be checked.

I think any such argument would have to be based on how many packages could benefit from it.

It must be an incredibly low risk package to add.

Dave

comment:6 follow-up: Changed 9 years ago by kcrisman

Whether or not it should be standard, I agree that we need to at least have it optional so that it becomes more 'Sage-ic' (totally not the right adjective, but I'm groping here) to test several fundamental python packages from within Sage. Also, it is needed to test some potential other packages, such as Brian (#9675), which is where I originally found out about this.

comment:7 in reply to: ↑ 6 ; follow-up: Changed 9 years ago by uri

Replying to kcrisman:

Whether or not it should be standard, I agree that we need to at least have it optional so that it becomes more 'Sage-ic' (totally not the right adjective, but I'm groping here) to test several fundamental python packages from within Sage. Also, it is needed to test some potential other packages, such as Brian (#9675), which is where I originally found out about this.

I installed the package and tried it to test Brian, and it worked perfectly. However, I think some changes should be made in order to fit the structure of Sage packages: all the code should be in a directory named src/, it lacks the .hg folder, the .hgignore and SPKG.txt files, etc. (see the page Producing New Sage Packages to see how it should be).

comment:8 in reply to: ↑ 7 ; follow-ups: Changed 9 years ago by leif

  • Type changed from defect to enhancement

Replying to uri:

[...] However, I think some changes should be made in order to fit the structure of Sage packages: all the code should be in a directory named src/, it lacks the .hg folder, the .hgignore and SPKG.txt files, etc. [...]

? Of course we first have to produce an spkg, be it optional or (later) standard.

Or did I miss something?

comment:9 in reply to: ↑ 8 ; follow-ups: Changed 9 years ago by kcrisman

[...] However, I think some changes should be made in order to fit the structure of Sage packages: all the code should be in a directory named src/, it lacks the .hg folder, the .hgignore and SPKG.txt files, etc. [...]

? Of course we first have to produce an spkg, be it optional or (later) standard.

Or did I miss something?

No, I think that uri was perhaps misunderstanding jason's directions to use nose outside of Sage to be a suggestion for how it would become a Sage package. No harm done in reminding us of the official rules, though :)

comment:10 in reply to: ↑ 9 Changed 9 years ago by leif

Replying to kcrisman:

No, I think that uri was perhaps misunderstanding jason's directions to use nose outside of Sage to be a suggestion for how it would become a Sage package.

Yeah, an spkg just containing spkg-install which is

wget http://code.google.com/p/python-nose/downloads/detail?name=nose-0.11.3.tar.gz
tar xzvf nose-0.11.3.tar.gz
cd nose-0.11.3
sage -python setup.py install  

:D

comment:11 Changed 9 years ago by leif

Ok, we should concatenate the lines with && to make it more robust...

comment:12 in reply to: ↑ 8 Changed 9 years ago by uri

Replying to leif:

Replying to uri:

[...] However, I think some changes should be made in order to fit the structure of Sage packages: all the code should be in a directory named src/, it lacks the .hg folder, the .hgignore and SPKG.txt files, etc. [...]

? Of course we first have to produce an spkg, be it optional or (later) standard.

Or did I miss something?

Oh, right, I missunderstood... sorry :)

comment:13 in reply to: ↑ 9 Changed 9 years ago by jason

Replying to kcrisman:

[...] However, I think some changes should be made in order to fit the structure of Sage packages: all the code should be in a directory named src/, it lacks the .hg folder, the .hgignore and SPKG.txt files, etc. [...]

? Of course we first have to produce an spkg, be it optional or (later) standard.

Or did I miss something?

No, I think that uri was perhaps misunderstanding jason's directions to use nose outside of Sage to be a suggestion for how it would become a Sage package. No harm done in reminding us of the official rules, though :)

Yep, the point of the instructions was to show that even without an spkg, using nose "is fairly simple without an spkg". Of course, you can't download something in the spkg, so those instructions would not work for an spkg. However, it would be a very generic spkg-install file that would basically do the normal standard checks and then run "python setup.py install".

comment:14 Changed 9 years ago by kcrisman

I tried using nose, and it was very interesting. Tested Brian on two different boxes, and then tested both numpy and scipy on OS X 10.6 - discovered no errors on the first, and a fair number on the second! So I think that there should be no doubt this could go straight to optional if an spkg was made.

I couldn't do

import matplotlib
matplotlib.test()

because for some reason our matplotlib doesn't have this method.

Much less importantly, I should also point out that on OS X wget is not a builtin, though I have an alias curl -O that seems to accomplish the same purpose. I just downloaded and double-clicked, actually :)

comment:15 Changed 7 years ago by kcrisman

  • Description modified (diff)

Nose is now on Github: https://github.com/nose-devs/nose.

comment:16 Changed 7 years ago by jhpalmieri

  • Cc jhpalmieri added

comment:17 Changed 7 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Status changed from new to needs_review

I've posted an spkg. It builds well enough for me, but I haven't actually used it to test anything except for itself (in spkg-check).

Changed 7 years ago by jhpalmieri

scripts repo

comment:18 Changed 7 years ago by jhpalmieri

  • Description modified (diff)

comment:19 Changed 7 years ago by jhpalmieri

By the way, version 1.1.2 was the most recent version for which I could find a tar ball. Or we could use the version from github; would that be better?

comment:20 Changed 7 years ago by jhpalmieri

  • Description modified (diff)

comment:21 follow-up: Changed 7 years ago by jhpalmieri

I've now also prepared a version from git. This one fails its own self tests, though ;)

Changed 7 years ago by jhpalmieri

patch for spkg, for review only

Changed 7 years ago by jhpalmieri

patch for spkg, for review only (version 1.1.3)

comment:22 in reply to: ↑ 21 Changed 7 years ago by kcrisman

I've now also prepared a version from git. This one fails its own self tests, though ;)

Interesting!

You can try using it with the brian optional spkg, and I believe numpy and mpl also use this?

comment:23 Changed 7 years ago by jason

ipython also uses nose.

comment:24 Changed 7 years ago by jhpalmieri

I tried running self-tests with numpy, using python -c 'import numpy; numpy.test()'. The good news: tests ran. The bad news: some tests failed. More bad news: this exited with a return code of zero, so it's hard to tell (within spkg-check, for example), that tests failed. I have the same issue with IPython.

comment:25 Changed 7 years ago by kcrisman

So is that a problem with nose or a problem with the numpy/ipython test suites? Sounds like nose is ready from this point of view :)

comment:26 Changed 7 years ago by jhpalmieri

  • Keywords sd41 added

comment:27 Changed 7 years ago by kcrisman

  • Description modified (diff)
  • Reviewers set to Karl-Dieter Crisman

This is ridiculous. Nose works fine at testing things on sage.math. It certainly finds various errors and warnings - apparently scipy generates a number as well, I just tried it.

The issue with it not passing its own tests is not so good, and I can confirm this in both cases. On the plus side, it only breaks the Sage installation process in the 1.1.3, so I would say let's go with the 1.1.2 for now.


I do get something weird, hopefully unrelated to nose itself, but instead related to our defaults for matplotlib.

kcrisman@sage:~/sage-5.1.beta1-boxen-x86_64-Linux$ ./sage -c 'import matplotlib; matplotlib.test()'
======================================================================
ERROR: Failure: OSError (No such file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/import matplotlib; matplotlib.test())
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/local/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/failure.py", line 39, in runTest
    raise self.exc_class(self.exc_val)
OSError: No such file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/import matplotlib; matplotlib.test()

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

sage: import matplotlib  
sage: matplotlib.test()
======================================================================
ERROR: Failure: ValueError (Unable to load tests from file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/sage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/local/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py", line 213, in loadTestsFromFile
    % filename)
ValueError: Unable to load tests from file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/sage

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
False

On a computer without nose:

sage: matplotlib.test()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Users/karl-dietercrisman/Downloads/sage-5.1.beta6/<ipython console> in <module>()

/Users/.../sage-5.1.beta6/local/lib/python2.7/site-packages/matplotlib/__init__.pyc in test(verbosity)
    986 def test(verbosity=0):
    987     """run the matplotlib test suite"""
--> 988     import nose
    989     import nose.plugins.builtin
    990     from testing.noseclasses import KnownFailure

ImportError: No module named nose

I think that in matplotlib's lib/init.py

    success = nose.run( defaultTest=default_test_modules,
                        config=config,
                        )

we aren't using the right default modules, they aren't imported or something. So it goes back to just looking at . for the default test module - I get the same thing.

sage: nose.run(defaultTest='.')                 
E
======================================================================
ERROR: Failure: ValueError (Unable to load tests from file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/sage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/local/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py", line 213, in loadTestsFromFile
    % filename)
ValueError: Unable to load tests from file /home/kcrisman/sage-5.1.beta1-boxen-x86_64-Linux/sage

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
False

Can you see a reason why this shouldn't have positive review? I just think this must be a problem in how we're dealing with mpl. I'm surprised it does this; we only removed the baseline images, not the testing scripts!

comment:28 Changed 7 years ago by mraum

  • Reviewers changed from Karl-Dieter Crisman to Karl-Dieter Crisman, Martin Raum

I have also reviewed this ticket, since I think it is important to have it. The matplotlib error is really a bug (or at least laxness) in matplotlib. If we waited for nosetest to be included until all spkg's pass, we would wait until the cows come home.

If nobody objects, I will set this to positive review by next week.

comment:29 Changed 7 years ago by mraum

  • Status changed from needs_review to positive_review

comment:30 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:31 Changed 7 years ago by schilly

spkg is put on the servers.

comment:32 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.7.beta2
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.