Opened 6 years ago

Closed 6 years ago

#16938 closed defect (fixed)

Sage debug version

Reported by: SimonKing Owned by:
Priority: critical Milestone: sage-6.4
Component: build Keywords: debug singular cygwin64
Cc: jpflori, vbraun, malb Merged in:
Authors: Simon King, Volker Braun, Jeroen Demeyer Reviewers: Jeroen Demeyer, Volker Braun
Report Upstream: Reported upstream. No feedback yet. Work issues:
Branch: c471edd (Commits) Commit: c471edd674e461bb2db75fc899d44fb75b745bba
Dependencies: #17088 Stopgaps:

Description (last modified by jdemeyer)

When building sage with SAGE_DEBUG=yes, then it turns out that the debug and the cygwin64 patches to the Singular spkg are in conflict. Hence, the build fails. Let's fix it!

I suppose the fix will eventually be in a stable upstream release.

There is also an issue with the new m4ri: https://bitbucket.org/malb/m4ri/issue/59/assertion-sizeof-mzd_t-64-fails-on-32-bit

Change History (116)

comment:1 Changed 6 years ago by SimonKing

The function choose_patches() of singular's spkg-install removes most of the files in $SRC/omalloc/ if we are in debug mode. However, the cygwin64 patch tries to patch some files in that folder.

In addition to that, the cygwin64 patch says

+ifeq ($(SINGUNAME),x86_64-Win)
+SO_SUFFIX = dll
+MODULE_SUFFIX    = dll
+LIBSINGULAR_FLAGS = -shared
+LIBSINGULAR_LIBS = -lsingfac -lsingcf -lntl -lreadline -lgmp -lomalloc 
+endif
+

So, if one would try to build the debug version in 64 bit cygwin, then one had -lomalloc, but that would not work, since omalloc is disabled on purpose, for the debug version.

What to do? In principle, we could put an alternative version of the cygwin64 patch into the patches/conditional folder, and if we are in debug version, we could let choose_patches() replace the regular cygwin patch version by the conditional version.

I will try to solve the problem accordingly, however, I have no windows box. Hence, someone else needs to test if a debug version of singular works in cygwin.

Last edited 6 years ago by SimonKing (previous) (diff)

comment:2 Changed 6 years ago by SimonKing

PS: Instead of omalloc, the debug version's memory manager is called xalloc. This is a hint for a debug version in cygwin: Perhaps it needs to say -lxalloc rather than -lomalloc?

comment:3 follow-up: Changed 6 years ago by SimonKing

Unfortunately, one important debug patch, namely singular_xalloc.patch, fails to apply.

comment:4 in reply to: ↑ 3 Changed 6 years ago by SimonKing

Replying to SimonKing:

Unfortunately, one important debug patch, namely singular_xalloc.patch, fails to apply.

The good news is that many of the failing patches went upstream!

comment:5 Changed 6 years ago by SimonKing

I modified the patches so that they apply. However, the built fails, as the function omalloc0 is used in one place, but is not defined when using xalloc instead of omalloc. I guess I have to ask Hans Schönemann for advice.

comment:6 follow-up: Changed 6 years ago by vbraun

IMHO we shouldn't waste time on a Cygwin debug version. Thats for way down the road. If it doesn't work now, so be it.

comment:7 in reply to: ↑ 6 Changed 6 years ago by SimonKing

Replying to vbraun:

IMHO we shouldn't waste time on a Cygwin debug version. Thats for way down the road. If it doesn't work now, so be it.

Cygwin itself is not in the game. Only the cygwin PATCH currently constitutes a problem. It is not a big deal to solve it: I modified the patches and spkg-install so that building the debug version starts (i.e., so that all patches and changes apply).

Meanwhile Hans gave me a hint on where to get a missing function. After fixing that, the debug version will hopefully build.

comment:8 Changed 6 years ago by SimonKing

With Hans' hint, building the debug version went a bit further. However, building still fails, with "cannot find -lomalloc_ndebug".

OOOOPS! Looking for -lomalloc_ndebug in our patches, I see that the sage_debug.patch looks like this:

  • kernel/Makefile.in

    diff -ru src/kernel/Makefile.in src.debug/kernel/Makefile.in
    old new  
    4949CXXFLAGS       = @CXXFLAGS@ ${PIPE}
    5050CXXTEMPLFLAGS  = @CXXTEMPLFLAGS@
    5151CPPFLAGS       = -I${srcdir} -I.. -I@prefix@  @CPPFLAGS@
    52 DEFS           = -DNDEBUG -DOM_NDEBUG -D@SING_UNAME@ @DEFS@
     52DEFS           = -DOM_NDEBUG -D@SING_UNAME@ @DEFS@
    5353LDFLAGS                = @LDFLAGS@
    5454LD_DYN_FLAGS   = @LD_DYN_FLAGS@
    5555SFLAGS         = @SFLAGS@
     
    6464LIBS           = -lm @NEED_LIBS@
    6565else
    6666# for the 2-0-* versions under Windows, we don't need gdbm, readline and ncurses
    67 LIBS           = -lsingfac -lsingcf -lntl -lgmp -lreadline -lncurses -lomalloc_ndebug
     67LIBS           = -lsingfac -lsingcf -lntl -lgmp -lreadline -lncurses -lomalloc
    6868#LIBS          = -lsingfac -lsingcf -lgmp
    6969endif
    7070MP_LIBS                = @MP_LIBS@

Isn't the diff file formatted wrongly? I am talking about the paths "src/kernel" and "src.debug/kernel", which do not exist.

comment:9 Changed 6 years ago by SimonKing

Concerning ndebug: There is quite a number of files in upstream/singular-3.1.6/latest/ mentioning -lomalloc_ndebug. Probably we should change all of them to only mention -lomalloc.

comment:10 Changed 6 years ago by SimonKing

  • Branch set to u/SimonKing/sage_debug_version

comment:11 Changed 6 years ago by SimonKing

  • Commit set to bd802db07ff44b2cbbff87e1e900cf30fef4c04a

I have pushed my changes, so that someone else may have a change to see how to solve the -lomalloc_ndebug problem.


New commits:

b426226Make it so that the patches to Singular apply with SAGE_DEBUG=yes
3f5ee73Define omalloc0
bd802dbSlight improvement of a patch

comment:12 follow-up: Changed 6 years ago by vbraun

Patches are applied with -p1, so the first path component is stripped off.

comment:13 in reply to: ↑ 12 Changed 6 years ago by SimonKing

Replying to vbraun:

Patches are applied with -p1, so the first path component is stripped off.

OK, but I think it doesn't hurt that in the latest commit I change the path names according to what is done in the other patches (source path a/... versus target path b/...).

I got more Info from Hans. There is a debug version of omalloc, but it would still be omalloc. Hence, valgrind would not see details but would only see large pages allocated. So, he recommends to stick with xalloc (which is a compatibility layer on top of malloc).

If we would have Singular 4-x in Sage (we only have 3-1-6), xalloc would be part of the sources. Hence, all our current patching for the debug version wouldn't be needed.

As for -lomalloc_ndebug: Hans suggests to change the makefile so that a module libomalloc_ndebug.so rather than libomalloc.so is created. To my understanding, it would also allow to remove other parts of our patches, where we replace -lomalloc_ndebug by -lomalloc.

comment:14 Changed 6 years ago by vbraun

libm4ri fails its self-tests when debug is on, this is #16966.

comment:15 Changed 6 years ago by git

  • Commit changed from bd802db07ff44b2cbbff87e1e900cf30fef4c04a to 9c318934ef2fc17bb341cc7353e48253228c6955

Branch pushed to git repo; I updated commit sha1. New commits:

9c31893Define a missing symbol and change the module name: Singular with xalloc builds

comment:16 Changed 6 years ago by SimonKing

  • Authors set to Simon King
  • Report Upstream changed from N/A to None of the above - read trac for reasoning.
  • Status changed from new to needs_review

Meanwhile I got Singular's xalloc version (which is better for valgrinding than the debug version of omalloc) to build. Now I am trying to build the rest of Sage, but expect that I will soon run into the issue tracked at #16966...

In the latest commit I added one missing function (which is empty in the debug version) and I changed the module name from libomalloc.so into libomalloc_ndebug.so. It may seem strange for someone that the DEBUG version is named ..._ndebug. If that's a problem then (s)he may change it, so that it still works...

I am trying to continue building, potentially with m4ri in non-debug mode. I guess for now one can say that it needs review.

comment:17 Changed 6 years ago by vbraun

For the record, m4ri builds fine if you don't set SAGE_CHECK=yes

comment:18 follow-up: Changed 6 years ago by vbraun

I get /usr/bin/ld: cannot find -lomalloc when building from scratch...

comment:19 in reply to: ↑ 18 Changed 6 years ago by SimonKing

  • Status changed from needs_review to needs_work

Replying to vbraun:

I get /usr/bin/ld: cannot find -lomalloc when building from scratch...

It has built for me (without SAGE_CHECK=yes), but I suppose that libomalloc was present from previous attempts to build. So, let's try "make distclean" and try again...

comment:20 Changed 6 years ago by git

  • Commit changed from 9c318934ef2fc17bb341cc7353e48253228c6955 to f4c67018882e40679019e17ab1d58d8d1b6cc3b7

Branch pushed to git repo; I updated commit sha1. New commits:

f4c6701Replace lomalloc by lomalloc_ndebug everywhere in debug version

comment:21 Changed 6 years ago by SimonKing

  • Status changed from needs_work to needs_review

After make distclean I could reproduce the build errors.

The latest commit provides a solution. Perhaps you don't like it, for aesthetic reasons ("the debug version uses a module called omalloc_ndebug"), but it works. I also had to resolve a conflict with cygwin_64.patch to achieve it.

comment:22 Changed 6 years ago by vbraun

I'll give it a spin.

IMHO we don't have to win beauty contests with the debug version...

comment:23 follow-up: Changed 6 years ago by vbraun

Still doesn't build for me:

g++ -O0 -g  -fPIC -I.. -I/home/vbraun/Sage/git-develop/local -pipe -I. -I.. -I/home/vbraun/Sage/git-develop/local -I/home/vbraun/Sage/git-develop/local/include -I/home/vbraun/Sage/git-develop/local/include   -fno-implicit-templates -I.. -I/home/vbraun/Sage/git-develop/local -Dx86_64_Linux -DHAVE_CONFIG_H \
  -o Singular \
  tesths.cc iparith.o mpsr_Tok.o claptmpl.o\
  grammar.o scanner.o attrib.o blackbox.o eigenval_ip.o extra.o fehelp.o feOpt.o ipassign.o ipconv.o ipid.o iplib.o ipprint.o ipshell.o newstruct.o lists.o sdb.o fglm.o interpolation.o silink.o ssiLink.o s_buff.o subexpr.o janet.o wrapper.o libparse.o sing_win.o gms.o pcv.o maps_ip.o walk.o walk_ip.o cntrlc.o misc_ip.o calcSVD.o pipeLink.o Minor.o MinorProcessor.o MinorInterface.o bigintm.o pyobject_setup.o denom_list.o minpoly.o countedref.o semaphore.o slInit_Dynamic.o -rdynamic -L/home/vbraun/Sage/git-develop/local/kernel -L../kernel -lkernel_g -L/home/vbraun/Sage/git-develop/local/lib  -L/home/vbraun/Sage/git-develop/local/lib -lflint -lmpfr -lmpir  -ldl -lm -lsingfac -lsingcf -lflint -lmpfr -lntl -lgmp -lreadline -ltermcap -lnsl -lm  -lnsl -lbsd  -lomalloc -lpthread  ../kernel/mmalloc.o 
/usr/bin/ld: cannot find -lomalloc
collect2: error: ld returned 1 exit status
make[1]: *** [Singular] Error 1
make[1]: Leaving directory `/home/vbraun/Sage/git-develop/local/var/tmp/sage/build/singular-3.1.6.p3/src/latest/Singular'
make: *** [install-nolns] Error 1
Unable to build and install Singular
Error building Singular (error in build_singular).

comment:24 in reply to: ↑ 23 Changed 6 years ago by SimonKing

Replying to vbraun:

Still doesn't build for me:

Seriously? It did build for me, even though I did make distclean after making the makefile make omalloc_ndebug instead of omalloc.

g++ -O0 -g  -fPIC -I.. -I/home/vbraun/Sage/git-develop/local -pipe -I. -I.. -I/home/vbraun/Sage/git-develop/local -I/home/vbraun/Sage/git-develop/local/include -I/home/vbraun/Sage/git-develop/local/include   -fno-implicit-templates -I.. -I/home/vbraun/Sage/git-develop/local -Dx86_64_Linux -DHAVE_CONFIG_H \
  -o Singular \
  tesths.cc iparith.o mpsr_Tok.o claptmpl.o\...

The name tesths.cc suggests that this is for testing. So, did you build with SAGE_CHECK and SAGE_DEBUG together?

comment:25 follow-up: Changed 6 years ago by vbraun

Are you using 32bit? I did not enable SAGE_CHECKS.

comment:26 Changed 6 years ago by vbraun

  • Branch changed from u/SimonKing/sage_debug_version to u/vbraun/sage_debug_version

comment:27 in reply to: ↑ 25 Changed 6 years ago by SimonKing

  • Commit changed from f4c67018882e40679019e17ab1d58d8d1b6cc3b7 to 25b66c3f6ad76e06a334b18c3724f4c58537c987

Replying to vbraun:

Are you using 32bit? I did not enable SAGE_CHECKS.

No, I am on 64 bit. Is the file you mentioned only built on 32 bit??


New commits:

25b66c3Switch yet another -lomalloc

comment:28 Changed 6 years ago by SimonKing

OK, the change sounds reasonable.

comment:29 Changed 6 years ago by vbraun

No, I'm on x86 too. The issue comes from the test "$ac_cv_sizeof_voidp" != 4; so I thought it might be a bitwidth thing.

comment:30 Changed 6 years ago by SimonKing

By the way: I did not run the full test suite yet (now I am rebuilding from scratch), but found a surprisingly high number of test errors, given that not so long time ago the debug version passed all tests.

comment:31 Changed 6 years ago by vbraun

Most failures seem to be from non-commutative stuff...

comment:32 Changed 6 years ago by vbraun

For example:

(gdb) break dReportError
Breakpoint 1 at 0x7fffcc699a71: file dError.c, line 28.
(gdb) cont
Continuing.

sage: F.<x,y,z> = FreeAlgebra(QQ, implementation='letterplace')
sage: I = F*[x*y+y*z,x^2+x*y-y*x-y^2]*F
sage: Q.<a,b,c> = F.quo(I); Q
Quotient of Free Associative Unital Algebra on 3 generators (x, y, z) over Rational Field by the ideal (x*y + y*z, x*x + x*y - y*x - y*y)
sage: a^3

Breakpoint 1, dReportError (fmt=0x7fffcc8b2679 "S_2_R[%d]=%d != T[%d].i_r=%d\n") at dError.c:28
28	dError.c: No such file or directory.
(gdb) bt
#0  dReportError (fmt=0x7fffcc8b2679 "S_2_R[%d]=%d != T[%d].i_r=%d\n") at dError.c:28
#1  0x00007fffcc5b860a in kTest_TS (strat=0x2d09920) at kutil.cc:861
#2  0x00007fffcc5b0598 in bbaShift (F=0x18d0638, Q=0x0, w=0x0, hilb=0x0, strat=0x2d09920, uptodeg=3, lV=3)
    at kstd2.cc:1724
#3  0x00007fffcc5a79ed in kStdShift (F=0x18d0638, Q=0x0, h=isHomog, w=0x0, hilb=0x0, syzComp=0, 
    newIdeal=0, vw=0x0, uptodeg=3, lV=3) at kstd1.cc:1912
#4  0x00007fffcc5b09ce in freegb (I=0x18d0638, uptodeg=3, lVblock=3) at kstd2.cc:1823
#5  0x00007fffcc4e2301 in jjSYSTEM (res=0x2d06588, args=0x18d0548) at extra.cc:1370
#6  0x00007fffcc4a06b5 in iiExprArithM (res=0x2d06588, a=0x18d0548, op=496) at iparith.cc:8321
#7  0x00007fffcb9c3eb9 in __pyx_f_4sage_4libs_8singular_8function_17KernelCallHandler_handle_call (
    __pyx_v_self=0x7fffb80d2880, __pyx_v_argument_list=0x7fffb80d1860, __pyx_optional_args=0x7fffffff7ef0)
    at build/cythonized/sage/libs/singular/function.cpp:12083
[...]
(gdb) cy bt
[...]
#78 0x00007ffff7cd7b2d in _reduce_() at /home/vbraun/Sage/git-develop/local/lib/python2.7/site-packages/sage/rings/quotient_ring_element.py:118
       118            self.__rep = I.reduce(self.__rep)
#83 0x00007fffb5ae04ce in reduce() at /home/vbraun/Sage/git-develop/src/sage/algebras/letterplace/letterplace_ideal.pyx:306
       306        def reduce(self, G):
#87 0x00007fffb5f90e45 in normal_form() at /home/vbraun/Sage/git-develop/src/sage/algebras/letterplace/free_algebra_element_letterplace.pyx:733
       733        def normal_form(self,I):
#93 0x00007fffb5ad9bb4 in groebner_basis() at /home/vbraun/Sage/git-develop/src/sage/algebras/letterplace/letterplace_ideal.pyx:187
---Type <return> to continue, or q <return> to quit---
       187        def groebner_basis(self, degbound=None):
#96 0x00007fffcb9c5095 in __call__() at /home/vbraun/Sage/git-develop/src/sage/libs/singular/function.pyx:1185
      1185        def __call__(self, *args, ring=None, bint interruptible=True, attributes=None):
#98 0x00007fffcb9ca76b in call_function() at /home/vbraun/Sage/git-develop/src/sage/libs/singular/function.pyx:1462
      1462                _res = self.call_handler.handle_call(argument_list, si_ring)
#99 0x00007fffcb9c3d15 in handle_call() at /home/vbraun/Sage/git-develop/src/sage/libs/singular/function.pyx:1093
      1093                iiExprArithM(res, argument_list.args, self.cmd_n)
#100 0x00007fffcc4a029d in iiExprArithM()

comment:33 Changed 6 years ago by vbraun

I can reproduce some failures just in Singular, so I've asked here: https://groups.google.com/d/msg/libsingular-devel/4fL9NKCVGw0/mlOd_vzNSG8J

comment:34 Changed 6 years ago by git

  • Commit changed from 25b66c3f6ad76e06a334b18c3724f4c58537c987 to a38e790babdd0c0cc6ced2d79dd16c0603072cc3

Branch pushed to git repo; I updated commit sha1. New commits:

a38e790Fix dangling C pointer to Python temporary

comment:35 Changed 6 years ago by git

  • Commit changed from a38e790babdd0c0cc6ced2d79dd16c0603072cc3 to 0802e935cab470a2ce9ee9102203d9b27b55b1af

Branch pushed to git repo; I updated commit sha1. New commits:

1cd65a8Remove some failing Singular debug statements
0802e93Reduce size of doctest

comment:36 Changed 6 years ago by vbraun

I think some internal Singular checks that get enabled with NDEBUG are too strict, I've added a patch removing these. I also almost despaired in fast_map until I noticed that idTest incorrectly assumes that currRing is set.

Now make ptest passes.

comment:37 Changed 6 years ago by git

  • Commit changed from 0802e935cab470a2ce9ee9102203d9b27b55b1af to 37fda1dff0d1f58d7b07489a250526d735a12c62

Branch pushed to git repo; I updated commit sha1. New commits:

37fda1dReduce scale of some more huge doctests

comment:38 Changed 6 years ago by vbraun

Computing the invariants of an order-480 group after an order-48 group really only checks that you have a fast computer.

This fixes the last timeout in the make ptestlong for me.

comment:39 Changed 6 years ago by vbraun

  • Authors changed from Simon King to Simon King, Volker Braun
  • Report Upstream changed from None of the above - read trac for reasoning. to Reported upstream. No feedback yet.

comment:40 Changed 6 years ago by vbraun

  • Cc jpflori added; jpflory removed

comment:41 Changed 6 years ago by SimonKing

Hi Volker, I am not happy with the two commits to reduce size of doctests.

It is said in the doc that below we do the Fateman benchmark over the finite field of order 32003, but I guess with your changes the statement becomes wrong, it isn't the Fateman benchmark if the exponent is only 10 and not 20.

Also I don't see why a test that is marked "long" anyway should be removed just because the debug version slows it down too much.

By the way, how long does this test take in non-debug version? It is stated that it used to take 12 seconds, four years ago. And 12 seconds is totally fine for a long test, IMHO.

comment:42 follow-up: Changed 6 years ago by vbraun

They are doctests, not benchmarks. Don't confuse the two. Really, showing off what size of a problem you can do in doctests is just an overall drag on development as people have to wait longer and longer for tests. IMHO 12 seconds is already on the long side, you should aim for <=5 for long tests.

Put differently, would you rather a) have a buildbot with SAGE_DEBUG, or b) doctest a huge computation and then discard the result because it can't do anything useful with the timings obtained.

comment:43 in reply to: ↑ 42 Changed 6 years ago by SimonKing

Replying to vbraun:

They are doctests, not benchmarks. Don't confuse the two. Really, showing off what size of a problem you can do in doctests is just an overall drag on development as people have to wait longer and longer for tests. IMHO 12 seconds is already on the long side, you should aim for <=5 for long tests.

Put differently, would you rather a) have a buildbot with SAGE_DEBUG, or b) doctest a huge computation and then discard the result because it can't do anything useful with the timings obtained.

I'd rather have c): A buildbot with SAGE_DEBUG that only runs the tests that are not marked long.

Last edited 6 years ago by SimonKing (previous) (diff)

comment:44 Changed 6 years ago by vbraun

I disagree. Doctests are not benchmarks. Doctests coverage is more important than benchvertising. We should have a framework for benchmarks in addition to doctests, but stuffing benchmarks into doctests is counterproductive.

comment:45 Changed 6 years ago by SimonKing

I can not tell why the patchbot scripts are complaining. In any case, I get the following test failures:

sage -t src/sage/stats/hmm/hmm.pyx
    [118 tests, 2 failures, 1.70 s]
...
sage -t src/sage/gsl/probability_distribution.pyx
    [207 tests, 3 failures, 11.09 s]
...
sage -t src/sage/plot/plot3d/transform.pyx
    [25 tests, 1 failure, 6.27 s]

Unfortunately it doesn't say what the failure is. Why does it not?

comment:46 follow-up: Changed 6 years ago by vbraun

Try sage -t --verbose. Also, I've never seen a failure to report whats wrong. Can you post the full log?

comment:47 in reply to: ↑ 46 ; follow-up: Changed 6 years ago by SimonKing

Replying to vbraun:

Try sage -t --verbose.

**********************************************************************
File "src/sage/stats/hmm/hmm.pyx", line 282, in sage.stats.hmm.hmm.DiscreteHiddenMarkovModel
Failed example:
    m
Expected:
    Discrete Hidden Markov Model with 2 States and 2 Emissions
    Transition matrix:
    [1.0134345614...e-70               1.0]
    [              1.0 3.997435271...e-19]
    Emission matrix:
    [7.3802215662...e-54               1.0]
    [              1.0  3.99743526...e-19]
    Initial probabilities: [0.0000, 1.0000]
Got:
    Discrete Hidden Markov Model with 2 States and 2 Emissions
    Transition matrix:
    [1.0134345614745788e-70                    1.0]
    [                   1.0 3.9974352713558623e-19]
    Emission matrix:
    [ 7.380221566254936e-54                    1.0]
    [                   1.0 3.9974352626002193e-19]
    Initial probabilities: [0.0000, 1.0000]
**********************************************************************
File "src/sage/stats/hmm/hmm.pyx", line 1236, in sage.stats.hmm.hmm.DiscreteHiddenMarkovModel.baum_welch
Failed example:
    m.baum_welch(v)
Expected:
    (-66.7823606592935..., 100)
Got:
    (-66.7823606592936, 100)
**********************************************************************
File "src/sage/gsl/probability_distribution.pyx", line 364, in sage.gsl.probability_distribution.RealDistribution
Failed example:
    T.cum_distribution_function_inv(.5)
Expected:
    3.532230067546424
Got:
    3.5322300675464247
**********************************************************************
File "src/sage/gsl/probability_distribution.pyx", line 402, in sage.gsl.probability_distribution.RealDistribution
Failed example:
    T.distribution_function(0)
Expected:
    0.3183098861837906
Got:
    0.3183098861837905
Trying (line 404):    T.cum_distribution_function(1)
Expecting:
    0.7499999999999998
**********************************************************************
File "src/sage/gsl/probability_distribution.pyx", line 404, in sage.gsl.probability_distribution.RealDistribution
Failed example:
    T.cum_distribution_function(1)
Expected:
    0.7499999999999998
Got:
    0.75
*********************************************************************
File "src/sage/plot/plot3d/transform.pyx", line 179, in sage.plot.plot3d.transform.rotate_arbitrary
Failed example:
    rotate_arbitrary((1,1,1), 2*pi/3) * vector(RDF, (1,2,3))  # rel tol 1e-15
Expected:
    (1.9999999999999996, 2.9999999999999996, 0.9999999999999999)
Got:
    (2.000000000000001, 3.0000000000000013, 1.000000000000001)
Tolerance exceeded in 1 of 3:
    0.9999999999999999 vs 1.000000000000001, tolerance 1e-15 > 1e-15

Also, I've never seen a failure to report whats wrong.

Yes, but I have always seen a failure to report what has failed.

But here, there is simply nothing in the log. It simply says that some tests failed, but not WHICH test failed, not what was the expected output and the actual output, not whether there has been a crash. Absolutely nothing.

Can you post the full log?

Not before Monday (bandwidth).

Anyway, the verbose output above indicates that we have numerical noise, not related with Singular. By the way, if that's relevant: In my local branch I have merged your gdb branch from #16978.

comment:48 Changed 6 years ago by vbraun

  • Branch changed from u/vbraun/sage_debug_version to u/vbraun/sage_debug_version_2

comment:49 Changed 6 years ago by vbraun

  • Commit changed from 37fda1dff0d1f58d7b07489a250526d735a12c62 to 3dedecd8dbe079567d53f37d3666e1419ed51a43
  • Status changed from needs_review to needs_work

Hans sent a patch for one of the debug build issues to libsingular-devel, I've added it and removed my patch that disabled various checks. If he manages to fix the rest then we might not need my patch...


New commits:

75940b2Reduce size of doctest
d737eb4Reduce scale of some more huge doctests
3dedecdFix erroneous "mixed poly/vector" error message

comment:50 Changed 6 years ago by git

  • Commit changed from 3dedecd8dbe079567d53f37d3666e1419ed51a43 to ceb83f9a60ddd6d4f60fcc6ba0e67a4cc3c49563

Branch pushed to git repo; I updated commit sha1. New commits:

ceb83f9Fix slimgb and FreeAlgebra errors

comment:51 Changed 6 years ago by git

  • Commit changed from ceb83f9a60ddd6d4f60fcc6ba0e67a4cc3c49563 to 9233d05bfaf8fd2d5a005d6880c30448fd750a77

Branch pushed to git repo; I updated commit sha1. New commits:

9233d05Fix quo_rem and macaulay_resultant errors

comment:52 follow-ups: Changed 6 years ago by vbraun

  • Status changed from needs_work to needs_review

I've reported the failures on libsingular-devel and Hans posted patches for all of them. Branch now compiles/doctests fine for me without disabling any debug statements.

comment:53 in reply to: ↑ 52 Changed 6 years ago by SimonKing

  • Description modified (diff)
  • Report Upstream changed from Reported upstream. No feedback yet. to Fixed upstream, in a later stable release.

Replying to vbraun:

I've reported the failures on libsingular-devel and Hans posted patches for all of them. Branch now compiles/doctests fine for me without disabling any debug statements.

Excellent!

comment:54 in reply to: ↑ 52 Changed 6 years ago by SimonKing

Replying to vbraun:

I've reported the failures on libsingular-devel and Hans posted patches for all of them. Branch now compiles/doctests fine for me without disabling any debug statements.

The branch did not change since four days. So, is it the case that the branch contains all of Hans' patches, or do you need to push another commit?

comment:55 Changed 6 years ago by vbraun

I was just busy elsewhere...

comment:56 Changed 6 years ago by vbraun

imho there is little to review here... it works, and most of the patches come straight from upstream.

comment:57 Changed 6 years ago by SimonKing

I'll try to review it, although I am one of the authors...

comment:58 Changed 6 years ago by jdemeyer

  • Branch changed from u/vbraun/sage_debug_version_2 to u/jdemeyer/ticket/16938
  • Created changed from 09/05/14 14:07:35 to 09/05/14 14:07:35
  • Modified changed from 09/30/14 12:30:41 to 09/30/14 12:30:41

comment:59 in reply to: ↑ 47 Changed 6 years ago by jdemeyer

  • Commit changed from 9233d05bfaf8fd2d5a005d6880c30448fd750a77 to 44ac240d3a42ae8e3cd3b1e1d0b725f937737d34

Replying to SimonKing:

But here, there is simply nothing in the log. It simply says that some tests failed, but not WHICH test failed, not what was the expected output and the actual output, not whether there has been a crash. Absolutely nothing.

I have heard occasional rumours that this happens sometimes, but I have no idea how to reproduce it.


New commits:

80ef8cdMerge remote-tracking branch 'origin/develop' into ticket/16938
44ac240Typos

comment:60 Changed 6 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to positive_review

comment:61 Changed 6 years ago by SimonKing

Thank you, Jeroen. I would have needed to run tests overnight to finish the review. But I do think that the patched patches look OK.

comment:62 follow-up: Changed 6 years ago by SimonKing

  • Status changed from positive_review to needs_info

The conway_polynomials package fails to build, even though I have SAGE_UPGRADING set to yes. See logs

Is there a problem with the debug? Or is it a problem with "make" and dependency checking?

comment:63 Changed 6 years ago by jdemeyer

Did you build everything with this ticket applied and with SAGE_DEBUG=yes?

comment:64 Changed 6 years ago by git

  • Commit changed from 44ac240d3a42ae8e3cd3b1e1d0b725f937737d34 to 0fb627916b73a741be1d942b0d94fd0046dc2536

Branch pushed to git repo; I updated commit sha1. New commits:

0fb6279Trac #16938: bump Singular version

comment:65 in reply to: ↑ 62 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

Is there a problem with the debug?

Looks like a genuine bug to me in libm4ri. I don't know why they need sizeof(mzd_t) == 64, but that equality is only true on 64-bit systems. Since two of the fields in mzd_t are pointers, this would give only 56 bytes on 32-bit systems.

I'll try to reproduce it and report upstream if it is confirmed.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:66 Changed 6 years ago by git

  • Commit changed from 0fb627916b73a741be1d942b0d94fd0046dc2536 to 53b73dbfaf33d8e28840d5b6ad91ce1bf941cffa

Branch pushed to git repo; I updated commit sha1. New commits:

53b73dbFix m4ri assertion sizeof(mzd_t) == 64

comment:67 in reply to: ↑ 65 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Is there a problem with the debug?

Looks like a genuine bug to me in libm4ri. I don't know why they need sizeof(mzd_t) == 64, but that equality is only true on 64-bit systems. Since two of the fields in mzd_t are pointers, this would give only 56 bytes on 32-bit systems.

I'll try to reproduce it and report upstream if it is confirmed.

Would be surprising to me. After all, in an earlier debug version on this ticket, Sage did build. Now it doesn't. Anyway, I will try with your latest commit.

comment:68 in reply to: ↑ 67 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

Would be surprising to me. After all, in an earlier debug version on this ticket, Sage did build.

Was that after #16981?

comment:69 in reply to: ↑ 68 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Would be surprising to me. After all, in an earlier debug version on this ticket, Sage did build.

Was that after #16981?

No idea. And I actually doubt that M4RI is to blame. After all, the error occurs in the line

>    4    from sage.all import save

when installing the Conway polynomial package.

comment:70 Changed 6 years ago by SimonKing

Note that in addition to the install log that I posted earlier, there also is a Crash Log that is the result of the failing import.

comment:71 in reply to: ↑ 69 Changed 6 years ago by jdemeyer

Replying to SimonKing:

the error occurs in the line

>    4    from sage.all import save

when installing the Conway polynomial package.

That's just the top of the traceback. Importing sage.all does a lot of things. The full traceback clearly shows that the problem is in m4ri:

Stack backtrace
---------------
[...]
#8  0xb72e9d07 in __assert_fail () from /lib/libc.so.6
No symbol table info available.
#9  0xb18e62f1 in mzd_init (r=7, c=7) at m4ri/mzd.c:151
        __PRETTY_FUNCTION__ = "mzd_init"
#10 0xb19270c4 in __pyx_pf_4sage_6matrix_17matrix_mod2_dense_17Matrix_mod2_dense___cinit__ (__pyx_v_self=0xad884034, __pyx_v_parent=0xace7b2f4, 
    __pyx_v_entries=0xad874fbc, __pyx_v_copy=0xb76a48ec <_Py_ZeroStruct>, 
    __pyx_v_coerce=0xb76a4900 <_Py_TrueStruct>, 
    __pyx_v_alloc=0xb76a4900 <_Py_TrueStruct>)
    at build/cythonized/sage/matrix/matrix_mod2_dense.c:3313
        __pyx_r = 1245376913
        __pyx_t_1 = 0x0
        __pyx_t_2 = 0x0
        __pyx_t_3 = 0x0
        __pyx_t_4 = 0
        __pyx_t_5 = 0x0
        __pyx_t_6 = 1
        __pyx_lineno = 0
        __pyx_filename = 0x0
        __pyx_clineno = 0
[...]

comment:72 Changed 6 years ago by jdemeyer

  • Description modified (diff)
  • Report Upstream changed from Fixed upstream, in a later stable release. to Reported upstream. No feedback yet.
  • Status changed from needs_info to needs_review

comment:73 Changed 6 years ago by SimonKing

  • Status changed from needs_review to needs_work

The build problem that I obtain is with the latest commit (53b73dbfaf33d8e28840d5b6ad91ce1bf941cffa), hence, it needs work.

comment:74 follow-up: Changed 6 years ago by jdemeyer

  • Status changed from needs_work to needs_review

Are you really really sure? The patch fixes the problem for me. Don't forget to rebuild m4ri!

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:75 in reply to: ↑ 74 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Are you really really sure? The patch fixes the problem for me. Don't forget to rebuild m4ri!

Then we come back to my question above: Is it a problem with dependency checking? I did make with SAGE_UPGRADING=yes, and it didn't work.

comment:76 Changed 6 years ago by SimonKing

For the record: Now I explicitly did sage -i m4ri before doing make again. Let's see if it works this time.

comment:77 Changed 6 years ago by SimonKing

It does start. Now let us wait for the tests...

comment:78 in reply to: ↑ 75 Changed 6 years ago by jdemeyer

Replying to SimonKing:

Is it a problem with dependency checking?

Not really. You built a broken version of m4ri before and that broken version will not automatically be updated when doing "make". However, as long as you don't first build a broken version with SAGE_DEBUG=yes without applying this patch, there is no problem.

One could argue that there is a problem with dependency checking in the sense that SAGE_DEBUG=yes does not automatically cause everything to get rebuilt. But that's really a different issue.

comment:79 Changed 6 years ago by SimonKing

Next, I was bitten again by the infamous "incremental documentation builds sometimes cause spurious". Sigh.

comment:80 follow-ups: Changed 6 years ago by vbraun

hashdist also takes build switches / environment variables into account for the dependency calculation, fwiw.

Until then I think its reasonable to require a "make distclean" to switch between debug and non-debug build.

comment:81 in reply to: ↑ 80 Changed 6 years ago by jdemeyer

Replying to vbraun:

hashdist

Is this another of the many proposals to change the Sage build system with something else? I feel like I have seen too many of these...

comment:82 follow-up: Changed 6 years ago by SimonKing

So far there is one error, and it seems to be a serious problem, namely a segmentation fault:

sage -t src/sage/modular/modsym/space.py
**********************************************************************
File "src/sage/modular/modsym/space.py", line 930, in sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module
Failed example:
    ModularSymbols(11, 2, base_ring=GF(4,'a')).cuspidal_submodule()._q_expansion_module(prec=4, algorithm="hecke")
Exception raised:
    Traceback (most recent call last):
      File "/home/king/Sage/debug/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/king/Sage/debug/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 849, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module[0]>", line 1, in <module>
        ModularSymbols(Integer(11), Integer(2), base_ring=GF(Integer(4),'a')).cuspidal_submodule()._q_expansion_module(prec=Integer(4), algorithm="hecke")
      File "/home/king/Sage/debug/sage/local/lib/python2.7/site-packages/sage/modular/modsym/ambient.py", line 1392, in cuspidal_submodule
        S = self.boundary_map().kernel()
      File "/home/king/Sage/debug/sage/local/lib/python2.7/site-packages/sage/modules/matrix_morphism.py", line 649, in kernel
        V = self.matrix().kernel()
      File "sage/matrix/matrix2.pyx", line 3755, in sage.matrix.matrix2.Matrix.left_kernel (build/cythonized/sage/matrix/matrix2.c:23920)
      File "sage/matrix/matrix2.pyx", line 3596, in sage.matrix.matrix2.Matrix.right_kernel (build/cythonized/sage/matrix/matrix2.c:23371)
      File "sage/matrix/matrix2.pyx", line 3244, in sage.matrix.matrix2.Matrix.right_kernel_matrix (build/cythonized/sage/matrix/matrix2.c:22751)
      File "sage/matrix/matrix2.pyx", line 6119, in sage.matrix.matrix2.Matrix.echelon_form (build/cythonized/sage/matrix/matrix2.c:42445)
      File "sage/matrix/matrix_mod2e_dense.pyx", line 946, in sage.matrix.matrix_mod2e_dense.Matrix_mod2e_dense.echelonize (build/cythonized/sage/matrix/matrix_mod2e_dense.c:7938)
      File "sage/ext/c_lib.pyx", line 97, in sage.ext.c_lib.sig_raise_exception (build/cythonized/sage/ext/c_lib.c:1168)
    SignalError: Segmentation fault
**********************************************************************
1 item had failures:
   1 of   3 in sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module
    [313 tests, 1 failure, 43.51 s]

comment:83 in reply to: ↑ 80 ; follow-up: Changed 6 years ago by SimonKing

Replying to vbraun:

Until then I think its reasonable to require a "make distclean" to switch between debug and non-debug build.

I did not switch between debug and non-debug build. I have one sage git repository which I have never touched without SAGE_DEBUG being set to yes. Nevertheless, things got wrong.

comment:84 in reply to: ↑ 83 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

Nevertheless, things got wrong.

...while testing a ticket in development. You cannot do development without things breaking now and then. The goal is that the end result of a ticket works, there are no guarantees about what happens in between.

comment:85 Changed 6 years ago by vbraun

  • Cc malb added

Martin, can you comment on the 64-bit alignment issue in m4ri?

comment:86 in reply to: ↑ 82 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

So far there is one error, and it seems to be a serious problem, namely a segmentation fault:

Works for me. The problem seems to be in m4rie, did you remember to rebuild with SAGE_UPGRADING=yes after rebuilding mpir?

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:87 in reply to: ↑ 86 Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

So far there is one error, and it seems to be a serious problem, namely a segmentation fault:

Works for me. Did you remember to rebuild with SAGE_UPGRADING=yes after rebuilding mpir?

I currently have SAGE_UPGRADING=yes by default.

comment:88 in reply to: ↑ 84 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Nevertheless, things got wrong.

...while testing a ticket in development. You cannot do development without things breaking now and then. The goal is that the end result of a ticket works, there are no guarantees about what happens in between.

OK, but I think that dependency checking should work, regardless whether it is a debug or a non-debug built.

Anyway, meanwhile I got loads of other errors. I will next try make distclean and make, with SAGE_DEBUG=yes and unsetting SAGE_UPGRADING.

comment:89 in reply to: ↑ 88 Changed 6 years ago by jdemeyer

Replying to SimonKing:

OK, but I think that dependency checking should work, regardless whether it is a debug or a non-debug built.

Dependency checking works (modulo #17072).

The problem is that building with SAGE_DEBUG=yes didn't work before, but the problem with m4ri wasn't seen at build-time of m4ri. Therefore, the build system thinks that the build of m4ri was successful, even though it was completely broken. This is a very specific set of circumstances, I don't think it requires a fix to the build system.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:90 follow-up: Changed 6 years ago by jdemeyer

Simon, just to be sure, I will try to reproduce your bug on a 32-bit system:

$ export SAGE_UPGRADING=yes SAGE_DEBUG=yes
$ git checkout 0fb627916b73a741be1d942b0d94fd0046dc2536
$ make    # should fail in conway_polynomials
$ git checkout 53b73dbfaf33d8e28840d5b6ad91ce1bf941cffa
$ ./sage -f m4ri
$ make ptest

comment:91 in reply to: ↑ 90 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Simon, just to be sure, I will try to reproduce your bug on a 32-bit system:

$ export SAGE_UPGRADING=yes SAGE_DEBUG=yes
$ git checkout 0fb627916b73a741be1d942b0d94fd0046dc2536
$ make    # should fail in conway_polynomials
$ git checkout 53b73dbfaf33d8e28840d5b6ad91ce1bf941cffa
$ ./sage -f m4ri
$ make ptest

I am on 64 bit, if I am not mistaken.

comment:92 in reply to: ↑ 91 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

I am on 64 bit, if I am not mistaken.

Your previous error was on a 32-bit system.

comment:93 in reply to: ↑ 92 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

I am on 64 bit, if I am not mistaken.

Your previous error was on a 32-bit system.

I did all tests related with this ticket on the same machine, namely my OpenSuse laptop, and I am pretty sure that it is x86_64. Although I can currently not recall how to find out the architecture. In any case, cat /proc/cpuinfo tells me that I have two Intel(R) Core(TM)2 Duo.

Last edited 6 years ago by SimonKing (previous) (diff)

comment:94 in reply to: ↑ 93 Changed 6 years ago by jdemeyer

Replying to SimonKing:

I did all tests related with this ticket on the same machine, namely my OpenSuse laptop, and I am pretty sure that it is x86_64.

Perhaps the hardware is 64-bit, but the kernel in the logs you posted is 32-bit.

Although I can currently not recall how to find out the architecture.

uname -m prints the "machine hardware name" which is probably what you want.

comment:95 Changed 6 years ago by SimonKing

uname -m
i686

whatever that means.

comment:96 follow-up: Changed 6 years ago by vbraun

That means 32-bit with a certain set of ISA extensions beyond the i386 (requires Pentium Pro / Pentium II and newer). If you have < 4GB then this is would be the recommended installation.

comment:97 in reply to: ↑ 96 Changed 6 years ago by SimonKing

Replying to vbraun:

That means 32-bit with a certain set of ISA extensions beyond the i386 (requires Pentium Pro / Pentium II and newer). If you have < 4GB then this is would be the recommended installation.

Interesting. Currently I have 4 GB. Can I change the installation without to much hassle when upgrading the memory in future?

comment:98 Changed 6 years ago by vbraun

I don't think any distro supports architecture changes. Just install x86_64 on a separate partition / btrfs volume and mound your /home.

comment:99 Changed 6 years ago by SimonKing

After make distclean, make test did almost work. I got no timeouts, and the following errors:

sage -t src/sage/stats/hmm/hmm.pyx  # 2 doctests failed
sage -t src/sage/schemes/elliptic_curves/period_lattice_region.pyx  # 1 doctest failed
sage -t src/sage/gsl/probability_distribution.pyx  # 3 doctests failed
sage -t src/sage/plot/plot3d/transform.pyx  # 1 doctest failed

Unfortunately, I again encounter the problem that the test log does not indicate what tests went wrong, it just gives the number of failing tests.

comment:100 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:101 Changed 6 years ago by jdemeyer

There is a very strange thing happening on this 32-bit system. When running doctests with make ptest, a lot of tests fail. When running those tests indivually, most of them pass (except for the ones mentioned by Simon). So there is certainly something fishy going on.

This might be due to https://savannah.gnu.org/bugs/?22010, I'm upgrading make to see if that fixes it.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:102 Changed 6 years ago by jdemeyer

I am investigating the doctest failures reported by Simon, hang on...

comment:103 Changed 6 years ago by jdemeyer

  • Dependencies set to #17088

comment:104 Changed 6 years ago by git

  • Commit changed from 53b73dbfaf33d8e28840d5b6ad91ce1bf941cffa to c471edd674e461bb2db75fc899d44fb75b745bba

Branch pushed to git repo; I updated commit sha1. New commits:

573cf86Fix numerical noise in debug build
15b7588Use integer arithmetic in PeriodicRegion.__div__
c471eddMerge branch 'u/jdemeyer/ticket/17088' of git://trac.sagemath.org/sage into ticket/16938

comment:105 follow-up: Changed 6 years ago by SimonKing

Can you elaborate on the "periodic lattice" commit?

comment:106 in reply to: ↑ 105 ; follow-up: Changed 6 years ago by jdemeyer

Replying to SimonKing:

Can you elaborate on the "periodic lattice" commit?

See #17088

comment:107 in reply to: ↑ 106 Changed 6 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Can you elaborate on the "periodic lattice" commit?

See #17088

OK. Running tests now. Hope it will work this time...

comment:108 follow-up: Changed 6 years ago by jdemeyer

I still get random segmentation faults with long tests in src/sage/matrix/matrix_integer_dense_hnf.py

comment:109 Changed 6 years ago by jdemeyer

If you're interested in debugging on Linux: see also #17089.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:110 in reply to: ↑ 108 ; follow-up: Changed 6 years ago by SimonKing

Replying to jdemeyer:

I still get random segmentation faults with long tests in src/sage/matrix/matrix_integer_dense_hnf.py

O_o...

I thought the point of a debug version is that all segmentation faults will be non-random, reproducible and accompanied by a useful C backtrace.

comment:111 in reply to: ↑ 110 Changed 6 years ago by jdemeyer

Replying to SimonKing:

accompanied by a useful C backtrace.

...at least we have this (modulo #17089)

comment:112 Changed 6 years ago by jdemeyer

The command which crashes is

sage: from sage.matrix.matrix_integer_dense_hnf import benchmark_hnf; benchmark_hnf([50,100],32)

comment:113 follow-up: Changed 6 years ago by SimonKing

I wonder whether random segmentation faults in the debug version should prevent this ticket from getting a positive review. After all, if segmentation faults show up then probably there is a problem in the non-debug version as well. This ticket is not about fixing the segmentation fault, but it is about providing the means to analyse segmentation fault.

comment:114 in reply to: ↑ 113 Changed 6 years ago by jdemeyer

  • Status changed from needs_work to needs_review

Replying to SimonKing:

This ticket is not about fixing the segmentation fault, but it is about providing the means to analyse segmentation fault.

Exactly, the segfault issue is #17090.

comment:115 Changed 6 years ago by vbraun

  • Authors changed from Simon King, Volker Braun to Simon King, Volker Braun, Jeroen Demeyer
  • Reviewers changed from Jeroen Demeyer to Jeroen Demeyer, Volker Braun
  • Status changed from needs_review to positive_review

lgtm

comment:116 Changed 6 years ago by vbraun

  • Branch changed from u/jdemeyer/ticket/16938 to c471edd674e461bb2db75fc899d44fb75b745bba
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.