Opened 9 years ago

Last modified 8 years ago

#14625 new defect

lrcalc calls exit()

Reported by: thansen Owned by: jdemeyer
Priority: major Milestone: sage-6.4
Component: packages: standard Keywords: spkg, library, lrcalc
Cc: thansen, zabrocki, aschilling, mhansen, sage-combinat, asbuch, mguaypaq, saliola, darij, tfeulner, tscrim, nthiery Merged in:
Authors: Reviewers:
Report Upstream: Reported upstream. Developers acknowledge bug. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

While packaging lrcalc for Debian, our qa tool lintian detected that the lrcalc library is calling exit(). And indeed, it calls exit and shuts down Sage every time it runs out of memory, in mathlib/alloc.c.

We already discussed this with the upstream author Anders Skovsted Buch and with Nicolas Thiéry. It's a bug in lrcalc, but since Anders has no time to fix it, one could fix it in Sage first. Normally one would make the library report an error and let the program using the library handle it. The problem is that lrcalc does not free its memory and returns to a clean state after an error. Anders says this would be difficult to achieve. A workaround would be to perform each calculation in a new thread that is thrown away in the case of an error.

Version: 1.1.6

Steps to reproduce:

Limit the memory for this shell to 900 MB so the memory will quickly be full:
ulimit -Sv 900000

Start Sage and run a lrcalc command that needs a lot of memory:
sage
sage: import sage.libs.lrcalc.lrcalc as lrcalc
sage: sage.libs.lrcalc.lrcalc.mult([20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1], [20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1])

Change History (16)

comment:1 Changed 9 years ago by thansen

  • Cc thansen added

comment:2 Changed 9 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:3 Changed 9 years ago by chapoton

  • Keywords lrcalc added

comment:4 Changed 9 years ago by zabrocki

  • Cc zabrocki aschilling mhansen sage-combinat asbuch mguaypaq saliola darij tfeulner tscrim added

Anne forwarded me this message from Anders and I am cc'ing all the relevant suspects and then some. This issue is relevant because of a bug found in the Schur product in symmetrica (reported in #15397). I am proposing that lrcalc be used to compute the product of two Schur functions.:

My latest version of lrcalc is 1.1.7. I'm not sure what has been included in sage. The latest version of the library requires that the calling program uses setjmp() to recover from a failed call to lrcalc. Probably sage has a wrapper function that calls the lrcalc library already. It should be modified as follows:

result_type * call_lrcalc_mult(args)
{
   if (setjmp(lrcalc_panic_frame))
     {
       /* if this code is executed, then lrcalc ran out of memory. */
       return NULL;
     }
   prd = mult(args);
   sageprd = < convert prd to sage structures >
   return sageprd;
}

But again, to obtain any real improvement, somebody has to figure out how much memory lrcalc can safely use, this might not be entirely trivial to determine.

comment:5 follow-up: Changed 9 years ago by zabrocki

Another message that Anne forwarded to me relevant to this ticket:

Hi Anne,

I did know about that one, and I already did some work on this so that the latest version of lrcalc does not call exit (I think). But this doesn't really make a difference, at least not under linux.

Lrcalc calls exit() whenever a call to malloc() returns a zero pointer (which means out-of-memory). But I am told that under linux, malloc() NEVER returns zero. In stead, if the kernel cannot fulfill a request for more memory, it simply kills off the program that requests it. So if that is true, it doesn't matter if lrcalc calls exit or not, the kernel will still kill the library and the rest of sage if lrcalc ever runs out of memory.

I made some changes to the library that makes it possible to stop in the middle of a computation and clean up gracefully, and this could be done if lrcalc ever uses "too much" memory. But the library would need sage to tell it exactly how much memory it is allowed to use. I don't have time to get involved with programming sage proper, so somebody else will have to figure out a way to determine how much memory can safely be consumed. If somebody can do this, then I should be able to make sure that lrcalc will return with an error if that amount of memory is ever reached.

Anders

comment:6 in reply to: ↑ 5 ; follow-up: Changed 9 years ago by jdemeyer

Replying to zabrocki:

My latest version of lrcalc is 1.1.7. I'm not sure what has been included in sage.

Sage has 1.1.6. If you do want to upgrade, I recommend to do it in a separate ticket.

But I am told that under linux, malloc() NEVER returns zero.

That's not entirely true. It's true that Linux allows "overcommitting" (allocating more memory than is available), but only to a certain extent. If you allocate an enormous amount of memory, malloc() will fail anyway. Second of all, this is a configuration parameter and this overcommitting can be disabled. Third, when running under ulimit -v, you can never allocate more memory than the limit.

As for returning an error value during a computation: you can use sig_error() for this (introduced in #9640 and more examples in #15352).

comment:7 in reply to: ↑ 6 ; follow-up: Changed 9 years ago by mhansen

Replying to jdemeyer:

That's not entirely true. It's true that Linux allows "overcommitting" (allocating more memory than is available), but only to a certain extent. If you allocate an enormous amount of memory, malloc() will fail anyway. Second of all, this is a configuration parameter and this overcommitting can be disabled. Third, when running under ulimit -v, you can never allocate more memory than the limit.

Even so, most default configurations do not have ulimit set, and there are plenty of cases where malloc succeeds that will eventually trigger the Linux OOM killer. It'd be nice to be able to avoid this and have an exception. But, if Anders already has some code to do this, I think the best way to do this would be to have a configurable value that you can increase within Sage if you need to.

comment:8 in reply to: ↑ 7 Changed 9 years ago by jdemeyer

Replying to mhansen:

Even so, most default configurations do not have ulimit set

[citation needed]

But, if Anders already has some code to do this, I think the best way to do this would be to have a configurable value that you can increase within Sage if you need to.

I would prefer to not worry about the OOM killer and simply use malloc() without limits.

comment:9 Changed 9 years ago by aschilling

  • Cc nthiery added

comment:10 follow-up: Changed 9 years ago by jdemeyer

Let me elaborate: I would not set a hard limit on the memory used by lrcalc, simply because no other part of Sage does so. Moreover, if you do want a hard limit, there needs to be a discussion on what the default limit should be...

comment:11 in reply to: ↑ 10 ; follow-up: Changed 9 years ago by aschilling

Replying to jdemeyer:

Let me elaborate: I would not set a hard limit on the memory used by lrcalc, simply because no other part of Sage does so. Moreover, if you do want a hard limit, there needs to be a discussion on what the default limit should be...

Further reply from Anders Buch:

"Hello,

The situation is that sage can call lrcalc, which can in turn consume a lot of memory if the input partitions are too large. If this happens under any os, then the whole sage process (and possibly the system too) will be brought to its knees, and in some cases killed by the os.

If lrcalc is free to call malloc as much as it wants, then this is the scenario that results when the input is too large. Personally I am fine with using programs that work like this, but it may not meet the needs of all users. However, since version 1.1.6 of the lrcalc library only calls exit() in an out-of-memory situation, I think it is silly to worry about that use of exit() if nothing else is done to avoid situations where sage crashes because lrcalc eats all the memory. It doesn't matter who kills the process, and if it is to be killed, then the user/system will suffer the least if it happens quickly!

I should also add that the partial fix to the lrcalc library that I have made includes keeping all memory allocated by lrcalc in a doubly linked list, so that everything can be freed up gracefully in case lrcalc has to stop a computation and return an error code. This means that the error-handling code will make lrcalc use more memory and run slightly slower.

Anders"

comment:12 in reply to: ↑ 11 Changed 9 years ago by jdemeyer

Replying to aschilling:

However, since version 1.1.6 of the lrcalc library only calls exit() in an out-of-memory situation, I think it is silly to worry about that use of exit() if nothing else is done to avoid situations where sage crashes because lrcalc eats all the memory.

For what it's worth, I disagree with this statement. If possible, out-of-memory situations should be handled gracefully. However, I do agree that it's not the highest priority issue.

comment:13 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:14 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:15 Changed 8 years ago by tscrim

With #16560, we're now bumped to version 1.1.7.

comment:16 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4
Note: See TracTickets for help on using tickets.