Opened 8 years ago

Closed 8 years ago

#11818 closed defect (fixed)

sage <script> does not run sage-cleaner

Reported by: vbraun Owned by:
Priority: major Milestone: sage-4.7.2
Component: scripts Keywords:
Cc: Merged in: sage-4.7.2.alpha3
Authors: John Palmieri Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jhpalmieri)

When you are running a long computation, you are probably putting your sage program in a script.sage file and then run sage script.sage in a screen'ed terminal. In that case, sage-cleaner is not run. So if you use @parallel, each fork creates a new subdirectory $DOT_SAGE/temp/hostname/pid that is not being cleaned up. This can easily reach a limit of the file system, for example ext2/ext3 are limited to 215 subdirectories per directory.

A typical error message then looks like

[Errno 31] Too many links: '/home/vbraun/.sage//temp/hostname/26457/'

Apply trac_11818_run_sage_setup.patch to the sage-scripts repository ($SAGE_LOCAL/bin)

Attachments (2)

trac_11818_run_sage_setup.patch (774 bytes) - added by vbraun 8 years ago.
John's patch
trac_11818-experimental.patch (1.5 KB) - added by jhpalmieri 8 years ago.
for Sage library, but currently not working properly

Download all attachments as: .zip

Change History (16)

comment:1 Changed 8 years ago by vbraun

Steps to reproduce:

[vbraun@volker-desktop ~]$ ls ~/.sage/temp/volker_desktop.stp.dias.ie/ | wc
      0       0       0
[vbraun@volker-desktop ~]$ cat > script.sage <<EOF
@parallel(verbose=True)
def f(i):
    return i

list(f(range(0,100)))
EOF
[vbraun@volker-desktop ~]$ sage script.sage
[vbraun@volker-desktop ~]$ ls ~/.sage/temp/volker_desktop.stp.dias.ie/ | wc
    101     101     505

comment:2 Changed 8 years ago by jhpalmieri

Is a change like this good enough?

  • sage-sage

    diff --git a/sage-sage b/sage-sage
    a b if [ $# -ge 1 ]; then 
    10211021       exit $?
    10221022   fi
    10231023   cd "$SAGE_LOCAL/bin/"
     1024   SAGE_BANNER="no" sage_setup
    10241025   sage-run "$CUR" "$@"
    10251026   exit $?
    10261027fi

comment:3 Changed 8 years ago by vbraun

  • Authors set to John Palmieri
  • Status changed from new to needs_review

Yes, that is it!

Changed 8 years ago by vbraun

John's patch

comment:4 Changed 8 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

I am giving this positive review :-)

comment:5 Changed 8 years ago by vbraun

  • Description modified (diff)

comment:6 Changed 8 years ago by jhpalmieri

  • Description modified (diff)

comment:7 Changed 8 years ago by leif

I'll see if I can include this patch bomb into Sage 4.7.2.alpha3...

comment:8 follow-up: Changed 8 years ago by leif

P.S.:

Some doctest in sage/parallel/* analoguous to what Volker did wouldn't be bad.

comment:9 Changed 8 years ago by vbraun

But during parallel doctests there should be a large and fluctuating number of directories in the temp dir. So I don't see an easy way of testing this...

comment:10 in reply to: ↑ 8 Changed 8 years ago by leif

Replying to leif:

P.S.:

Some doctest in sage/parallel/* analoguous to what Volker did wouldn't be bad.

... which perhaps isn't that easy.

comment:11 Changed 8 years ago by jhpalmieri

I had an idea how to doctest this, but it passes doctests regardless of whether the patch here has been applied. I'll post it here in case you have any suggestions.

Changed 8 years ago by jhpalmieri

for Sage library, but currently not working properly

comment:12 follow-up: Changed 8 years ago by vbraun

When you are doctesting this, the sage-cleaner was already started by the doctesting process. In a single-threaded doctest you could kill the sage-cleaner and check that it is being restarted, but with parallel testing that would just be a race condition.

comment:13 in reply to: ↑ 12 Changed 8 years ago by jhpalmieri

Replying to vbraun:

When you are doctesting this, the sage-cleaner was already started by the doctesting process.

Right, I was just coming to the same conclusion, although I'm not sure where sage-cleaner is actually started when you run "sage -t FILE".

In any case, it looks difficult, or at least annoying, to test this from within Sage's doctesting framework.

comment:14 Changed 8 years ago by leif

  • Merged in set to sage-4.7.2.alpha3
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.