Opened 12 years ago

Closed 12 years ago

#7690 closed defect (fixed)

maxima stats too many files on startup, which is a performance issue

Reported by: was Owned by: tbd
Priority: blocker Milestone: sage-duplicate/invalid/wontfix
Component: packages: standard Keywords:
Cc: Merged in:
Authors: Reviewers:
Report Upstream: Fixed upstream, in a later stable release. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

Hi,

This email is a followup about  "1. maxima opens the root directory /
and stats each file found there. Then it does the same thing for the
/u (home) directory..."

Thanks to everybody that responded to my query below.  The
"MAXIMA-SHAREDIR" variable looks fine according to the output of
"maxima --directories".
Off list, Andrej Vodopivec remarked "It could be share-subdirs-list in
init-cl.lisp. If that is true, then it should be easy to remove the
call to share-subdirs-list.".    I tried doing what Andrej suggested,
and it worked perfectly.  Before doing that, Maxima would state about
4000 files (including all users' home directories) on startup, and
afterwards it stat'd only about 70 files.   The difference in
performance on some NSF filesystems is huge -- a second versus
potentially *minutes*.    Even the difference on
sage.math.washington.edu is quite noticeable (a very fast machine with
a fast network).  Looking at the output of makes this very clear:

   strace maxima --directories  > out 2>&1; grep stat out|wc -l


For now we'll be patching the Maxima in Sage so that
share-subdirs-list (in init-cl.lisp) falls back to the old "default
behavior" instead of the new behavior that was introduced in the
recent rewrite of init-cl.lisp.    I really hope whoever rewrote
init-cl.lisp can think about the significant performance regression
that was caused, and find a better solution.

Thanks again for all the incredibly helpful feedback!

 -- William

By the way, what I did to init-cl.lisp was stupid. I changed

#+ecl
(defun share-subdirs-list ()
  ;; This doesn't work yet on windows.  Give up in that case and use
  ;; the default list.
  (if (string= *autoconf-win32* "true")

to

#+ecl
(defun share-subdirs-list ()
  ;; This doesn't work yet on windows.  Give up in that case and use
  ;; the default list.
  (if (string= *autoconf-win32* "false")

Change History (6)

comment:1 Changed 12 years ago by was

  • Report Upstream changed from N/A to Fixed upstream, in a later stable release.
On Wed, Dec 16, 2009 at 6:14 AM, Robert Dodier <robert.dodier@gmail.com> wrote:

    Hello,

    We got a report about Maxima calling stat on a lot
    of files when the program is launched.
    Aside from all the files in the Maxima share directory,
    which we expect Maxima to look at, stat was called
    on a lot of other files as well (stuff outside the Maxima
    directory structure).


An unfortunate mistake: DIRECTORY used stat() on all entries in a directory, not only those that matched the mask. It was just a matter of switching lines. Now things are better. Thanks a lot for reporting. 

$ echo '(directory "/Users/jjgarcia/tmp/")(quit)' > foo.lsp; sudo dtrace -n 'pid$target::safe_stat:entry  { printf("%s\n", copyinstr((uintptr_t)arg0)); }' -c "ecl -norc -load foo"
dtrace: description 'pid$target::safe_stat:entry  ' matched 1 probe
;;; Loading #P"/Users/jjgarcia/foo.lsp"
dtrace: pid 39532 has exited
CPU     ID                    FUNCTION:NAME
  0  19180                  safe_stat:entry jjgarcia

  0  19180                  safe_stat:entry tmp

comment:2 Changed 12 years ago by jhpalmieri

  • Milestone changed from sage-4.4 to sage-5.0

No patch available, so I'm deferring this to Sage 5.0.

comment:3 Changed 12 years ago by jhpalmieri

  • Milestone changed from sage-5.0 to sage-4.4.1

comment:4 Changed 12 years ago by jason

This might be fixed by #8808

comment:5 Changed 12 years ago by was

  • Milestone changed from sage-4.4.3 to sage-5.0

No patch, so deferring...

comment:6 Changed 12 years ago by rlm

  • Milestone changed from sage-5.0 to sage-duplicate/invalid/wontfix
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.