Opened 8 years ago

Closed 5 years ago

Last modified 5 years ago

#12729 closed enhancement (invalid)

decrease the size of SAGE_ROOT/install.log

Reported by: jhpalmieri Owned by: GeorgSWeber
Priority: minor Milestone: sage-duplicate/invalid/wontfix
Component: build Keywords:
Cc: kini, roed, leif Merged in:
Authors: Reviewers: John Palmieri, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #12714, #12369 Stopgaps:

Description (last modified by chapoton)

The main install.log file is huge, and most of its contents are reproduced, more helpfully, in the individual log files in spkg/log/. The attached patch removes most of its contents.

I modified the mpir package so it would fail, and then built Sage with this patch. The install.log file looked like this:

Installing GCC because your 'gcc' is not so recent.
Installing GCC because a Fortran compiler is missing.
*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***

    [...snip...]

***********************************************
dir-0.1: installing. Log file: SAGE_ROOT/spkg/logs/dir-0.1.log.
  dir-0.1: installed successfully.
prereq-0.9: installing. Log file: SAGE_ROOT/spkg/logs/prereq-0.9.log.
  prereq-0.9: installed successfully.
bzip2-1.0.5: installing. Log file: SAGE_ROOT/spkg/logs/bzip2-1.0.5.log.
  bzip2-1.0.5: installed successfully.
patch-2.5.9.p2: installing. Log file: SAGE_ROOT/spkg/logs/patch-2.5.9.p2.log.
  patch-2.5.9.p2: installed successfully.
iconv-1.13.1.p3: installing. Log file: SAGE_ROOT/spkg/logs/iconv-1.13.1.p3.log.
zlib-1.2.5.p0: installing. Log file: SAGE_ROOT/spkg/logs/zlib-1.2.5.p0.log.
  iconv-1.13.1.p3: installed successfully.
mpir-2.1.3.p9: installing. Log file: SAGE_ROOT/spkg/logs/mpir-2.1.3.p9.log.
  zlib-1.2.5.p0: installed successfully.
make[1]: *** [installed/mpir-2.1.3.p9] Error 1

real 0m38.531s
user 0m17.554s
sys 0m10.304s

On the screen, you do not see the list of environment variables or the status updates for each package ("PKG: installing. Log file ..."); instead, you see the ongoing build process for each package, just the way things are now.


To make install.log small, apply trac_12729-small-install-log.patch to the root repository.

To instead cut down on what is printed to the screen, apply trac_12729-less-output.patch to the root repo.

Note: this depends on #12369, so it might be best to grab the tarball in that ticket's description. You can then apply this right after unpacking the tarball.

Attachments (2)

trac_12729-small-install-log.patch (32.8 KB) - added by jhpalmieri 8 years ago.
root repo
trac_12729-less-output.patch (33.5 KB) - added by jhpalmieri 8 years ago.
root repo

Download all attachments as: .zip

Change History (27)

comment:1 Changed 8 years ago by jhpalmieri

  • Description modified (diff)
  • Status changed from new to needs_review

Changed 8 years ago by jhpalmieri

root repo

comment:2 Changed 8 years ago by jhpalmieri

  • Dependencies changed from #12369 to #12714, #12369

(As of this writing, the tarball in #12369 may need to be updated (taking #12714 into account) before the patch here applies cleanly. I'm making #12714 a dependency for this ticket to make this explicit, even though #12714 is already a dependency for #12369.)

comment:3 Changed 8 years ago by jhpalmieri

  • Cc leif added

comment:4 Changed 8 years ago by leif

Wow, the dependencies are really weird... ;-)

I was primarily thinking of by default reducing the verbosity level of ECL and setuptools etc. when I mentioned this recently. (IIRC I gave some figures on the ticket we made tar extraction less verbose.)

comment:5 Changed 8 years ago by leif

To me it would also make more sense to reduce the (default) screen output; the size of install.log doesn't bother me much, although it doesn't hurt if that gets "more readable" as well.

(For debugging parallel builds, it might still be necessary to [optionally] have the full install log as it is now.)

comment:6 Changed 8 years ago by jhpalmieri

Of course, just getting rid of MoinMoin (#12713) will cut down considerably on the size of the log file...

We could also add some more environment variables (:p) to control the output from the make process: what gets written to install.log, what gets written to the screen, etc. If we took my patch and interchanged various roles, writing just a little information to the screen and (optionally) writing more in install.log, there would be times when the installation might look stalled, especially with ATLAS or gcc. I guess we could write

    PKG: installing. Please wait; installation has not stalled. Log file: ....

but the current level of feedback might be better.

comment:7 follow-up: Changed 8 years ago by kini

One easy way to decrease the size of install.log would be to bzip2 it... :)

comment:8 Changed 8 years ago by jhpalmieri

Here's a different patch. With it, install.log looks like it does now (large, with the output from the different processes all mixed together). The screen looks like this:

$ make
cd spkg && env SAGE_PARALLEL_SPKG_BUILD='' ./install all 2>&1

Building Sage, writing log to install.log. Each component of Sage
also logs its output into an individual log file, as indicated.

dir-0.1: installing. Log file: spkg/logs/dir-0.1.log.
  dir-0.1: installed successfully.
prereq-0.9: installing. Log file: spkg/logs/prereq-0.9.log.
  prereq-0.9: installed successfully.
bzip2-1.0.5: installing. Log file: spkg/logs/bzip2-1.0.5.log.
  bzip2-1.0.5: installed successfully.
patch-2.5.9.p2: installing. Log file: spkg/logs/patch-2.5.9.p2.log.
  patch-2.5.9.p2: installed successfully.
zlib-1.2.5.p0: installing. Log file: spkg/logs/zlib-1.2.5.p0.log.
iconv-1.13.1.p3: installing. Log file: spkg/logs/iconv-1.13.1.p3.log.
  iconv-1.13.1.p3: installed successfully.
mpir-2.1.3.p9: installing. Log file: spkg/logs/mpir-2.1.3.p9.log.
  zlib-1.2.5.p0: installed successfully.
make[1]: *** [installed/mpir-2.1.3.p9] Error 1

real	0m28.860s
user	0m17.394s
sys	0m9.852s
Error building Sage.
make: *** [build] Error 1
$ 

Changed 8 years ago by jhpalmieri

root repo

comment:9 Changed 8 years ago by jhpalmieri

  • Description modified (diff)

comment:10 in reply to: ↑ 7 Changed 8 years ago by leif

Replying to kini:

One easy way to decrease the size of install.log would be to bzip2 it... :)

You mean

nohup make build & ; tail -f install.log | bzip2 -c9

?

comment:11 Changed 8 years ago by leif

Since we're going to generate the effective Makefile anyway, I wouldn't patch deps, but only put dependencies (as its name suggests) into it, and flexibly generate the build rules on-the-fly, depending on what output / verbosity level is currently requested.

I.e., deps would just contain lines like

$(INST)/$(ATLAS): $(BASE) $(INST)/$(LAPACK) $(INST)/$(PYTHON) $(INST)/$(PATCH)
 
$(INST)/$(BOEHM_GC): $(BASE) $(INST)/$(PATCH)

$(INST)/$(BOOST_CROPPED): $(BASE) $(INST)/$(PATCH)
 
$(INST)/$(CLIQUER): $(BASE) $(INST)/$(PATCH)
 
$(INST)/$(TERMCAP): $(BASE) $(INST)/$(PATCH)

or, preferably:

$(ATLAS): $(BASE) $(LAPACK) $(PYTHON) $(PATCH)
 
$(BOEHM_GC): $(BASE) $(PATCH)

$(BOOST_CROPPED): $(BASE) $(PATCH)
 
$(CLIQUER): $(BASE) $(PATCH)
 
$(TERMCAP): $(BASE) $(PATCH)

Or, for the example given:

$(ATLAS) $(BOEHM_GC) $(BOOST_CROPPED) $(CLIQUER) $(TERMCAP): $(BASE) $(PATCH)

$(ATLAS): $(LAPACK) $(PYTHON)

(In general, one could write a single make rule for building / installing all spkgs, probably with some changes to the directory structure though.)

comment:12 Changed 8 years ago by jdemeyer

I like the idea of the patch, but not the actual implementation, in particular the fact that spkg/standard/deps has to be changed and that file descriptor numbers are hard-coded.

I might think about alternatives, but in any case this is a low-priority ticket for me.

comment:13 Changed 8 years ago by jhpalmieri

I like Leif's idea to do the modifications when creating the Makefile from deps. But it's also low priority for me.

comment:14 follow-up: Changed 7 years ago by jdemeyer

Now that I think about this again, it is sometimes useful to have the full install.log like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".

comment:15 in reply to: ↑ 14 ; follow-up: Changed 7 years ago by leif

Replying to jdemeyer:

Now that I think about this again, it is sometimes useful to have the full install.log like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".

Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.

The average userTM will hardly be interested in what we produce now (while some kind of install.log could still be complete, also configurably); buildbots should certainly create "full" logs.

If a user disables all full logs but afterwards reports some hardly reproducable error (presumably due to some race condition), he and we will have to live with that; that's not much different to the frequent case where the logs (or further details regarding the build conditions) somehow got lost...

comment:16 in reply to: ↑ 15 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_info

Replying to leif:

Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.

Sure, it's feasible but why would you ever be bothered by how much stuff is written to the screen?

The average userTM will hardly be interested in what we produce now

Which also implies that the average user doesn't care. Hence, the average user would be happy with the status quo.

comment:17 Changed 7 years ago by jhpalmieri

First, I think this is not ready for review any more; it should be configurable, as Leif says. Second, working on this has extremely low priority. Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.

comment:18 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:19 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:20 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:21 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:22 in reply to: ↑ description Changed 5 years ago by jdemeyer

After reading through the comments on this ticket again, I am still convinced it should be closed as "wontfix". The few megabytes that this file takes is not a major problem and, as some comments note, the full log is still useful for debugging race conditions during a parallel build.

Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.

This is now clearer since #14263: a list of failed packages (with log files) is output after the build fails.

Last edited 5 years ago by jdemeyer (previous) (diff)

comment:23 Changed 5 years ago by jdemeyer

  • Authors John Palmieri deleted
  • Milestone changed from sage-6.4 to sage-duplicate/invalid/wontfix
  • Reviewers set to John Palmieri, Jeroen Demeyer
  • Status changed from needs_info to positive_review

comment:24 Changed 5 years ago by vbraun

  • Resolution set to invalid
  • Status changed from positive_review to closed

comment:25 Changed 5 years ago by chapoton

  • Description modified (diff)
Note: See TracTickets for help on using tickets.