decrease the size of SAGE_ROOT/install.log

[jhpalmieri]

The main install.log file is huge, and most of its contents are reproduced, more helpfully, in the individual log files in spkg/log/. The attached patch removes most of its contents.

I modified the mpir package so it would fail, and then built Sage with this patch. The install.log file looked like this:

Installing GCC because your 'gcc' is not so recent.
Installing GCC because a Fortran compiler is missing.
*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***

    [...snip...]

***********************************************
dir-0.1: installing. Log file: SAGE_ROOT/spkg/logs/dir-0.1.log.
  dir-0.1: installed successfully.
prereq-0.9: installing. Log file: SAGE_ROOT/spkg/logs/prereq-0.9.log.
  prereq-0.9: installed successfully.
bzip2-1.0.5: installing. Log file: SAGE_ROOT/spkg/logs/bzip2-1.0.5.log.
  bzip2-1.0.5: installed successfully.
patch-2.5.9.p2: installing. Log file: SAGE_ROOT/spkg/logs/patch-2.5.9.p2.log.
  patch-2.5.9.p2: installed successfully.
iconv-1.13.1.p3: installing. Log file: SAGE_ROOT/spkg/logs/iconv-1.13.1.p3.log.
zlib-1.2.5.p0: installing. Log file: SAGE_ROOT/spkg/logs/zlib-1.2.5.p0.log.
  iconv-1.13.1.p3: installed successfully.
mpir-2.1.3.p9: installing. Log file: SAGE_ROOT/spkg/logs/mpir-2.1.3.p9.log.
  zlib-1.2.5.p0: installed successfully.
make[1]: *** [installed/mpir-2.1.3.p9] Error 1

real 0m38.531s
user 0m17.554s
sys 0m10.304s

On the screen, you do not see the list of environment variables or the status updates for each package ("PKG: installing. Log file ..."); instead, you see the ongoing build process for each package, just the way things are now.

---

To make install.log small, apply trac_12729-small-install-log.patch​ to the root repository.

To instead cut down on what is printed to the screen, apply trac_12729-less-output.patch​ to the root repo.

Note: this depends on #12369, so it might be best to grab the tarball in that ticket's description. You can then apply this right after unpacking the tarball.

[jhpalmieri]

root repo

[jhpalmieri]

(As of this writing, the tarball in #12369 may need to be updated (taking #12714 into account) before the patch here applies cleanly. I'm making #12714 a dependency for this ticket to make this explicit, even though #12714 is already a dependency for #12369.)

[leif]

Wow, the dependencies are really weird... ;-)

I was primarily thinking of by default reducing the verbosity level of ECL and setuptools etc. when I mentioned this recently. (IIRC I gave some figures on the ticket we made tar extraction less verbose.)

[leif]

To me it would also make more sense to reduce the (default) screen output; the size of install.log doesn't bother me much, although it doesn't hurt if that gets "more readable" as well.

(For debugging parallel builds, it might still be necessary to [optionally] have the full install log as it is now.)

[jhpalmieri]

Of course, just getting rid of MoinMoin (#12713) will cut down considerably on the size of the log file...

We could also add some more environment variables (:p) to control the output from the make process: what gets written to install.log, what gets written to the screen, etc. If we took my patch and interchanged various roles, writing just a little information to the screen and (optionally) writing more in install.log, there would be times when the installation might look stalled, especially with ATLAS or gcc. I guess we could write

    PKG: installing. Please wait; installation has not stalled. Log file: ....

but the current level of feedback might be better.

[kini]

One easy way to decrease the size of install.log would be to bzip2 it... :)

[jhpalmieri]

Here's a different patch. With it, install.log looks like it does now (large, with the output from the different processes all mixed together). The screen looks like this:

$ make
cd spkg && env SAGE_PARALLEL_SPKG_BUILD='' ./install all 2>&1

Building Sage, writing log to install.log. Each component of Sage
also logs its output into an individual log file, as indicated.

dir-0.1: installing. Log file: spkg/logs/dir-0.1.log.
  dir-0.1: installed successfully.
prereq-0.9: installing. Log file: spkg/logs/prereq-0.9.log.
  prereq-0.9: installed successfully.
bzip2-1.0.5: installing. Log file: spkg/logs/bzip2-1.0.5.log.
  bzip2-1.0.5: installed successfully.
patch-2.5.9.p2: installing. Log file: spkg/logs/patch-2.5.9.p2.log.
  patch-2.5.9.p2: installed successfully.
zlib-1.2.5.p0: installing. Log file: spkg/logs/zlib-1.2.5.p0.log.
iconv-1.13.1.p3: installing. Log file: spkg/logs/iconv-1.13.1.p3.log.
  iconv-1.13.1.p3: installed successfully.
mpir-2.1.3.p9: installing. Log file: spkg/logs/mpir-2.1.3.p9.log.
  zlib-1.2.5.p0: installed successfully.
make[1]: *** [installed/mpir-2.1.3.p9] Error 1

real	0m28.860s
user	0m17.394s
sys	0m9.852s
Error building Sage.
make: *** [build] Error 1
$ 

[jhpalmieri]

root repo

[leif]

Replying to kini:

One easy way to decrease the size of install.log would be to bzip2 it... :)

You mean

nohup make build & ; tail -f install.log | bzip2 -c9

?

[leif]

Since we're going to generate the effective Makefile anyway, I wouldn't patch deps, but only put dependencies (as its name suggests) into it, and flexibly generate the build rules on-the-fly, depending on what output / verbosity level is currently requested.

I.e., deps would just contain lines like

$(INST)/$(ATLAS): $(BASE) $(INST)/$(LAPACK) $(INST)/$(PYTHON) $(INST)/$(PATCH)
 
$(INST)/$(BOEHM_GC): $(BASE) $(INST)/$(PATCH)

$(INST)/$(BOOST_CROPPED): $(BASE) $(INST)/$(PATCH)
 
$(INST)/$(CLIQUER): $(BASE) $(INST)/$(PATCH)
 
$(INST)/$(TERMCAP): $(BASE) $(INST)/$(PATCH)

or, preferably:

$(ATLAS): $(BASE) $(LAPACK) $(PYTHON) $(PATCH)
 
$(BOEHM_GC): $(BASE) $(PATCH)

$(BOOST_CROPPED): $(BASE) $(PATCH)
 
$(CLIQUER): $(BASE) $(PATCH)
 
$(TERMCAP): $(BASE) $(PATCH)

Or, for the example given:

$(ATLAS) $(BOEHM_GC) $(BOOST_CROPPED) $(CLIQUER) $(TERMCAP): $(BASE) $(PATCH)

$(ATLAS): $(LAPACK) $(PYTHON)

(In general, one could write a single make rule for building / installing all spkgs, probably with some changes to the directory structure though.)

[jdemeyer]

I like the idea of the patch, but not the actual implementation, in particular the fact that spkg/standard/deps has to be changed and that file descriptor numbers are hard-coded.

I might think about alternatives, but in any case this is a low-priority ticket for me.

[jhpalmieri]

I like Leif's idea to do the modifications when creating the Makefile from deps. But it's also low priority for me.

[jdemeyer]

Now that I think about this again, it is sometimes useful to have the full install.log like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".

[leif]

Replying to jdemeyer:

Now that I think about this again, it is sometimes useful to have the full install.log like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".

Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.

The average user^TM will hardly be interested in what we produce now (while some kind of install.log could still be complete, also configurably); buildbots should certainly create "full" logs.

If a user disables all full logs but afterwards reports some hardly reproducable error (presumably due to some race condition), he and we will have to live with that; that's not much different to the frequent case where the logs (or further details regarding the build conditions) somehow got lost...

[jdemeyer]

Replying to leif:

Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.

Sure, it's feasible but why would you ever be bothered by how much stuff is written to the screen?

The average user^TM will hardly be interested in what we produce now

Which also implies that the average user doesn't care. Hence, the average user would be happy with the status quo.

[jhpalmieri]

First, I think this is not ready for review any more; it should be configurable, as Leif says. Second, working on this has extremely low priority. Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.

[jdemeyer]

After reading through the comments on this ticket again, I am still convinced it should be closed as "wontfix". The few megabytes that this file takes is not a major problem and, as some comments note, the full log is still useful for debugging race conditions during a parallel build.

Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.

This is now clearer since #14263: a list of failed packages (with log files) is output after the build fails.