#12729 closed enhancement (invalid)
decrease the size of SAGE_ROOT/install.log
Reported by: | jhpalmieri | Owned by: | GeorgSWeber |
---|---|---|---|
Priority: | minor | Milestone: | sage-duplicate/invalid/wontfix |
Component: | build | Keywords: | |
Cc: | kini, roed, leif | Merged in: | |
Authors: | Reviewers: | John Palmieri, Jeroen Demeyer | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #12714, #12369 | Stopgaps: |
Description (last modified by )
The main install.log file is huge, and most of its contents are reproduced, more helpfully, in the individual log files in spkg/log/. The attached patch removes most of its contents.
I modified the mpir package so it would fail, and then built Sage with this patch. The install.log file looked like this:
Installing GCC because your 'gcc' is not so recent. Installing GCC because a Fortran compiler is missing. *** ALL ENVIRONMENT VARIABLES BEFORE BUILD: *** [...snip...] *********************************************** dir-0.1: installing. Log file: SAGE_ROOT/spkg/logs/dir-0.1.log. dir-0.1: installed successfully. prereq-0.9: installing. Log file: SAGE_ROOT/spkg/logs/prereq-0.9.log. prereq-0.9: installed successfully. bzip2-1.0.5: installing. Log file: SAGE_ROOT/spkg/logs/bzip2-1.0.5.log. bzip2-1.0.5: installed successfully. patch-2.5.9.p2: installing. Log file: SAGE_ROOT/spkg/logs/patch-2.5.9.p2.log. patch-2.5.9.p2: installed successfully. iconv-1.13.1.p3: installing. Log file: SAGE_ROOT/spkg/logs/iconv-1.13.1.p3.log. zlib-1.2.5.p0: installing. Log file: SAGE_ROOT/spkg/logs/zlib-1.2.5.p0.log. iconv-1.13.1.p3: installed successfully. mpir-2.1.3.p9: installing. Log file: SAGE_ROOT/spkg/logs/mpir-2.1.3.p9.log. zlib-1.2.5.p0: installed successfully. make[1]: *** [installed/mpir-2.1.3.p9] Error 1 real 0m38.531s user 0m17.554s sys 0m10.304s
On the screen, you do not see the list of environment variables or the status updates for each package ("PKG: installing. Log file ..."); instead, you see the ongoing build process for each package, just the way things are now.
To make install.log small, apply trac_12729-small-install-log.patch to the root repository.
To instead cut down on what is printed to the screen, apply trac_12729-less-output.patch to the root repo.
Note: this depends on #12369, so it might be best to grab the tarball in that ticket's description. You can then apply this right after unpacking the tarball.
Attachments (2)
Change History (27)
comment:1 Changed 9 years ago by
- Description modified (diff)
- Status changed from new to needs_review
Changed 9 years ago by
comment:2 Changed 9 years ago by
- Dependencies changed from #12369 to #12714, #12369
comment:3 Changed 9 years ago by
- Cc leif added
comment:4 Changed 9 years ago by
Wow, the dependencies are really weird... ;-)
I was primarily thinking of by default reducing the verbosity level of ECL and setuptools etc. when I mentioned this recently. (IIRC I gave some figures on the ticket we made tar extraction less verbose.)
comment:5 Changed 9 years ago by
To me it would also make more sense to reduce the (default) screen output; the size of install.log
doesn't bother me much, although it doesn't hurt if that gets "more readable" as well.
(For debugging parallel builds, it might still be necessary to [optionally] have the full install log as it is now.)
comment:6 Changed 9 years ago by
Of course, just getting rid of MoinMoin (#12713) will cut down considerably on the size of the log file...
We could also add some more environment variables (:p) to control the output from the make process: what gets written to install.log, what gets written to the screen, etc. If we took my patch and interchanged various roles, writing just a little information to the screen and (optionally) writing more in install.log, there would be times when the installation might look stalled, especially with ATLAS or gcc. I guess we could write
PKG: installing. Please wait; installation has not stalled. Log file: ....
but the current level of feedback might be better.
comment:7 follow-up: ↓ 10 Changed 9 years ago by
One easy way to decrease the size of install.log would be to bzip2 it... :)
comment:8 Changed 9 years ago by
Here's a different patch. With it, install.log looks like it does now (large, with the output from the different processes all mixed together). The screen looks like this:
$ make cd spkg && env SAGE_PARALLEL_SPKG_BUILD='' ./install all 2>&1 Building Sage, writing log to install.log. Each component of Sage also logs its output into an individual log file, as indicated. dir-0.1: installing. Log file: spkg/logs/dir-0.1.log. dir-0.1: installed successfully. prereq-0.9: installing. Log file: spkg/logs/prereq-0.9.log. prereq-0.9: installed successfully. bzip2-1.0.5: installing. Log file: spkg/logs/bzip2-1.0.5.log. bzip2-1.0.5: installed successfully. patch-2.5.9.p2: installing. Log file: spkg/logs/patch-2.5.9.p2.log. patch-2.5.9.p2: installed successfully. zlib-1.2.5.p0: installing. Log file: spkg/logs/zlib-1.2.5.p0.log. iconv-1.13.1.p3: installing. Log file: spkg/logs/iconv-1.13.1.p3.log. iconv-1.13.1.p3: installed successfully. mpir-2.1.3.p9: installing. Log file: spkg/logs/mpir-2.1.3.p9.log. zlib-1.2.5.p0: installed successfully. make[1]: *** [installed/mpir-2.1.3.p9] Error 1 real 0m28.860s user 0m17.394s sys 0m9.852s Error building Sage. make: *** [build] Error 1 $
comment:9 Changed 9 years ago by
- Description modified (diff)
comment:10 in reply to: ↑ 7 Changed 9 years ago by
Replying to kini:
One easy way to decrease the size of install.log would be to bzip2 it... :)
You mean
nohup make build & ; tail -f install.log | bzip2 -c9
?
comment:11 Changed 9 years ago by
Since we're going to generate the effective Makefile
anyway, I wouldn't patch deps
, but only put dependencies (as its name suggests) into it, and flexibly generate the build rules on-the-fly, depending on what output / verbosity level is currently requested.
I.e., deps
would just contain lines like
$(INST)/$(ATLAS): $(BASE) $(INST)/$(LAPACK) $(INST)/$(PYTHON) $(INST)/$(PATCH) $(INST)/$(BOEHM_GC): $(BASE) $(INST)/$(PATCH) $(INST)/$(BOOST_CROPPED): $(BASE) $(INST)/$(PATCH) $(INST)/$(CLIQUER): $(BASE) $(INST)/$(PATCH) $(INST)/$(TERMCAP): $(BASE) $(INST)/$(PATCH)
or, preferably:
$(ATLAS): $(BASE) $(LAPACK) $(PYTHON) $(PATCH) $(BOEHM_GC): $(BASE) $(PATCH) $(BOOST_CROPPED): $(BASE) $(PATCH) $(CLIQUER): $(BASE) $(PATCH) $(TERMCAP): $(BASE) $(PATCH)
Or, for the example given:
$(ATLAS) $(BOEHM_GC) $(BOOST_CROPPED) $(CLIQUER) $(TERMCAP): $(BASE) $(PATCH) $(ATLAS): $(LAPACK) $(PYTHON)
(In general, one could write a single make
rule for building / installing all spkgs, probably with some changes to the directory structure though.)
comment:12 Changed 9 years ago by
I like the idea of the patch, but not the actual implementation, in particular the fact that spkg/standard/deps
has to be changed and that file descriptor numbers are hard-coded.
I might think about alternatives, but in any case this is a low-priority ticket for me.
comment:13 Changed 9 years ago by
I like Leif's idea to do the modifications when creating the Makefile from deps
. But it's also low priority for me.
comment:14 follow-up: ↓ 15 Changed 8 years ago by
Now that I think about this again, it is sometimes useful to have the full install.log
like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".
comment:15 in reply to: ↑ 14 ; follow-up: ↓ 16 Changed 8 years ago by
Replying to jdemeyer:
Now that I think about this again, it is sometimes useful to have the full
install.log
like we have now, for debugging race conditions that exist in parallel builds. With this patch applied, it would have been much harder to diagnose #13407 for example. So consider this as a mild proposal to close this ticket as "wontfix".
Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.
The average userTM will hardly be interested in what we produce now (while some kind of install.log
could still be complete, also configurably); buildbots should certainly create "full" logs.
If a user disables all full logs but afterwards reports some hardly reproducable error (presumably due to some race condition), he and we will have to live with that; that's not much different to the frequent case where the logs (or further details regarding the build conditions) somehow got lost...
comment:16 in reply to: ↑ 15 Changed 8 years ago by
- Status changed from needs_review to needs_info
Replying to leif:
Well, as mentioned, the amount of output (especially to the screen) should be configurable, which is IMHO feasable.
Sure, it's feasible but why would you ever be bothered by how much stuff is written to the screen?
The average userTM will hardly be interested in what we produce now
Which also implies that the average user doesn't care. Hence, the average user would be happy with the status quo.
comment:17 Changed 8 years ago by
First, I think this is not ready for review any more; it should be configurable, as Leif says. Second, working on this has extremely low priority. Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.
comment:18 Changed 8 years ago by
- Milestone changed from sage-5.11 to sage-5.12
comment:19 Changed 7 years ago by
- Milestone changed from sage-6.1 to sage-6.2
comment:20 Changed 7 years ago by
- Milestone changed from sage-6.2 to sage-6.3
comment:21 Changed 7 years ago by
- Milestone changed from sage-6.3 to sage-6.4
comment:22 in reply to: ↑ description Changed 6 years ago by
After reading through the comments on this ticket again, I am still convinced it should be closed as "wontfix". The few megabytes that this file takes is not a major problem and, as some comments note, the full log is still useful for debugging race conditions during a parallel build.
Third, I can think of one problem this could solve: you try to build Sage in parallel, and it fails to build. Which package failed? You can figure that out, but perhaps it should be more obvious.
This is now clearer since #14263: a list of failed packages (with log files) is output after the build fails.
comment:23 Changed 6 years ago by
- Milestone changed from sage-6.4 to sage-duplicate/invalid/wontfix
- Reviewers set to John Palmieri, Jeroen Demeyer
- Status changed from needs_info to positive_review
comment:24 Changed 6 years ago by
- Resolution set to invalid
- Status changed from positive_review to closed
comment:25 Changed 6 years ago by
- Description modified (diff)
root repo