Opened 15 years ago
Last modified 8 months ago
#21 needs_work enhancement
command line option parsing
Reported by: | was | Owned by: | jdemeyer |
---|---|---|---|
Priority: | minor | Milestone: | sage-6.4 |
Component: | user interface | Keywords: | |
Cc: | kini, saraedum, iandrus | Merged in: | |
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #9958 | Stopgaps: |
Description (last modified by )
We should improve and/or modernize and/or revise Sage's command-line parsing.
Two ideas, which could be debated endlessly and could also be implemented independently of each other:
- use Python's argparse module to handle the parsing, rather than a shell script
- change Sage's options from flags to subcommands:
sage --package ...
would be changed tosage package ...
, etc. (comment:56 and comment:57 list some possibilities)
Attachments (9)
Change History (86)
comment:1 Changed 14 years ago by
- Type changed from defect to enhancement
comment:2 Changed 14 years ago by
- Milestone set to Sage-2.10
This should be fixable, but the long term goal is to do a proper rewrite of the command line options.
Cheers,
Michael
comment:3 Changed 13 years ago by
- Owner changed from somebody to gfurnish
- Status changed from new to assigned
comment:4 Changed 13 years ago by
- Component changed from basic arithmetic to interfaces
comment:5 Changed 13 years ago by
- Owner changed from gfurnish to mabshoff
- Status changed from assigned to new
See also #180 for a bunch of related failures due to the option parsing being dumb :o
Cheers,
Michael
comment:6 Changed 13 years ago by
- Status changed from new to assigned
comment:7 Changed 11 years ago by
- Report Upstream set to N/A
Note that sage -bn now builds, then does notebook, though of course it doesn't fix the issue here.
Changed 11 years ago by
comment:8 Changed 11 years ago by
- Status changed from new to needs_review
Here are patches. After applying "trac_21-scripts.patch", you may need to make "SAGE_ROOT/local/bin/sage-sage.py" executable. The build process works for me with these patches. For the standard packages, the third line in
if [ "$SAGE_LOCAL" = "" ]; then echo "SAGE_LOCAL undefined ... exiting"; echo "Maybe run 'sage -sh'?" exit 1 fi
should be changed to "Maybe run 'sage --sh'?", but this doesn't affect the functioning of the packages, and otherwise, they don't need changing. I haven't looked at optional packages.
This approaches uses Python's optparse to parse command-line options. If someone wants to write a version using shflags or some other package, go ahead.
I propose the following approach, whether using these patches or other ones:
- first, we include new command-line options but don't turn them on by default, instead printing a message like this one when you type "sage [...]" with a nonempty argument:
Note: Using old-style Sage command-line options. To try out Sage's experimental GNU/Posix-style command-line options (for example, 'sage --notebook' instead of 'sage -notebook'), set the environment variable $SAGE_NEW_OPTIONS to something nonempty. To bypass this message, set the environment variable $SAGE_SKIP_OPTIONS_MESSAGE to something nonempty.
Running "sage" (with no arguments) would not trigger this message. (Perhaps we could only turn this on in prerelease (alpha and rc) versions? Alternatively, a change like this could go with the version 5.0 release.)
- after a while, we switch this to
Warning: Using old-style Sage command-line options. Sage is changing to use conventional GNU/Posix-style command-line options (for example, 'sage --notebook instead of 'sage -notebook). This change will become the default soon. Meanwhile, to use this new style (and therefore to avoid seeing this message), set the environment variable $SAGE_NEW_OPTIONS to something nonempty.
perhaps with no easy way of disabling this message while using old-style options.
- finally, we turn on the new options, perhaps with an environment variable $SAGE_OLD_OPTIONS to use the old ones, with the understanding that any changes in command-line options may not be maintained for the old version.
See sage-devel for some discussion.
comment:9 Changed 11 years ago by
- Priority changed from minor to critical
comment:10 follow-up: ↓ 11 Changed 11 years ago by
I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:
Another possibility might be to first check for "--gp", "--gap", etc., and do those before doing the general option parsing. I.e., just do what you already planned, but with one optimization to deal with this use case.
This is to speed up access to these programs: do a check like this in a shell script, and then pass the rest of the arguments to Python's optparse using the script included in this patch, or one like it. Then you avoid the slight delay involved in starting up Python if you want to run "gp". It would be nice to have a shell script which had a list of strings "gp", "gap", etc., checked to see if the first(?) argument was "--STR" for STR in this list, and if so, run the appropriate program from SAGE_LOCAL/lib, passing the rest of the line as arguments. Having one list containing all of these strings would make it easy to customize.
Changed 11 years ago by
comment:11 in reply to: ↑ 10 Changed 11 years ago by
Replying to jhpalmieri:
I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:
Another possibility might be to first check for "--gp", "--gap", etc., and do those before doing the general option parsing. I.e., just do what you already planned, but with one optimization to deal with this use case.
Okay, here's a new version which does this: it adds a file sage-sage-quickstart which gets run first, implementing the above idea. Then if SAGE_NEW_OPTIONS is nonempty, it calls sage-sage.py, the Python/optparse version with GNU/Posix standard command-line options. Otherwise, it calls the old parser sage-sage.
For the record, the commands in sage-sage-quickstart are: axiom, ecl/lisp, gap, gp, hg, ipython, maxima, mwrank, python, R, singular. Are any others particularly sensitive to startup times? (Using python adds something less than .1 second on my two-year old iMac, so we're not talking about a lot of time, in any case.)
comment:12 Changed 11 years ago by
Note that Python 2.7 will include the argparse module, which might be easier to use than optparse.
comment:13 Changed 11 years ago by
- Status changed from needs_review to positive_review
Wow, this is really fantastic.
comment:14 Changed 11 years ago by
- Status changed from positive_review to needs_work
comment:15 Changed 11 years ago by
- Status changed from needs_work to needs_review
I'm changing this back to needs review. I realized when trying to apply it that I had got confused about how to even apply this.
John, can you please post clear directions about how to use the patches attached to this ticket?
sage: hg_scripts.apply('http://trac.sagemath.org/sage_trac/attachment/ticket/21/trac_21-scripts.patch') Attempting to load remote file: http://trac.sagemath.org/sage_trac/raw-attachment/ticket/21/trac_21-scripts.patch Loading: [..........] cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg status cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg status cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg import "/scratch/wstein/sage/temp/sage.math.washington.edu/22653/tmp_0.patch" applying /scratch/wstein/sage/temp/sage.math.washington.edu/22653/tmp_0.patch patching file sage-pkg Hunk #1 FAILED at 40 Hunk #2 FAILED at 63 2 out of 2 hunks FAILED -- saving rejects to file sage-pkg.rej patching file sage-run Hunk #1 FAILED at 17 1 out of 1 hunks FAILED -- saving rejects to file sage-run.rej patching file sage-sage Hunk #2 FAILED at 34 Hunk #3 FAILED at 196 Hunk #4 FAILED at 212 Hunk #5 FAILED at 424 Hunk #6 FAILED at 450 Hunk #8 FAILED at 608 Hunk #9 FAILED at 744 Hunk #10 FAILED at 767 Hunk #11 succeeded at 894 with fuzz 1 (offset 0 lines). 8 out of 11 hunks FAILED -- saving rejects to file sage-sage.rej abort: patch failed to apply
comment:16 Changed 11 years ago by
John, can you please post clear directions about how to use the patches attached to this ticket?
Sorry, some parts needed rebasing. I think it's okay now. I've modified the summary with instructions for how to apply the patches.
comment:17 Changed 11 years ago by
- Description modified (diff)
comment:18 Changed 10 years ago by
- Status changed from needs_review to needs_work
It appears like sage-sage.py managed to get itself into 4.4.3 (William, could this have happened when you were trying to apply it). Also, sage-apply-ticket has greatly changed since the patches were posted, so I am changing this back to needs work.
comment:19 Changed 10 years ago by
As far as I know, the file sage-sage.py was accidentally added in 4.4.3, but the release manager should probably delete it: it's not used anywhere.
I don't have the time to rebase it right now, so if anyone else wants to work it, that would be great.
comment:20 Changed 10 years ago by
I've uploaded a patch rebased on 4.6.1.alpha1 (I also added in #8654 while I was at it). One thing I noted during rebasing it is that -tp no longer works (and hence -btp which was added since 4.4.3), I don't know optparse well enough to come up with a solution (if we want to provide one). The documentation patches will need to be rebased as well.
For anyone who wants to apply this, make sure to remove sage-sage.py before applying, the file is currently just hanging around not doing anything - mercurial doesn't even know about it.
comment:21 Changed 10 years ago by
- Cc kini added
comment:22 Changed 9 years ago by
- Dependencies set to #9958
comment:23 Changed 9 years ago by
I think waiting until 2.7 is fine.
comment:24 Changed 9 years ago by
Can we edit the title and description of this ticket to reflect what we're actually trying to do here? (I would, but preferably someone who has been working on it should do it.) The goal is to completely replace the command line handler (currently a shell script) with a Python script which uses argparse for extensibility, right?
comment:25 Changed 9 years ago by
- Cc saraedum added
comment:26 follow-up: ↓ 27 Changed 9 years ago by
People CC'd to this ticket probably already know this, but the above mentioned #9958 is merged, so we can use Python 2.7 default modules such as argparse now.
comment:27 in reply to: ↑ 26 Changed 9 years ago by
comment:28 follow-up: ↓ 29 Changed 9 years ago by
Capital! :) As it's a large project maybe you could post WIPs once in a while?
I was planning on diving into the scripts dir myself and trying to work on this, but I guess it should be an order of magnitude easier for you, haha. Still, let me know if I can help with anything.
comment:29 in reply to: ↑ 28 Changed 9 years ago by
Replying to kini:
Capital! :) As it's a large project maybe you could post WIPs once in a while?
No problem, once I have something of any real substance, I'll be sure to post it somewhere.
I was planning on diving into the scripts dir myself and trying to work on this, but I guess it should be an order of magnitude easier for you, haha. Still, let me know if I can help with anything.
Well, I'm just starting and have had a busy week with other stuff, so you would probably be in about as good of position as myself, especially since we need to rebase this off of the 5.0 dev builds, which changed a lot of that stuff anyway.
I'm finding myself annoyed at spkg/script, and all the little special cases that we have. I really just want to rip that out and try to have a (more) unified design to our parsing. My current fancy is to introduce a bunch of subcommands, like mercurial or aptitude. Some of the ones I've thought about:
% sage ARGS # this would be for running sage scripts, or a couple of oddball arguments % sage notebook ARGS % sage pkg ARGS # this would include spkg stuff % sage pkg install # since install has some special flags like -f or -s % sage test ARGS % sage build ARGS % sage {python,sqlite3,R,gp,...} ARGS # we can consider these programs as subcommands of sage
I haven't fully worked out what this would look like with all the arguments (such as debugging), but IMO it would greatly clean up our command line tools. Also, this would simplify many aspects of the implementation, although some hacking of argparse will still have to be done (it currently doesn't support optional subparsers, see http://bugs.python.org/issue9253).
I probably should bring this up on the devel list, but I'm tired and should go to bed before I have to be up in the morning :/.
comment:30 Changed 9 years ago by
- Milestone changed from sage-5.1 to sage-6.0
We're going to try to get this in as a Sage 6.0 goal along with #13015.
comment:31 Changed 9 years ago by
patchbot: apply 21-scripts-4.6.1.alpha1.patch
(this doesn't actually work but at least this way the patchbot won't try to apply the other stuff, and even crash, apparently)
comment:32 Changed 9 years ago by
- Cc iandrus added
comment:33 Changed 8 years ago by
- Owner changed from mabshoff to jdemeyer
Going with argparse
is probably a good idea, but I would still like a special "pre-parser" in bash
to handle to options
sage --sh sage -i sage -f
(or whatever version they will become)
I think these must be available from the start, before Python has been built.
comment:34 Changed 8 years ago by
ohanar and I talked about this a bit at SD40.5. What do you think about having a totally separate executable (rather than a preparser) called sage-sh
? Stuff like the current sage -gap
could be called rapidly with sage-sh -c gap
, or slowly via python with sage gap
or what-have-you.
comment:35 follow-up: ↓ 40 Changed 8 years ago by
William, among others, as alluded to above, was pretty insistent that commands like sage --gp
execute quickly, without the overhead that starting Python entails. Hence the approach in my patches on this ticket. If you want to insist that people use sage-sh -c gp
instead, I don't think it can get a positive review unless the approach gets a positive response on sage-devel.
comment:36 Changed 8 years ago by
Rather than building on your patches on this ticket, we're proposing completely rewriting the command line interface of Sage, and that includes changing invocation methods. That's why we've set the milestone to sage-6.0 - the idea is to tie it with the layout restructuring and new development interface that will come with the git transition (#13015). We figured that the git transition is a big enough change to warrant a major version bump (though that might end up being sage-7.0 or whatever, depending on the timeline).
So, of course, this will all go through sage-devel in time, once we figure out exactly what we're proposing.
comment:37 Changed 8 years ago by
Oh, I understood that you were starting from scratch. I was responding to your proposal about sage-sh
, suggesting that not all users would be happy with that solution.
comment:38 Changed 8 years ago by
If you need a separate bash
script anyway to parse some options like --sh
, then it's a trivial exercise to parse more options (like --gp
) in that same script.
comment:39 Changed 8 years ago by
jdemeyer: I think you misunderstand. Currently, sage
is a bash script, which checks some options, then starts up Python to handle the rest (separately in various different code branches, no less). What we're proposing is the following scenario:
sage
is a Python script. sage-sh
is a bash script. sage-sh
does not parse any arguments - it just sources sage-env, does whatever other setup required, and dumps you in a shell. Any arguments to sage-sh
are just passed to that shell. As such, -c
would allow you to specify a command to execute in the Sage shell, such as gap
or gp
. sage-sh
does not load Python unless you happen to give it -c python
as an argument.
sage
is directly run by Python because it is a python script. sage --sh
(or probably sage sh
because of ohanar's subcommands idea) will cause Python to start up, parse the arguments, and then exec the bash script sage-sh
(i.e. be very slow).
The argument would be that if you need something to happen fast and want to avoid starting up Python, or are the release manager and are doing things with Sage in a state where Python doesn't exist, you're probably writing scripts anyway and can afford to write out sage-sh -c sage-pkg
(or whatever the package manager command ends up being) inside your script, or make an alias in your shell. If you're a normal user just typing on the command line, you won't care that Python had to start up just to parse your command line arguments.
The benefit of doing this is that the entire normal startup chain of Sage can be made pure Python, which is easier to maintain and might be more portable (?).
comment:40 in reply to: ↑ 35 Changed 8 years ago by
Doesn't this contradict?
Replying to jhpalmieri:
William, among others, as alluded to above, was pretty insistent that commands like
sage --gp
execute quickly, without the overhead that starting Python entails.
Replying to kini:
sage
is a Python script.
comment:41 Changed 8 years ago by
1) That comment is 3 years old and I imagine it's up for discussion by now.
2) Replace sage --gp
with sage-sh -c gp
and all is well. I doubt William, among others, is talking about the exact command sage --gp
; as long as there is a good way to start up Sage's gp quickly without loading Python, we should be fine (?).
comment:42 follow-ups: ↓ 43 ↓ 45 Changed 8 years ago by
OK I see. So we would have two "top-level" scripts then:
./sage
(Python-based) and ./sage-sh
(bash
-based). Yes, that would be fine for me.
But what about sage-env
then? That's needed by sage-sh
, so it cannot be Python-based.
comment:43 in reply to: ↑ 42 ; follow-up: ↓ 44 Changed 8 years ago by
Replying to jdemeyer:
But what about
sage-env
then? That's needed bysage-sh
, so it cannot be Python-based.
Definitely short term I don't think it is realistic to make the entire startup entirely python because we don't have any python module to setup a suitable environment, however I think it would be good to still make separate out sage-sh:
Specifically I would have the root level sage script be (more or less)
#!/usr/bin/env sh "$0-sh" -c "sage $*"
and have SAGE_LOCAL/bin/sage
be a pure python script.
comment:44 in reply to: ↑ 43 Changed 8 years ago by
Given that part of the startup needs to be bash
anyway (I actually think bash
is a good language to implement sage-env
), perhaps the two-pass argument parsing (as proposed in the comments of 3 years ago) would be best. Have a small sage
script written in bash
which processes just a few options and when options aren't recognized, run all the Python argparse
machinery. What could be the problem with that?
comment:45 in reply to: ↑ 42 ; follow-ups: ↓ 46 ↓ 48 Changed 8 years ago by
Replying to jdemeyer:
But what about
sage-env
then? That's needed bysage-sh
, so it cannot be Python-based.
Right, we might have a very thin bash wrapper that loads sage-env
(which will be a bash script) before the main Python script. Or, since sage-env
ideally should just set up environment variables and do nothing else (right?), we could turn it into a config file that was read independently by sage-sh
and by sage
. This would also allow us to rely less on environment variables for random things seemingly unrelated to the shell.
The problem with having two-pass argument parsing is that it separates the processing of arguments into multiple areas, making the architecture of the startup process needlessly complex. It is also pretty ugly to actually do this in the standard option parsing way because either you start to want to enforce arbitrary argument orders like we currently do (sage -tp
works and sage -pt
doesn't, sage -br
works and sage -rb
doesn't, etc.), or now the bash script needs to basically reimplement optparse/argparse in bash in order to correctly read the flags it's looking for.
In any case, if I as a new Sage developer want to know or modify what option --foo
does, there should be one obvious place to look for it. Making sage-sh
parse arguments also means that we are shadowing arguments that could be passed on to the shell, etc. etc. Splitting argument parsing into two places is just generally a nasty design IMHO.
Why does part of the startup need to be bash, other than because of sage-env
?
comment:46 in reply to: ↑ 45 ; follow-up: ↓ 47 Changed 8 years ago by
Replying to kini:
Right, we might have a very thin bash wrapper that loads
sage-env
(which will be a bash script) before the main Python script. Or, sincesage-env
ideally should just set up environment variables and do nothing else (right?), we could turn it into a config file that was read independently bysage-sh
and bysage
. This would also allow us to rely less on environment variables for random things seemingly unrelated to the shell.
Are you saying that a configuration file should store the current environment? And that anytime it is changed (such as if the root directory of sage is moved) that this should be updated?
The problem with having two-pass argument parsing is that it separates the processing of arguments into multiple areas, making the architecture of the startup process needlessly complex. It is also pretty ugly to actually do this in the standard option parsing way because either you start to want to enforce arbitrary argument orders like we currently do (
sage -tp
works andsage -pt
doesn't,sage -br
works andsage -rb
doesn't, etc.), or now the bash script needs to basically reimplement optparse/argparse in bash in order to correctly read the flags it's looking for.
Also
- argparse/optparse handles help functionality (consistent formatting), so all pre-parsed commands would still need stubs in argparse/optparse (and -1 for code duplication)
- argparse (and maybe optparse) matches subcommands so long as they are not ambiguous. So if (for instance) sage only had the subcommands foo and bar then
sage f [args]
would be expanded tosage foo [args]
. (this is fairly standard for software with subcommands) This functionality would be inconsistent if there were any pre-parsed commands.
In any case, if I as a new Sage developer want to know or modify what option
--foo
does, there should be one obvious place to look for it. Makingsage-sh
parse arguments also means that we are shadowing arguments that could be passed on to the shell, etc. etc. Splitting argument parsing into two places is just generally a nasty design IMHO.
+1
Why does part of the startup need to be bash, other than because of
sage-env
?
For one of two reasons:
- python may not be in
PATH
because python is not currently a dependency - even if python is in
PATH
, sage may not work with the default python
comment:47 in reply to: ↑ 46 Changed 8 years ago by
Replying to ohanar:
Replying to kini:
Right, we might have a very thin bash wrapper that loads
sage-env
(which will be a bash script) before the main Python script. Or, sincesage-env
ideally should just set up environment variables and do nothing else (right?), we could turn it into a config file that was read independently bysage-sh
and bysage
. This would also allow us to rely less on environment variables for random things seemingly unrelated to the shell.Are you saying that a configuration file should store the current environment? And that anytime it is changed (such as if the root directory of sage is moved) that this should be updated?
It should store the current startup environment. It would change if the root directory of Sage is moved, for example, yes. But if a user decided to change an environment variable in a Sage session with os.environ
that wouldn't become reflected in the file, of course.
Why does part of the startup need to be bash, other than because of
sage-env
?For one of two reasons:
- python may not be in
PATH
because python is not currently a dependency- even if python is in
PATH
, sage may not work with the default python
Oh, right, of course. So then yes, sage
should be bootstrapped in the way you described above.
comment:48 in reply to: ↑ 45 Changed 8 years ago by
Replying to kini:
Or, since
sage-env
ideally should just set up environment variables and do nothing else
Well, some of these environment variables are conditional, so it's not that easy to have a file which works both from bash and from Python. And I certainly don't see it as a problem that sage-env
remains in bash
as it is now.
The problem with having two-pass argument parsing is that it separates the processing of arguments into multiple areas, making the architecture of the startup process needlessly complex. It is also pretty ugly to actually do this in the standard option parsing way because either you start to want to enforce arbitrary argument orders like we currently do (
sage -tp
works andsage -pt
doesn't,sage -br
works andsage -rb
doesn't, etc.), or now the bash script needs to basically reimplement optparse/argparse in bash in order to correctly read the flags it's looking for.
All these arguments are essentially irrelevant if the first pass needs to support just very few options. Things like ./sage -tp
or ./sage -pt
would be handled anyway by the argparse
script.
Of course it's bad design to have two-pass argument parsing, but it would be so nice to keep ./sage -i
and ./sage --sh
working.
Why does part of the startup need to be bash, other than because of
sage-env
?
I think sage-env
is the main reason.
comment:49 Changed 7 years ago by
- Milestone changed from sage-6.0 to sage-6.1
comment:50 Changed 7 years ago by
- Milestone changed from sage-6.1 to sage-6.2
comment:51 Changed 7 years ago by
- Milestone changed from sage-6.2 to sage-6.3
comment:52 Changed 7 years ago by
- Milestone changed from sage-6.3 to sage-6.4
comment:53 Changed 6 years ago by
The command line interface continues to evolve; can someone (who cares) give a summary of what still would be needed? comment:24 still hasn't been resolved, and comment:33 (as well as quick, non-Python-starting, use of sage -maxima
and friends) seems quite relevant.
comment:54 Changed 6 years ago by
- Component changed from interfaces to user interface
comment:55 follow-up: ↓ 59 Changed 4 years ago by
Still reading up on this ticket, and don't have any comments to add yet to the existing discussion. But one question I have in general: Is anyone opposed at all to the idea of creating a new sub-command based interface, more like git, than the slightly unusual interface that uses single-character flags for subcommands? E.g. replace sage -t
with sage test
. Yes, it's more to type, but only by two characters, and is much less unusual. The old -t
could still be supported very easily for backwards compatibility, but perhaps with a deprecation warning.
comment:56 Changed 4 years ago by
William Stein and I were just talking about this idea yesterday. Something like this?
sage FILE.[sage|py|spyx] sage help sage help --advanced? sage -c <CMD> sage package config sage package name <TARBALL> sage package list sage package list standard sage package list optional sage package list experimental sage package apropos sage package download sage package update sage package fix-checksum sage package create sage install <PKGS> options: [-f, --force] [-c, --check] [-d, --download] [-s, --save] [-y, --yes] [-n, --no] [--no-dependency] [-i, --info] sage pip sage gap sage gap3 sage gp sage maxima sage python sage python3 sage ipython sage ipython3 (not yet implemented) sage R sage singular sage git sage cython sage cleaner sage ecl sage gdb sage kash sage lisp sage M2 sage mwrank sage polymake sage scons sage sqlite3 sage twistd sage shell sage notebook=[default|ssagenb|jupyter|export|jupyterlab|ipython] options: --log=... sage notebook rst2ipynb ... sage notebook rst2txt ... sage notebook rst2sws ... sage notebook sws2rst ... sage test FILES sage test --all options: --long, --verbose, --optional, --sagenb, --help, [-p|--parallel], --randorder[=seed], --new, --initial, --debug, --failed, --warn-long [timeout] sage preparse <FILE> sage startuptime sage coverage [-a, --all] sage search? search_src? search_doc? grep? grep_doc? sage sdist sage valgrind [--cacherind] [--callgrind] [--massif] [--memcheck] [--omega] sage docbuild options: (run sage --docbuild --help to see) sage --nodotsage sage --root sage -q sage --min sage [-v, --version] sage dumpversion? sage fixdoctests ... sage build sage build --force sage build test? (currently sage -bt ...) sage build run? (currently sage -br)
Maybe some of these can be removed. Maybe some can be consolidated: do we need separate commands for gap, gp, maxima, ecl, R, etc., or can they be combined under a single command, like "sage run <PROGRAM>"? There is endless bikeshedding available.
comment:57 Changed 4 years ago by
Yes, something quite like that.
And I was thinking of writing up some kind of declarative list(s) of subcommands. In particular I was thinking two separate lists:
- One list of sage-specific sub-commands (such as
sage package
in your example above), which would automatically be translated to running individual scripts that implement them that would be namedsage-<subcommand>
. This is mostly howgit
works as well.
- One list of programs installed in the Sage distribution (
sage sh
,sage gap
, etc.) that can be launched from the interface. In principle one could make this automatic but I think it's better to have a hard-coded list. I don't think asage run
is really necessary.sage sh <whatever>
is essentially the same as this, but I think it's still convenient to have shortcuts for common programs included in the Sage distribution.
Although somewhat redundant, because it's common I would also have sage --help
as an alias for sage help
and sage --help-advanced
for sage help --advanced
, though one could bikeshed about whether those should do the exact same thing or not.
I might also hide more of the development-specific commands (sage coverage
, sage startuptime
, etc.) behind a sub-command.
comment:58 Changed 4 years ago by
(I should add, that's a very nice mock-up of what such an interface would look like, so thank you for that.)
comment:59 in reply to: ↑ 55 ; follow-up: ↓ 62 Changed 4 years ago by
Replying to embray:
E.g. replace
sage -t
withsage test
.
What about sage -btp
? I use that all the time. I would hate it if that would become
sage buildtest -p
or worse, make build && sage test -p
.
comment:60 Changed 4 years ago by
We should also think to what extent the build system should be exposed under the sage
command. For example, we now have
make FOO # Build dependencies of FOO + FOO sage -i FOO # Build toolchain + dependencies of FOO + FOO sage -f FOO # Build toolchain + dependencies of FOO + *rebuild* FOO sage -p FOO # Build FOO *without* dependencies
and
make sagelib # Build Sage library with dependencies sage -i sagelib # Build Sage library with toolchain and dependencies sage -b # Build Sage library *without* dependencies sage -f sagelib # Rebuild all of the Sage library with toolchain and dependencies sage -ba # Rebuild all of the Sage library *without* dependencies
This is all for historical and accidental reasons, but this ticket should clean that up too.
comment:61 Changed 4 years ago by
Needless to say, many people don't even know the subtle differences between the above commands.
comment:62 in reply to: ↑ 59 Changed 4 years ago by
Replying to jdemeyer:
Replying to embray:
E.g. replace
sage -t
withsage test
.What about
sage -btp
? I use that all the time. I would hate it if that would becomesage buildtest -p
or worse,make build && sage test -p
.
I was actually thinking of allowing subcommands to be chained, like in setup.py
. So sage build test
, where each can take optional flags if desired.
comment:63 Changed 8 months ago by
- Description modified (diff)
comment:64 Changed 8 months ago by
- Description modified (diff)
comment:65 Changed 8 months ago by
- Description modified (diff)
comment:66 Changed 8 months ago by
Strong -1 "wontfix" for the idea of making an incompatible change toward using "subcommands" after 14 years of the existence of the sage
script.
comment:67 Changed 8 months ago by
I think the current interface of the sage
script is pretty clumsy and dated by modern standards. It can and should be made more user-friendly. Just as one example of an advantage of subcommands is it makes the help documentation vastly more digestible. I wouldn't propose changing it without maintaining backwards compatibility though.
comment:68 follow-ups: ↓ 70 ↓ 73 Changed 8 months ago by
+1 from me to changing the sage script to support subcommands, especially if we can somehow do it in a way that preserves compatibility with the current parsing. I wonder to what extent the following is possible:
- 1. run the current shell script and if it "works" then done (with maybe some slight tightening)
- 2. parse using subcommands.
Or something else, e.g,. if you do "sage [explicit list of subcommands]" uses the subcommand approach; otherwise, fall back to the current parser.
I wrote at least the first version of the current sage command line parser, and frankly I didn't know what I was doing at the time, and just sort of stupidly copied random bits and pieces of design from programs I had used. Using python's subcommands support is a lot more systematic, and can also result in very nice modular code.
comment:69 Changed 8 months ago by
Since all of the sage script's current "subcommands" (e.g. sage -b
, sage -i
) all start with hyphens, and new subcommands with be non-hyphenated, I think it would work to support both without too much ambiguity but I'd be interested in a counter-example. sage <filename.{py,sage}>
would still work since no sub-command would be confused with a runnable script filename.
comment:70 in reply to: ↑ 68 Changed 8 months ago by
Replying to was:
Using python's subcommands support is a lot more systematic, and can also result in very nice modular code.
I might still just write it as a shell script. Reason being, based on my experience implementing CLIs in Python, it tends to be much much slower to run a single command. At the very least I would do this for the top-level sage
script. Most subcommands would delegate to another program which might be another shell script, or could be written in Python (as is already the case). For subcommands written in Python it's not always so bad as long as most operations you would do with that command are long enough to make the Python interpreter startup time negligible. Those could also have further subcommands.
comment:71 Changed 8 months ago by
I might still just write it as a shell script.
+1
I should have just said "in my experience, structuring command line parsing code as subcommands (implemented in any language) can result in very nice modular code."
comment:72 Changed 8 months ago by
- Priority changed from critical to minor
comment:73 in reply to: ↑ 68 ; follow-up: ↓ 74 Changed 8 months ago by
Replying to was:
+1 from me to changing the sage script to support subcommands,
One very frustrating part of subcommands is the failure of standard tab-completion to work with it:
for instance, for jupyter notebook sheet.ipynb
. If there were just a command jupyter-notebook
, it would be much easier and faster to type. Particularly with jupyter, which you nearly always use to start its notebook, it's rather frustrating. For git somehow it feels a little more natural, probably because there is naturally a larger variety of actions you want to take through it. It also helps that most preconfigured bash tabcompletions are aware of git subcommands. (although still, distinct commands git-push, git-branch, git-pull would be faster)
comment:74 in reply to: ↑ 73 ; follow-up: ↓ 75 Changed 8 months ago by
Replying to nbruin:
Replying to was:
+1 from me to changing the sage script to support subcommands,
One very frustrating part of subcommands is the failure of standard tab-completion to work with it:
The question so far has been (for example) sage --notebook
vs. sage notebook
, and tab-completion won't work with either. Are you suggesting adding sage-notebook
and other scripts?
comment:75 in reply to: ↑ 74 ; follow-ups: ↓ 76 ↓ 77 Changed 8 months ago by
Replying to jhpalmieri:
The question so far has been (for example)
sage --notebook
vs.sage notebook
, and tab-completion won't work with either. Are you suggesting addingsage-notebook
and other scripts?
From a tab-completion point of view that would make sense, yes. (that, or learn how to extend the tab completion patterns). For this particular example, I'd use jupyter notebook
anyway, with the sage kernel installed in the system jupyter server.
The traditional reason for having dashes in front of options/subcommands is to remove ambiguity from sage notebook
(to run the file notebook) and sage notebook
(to start the notebook). For that reason, I think we can only have subcommands for sage
if sage <file>
would have no meaning. I don't think we can discard this main function of sage
. I think with jupyter, where there is one VERY common use, it's already a mistake to go with a subcommand design.
Extrapolating from that, I think that using subcommands for the sage
script is also the wrong fit. It works well with git
, but users really interact differently with git.
comment:76 in reply to: ↑ 75 Changed 8 months ago by
Replying to nbruin:
The traditional reason for having dashes in front of options/subcommands is to remove ambiguity from
sage notebook
(to run the file notebook) andsage notebook
(to start the notebook). For that reason, I think we can only have subcommands forsage
ifsage <file>
would have no meaning. I don't think we can discard this main function ofsage
. I think with jupyter, where there is one VERY common use, it's already a mistake to go with a subcommand design.Extrapolating from that, I think that using subcommands for the
sage
script is also the wrong fit. It works well withgit
, but users really interact differently with git.
I fully agree.
comment:77 in reply to: ↑ 75 Changed 8 months ago by
Replying to nbruin:
(that, or learn how to extend the tab completion patterns).
FWIW, there are tools which can do this for you if your command line subcommands and options are all handled by argparse (rather than e.g. a top level bash script that calls out to python programs that use argparse for each subcommand, as embray suggested).
argcomplete will dynamically provide on-the-fly completion candidates by actually running the argument parsing logic from the sage
program every time you hit tab in the shell. This is always accurate but could be slow if sage
takes a long time to get to the line of code where the argument parser is run (e.g. if it has some heavy imports).
shtab also runs the argument parsing logic from the sage
program, but it statically generates a bash completion script which you can then register with bash-completion
by putting it in a relevant place (if you don't have root access, this can be ~/.local/share/bash-completion/completions/
). Then when you press tab in the shell, completions should be pretty instantaneous, but the completion script needs to be kept up to date with the command line interface of sage
.
If you use a shell other than bash, it may be harder. zsh, at least, is supported by shtab and to some extent argcomplete as well.
I suggest that shtab
be run as part of the build process of Sage and that the resulting bash completion script be installed as part of the installation process. That should give out-of-the-box completion functionality to the majority of users, which would be nice. (Again, though, this would only work if the top-level sage
does argument parsing in Python with argparse
, and I understand that might not end up being the case for other reasons.)
no -- you can't combine command line options like that. this isn't a bug but a not implemented yet issue.