Opened 11 years ago

Closed 11 years ago

#8474 closed defect (fixed)

Detect whether a program is in the path

Reported by: jhpalmieri Owned by: drkirkby
Priority: blocker Milestone: sage-4.3.4
Component: porting Keywords:
Cc: drkirby, mhansen Merged in: sage-4.3.4.alpha1
Authors: John Palmieri Reviewers: David Kirkby
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:


In various places in the Sage library, we test for the existence of programs using code like this:

import os
if os.system('which program') == 0:
    # program exists
    # it doesn't

On Solaris, executing "which program" seems to return 0 regardless of whether the program actually exists, and so any code like this is broken. For example, try this on t2.math:

sage: from sage.misc.latex import have_latex
sage: have_latex()
sage: import os
sage: os.system('which latex')                                                                    
no latex in /usr/local/sage- /usr/local/sage- /usr/local/sage- /usr/local/gcc-4.4.1-sun-linker/bin /usr/local/bin2 /usr/bin /usr/ccs/bin /usr/local/bin /usr/sfw/bin /bin /usr/sbin

So we should have a function which replaces this, and we should use it in the Sage library. On IRC, mhansen says

<mhansen> I think you can use "type" on Solaris

The "type" command actually works for me on several different platforms, and so it's what the patch uses.

This needs testing on lots more different platforms to make sure it's portable.

Attachments (1)

trac_8474-have-program.patch (6.3 KB) - added by jhpalmieri 11 years ago.

Download all attachments as: .zip

Change History (10)

comment:1 Changed 11 years ago by jhpalmieri

  • Status changed from new to needs_review

This should also fix the problem at #8463.

comment:2 Changed 11 years ago by drkirkby

  • Status changed from needs_review to needs_work

Hi, I believe this needs changing a little, as type is nothing really like which. Here are a few specific points though.

  • 'which' is not part of POSIX, so as such I think it is best avoided. It is not portable.
  • 'which' does exist on Solaris, so it should be ok(ish), though it would not be my first choice. However, the documented behavior (in the Solaris man page) of which is to exit with an exit status of 0 if the command is found, and an exit status of >0 if the command is not found. So it should be ok for Solaris, even though I have reservations it is not a good command to use due to its non-portability.
    drkirkby@hawk:~$ man which
    User Commands                                            which(1)
         which - locate a command and display its pathname or alias
         which [name]...
         The following exit values are returned:
         0      Successful completion.
         >0     One or more name operands  were  not  located  or  an
                error occurred.

Although I thought it highly unlikely to be a bug in Sun's implementation 'which', I decided to check. But that is not the problem, as the following code shows:

drkirkby@hawk:~$ cat testit2
which $1
if [ $? -ne 0 ]; then
    echo " can't find $1"
  echo "found $1"

Running that script twice, first with a command one expects to find (ls), then one does not (tokoklklstgjglkjhs), we find it behaves as you expect, and how it is documented to behave.

drkirkby@hawk:~$ ./testit2 ls
found ls
drkirkby@hawk:~$ ./testit2 tokoklklstgjglkjhs
no tokoklklstgjglkjhs in /export/home/drkirkby/bins-for-sage /usr/local/gcc-4.3.4-GNU-assembler-Sun-linker/bin /usr/csw/bin/ /usr/local/bin /usr/bin /usr/X11/bin /usr/openwin/bin
 can't find tokoklklstgjglkjhs

So 'which' is behaving on Solaris in the way documented in the Solaris man page! It's also as documented in the Linux man page, so I suspect the problem might lie elsewhere.

  • I have reservations about the use of type too, as the output is nothing like that of which, as it does not show the path and only the path. It shows more information, in a more confusing way.

It will tell you if the command exits or not, but does not give the output in the same way as which

drkirkby@hawk:~$ which ls
drkirkby@hawk:~$ type ls
ls is /usr/bin/ls
  • After spending 10-15 minutes reading the POSIX specification, I've come to the conclusion what we should really be using is command -v to find if a command exists, and if so its path.


    command - execute a simple command


    command [-p] command_name [argument...]

    command [-p][-v|-V] command_name


    The command utility shall cause the shell to treat the arguments as a simple command, suppressing the shell function lookup that is described in Command Search and Execution , item 1b.

    If the command_name is the same as the name of one of the special built-in utilities, the special properties in the enumerated list at the beginning of Special Built-In Utilities shall not occur. In every other respect, if command_name is not the name of a function, the effect of command (with no options) shall be the same as omitting command.

    When the -v or -V option is used, the command utility shall provide information concerning how a command name is interpreted by the shell.


    The command utility shall conform to XBD Utility Syntax Guidelines .

    The following options shall be supported:

        Perform the command search using a default value for PATH that is guaranteed to find all of the standard utilities.
        Write a string to standard output that indicates the pathname or command that will be used by the shell, in the current shell execution environment (see Shell Execution Environment ), to invoke command_name, but do not invoke command_name. 

    When the -v or -V options are specified, the following exit values shall be returned:

        Successful completion.
        The command_name could not be found or an error occurred. 

So using command -v has many advantages.

  • It is part of POSIX
  • It's output is the same format as the 'which' command.
  • It works on every system I have tried it on - these include
    • Linux
    • OS X
    • Solaris
    • HP-UX
    • FreeBSD

So, in summary, I believe command -v is the most appropriate, but I can't understand why which is not working, as it is documented to work and I've tested it and found it does work.



comment:3 Changed 11 years ago by drkirkby

Just to show you what I mean, here is the output when testing for an existing file, and a non-existing file. Note there is a different response from Solaris 10 (on SPARC) to all the others if the command is not found, but the exit code remains non-zero on Solaris 10 (SPARC).

If the command is found on Solaris 10, then the output is exactly the same as on every other platform, and exactly the same as 'which'. But I think 'command -v' is preferable, as it is more portable.

On Solaris 10 (SPARC processor):

drkirkby@redstart:~$ command -v ls
drkirkby@redstart:~$ command -v fdsfdgddgt
-bash: command: fdsfdgddgt: not found

On HP-UX (PA-RISC processor):

-bash-4.0$ command -v ls                                                
-bash-4.0$ command -v lssdsssdfdf

On OpenSolaris (Intel Xeon processor):

drkirkby@hawk:~$ command -v ls
drkirkby@hawk:~$ command -v klsddshfsd

On OS X (Intel processor of some sort):

[kirkby@bsd ~]$ command -v ls
[kirkby@bsd ~]$ command -v reererer
[kirkby@bsd ~]$ 

On Linux (Intel processor of some sort)

kirkby@sage:~$ command -v ls
kirkby@sage:~$ command -v fdlkskld

On FreeBSD (VirtualBox virtual machine, running on OpenSolaris host):

eagle# command -v ls
eagle# command -v sdfdsf

comment:4 Changed 11 years ago by jhpalmieri

  • Status changed from needs_work to needs_review

Here's a new patch which uses "command -v". On my mac, on sage.math, and on t2.math, "type" and "command -v" behave essentially the same, more or less like this:

sage: from subprocess import call, PIPE
sage: call('type ' + 'ls', shell=True, stdout=PIPE, stderr=PIPE)  # note the space in 'type '
sage: call('type ' + 'lljsdfs', shell=True, stdout=PIPE, stderr=PIPE)       
sage: call('command -v ' + 'ls', shell=True, stdout=PIPE, stderr=PIPE)      
sage: call('command -v ' + 'llkjsdfs', shell=True, stdout=PIPE, stderr=PIPE)

The only difference on the platforms is the value of the nonzero return code: sometimes it's 1, sometimes is 127, depending on the platform and the command. It's always nonzero, though, when the program doesn't exist.

(With these arguments, the command "call" calls a program by passing to the shell without printing standard output or standard error, and it returns a code which is zero if the program exits correctly, nonzero otherwise, and I think the return codes have something to do with the system return codes, but they're not necessarily the same. See

Note that I don't care about the output of the functions, so your concern about "type" in that regard is not a big deal. However, if "command -v" is Posix standard, we can switch to that, since it seems to behave the same way.

Regardless, I think you'll agree that it would be good to have one portable way to do this, in one place in the Sage library, so functions like "have_latex" and "have_chomp" will work right on Solaris, linux, Mac, etc.

comment:5 Changed 11 years ago by jhpalmieri

Just to emphasize: on t2.math, I get this behavior (in contrast to using "command -v" or "type"):

sage: from subprocess import call, PIPE
sage: call('which ' + 'ls', shell=True, stdout=PIPE, stderr=PIPE)
sage: call('which ' + 'lljsdfs', shell=True, stdout=PIPE, stderr=PIPE)

I don't know why. The same happens with os.system('which lljsdfs'), and this is what is used in the Sage library. ("call" is similar in behavior to "os.system", but I think "call" is slightly preferred. It also gives better control of stdin, stdout, and stderr.) So I do think this is what's wrong with #8463. As I said in the description of this ticket, the function "have_latex" returns True on t2.math, and I would guess that if I could install Sage 4.3.4.alpha0 on t2.math, then the comparable "have_chomp" functions would return True as well. Therefore Sage would try to call chomp when doing homology calculations, and since the program isn't there, it would crash.

comment:6 Changed 11 years ago by mhansen

Note that on t2 from the command-line

mhansen@t2:~$ which asdfasdfasfdsfsadf
no asdfasdfasfdsfsadf in /scratch/mhansen/bin /usr/local/gcc-4.4.1-sun-linker/bin /usr/local/bin2 /usr/bin /usr/ccs/bin /usr/local/bin /usr/sfw/bin /bin /usr/sbin
mhansen@t2:~$ echo $?

comment:7 Changed 11 years ago by jhpalmieri

I installed Solaris on a virtual machine on my Mac:

$ uname -a
SunOS unknown 5.10 Generic_141445-09 i86pc i386 i386pc

and I get the same results that mhansen reported on t2. Also, the man page for "which" doesn't discuss error codes. It says, among other things

     which is not a shell built-in command; it is the  UNIX  com-
     mand, /usr/bin/which

I haven't tried to install Sage on this (for one thing, I don't have gcc installed and I don't know how to get it), but I get similar results in python: running "call('which lsjdflsjdflkjs', ...)" returns 0, same as "call('which ls', ...)".

comment:8 Changed 11 years ago by drkirkby

  • Reviewers set to David Kirkby
  • Status changed from needs_review to positive_review

OK, I think I understand this better now.

I just noticed the example above where I showed 'which' behaving as expected was actually on OpenSolaris x64 (hostname hawk). However, I just tried it on Solaris 10 (SPARC), and get the same behavior. i.e. my script works as expected.

I just tried what Mike did, using 'which ffddfdfd' on Solaris 10 (SPARC), Solaris 10 x64 (virtual machine) and OpenSolaris (x64). What I found was

  • 'which' has an exit code of 0 on Solaris 10 (SPARC).
  • 'which' has an exit code of 0 on Solaris 10 x86 (virtual machine).
  • 'which' an exit code of 1 on OpenSolaris.

However, the man pages on OpenSolaris and Solaris 10 are different. The Solaris 10 man page does not discuss exit codes, but the OpenSolaris man page does. (This is one of the problems of using non-POSIX functions - their behavior is not well defined). I installed the patch and run the tests. All thse doctests then pass - there are still some other failures.

I'm not a Python guru, but the patch all looks logical to me. So I'm going to give it positive review.

One very minor niggle - you might want to change the example in the docstring. I think

   sage: have_program('ls') 
   sage: have_program('there_is_not_a_program_with_this_name') 

would be a bit preferable to

   sage: have_program('command') 
   sage: have_program('there_is_not_a_program_with_this_name') 

as far more people will know what 'ls' is. I doubt many will realise there is such a command called 'command'. I thought I knew Unix pretty well, but I was unaware of it until today. But it is a very minor point.

So positive review from me. Change that docstring if you want, but otherwise leave it unchanged.

I've also noticed a rather annoying message about a lack of xdg-open before. It looks like it might have been related - I see you have fixed that too, so hopefully I won't see that any more.

I agree it is more logical to have one library function that works on any system, rather than loads of different implementations which are not portable.

Thank you very much John.

Note to the release manager When this is merged, #8463 may be closed.


Changed 11 years ago by jhpalmieri

comment:9 Changed 11 years ago by mhansen

  • Merged in set to sage-4.3.4.alpha1
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.