src/bin/sage-list-packages: Make it work if SAGE_ROOT is not available

[mkoeppe]

In this case we can still retrieve the installed versions from the installation records $SAGE_LOCAL/var/lib/sage/installed (in the case of the sage distribution), but we will not be able to provide all details such as remote_version.

[mkoeppe]

New commits:

​8ccc157	sage.misc.package: Deprecate standard_packages, optional_packages, experimental_packages
​c029fdf	sage.misc.package.pip_installed_packages: Normalize package names
​436cc51	sage.misc.package.list_packages: Merge package lists from installation records and build/pkgs
​b577ddc	sage.misc.package.installed_packages: Omit .dummy and other dot files
​3cc5edf	sage.misc.package.is_package_installed: Remove outdated code

[gh-tobiasdiez]

I think the previous code was easier to understand: if SAGE_PKGS is None, we don't need to go through the following for loop. But now one has to understand that the iteration is over an empty list. Better to exit the method early.

-    if not SAGE_PKGS:
-        return {}
-
-    try:
-        lp = os.listdir(SAGE_PKGS)
-    except FileNotFoundError:
-        return {}
+    if SAGE_PKGS:
+        try:
+            lp = os.listdir(SAGE_PKGS)
+        except FileNotFoundError:
+            pass
     for ...

Moreover, I was wondering if the name normalization is ok as it changes the output of a public facing method and thus might break user scripts.

Otherwise it looks good to me.

[mkoeppe]

Replying to gh-tobiasdiez:

I was wondering if the name normalization is ok as it changes the output of a public facing method and thus might break user scripts.

Good point, I'll make this normalization optional

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​97942e3	sage.misc.package.pip_installed_packages: Only normalize when optional arg normalization='spkg'
​6706182	sage.misc.package.list_packages: Return early if SAGE_PKGS is not available
​9258849	sage.misc.package.list_packages: Streamline handling of exclude_pip
​7f80e3e	sage.misc.package.list_packages: Actually add new items to the list

[mkoeppe]

Replying to gh-tobiasdiez:

I think the previous code was easier to understand: if SAGE_PKGS is None, we don't need to go through the following for loop. But now one has to understand that the iteration is over an empty list. Better to exit the method early.

Thanks, done.

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[gh-tobiasdiez]

Code-wise it looks good to me now, thanks for the update!

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​78bae1f

Merge tag '9.3.beta3' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available

[mkoeppe]

Merged current beta

[vbraun]

**********************************************************************
File "src/sage/misc/package.py", line 384, in sage.misc.package.is_package_installed
Failed example:
    for pkg in list_packages(pkg_sources=('pip'), local=True):  # optional - build
        assert not is_package_installed(pkg), "pip package is installed: {}".format(pkg)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.is_package_installed[3]>", line 2, in <module>
        assert not is_package_installed(pkg), "pip package is installed: {}".format(pkg)
    AssertionError: pip package is installed: zope_interface
**********************************************************************
File "src/sage/misc/package.py", line 454, in sage.misc.package.standard_packages
Failed example:
    installed[0], installed[-1]  # optional - build
Expected:
    ('alabaster', 'zn_poly')
Got:
    ('alabaster', 'zope_interface')
**********************************************************************

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​945abb9	build/pkgs/zope_interface: Remove
​c46e7e4	src/sage/misc/package.py: Remove zope_interface from doctests
​3bf4ff3	Merge branch 'u/isuruf/environment_yaml' of git://trac.sagemath.org/sage into t/30900/remove_last_traces_of_zope_interface
​24d7971	build/pkgs/zope_interface/distros/conda.txt: Remove
​a50ddf8	Merge branch 't/28745/environment_yaml' into t/30900/remove_last_traces_of_zope_interface
​3cf5ee4	Merge commit 'a50ddf88975086b14a49895e371477df00fd57b5' of git://trac.sagemath.org/sage into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​b4927e1	sage.misc.package: Remove/adjust non-robust doctests

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e5fe752	Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​78ff9d5	src/sage/misc/package.py: Add one more # optional - build

[mkoeppe]

Let's please get this ticket in.

[gh-kliem]

I would expect an empty line before and after

        def normalize(name):
            if normalization is None:
                return name
            elif normalization == 'spkg':
                return name.lower().replace('-', '_').replace('.', '_')
            else:
                raise NotImplementedError(f'normalization {normalization} is not implemented')

[gh-kliem]

Why do we do it as a string here?

pkg['remote_version'] = 'none'

[mkoeppe]

Replying to gh-kliem:

Why do we do it as a string here?

pkg['remote_version'] = 'none'

This matches what m4/sage_spkg_collect.m4 reports as the version of a package without package-version.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64bde5f	Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​e9a7572	src/sage/misc/package.py: Improve source formatting

[gh-kliem]

LGTM.

[mkoeppe]

Thanks!