Opened 4 years ago

Closed 2 years ago

#26160 closed defect (invalid)

py3: race condition with .pth files

Reported by: jhpalmieri Owned by:
Priority: minor Milestone: sage-duplicate/invalid/wontfix
Component: build Keywords: python3
Cc: chapoton, jdemeyer, embray Merged in:
Authors: Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jhpalmieri)

I am seeing this in the occasional log file, here fpylll-0.4.1dev.log:

Error processing line 1 of /Users/jpalmier/Desktop/Sage/sage_builds/PYTHON3/sage-8.4.beta2/local/lib/python3.6/site-packages/matplotlib-2.2.2-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "/Users/jpalmier/Desktop/Sage/sage_builds/PYTHON3/sage-8.4.beta2/local/lib/python3.6/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 568, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored

The installation reports success, despite this warning.

I'm guessing that the matplotlib installation is writing the .pth file while the fpylll installation is trying to read it. I've only seen this with a Python 3 build, so it may be Python 3 specific.

Change History (12)

comment:1 Changed 4 years ago by jdemeyer

Are you sure that the race condition is related to the .pth file? I would guess that this is really #18438 which just happened to hit while processing a .pth file.

comment:2 Changed 4 years ago by jdemeyer

Or maybe it's #26018...

comment:3 Changed 4 years ago by jhpalmieri

It's not #18438: the affected packages have $(PYTHON) as a dependency, so their installations shouldn't even start until Python 2 and 3 are built. (Also, the time stamps on the log files are wrong.) I don't know how to tell if it's #26018, but I consistently get an error like

Error processing line 1 of /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.4.beta0/local/lib/python3.6/site-packages/matplotlib-2.1.0-py3.6-nspkg.pth:

(Sometimes it's been the matplotlib .pth file, sometimes zope.interface. Maybe there have been others, too.) In all cases, the timestamps show that the two log files (the one with the error and the one with the referenced .pth file) were last modified at times very close to each other.

I have rebuilt Sage several times on several machines, all OS X in case that matters, and I have seen this only when first doing ./configure --with-python=3, and I see it consistently in that case. So either it's a coincidence and I just haven't seen it with a Python 2 build, or it's #26018 in a different guise, or it's something new. Any suggestions on how to diagnose it?

comment:4 Changed 4 years ago by jhpalmieri

  • Cc embray added; ebray removed
  • Description modified (diff)
  • Keywords python3 added
  • Summary changed from Race condition with .pth files to py3: race condition with .pth files

I see this consistently with Python 3 builds and not otherwise, so I am changing the title. I'm curious if it is an issue only with OS X and Python 3, or with Python 3 on any platform.

comment:5 Changed 4 years ago by jhpalmieri

One way to see this problem: ./configure --with-python=3 followed by make. Note that make build will succeed but make will fail because building the docs fails. That failure triggers an examination of the log files, and on OS X it displays a report of these apparent "failures" when installing a few packages.

comment:6 Changed 4 years ago by embray

I'm still pretty sure this is some version of #26018. Have you tried applying my fix from that ticket (which has been needing review for some time anyways...)?

comment:7 Changed 4 years ago by embray

Though actually I think #26018 wouldn't solve this after all. I think you might be right that it's a different race condition related to writing .pth files. In #26018 we talked about maybe putting a lock around writing anything to a Python site-packages while simultaneously building other packages. I found a different workaround that did not require that, but I don't think my workaround in #26018 will solve this.

The problem with .pth files is that they are automatically loaded when starting up the Python interpreter. So I could see that if a package that uses Sage's Python to install (such as fpylll) starts up the interpreter while some other package is in the middle of writing a .pth file this could blow up in a weird way.

So it's still more or less the same issue as #26018, but my solution there probably won't solve this case.

Last edited 4 years ago by embray (previous) (diff)

comment:8 Changed 4 years ago by jhpalmieri

I can confirm: the branch as #26018 did not help (I mean, after I corrected the dependencies on pip on that branch so it would actually build).

comment:9 Changed 2 years ago by mkoeppe

  • Status changed from new to needs_info

Is this problem still present?

comment:10 Changed 2 years ago by jhpalmieri

  • Milestone changed from sage-8.4 to sage-duplicate/invalid/wontfix
  • Status changed from needs_info to needs_review

I haven't seen it any time in recent memory, so I guess not. Let's close it.

comment:11 Changed 2 years ago by mkoeppe

  • Reviewers set to Matthias Koeppe
  • Status changed from needs_review to positive_review

Sounds good.

comment:12 Changed 2 years ago by chapoton

  • Resolution set to invalid
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.