Opened 10 months ago

Last modified 2 months ago

#26990 new defect

Race when copying package files

Reported by: vbraun Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: build Keywords:
Cc: embray Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by vbraun)

There seems to be a race when copying files as cp is not atomic. I'm not sure if the copying holds a lock. Either the copying or the *.pyc file generation by the Python interpreter in another process conflicted here:

[future-0.17.1] Successfully installed future-0.17.1
[future-0.17.1] Cleaning up...
[future-0.17.1] Removed build tracker '/tmp/pip-req-tracker-ruDJcT'
[future-0.17.1] 
[future-0.17.1] real	0m4.907s
[future-0.17.1] user	0m2.226s
[future-0.17.1] sys	0m0.382s
[future-0.17.1] Copying package files from temporary location /home/chapoton/sage/local/var/tmp/sage/build/future-0.17.1/inst to /home/chapoton/sage/local
[future-0.17.1] cp: cannot create regular file '/home/chapoton/sage/local/./lib/python2.7/site-packages/future/__init__.pyc': File exists
[future-0.17.1] ************************************************************************
[future-0.17.1] Error copying files for future-0.17.1.
[future-0.17.1] ************************************************************************
[future-0.17.1] Please email sage-devel (http://groups.google.com/group/sage-devel)
[future-0.17.1] explaining the problem and including the log file
[future-0.17.1]   /home/chapoton/sage/logs/pkgs/future-0.17.1.log
[future-0.17.1] Describe your computer, operating system, etc.
[future-0.17.1] ************************************************************************

See also: https://unix.stackexchange.com/questions/116280/cannot-create-regular-file-filename-file-exists

Change History (11)

comment:1 Changed 10 months ago by vbraun

  • Description modified (diff)

comment:2 Changed 10 months ago by embray

  • Milestone changed from sage-8.6 to sage-duplicate/invalid/wontfix
  • Resolution set to duplicate
  • Status changed from new to closed

This is basically a duplicate of #26018, particularly comment:4:ticket:26018, though I'm not positive if the branch there fixes it for all cases.

comment:3 Changed 10 months ago by embray

Actually I'm not sure this is #26018, but it still looks like a fluke. How could you get:

Error copying files for future-0.17.1.

after

Successfully installed future-0.17.1

unless somehow the same package was being installed twice simultaneously?

Last edited 10 months ago by embray (previous) (diff)

comment:4 Changed 10 months ago by vbraun

  • Resolution duplicate deleted
  • Status changed from closed to new

The Error copying files for future-0.17.1. is just the error branch in build/bin/sage-spkg

Version 0, edited 10 months ago by vbraun (next)

comment:5 Changed 10 months ago by embray

I understand that, but the error case is:

 903     $SAGE_SUDO cp -Rp "$PREFIX/." "$SAGE_LOCAL"
 904     if [ $? -ne 0 ]; then
 905         error_msg "Error copying files for $PKG_NAME."
 906         exit 1
 907     fi

i.e. it should have exited the script before the success message was ever echoed at all.

comment:6 Changed 10 months ago by vbraun

The first 'Successfully installed' is printed by pip, which is also why its before the 'Removed build tracker...'

comment:7 Changed 10 months ago by embray

Yes, I just realized that. It's the same exact message so it's confusing. In that case I do believe this is a duplicate.

comment:8 Changed 10 months ago by embray

  • Resolution set to duplicate
  • Status changed from new to closed

So this is still a duplicate of #26018. The problem with the future package in particular is that it (confusingly) provides stand-ins for several packages and modules in the Python 3 stdlib, such that they end up getting imported during the build process by other packages (such as pip) that support Python 3.

So I need to update #26018 to include future in the list of "python toolchain packages" that are not installed in parallel.

comment:9 Changed 3 months ago by vbraun

#26018 seems to have stalled

I'm also not convinced that its the right fix, imho we just shoudn't copy over pyc files for installation (they contain hardcoded absolute paths). Nothing good can come out of this.

comment:10 Changed 3 months ago by vbraun

  • Resolution duplicate deleted
  • Status changed from closed to new

comment:11 Changed 2 months ago by embray

That's a good point about the filename pyc files, but I'm not sure that's really the issue here. Besides, I believe there are ways (e.g., maybe with the compileall module) to deal with that, though I haven't tried.

#26018 doesn't appear to be stalled for any good reason. I set it to needs_review 7 months, no one did anything, and now it needs to be rebased again.

Note: See TracTickets for help on using tickets.