Opened 4 years ago

Closed 4 years ago

#18256 closed enhancement (wontfix)

Recythonize only if source files really changed

Reported by: rws Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: build Keywords:
Cc: Merged in:
Authors: Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

Cython uses timestamps for dependency checking, often causing recythonizing when it's not needed.

Change History (15)

comment:1 Changed 4 years ago by vdelecroix

This was already reported upstream in #17851. We should not have a Sage specific solution for that! Of course we would have to set again the cache once this will be fixed upstream...

comment:2 follow-up: Changed 4 years ago by jdemeyer

If the Cython version number was touched, it means that Cython really changed (otherwise, we wouldn't have changed the Cython version number...). So I don't really understand the issue.

comment:3 in reply to: ↑ 2 ; follow-up: Changed 4 years ago by rws

Replying to jdemeyer:

If the Cython version number was touched, it means that Cython really changed (otherwise, we wouldn't have changed the Cython version number...). So I don't really understand the issue.

Maybe I'm stupid but somewhere the dependencies are described, e.g., in a Makefile or other config, which trigger the complete build. Instead, you trigger it when the version string changes, not when the timestamp changes. This results in identical behaviour on upgrade, but spares full recythonize when I do an old branch checkout followed by develop merge.

comment:4 in reply to: ↑ 3 ; follow-up: Changed 4 years ago by jdemeyer

Replying to rws:

Replying to jdemeyer:

If the Cython version number was touched, it means that Cython really changed (otherwise, we wouldn't have changed the Cython version number...). So I don't really understand the issue.

Maybe I'm stupid but somewhere the dependencies are described, e.g., in a Makefile or other config, which trigger the complete build. Instead, you trigger it when the version string changes, not when the timestamp changes. This results in identical behaviour on upgrade, but spares full recythonize when I do an old branch checkout followed by develop merge.

???

I really don't understand what you are saying. Could you please clearly indicate the problem (not an analysis or a fix, just the problem).

comment:5 in reply to: ↑ 4 ; follow-up: Changed 4 years ago by rws

Replying to jdemeyer:

I really don't understand what you are saying. Could you please clearly indicate the problem (not an analysis or a fix, just the problem).

I frequently checkout older branches to do a fix or a review. After checkout I often merge develop. Now when I make, due to (my guess) the change back and forth in cython version a full recythonize is triggered. But it is IMHO not necessary because I merged develop and so, only some Python code really changed, or only specific pyx files. The proposal here is to optimize the make behaviour in such cases.

comment:6 Changed 4 years ago by jdemeyer

  • Component changed from cython to build
  • Description modified (diff)
  • Summary changed from recythonize only if cython really changed to Rebuild packages only if the version really changed

This has nothing to do with Cython, but with make.

comment:7 Changed 4 years ago by jdemeyer

  • Keywords cythonize removed

comment:8 Changed 4 years ago by jdemeyer

To clarify: this would happen with every package upgrade. Since building Cython and re-cythonizing takes a long time, you just notice it more when it's Cython which is being rebuilt.

comment:9 Changed 4 years ago by jdemeyer

  • Description modified (diff)

Hmm no, that's not true. There is logic in src/bin/sage-spkg to not re-build packages if you're installing the version which is already installed.

comment:10 in reply to: ↑ 5 ; follow-up: Changed 4 years ago by jdemeyer

Replying to rws:

a full recythonize is triggered.

Are you sure it's really a full recythonize and not just a rebuild of all files depending on a common file like src/ext/interrupt.pxi? It's true that Cython (just like make) uses timestamps to trigger rebuilds, so if the timestamp of src/ext/interrupt.pxi changes, then all Cython extensions depending on that (i.e. almost all of them) would be recythonized.

If this is the problem, I would like to close this ticket as wontfix since it's a feature, not a bug, to use timestamps for dependency checking. And even if it would be a bug, it's in upstream Cython, not Sage.

comment:11 in reply to: ↑ 10 Changed 4 years ago by rws

Replying to jdemeyer:

...so if the timestamp of src/ext/interrupt.pxi changes, then all Cython extensions depending on that (i.e. almost all of them) would be recythonized.

Then that seems to have been the reason. If there is no way to make that more fine-grained I agree with wontfix here.

comment:12 Changed 4 years ago by jdemeyer

  • Milestone changed from sage-6.7 to sage-duplicate/invalid/wontfix
  • Reviewers set to Jeroen Demeyer
  • Status changed from new to needs_review

comment:13 Changed 4 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:14 Changed 4 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from Rebuild packages only if the version really changed to Recythonize only if source files really changed

comment:15 Changed 4 years ago by vbraun

  • Resolution set to wontfix
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.