Opened 6 years ago
Closed 6 years ago
#18560 closed enhancement (fixed)
Upgrade arb to 2.6.0
Reported by:  cheuberg  Owned by:  

Priority:  major  Milestone:  sage6.8 
Component:  packages: optional  Keywords:  arb 
Cc:  mmezzarobba, fredrik.johansson, jhpalmieri  Merged in:  
Authors:  Clemens Heuberger  Reviewers:  Jeroen Demeyer 
Report Upstream:  N/A  Work issues:  
Branch:  922b674 (Commits)  Commit:  922b67430ff5863f5dd2426bf42aa3b9bde83cfd 
Dependencies:  Stopgaps: 
Description (last modified by )
Upgrade the optional arb spkg to version 2.6.0.
Tar.gz file at https://www.math.aau.at/user/cheuberg/sage/arb2.6.0.tar.gz
Change History (35)
comment:1 Changed 6 years ago by
 Branch set to u/cheuberg/libs/arb2.6.0
 Commit set to adad105f55f46fbefc951bda91950513219cf3bc
 Keywords arb added
 Status changed from new to needs_review
comment:2 followup: ↓ 3 Changed 6 years ago by
 Commit changed from adad105f55f46fbefc951bda91950513219cf3bc to 8d0b5995900b15f4594421f9baac0fe0faaa6574
Branch pushed to git repo; I updated commit sha1. New commits:
8d0b599  Trac #18560: fix doctests after arb upgrade

comment:3 in reply to: ↑ 2 ; followup: ↓ 4 Changed 6 years ago by
comment:4 in reply to: ↑ 3 ; followup: ↓ 5 Changed 6 years ago by
Replying to mmezzarobba:
Perhaps it would be better to add tolerances so that the tests pass with both versions of arb. What do you think?
 I do not know whether the doctest framework can handle this  how would it cope with the special representation of a
RealBall
?  I think that at least some kind of plausibility checking is interesting  if the old and new balls do not intersect, we'd be in trouble.
 On the other hand, changing doctests with each new revision of arb seems to be uncomfortable, too.
comment:5 in reply to: ↑ 4 Changed 6 years ago by
Replying to cheuberg:
Replying to mmezzarobba:
Perhaps it would be better to add tolerances so that the tests pass with both versions of arb. What do you think?
 I do not know whether the doctest framework can handle this  how would it cope with the special representation of a
RealBall
?
The tolerance simply applies independently to the center and the radius, so that in many cases # abs tol 1e15
will do something reasonable.
comment:6 Changed 6 years ago by
This builds and passes its test suite on OS X 10.10.3.
comment:7 followup: ↓ 9 Changed 6 years ago by
 Status changed from needs_review to needs_work
bash3.2$ ./sage i arb Found local metadata for arb2.6.0 Found local sources at /Users/jdemeyer/sage/upstream/arb2.6.0.tar.gz Checksum: e8c367b9f617262b4a7e9956cf49b6acf5c0db24 vs 096b61d0576821d9ef6c1d0880df3d225f8bfdf3 Invalid checksum for /Users/jdemeyer/sage/upstream/arb2.6.0.tar.gz
comment:8 Changed 6 years ago by
 Commit changed from 8d0b5995900b15f4594421f9baac0fe0faaa6574 to 6b54edc6663dbe00c9fa185ccd3d4af02b41d0ea
comment:9 in reply to: ↑ 7 Changed 6 years ago by
Replying to jdemeyer:
bash3.2$ ./sage i arb Found local metadata for arb2.6.0 Found local sources at /Users/jdemeyer/sage/upstream/arb2.6.0.tar.gz Checksum: e8c367b9f617262b4a7e9956cf49b6acf5c0db24 vs 096b61d0576821d9ef6c1d0880df3d225f8bfdf3 Invalid checksum for /Users/jdemeyer/sage/upstream/arb2.6.0.tar.gz
I do not understand what happened. I just tried it again:
$ wget https://github.com/fredrikjohansson/arb/archive/2.6.0.tar.gz ... $ sha1sum 2.6.0.tar.gz e8c367b9f617262b4a7e9956cf49b6acf5c0db24 2.6.0.tar.gz
which is what the checksum should be.
I now put a copy of the tar.gz file to http://www.math.aau.at/user/cheuberg/sage/arb2.6.0.tar.gz , could you please try that file (which also has a better file name for our purposes)? Thank you.
As I cannot reproduce the problem, I set the ticket back to needs_review.
comment:10 Changed 6 years ago by
 Status changed from needs_work to needs_review
comment:11 Changed 6 years ago by
 Description modified (diff)
comment:12 Changed 6 years ago by
 Description modified (diff)
comment:13 Changed 6 years ago by
OK, the problem was that curl
(which I used to download the package) doesn't follow redirects.
comment:14 followup: ↓ 19 Changed 6 years ago by
Why are you choosing absolute tolerances over relative tolerances in the doctests?
comment:15 Changed 6 years ago by
Can you make the files spkginstall
and spkgcheck
executable please?
comment:16 followup: ↓ 20 Changed 6 years ago by
About the doctests: I would actually prefer to just change the doctests without tolerance. It's kind of ugly to have to add tolerances everywhere. And most tests don't change, so it's not so bad.
comment:17 Changed 6 years ago by
 Status changed from needs_review to needs_work
comment:18 Changed 6 years ago by
 Commit changed from 6b54edc6663dbe00c9fa185ccd3d4af02b41d0ea to 3b534cbb91702fb1f7fd1dfd4b69974bb0411b4e
Branch pushed to git repo; I updated commit sha1. New commits:
3b534cb  Trac #18560: spkg{check,install} executable

comment:19 in reply to: ↑ 14 Changed 6 years ago by
Replying to jdemeyer:
Why are you choosing absolute tolerances over relative tolerances in the doctests?
In most cases, the relative changes in the radius is around 0.1, but I would not like to allow a relative tolerance of 0.1 for the center of the ball. Unless there is a way of specifying different relative tolerances for radius and center of the ball?
comment:20 in reply to: ↑ 16 ; followup: ↓ 21 Changed 6 years ago by
Replying to jdemeyer:
About the doctests: I would actually prefer to just change the doctests without tolerance. It's kind of ugly to have to add tolerances everywhere. And most tests don't change, so it's not so bad.
It might well be the case that the tolerances I introduced above might be too small when the upgrade to the next arb version or that other tests require new tolerances. At the moment, we could easily go back to the modified doctests I already reverted once. And switching back to tolerances when we realize that we keep changing doctests with every arb version would not to be hard, either.
OTOH: I am not sure whether all machines will return the same error in all computations, but that's perhaps something that Fredrik could comment on?
Marc, you suggested using tolerances. Any further comments on this issue?
comment:21 in reply to: ↑ 20 ; followup: ↓ 22 Changed 6 years ago by
Replying to cheuberg:
OTOH: I am not sure whether all machines will return the same error in all computations
As far as I understand, yes, they will.
Marc, you suggested using tolerances. Any further comments on this issue?
Not really. I agree with you on the choice of absolute tolerances, and I don't understand Jeroen's argument against tolerances, but honestly I don't care much.
comment:22 in reply to: ↑ 21 ; followup: ↓ 26 Changed 6 years ago by
Replying to mmezzarobba:
I don't understand Jeroen's argument against tolerances, but honestly I don't care much.
The problem with tolerances is:
 since a ball is defined by 2 numbers, it will always be awkward to use tolerances since you cannot easily give different tolerances for the center and radius.
 it makes reviewing harder, since a reviewer must check whether all the given tolerances make sense
arb
is supposed to be systemindependent, so we actually want to know if even the last digit of some result is wrong.
comment:23 followups: ↓ 24 ↓ 25 Changed 6 years ago by
Results can differ on 32bit and 64bit system but this should not happen often. Gratuitous nondeterminism is avoided, but it's safest to think of arb as actually employing randomization, which unfortunately doesn't play well with doctesting. Two reasonable workarounds would be:
 Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/ 4.11e10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e10.
 Implement a doctest directive that parses intervals and checks whether the actual and expected outputs overlap (this could also take a tolerance and verify that the new actual output is not too imprecise, e.g. 53bit pi suddenly coming out as [3.1 +/ 0.1]).
comment:24 in reply to: ↑ 23 ; followup: ↓ 29 Changed 6 years ago by
The two statements seem to contradict each other:
Replying to fredrik.johansson:
Results can differ on 32bit and 64bit system but this should not happen often. Gratuitous nondeterminism is avoided
it's safest to think of arb as actually employing randomization
comment:25 in reply to: ↑ 23 ; followup: ↓ 27 Changed 6 years ago by
Replying to fredrik.johansson:
 Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/ 4.11e10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e10.
If we go with this, I would not make a difference between normal interactive Sage output and doctests. Just print balls with fewer digits always, like we already do for RR
.
 Implement a doctest directive that parses intervals and checks whether the actual and expected outputs overlap (this could also take a tolerance and verify that the new actual output is not too imprecise, e.g. 53bit pi suddenly coming out as [3.1 +/ 0.1]).
This would require writing a much more complicated doctest parser (remember you don't just want balls, but also polynomials/lists/matrices with ball coefficients and then you'll also want support for complex numbers, mpfi intervals and who knows what), I don't think this is a realistic solution.
comment:26 in reply to: ↑ 22 Changed 6 years ago by
comment:27 in reply to: ↑ 25 ; followup: ↓ 28 Changed 6 years ago by
Replying to jdemeyer:
Replying to fredrik.johansson:
 Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/ 4.11e10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e10.
If we go with this, I would not make a difference between normal interactive Sage output and doctests. Just print balls with fewer digits always, like we already do for
RR
.
I do not like the idea of always printing less information than we actually have. After all, we precisely know the error, so dilluting it seems somehow crude.
comment:28 in reply to: ↑ 27 Changed 6 years ago by
Replying to cheuberg:
I do not like the idea of always printing less information than we actually have.
On the other hand, when do you really care about seeing the last few digits in a manydigit number? Like I said, we are already printing elements of RR
like that and I don't think anybody ever complained about it.
comment:29 in reply to: ↑ 24 ; followup: ↓ 30 Changed 6 years ago by
Replying to jdemeyer:
The two statements seem to contradict each other:
Replying to fredrik.johansson:
Results can differ on 32bit and 64bit system but this should not happen often. Gratuitous nondeterminism is avoided
it's safest to think of arb as actually employing randomization
What I mean is that you might run through billions of values before you see a difference, but if you assume that a difference *never* occurs and implicitly base your code on this assumption (perhaps forgoing using the radius information before converting output to floatingpoint numbers), then it could potentially bite you in a serious way when it does.
comment:30 in reply to: ↑ 29 ; followup: ↓ 33 Changed 6 years ago by
Replying to fredrik.johansson:
it could potentially bite you in a serious way when it does.
Well, doctests never "bite you in a serious way" so I'm willing to take my chances here.
If the problem turns out to be more common then expected, we can still change our strategy.
comment:31 followup: ↓ 32 Changed 6 years ago by
As for the number of digits to print, why not print many digits by default but do something like this in (some of the) doctests:
RBF = RealBallField(53, print_digits=10)
comment:32 in reply to: ↑ 31 Changed 6 years ago by
Replying to fredrik.johansson:
As for the number of digits to print, why not print many digits by default but do something like this in (some of the) doctests:
RBF = RealBallField(53, print_digits=10)
1 because it clutters the doctests for no obvious advantage.
comment:33 in reply to: ↑ 30 Changed 6 years ago by
 Branch changed from u/cheuberg/libs/arb2.6.0 to u/cheuberg/libs/arb2.6.0notolerance
 Commit changed from 3b534cbb91702fb1f7fd1dfd4b69974bb0411b4e to 922b67430ff5863f5dd2426bf42aa3b9bde83cfd
 Status changed from needs_work to needs_review
Replying to jdemeyer:
Well, doctests never "bite you in a serious way" so I'm willing to take my chances here.
If the problem turns out to be more common then expected, we can still change our strategy.
There is only one way to find out how common that problem will be ...
Thus I attach a branch without tolerances, but with modified doctests (as before), plus spkg{check,install}
executable as discussed above.
New commits:
922b674  Trac #18560: spkg{check,install} executable

comment:34 Changed 6 years ago by
 Reviewers set to Jeroen Demeyer
 Status changed from needs_review to positive_review
comment:35 Changed 6 years ago by
 Branch changed from u/cheuberg/libs/arb2.6.0notolerance to 922b67430ff5863f5dd2426bf42aa3b9bde83cfd
 Resolution set to fixed
 Status changed from positive_review to closed
New commits:
Trac 18560: Upgrade arb to 2.6.0