Global function fields

[klee]

Function fields part of Sage has long been neglected and has little significant functionality besides basic arithmetic and frameworks. This ticket improves the situation by two tasks:

1. Add global function fields (with complete functionality for the theory of AG codes)

2. Clean existing code (mostly revising docstrings)

A pdf for a short introduction to global function fields in Sage is attached.

---

Now the long-term process to merge the global function field machinery to Sage has finished. This ticket played the role of meta-ticket that tracks the multi-step process:

Step	Ticket	Comments
0	#24591	Trim the existing code. Add several classes that are now just stubs.
1	#25435	Add orders and ideals.
2	#26616	Add places.
3	#27121	Add divisors.
4	#27370	Add differentials.
5	#27418	Add completions.
6	#27558	Last fixes and additions.

---

The author was supported by Chosun University 2016 and by NRF of Korea 2017, 2018, 2019

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​4bb538e	Improve docstrings
​2847da0	Trim code and docstrings
​939fda6	Refine docstrings
​2ee70bd	Merge branch 'derivations_trac16561'
​a910bfa	Improve docstrings
​931c72c	Merge branch 'develop'
​665f508	Add new features and revise docstrings
​b1d1efd	Merge branch 'develop'
​4b5d8aa	Raise doctest coverage by adding lots of them
​269da47	Add more maps

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6fa87ce

Remove redundant a or an

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1a0414c	Merge branch 'develop'
​da7cb4d	Improve _test_derivative test
​7ef1bec	Merge branch 'trac16561'

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​b292fb2	Use reverse_rows_and_columns
​df8ce31	Fixes in docstrings and code
​0e57cc1	Improve docstrings
​ad9a8c4	Fix a bug in include_zero_rows
​1894770	Relocate _hermite_form_euclidean
​bf6b339	Fix a bug; add an example for ZZ
​464f6de	Fix a docstring failure
​734c862	Merge branch 'develop' into hermite_trac22841
​268f049	Compute U also by low level operations
​1034aac	Merge branch 'hermite_trac22841' into function_fields_trac22982

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a5ceb84

Merge branch 'develop' into function_fields_trac22982

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​4cd0abc	Merge branch 'develop' into hermite_trac22841
​a96c3be	Remove reversed hermite form methods
​f1c8202	Set hermite form immutable consistently
​9c8408f	More static typing and setting immutable
​3d7fd03	Fix EXAMPLE to EXAMPLES
​ff0b73e	Improve normalization
​bc890de	Merge branch 'hermite_trac22841' into function_fields_trac22982
​1b4881b	Fix reverse hermite form code
​e59bd0d	Merge branch 'develop' into function_fields_trac22982
​ae89ab2	Fix richcmp deprecation warnings

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d893d55	Merge branch 'develop' into function_fields_trac22982
​d12fd77	Merge branch 'develop' into hermite_trac22841
​dd0d065	Add an example using normalization argument
​7933860	Merge branch 'develop' into hermite_trac22841
​032ee67	Merge branch 'hermite_trac22841' into function_fields_trac22982
​8143ac6	Fix a doctest failure

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​40af34a

Fix a doctest that takes too long

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7c1505d	Remove __nonzero__ for compatibility with python3
​5ae6dc4	Fix blocks

[klee]

A short introduction to global function fields in Sage

[saraedum]

It's great to see progress in the function field code. What are your plans for this branch? Are you in contact with the people who were working on divisors recently at Sage Days 86.5?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​27b68b2

Fix an EXAMPLE block

[klee]

Replying to saraedum:

What are your plans for this branch?

The branch is essentially a patch bomb, and is not suitable for a regular review yet. But I see no way to split the code as it consists of several parts dependent on one another. My plan is to have people to play with it for some time, and I do bug fixes and improve documentation until there appears someone who can review the whole code. I don't know how long this procedure would take. Though I want it to be merged to Sage as soon as possible from my heart :-)

I have done this task during my sabbatical year, which is going to end soon. After that, I will not have much time but to maintain the code just not to be bitrotting.

Are you in contact with the people who were working on divisors recently at Sage Days 86.5?

No. I won't have time to cooperate with other people.

I looked at their wiki page. It seems that his(?) task in "basic arithmetic in function fields" may overlap with what I did here. I implemented fractional ideals in the classical way using hermite normal form while he uses OM representation, about which I know nothing. Practically he may use my branch as a tool/base to experiment and build his system. It seems his approach is more suitable for global function fields defined by more complicated equations. Thus it could be the case that the classical approach and the OM representation approach complement each other.

[klee]

Some docstrings are not in good style.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f0a1e0e

Merge branch 'develop' into function_fields_trac22982

[jsrn]

Great that you are working on this! This is something I also really want in Sage - I will try to play around with your implementation and provide feedback when I can.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7543316	Format True, False, and None
​fc92062	Merge branch 'develop' into function_fields_trac22982

[chapoton]

There remains a call to cmp, not compatible with python3 (see patchbot report)

[danielaugot]

Replying to jsrn:

Great that you are working on this! This is something I also really want in Sage - I will try to play around with your implementation and provide feedback when I can.

Yes this is great ! Thank you for this. I will try this and tell the news to colleagues. If it is good, it should be integrated in Sage, even if reviewing the ticket takes time.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7336f26	Remove cmp from ideal.py
​f6ef7cb	Merge branch 'develop' into function_fields_trac22982

[klee]

Replying to danielaugot:

If it is good, it should be integrated in Sage, even if reviewing the ticket takes time.

Compared with Magma, the biggest problem is speed. I worked hard to improve this, but I guess there is still wide room for improvement, though certainly there is a limit somewhere due to Python vs C.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4aae960

Remove redundant code in ideal.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f6f29d4

Merge branch 'develop' into function_fields_trac22982

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d6b9060

Merge branch 'develop' into function_fields_trac22982

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9bf1a41	Merge branch 'function_fields_trac22982' into function_fields_trac22982_dev
​7b7b31f	Merge branch 'function_fields_trac22982'
​8b590ab	Resolve conflicts

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​18bd03c

Merge branch 'function_fields_trac22982' into function_fields_trac22982_dev

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4e252a1

Merge branch 'function_fields_trac22982' into function_fields_trac22982_dev

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​0361682	24170: extend vector_space method of finite fields
​2d3ec2c	24170: implement _call_ instead of local functions
​6024257	Allow morphism as subfield and add inclusion_map
​cef982d	Merge branch 'finite_field_vector_space_trac24170' into function_fields_trac22982_dev
​e9dcfeb	Merge develop
​6e5577b	Use vector_space
​f468265	Improve residue_field() methods
​02b6c3e	Merge develop
​d205018	Set C and Cinv immutable
​1a4999c	Merge trac24170

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​557f689	Merge develop
​3bc8422	Change hasse to higher
​0159b20	Add VectorSpace to FunctionFieldIsomorphism
​015a0be	Change Isomorphism morphism to Isomorphism

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c64bd0f

Fix Isomorphism morphism to Isomorphism

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a1d8ae9

Merge develop

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8cb06e0	Merge develop
​0e4c2a1	Fix a doctest failure

[klee]

Perhaps it would be a good thing to quickly merge this branch to Sage, so that more people try it and we have more chances to fix and improve this part of Sage...

[jsrn]

Replying to klee:

Perhaps it would be a good thing to quickly merge this branch to Sage, so that more people try it and we have more chances to fix and improve this part of Sage...

Hi Kwankyu, I really appreciate your massive effort here, but I don't think it is realistic that this patch bomb will get quickly merged. Bugs are much better fixed before being introduced into Sage. Why is it not possible to break this ticket into a number of smaller ones? Each one can then be reviewed with the proper care. Some tickets could basically contain just the class structure of some object, while complicated algorithms for a specific task could be tickets by themselves. Possibly keeping this ticket as a wontfix reference ticket, reviewers of the smaller ones can see what your aim is. You could give an overview of the set of tickets on a dedicated subpage on the wiki.

I want to help out with testing and reviewing, but I don't see any time that I could take out 2 weeks straight to review this beast...

Best, Johan

[klee]

Replying to jsrn:

Replying to klee:

Perhaps it would be a good thing to quickly merge this branch to Sage, so that more people try it and we have more chances to fix and improve this part of Sage...

Hi Kwankyu, I really appreciate your massive effort here, but I don't think it is realistic that this patch bomb will get quickly merged. Bugs are much better fixed before being introduced into Sage. Why is it not possible to break this ticket into a number of smaller ones? Each one can then be reviewed with the proper care. Some tickets could basically contain just the class structure of some object, while complicated algorithms for a specific task could be tickets by themselves. Possibly keeping this ticket as a wontfix reference ticket, reviewers of the smaller ones can see what your aim is. You could give an overview of the set of tickets on a dedicated subpage on the wiki.

Thanks for the comments, Johan. You are right. That is what should happen, as no reviewer wants to swallow a patch bomb.

I really had no idea about how to spend a pleasant time splitting the big code, which seems an organic whole, into smaller chunks, even with my humble git skill.

So while having dinner, I thought about this problem. And now I think I can do something about it. It seems I have to plan carefully before making tickets.

I want to help out with testing and reviewing, but I don't see any time that I could take out 2 weeks straight to review this beast...

Let me spend some days to plan this out, so that you won't be deprived of 2 weeks straight :-)

[jsrn]

That sounds great :-) Realistically, I won't have much time in most of January (important conference deadlines), but I hope to have some time for smaller tickets in February. You should probably expect that getting all of the functionality of this ticket into Sage will take quite a while, though.

Some tips from my experience with implementing coding theory during the "ACTIS project": expect that the review process will cause the design to change, and later tickets can be lightly or massively impacted. If you cut this ticket into, say 10 tickets, then consider not setting up all 10 tickets at once with full dependency lattice. The later tickets won't be looked at before the early ones, and each single change to an early ticket would have to propagate down, causing lots of extra work and headache.

Best, Johan

[klee]

Replying to jsrn:

That sounds great :-) Realistically, I won't have much time in most of January (important conference deadlines), but I hope to have some time for smaller tickets in February. You should probably expect that getting all of the functionality of this ticket into Sage will take quite a while, though.

I don't intend to rush. Don't worry.

Some tips from my experience with implementing coding theory during the "ACTIS project": expect that the review process will cause the design to change, and later tickets can be lightly or massively impacted. If you cut this ticket into, say 10 tickets, then consider not setting up all 10 tickets at once with full dependency lattice. The later tickets won't be looked at before the early ones, and each single change to an early ticket would have to propagate down, causing lots of extra work and headache.

Right. Thanks for the tips. Let's see :-)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8250e96	Merge develop
​3b37339	Add back proof=True to _factor method

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​50efcbb	Rename ideal order element modules
​8588d76	Revert "Rename ideal order element modules"
​ec33bf8	Merge develop
​3c46c03	Remove orders code
​52a39af	Revert "Remove orders code"
​b06bdec	Merge branch 'function_fields_trac22982_5'
​6f23de4	Merge branch 'function_fields_trac22982_4'
​e281202	Merge branch 'function_fields_trac22982_3'
​0642c14	Merge branch 'function_fields_trac22982_2'
​8c797ac	Merge branch 'function_fields_trac22982_1'

[gh-BrentBaccala]

Is there a current branch for this ticket? u/klee/22982 seems to have been deleted and then recreated.

[klee]

Replying to gh-BrentBaccala:

Is there a current branch for this ticket? u/klee/22982 seems to have been deleted and then recreated.

It was deleted as it would be broken into several small patches to facilitate easy review. The whole patch working with the latest stable Sage is available as u/klee/22982_stable.

[gh-BrentBaccala]

Replying to klee:

Replying to gh-BrentBaccala:

Is there a current branch for this ticket? u/klee/22982 seems to have been deleted and then recreated.

It was deleted as it would be broken into several small patches to facilitate easy review. The whole patch working with the latest stable Sage is available as u/klee/22982_stable.

I'd like to get the divisor basis code (the Hess algorithm) working with function fields over QQ (and QQbar), but I'll wait until this patch gets incorporated into the main branch before trying to add to it.

[klee]

Replying to gh-BrentBaccala:

I'd like to get the divisor basis code (the Hess algorithm) working with function fields over QQ (and QQbar), but I'll wait until this patch gets incorporated into the main branch before trying to add to it.

That would be great. I guess that a crucial step for your aim is to compute the (finite) maximal order, and perhaps also to factor ideals in the order. After that, most of the code in this patch would just carry over. In particular, the Hess algorithm code in this patch would work over Q.

The current pace of merging this patch to Sage is very slow though. I will think about a least painful way to provide this patch so that you can base your code on this patch.

[gh-BrentBaccala]

Replying to klee:

The whole patch working with the latest stable Sage is available as u/klee/22982_stable.

I suggest double-checking that. There's a lot of diffs between u/klee/22982_stable and 8.2.

[klee]

Replying to gh-BrentBaccala:

Replying to klee:

The whole patch working with the latest stable Sage is available as u/klee/22982_stable.

I suggest double-checking that. There's a lot of diffs between u/klee/22982_stable and 8.2.

Thanks!

[klee]

Replying to gh-BrentBaccala:

Replying to klee:

The whole patch working with the latest stable Sage is available as u/klee/22982_stable.

I suggest double-checking that. There's a lot of diffs between u/klee/22982_stable and 8.2.

Now corrected.

[gh-BrentBaccala]

Replying to klee:

Now corrected.

Looks good!

Suggestion:

In the documentation for class FunctionFieldPlace, prime is defined as "prime ideal associated with the place". I assume this means a prime ideal of an order (as opposed to an ideal of the coordinate ring). I suggest documenting that more clearly, and perhaps checking when an instance is created if it's the right kind of ideal and throwing an exception otherwise.

I can see someone trying to do this:

sage: R.<x,y> = QQ[]
sage: C = Curve(y^2-x^3-1)

sage: FunctionFieldPlace(R*(x-2,y-3))

or more sensibly this:

sage: R.<x,y,z> = QQ[]
sage: C = Curve((y^2-x^3-1).homogenize(z))

sage: FunctionFieldPlace(R*(x-2,y-3).homogenize(z))

Also, is there a way to construct a FunctionField from a Curve?

[klee]

Replying to gh-BrentBaccala:

Replying to klee:

Now corrected.

Looks good!

Suggestion:

In the documentation for class FunctionFieldPlace, prime is defined as "prime ideal associated with the place". I assume this means a prime ideal of an order (as opposed to an ideal of the coordinate ring). I suggest documenting that more clearly, and perhaps checking when an instance is created if it's the right kind of ideal and throwing an exception otherwise.

I can see someone trying to do this:

sage: R.<x,y> = QQ[]
sage: C = Curve(y^2-x^3-1)

sage: FunctionFieldPlace(R*(x-2,y-3))

For a function field, there is no coordinate ring, and you are not supposed to use the class FunctionFieldPlace directly, just like you use the constructor Curve rather than a curve class.

or more sensibly this:

sage: R.<x,y,z> = QQ[]
sage: C = Curve((y^2-x^3-1).homogenize(z))

sage: FunctionFieldPlace(R*(x-2,y-3).homogenize(z))

Also, is there a way to construct a FunctionField from a Curve?

The function field code is not yet integrated with the curve code. This is something to be done later.

[gh-BrentBaccala]

Replying to klee:

The function field code is not yet integrated with the curve code. This is something to be done later.

Could we discuss that a bit, since that's a major use case and might affect how the code is designed?

There's two common things to do with a place on a curve that don't seem very easy with the current code:

1. get the coordinates of the point under the place

We've got place_below in FFP_Global, that I think should get us to a prime ideal in the coordinate ring, and from there we can get to coordinates of the point, but I don't see anything more straightforward than that

2. compute a series expansion of a function in terms of a uniformizing variable

We got local_uniformizer in FFP_Rational, that returns a uniformizing variable, but I don't think there's a method to do a series expansion

Also, what's the difference between a global place and a rational place?

[klee]

Replying to gh-BrentBaccala:

Replying to klee:

The function field code is not yet integrated with the curve code. This is something to be done later.

Could we discuss that a bit, since that's a major use case and might affect how the code is designed?

There's two common things to do with a place on a curve that don't seem very easy with the current code:

1. get the coordinates of the point under the place

This would be part of the code integrating a curve with its function field.

We've got place_below in FFP_Global, that I think should get us to a prime ideal in the coordinate ring, and from there we can get to coordinates of the point, but I don't see anything more straightforward than that.

No. place_below is nothing to do with the coordinate ring. A function field is an extension of a rational function field. place below return the place of the rational function field below a place of the function field.

2. compute a series expansion of a function in terms of a uniformizing variable

We got local_uniformizer in FFP_Rational, that returns a uniformizing variable, but I don't think there's a method to do a series expansion

I recommend you to read the attached pdf for short introduction to the function field code. Yes, there is a method for series expansion of functions at a place.

Also, what's the difference between a global place and a rational place?

You seem to mean places of global function fields and places of rational function fields.

If you consult Stichtenoth's book Algebraic Function Fields and Codes, then you would be better prepared to understand and use the function field code of this ticket. That book also explains how curve and function field are related.

[gh-BrentBaccala]

There are some changes that I'm considering making here:

1. some doctest failures, due mostly to packages being renamed

2. a Kash interface to implement a lot of these functions (probably selected with implementation='kash')

Should I make those changes to this ticket? Change this git branch to be public? Or is there something else you'd suggest?

[gh-BrentBaccala]

Replying to klee:

Replying to gh-BrentBaccala:

2. compute a series expansion of a function in terms of a uniformizing variable

We got local_uniformizer in FFP_Rational, that returns a uniformizing variable, but I don't think there's a method to do a series expansion

I recommend you to read the attached pdf for short introduction to the function field code. Yes, there is a method for series expansion of functions at a place.

I read through the PDF, and I'm sorry, I don't see the method for series expansion. Could you please point it out to me?

[klee]

Replying to gh-BrentBaccala:

There are some changes that I'm considering making here:

1. some doctest failures, due mostly to packages being renamed

I guess you mean doctest failures in the stable branch of this ticket. The branch is provided for those who may want to try the global function field code before a full merge of the code to Sage, which would take quite some time. You should expect that it is not completely "stable", though I put some efforts to make it so.

Actual long-term merge process is going on through the child tickets as described in the ticket description. So all necessary fixes would go to the branches in the child tickets.

2. a Kash interface to implement a lot of these functions (probably selected with implementation='kash')

The proper parameter name for that is "algorithm" in Sage.

Should I make those changes to this ticket? Change this git branch to be public?

No. Sage development occurs incrementally. You should base your code on either the Sage develop branch or possibly a branch of other ticket. For that, you create your own ticket. It is possible that you make your branch from a branch of some child ticket of the present ticket. In that case, you have to fill the dependencies field of your ticket.

[klee]

Replying to gh-BrentBaccala:

Replying to klee:

Replying to gh-BrentBaccala:

2. compute a series expansion of a function in terms of a uniformizing variable

We got local_uniformizer in FFP_Rational, that returns a uniformizing variable, but I don't think there's a method to do a series expansion

I recommend you to read the attached pdf for short introduction to the function field code. Yes, there is a method for series expansion of functions at a place.

I read through the PDF, and I'm sorry, I don't see the method for series expansion. Could you please point it out to me?

The pdf is for a general introduction. For documentation, you should consult the Sage reference manual. Check out the "completion" method of function fields.

[gh-BrentBaccala]

I've created a new ticket #25494 that (partially) implements function fields using kash. Both prime finite fields and QQ are supported as the underlying constant field (no algebraic extensions of them, though). A lot of the methods aren't implemented, but enough are done to allow Riemann-Roch basis calculations:

sage: F.<x> = FunctionField(QQ, implementation='kash')
sage: R.<Y> = F[]
sage: L.<y> = F.extension(Y^2 - x^8 - 1)
sage: O = L.maximal_order()
sage: I = O.ideal(x, y-1)
sage: P = I.place()
sage: D = P.divisor()
sage: D.basis_function_space()
[1]
sage: (4*D).basis_function_space()
[1, 1/x^4*y + 1/x^4]

It actually uses Kwankyu Lee's Sage implementation of the Hess algorithm. Although kash implements the Hess algorithm, it turns out that klee's code works fine; all that kash is needed for is constructing and manipulating orders and ideals.

Since kash is closed source, and doesn't support anything like QQbar (a major need for my application), this code is probably only useful to verify our results. We still need a native Sage implementation.

Would you like to add a link to #25494 on this ticket's description?

[gh-BrentBaccala]

Some of #25494's code should probably be merged into #22982. The commit labeled "refactor divisor methods" moves the divisor, divisor_of_zeros, and divisor_of_poles methods into superclasses. Is there a reason not to do this? It avoids some code duplication.

[gh-BrentBaccala]

Also, what is the gens method supposed to do for ideals? Looking at the test cases in ideal.py, it seems like it returns some kind of basis. My version just returns the generators used to create the ideal, and basis needs to be called for an actual basis. Which is correct?

[klee]

Replying to gh-BrentBaccala:

Would you like to add a link to #25494 on this ticket's description?

No, it should be the other way around. I mean that #25494 should contain a link to this ticket if it depends on this ticket.

[klee]

Replying to gh-BrentBaccala:

Also, what is the gens method supposed to do for ideals? Looking at the test cases in ideal.py, it seems like it returns some kind of basis. My version just returns the generators used to create the ideal, and basis needs to be called for an actual basis. Which is correct?

This comment should be addressed to #25435. Also you could be more specific in your question; gens method of what class do you mean?

[klee]

Replying to gh-BrentBaccala:

Some of #25494's code should probably be merged into #22982. The commit labeled "refactor divisor methods" moves the divisor, divisor_of_zeros, and divisor_of_poles methods into superclasses. Is there a reason not to do this? It avoids some code duplication.

As you see, this ticket is just a meta ticket. You comment should be addressed to a (not yet created) sub ticket of this ticket, merging divisor part of the code. I am still waiting for a review of the ticket #25435 dealing with orders and ideals before I create a ticket for divisors.

[gh-BrentBaccala]

Replying to klee:

Replying to gh-BrentBaccala:

Would you like to add a link to #25494 on this ticket's description?

No, it should be the other way around. I mean that #25494 should contain a link to this ticket if it depends on this ticket.

#25494 does contain a link to #22982 in its "dependencies".

I suggest putting a link in this ticket (just like you have links to the sub tickets) so that people will know that we've got a version of this code that works over QQ.

[dimpase]

I don't think working on kash makes sense - it's basically dead, it seems. The last release was 10 years ago, and it's binary only!

[klee]

Replying to dimpase:

I don't think working on kash makes sense - it's basically dead, it seems. The last release was 10 years ago, and it's binary only!

You read wrong some previous comments. This ticket is not about kash at all!