Opened 3 years ago

Closed 3 years ago

#21454 closed enhancement (fixed)

Create master references/bibliography file

Reported by: jhpalmieri Owned by:
Priority: major Milestone: sage-7.4
Component: documentation Keywords:
Cc: andrew.mathas, egourgoulhon Merged in:
Authors: John Palmieri, Johan Rosenkilde Reviewers: Johan Rosenkilde, John Palmieri
Report Upstream: N/A Work issues:
Branch: 186922c (Commits) Commit: 186922cd72d5da9fb69c8d41575cbd3acebc278c
Dependencies: Stopgaps:

Description

The eventual goal is to move all references to a central file. This deals with many of the references, but not all of them. (So far, it hasn't dealt with the references in combinat, graphs, rings, or schemes.)

Change History (32)

comment:1 Changed 3 years ago by jhpalmieri

A patch bomb. Wheeee!

comment:2 Changed 3 years ago by jhpalmieri

  • Branch set to u/jhpalmieri/references

comment:3 Changed 3 years ago by jhpalmieri

  • Commit set to b87483d2b84fcab5344b1084805268f55dc1a847
  • Status changed from new to needs_review

New commits:

b87483dCreate master references file.

comment:4 Changed 3 years ago by git

  • Commit changed from b87483d2b84fcab5344b1084805268f55dc1a847 to 631d85373734e3468e6ae277f03e95bade512f4a

Branch pushed to git repo; I updated commit sha1. New commits:

631d853A few small changes to the developer's guide

comment:5 follow-up: Changed 3 years ago by andrew.mathas

I started looking at the combinat files (looking only so far), and see many REFERENCE blocks with entries like:

:wikipedia:`Ordered_partition_of_a_set`

What should be done with these? Probably they should be left in place and only papers moved, although some papers only had arxiv or DOI links rather than full journal references.

Perhaps I was unlucky in the files that I looked at, but many of the files that I lookedat are using the references incorrectly, which means that there is little hope of scripting this...

Last edited 3 years ago by andrew.mathas (previous) (diff)

comment:6 Changed 3 years ago by andrew.mathas

  • Cc andrew.mathas added

comment:7 in reply to: ↑ 5 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to andrew.mathas:

I started looking at the combinat files (looking only so far), and see many REFERENCE blocks with entries like:

:wikipedia:`Ordered_partition_of_a_set`

What should be done with these?

They could be moved to a master list or left in place. Probably better to move them if they have a valid citation key, because part of the point of the master list is to avoid duplications.

Probably they should be left in place and only papers moved, although some papers only had arxiv or DOI links rather than full journal references.

Perhaps I was unlucky in the files that I looked at, but many of the files that I lookedat are using the references incorrectly, which means that there is little hope of scripting this...

In addition, some of the references are incomplete (preprints which have now been published, published papers with no year listed, etc.). There are also badly used references (e.g. things of the form [Ref] missing the trailing underscore).

So I don't know how to script it. I also don't know how to break it into small pieces in any good way: the master bibliography file is going to be touched by every change to the references, so even if we break this into lots of tickets, they would probably have to be merged in a specific order.

comment:8 in reply to: ↑ 7 Changed 3 years ago by jsrn

This is very impressive John!

Replying to jhpalmieri:

They could be moved to a master list or left in place. Probably better to move them if they have a valid citation key, because part of the point of the master list is to avoid duplications.

I agree.

Perhaps I was unlucky in the files that I looked at, but many of the files that I lookedat are using the references incorrectly, which means that there is little hope of scripting this...

In addition, some of the references are incomplete (preprints which have now been published, published papers with no year listed, etc.). There are also badly used references (e.g. things of the form [Ref] missing the trailing underscore).

More an argument for doing this task earlier rather than later :-)

Perhaps grep for REFERENCES, parse stuff "<whitespace>[...]" in the lines following REFERENCES before next """, then re-grep that file for [...] without the underscore. Should not be too bad in a Perl script.

So I don't know how to script it. I also don't know how to break it into small pieces in any good way: the master bibliography file is going to be touched by every change to the references, so even if we break this into lots of tickets, they would probably have to be merged in a specific order.

Perhaps something like:

  1. Ticket for creating master bibliography with a line ### sage.<module> for each module of Sage.
  1. A ticket for each module (or perhaps multiple small modules in one ticket), moving refs into the respective part of ### sage.<module>. All these tickets should be mergeable in arbitrary order.
  1. A ticket depending on all the previous which resorts the master bibliography file and removes the module lines.

But this is possibly a total nightmare on top of the diff of this ticket?

Best, Johan

comment:9 Changed 3 years ago by dimpase

Perhaps one should start by fixing all these missing '_' (which might require fixing repeated ref tags). Not on this ticket though.

comment:10 Changed 3 years ago by jsrn

In some of the modules you already went through, e.g. coding.linear_code.py there's still references left, e.g. [Du2003] and old-style (non-ReST) references such as - I. Duursma, .... Then [Du2003]_ is even there twice -- I didn't think that was legal Sphinx!

Should we aim at removing *all* REFERENCE blocks scattered around in source files? That way it's easy to at least grep for what is missing. In that case, I think the most maintainable way forward is to go module by module.

I wrote a Perl script which does what I suggested above. It finds all ReST citations (.. [blabla] ...) and reports whenever one is used without an underscore. It also reports all non-ReST citations (- blablabla ...). But I'm not sure anymore that it's useful

Best, Johan

Last edited 3 years ago by jsrn (previous) (diff)

comment:11 Changed 3 years ago by jhpalmieri

Regarding [Du2003]: are you concerned because it ought to be [Duu2003]? I hadn't gotten around to changing that one yet.

Regarding REFERENCE blocks: some of them are still useful, for example to list basic references in the introductory documentation at the top of a file.

Regarding finding references, we should not count on them being in REFERENCE[S] blocks; indeed some are not. So I've just been searching using the regular expression '^[ ]*\.\.[ ]+\['.

Regarding wikipedia references, after thinking about it more, they could probably be removed and just made into inline references of the form :wikipedia:`blah` -- creating a separate citation often doesn't add anything. But this could also be done on another ticket.

Finally, we could view this ticket (or some subset of the changes here) to be a starting point, and then follow up on other tickets.

comment:12 Changed 3 years ago by jsrn

  • Branch changed from u/jhpalmieri/references to u/jsrn/references

comment:13 Changed 3 years ago by jsrn

  • Commit changed from 631d85373734e3468e6ae277f03e95bade512f4a to dda2f1849173ccaee531f44dfa01d47f388c50a7

I merged in Sage 7.4.beta4 and fixed merge conflicts. I also fixed some things which I believe caused compilation errors on my machine.

Can it become a problem that this index file lies in en/reference? If we need a reference in e.g. a tutorial, can it cause problems to do a local reference there? Can it use overlapping labels?

In the Developer's guide, does it make sense to recommend people to make REFERENCES blocks? What are they good for, when every reference will be a hyperlink to the master bib anyway? If we decide to keep it anyway, the example should be changed to follow the naming conventions. Also, perhaps we should add a warning about the trailing underscore after reference links (since omitting them doesn't cause compilation error).

Best, Johan


New commits:

d37b10eMerge branch 'u/jhpalmieri/references' of git://trac.sagemath.org/sage into 21454_master_ref_bib
dda2f18Some more fixes. Compiles now

comment:14 follow-up: Changed 3 years ago by jhpalmieri

Thank you for the fixes. I don't know what style we should use for an author like van Lint: "Lin1999" or "vL1999" or "vLi1999" or ???

For the developer's guide, I think that there are some uses to REFERENCES blocks, or at least when I did the original work, I found some such blocks which it was not clear how to eliminate. As I tried to say in the changes in the developer's guide, if you have a long discussion of some mathematical subject, for example at the top of a file, it can be useful to add a list of general references at the end of that discussion, especially if you don't cite those references explicitly in the text. We could change that to "For more information, see [REF1]_ and [REF2]_" and then change the developer's guide accordingly. I don't have strong feelings about it.

Adding a warning about trailing underscores is a good idea.

comment:15 Changed 3 years ago by git

  • Commit changed from dda2f1849173ccaee531f44dfa01d47f388c50a7 to 609571e4860c8d03aae5757c5c4bf28348775cbd

Branch pushed to git repo; I updated commit sha1. New commits:

609571ePolishing the text in the Developer's manual

comment:16 in reply to: ↑ 14 ; follow-up: Changed 3 years ago by jsrn

Replying to jhpalmieri:

Thank you for the fixes. I don't know what style we should use for an author like van Lint: "Lin1999" or "vL1999" or "vLi1999" or ???

A Google search reveals that to be a complicated kettle of fish, see http://academia.stackexchange.com/questions/15326/how-to-deal-with-particles-in-a-last-name-in-a-reference-list.

As I intuitively did with "van Lint", one standard would put it under "L" since "van" is a particle. However, had van Lint come from Belgium, the spelling would likely be Van Lint and it should have been under "V". For italians, "di" and "Di" differ historically (something with nobility), and should be put under "D" or not correspondingly. More horrors abide.

For the developer's guide, I think that there are some uses to REFERENCES blocks, or at least when I did the original work, I found some such blocks which it was not clear how to eliminate. As I tried to say in the changes in the developer's guide, if you have a long discussion of some mathematical subject, for example at the top of a file, it can be useful to add a list of general references at the end of that discussion, especially if you don't cite those references explicitly in the text. We could change that to "For more information, see [REF1]_ and [REF2]_" and then change the developer's guide accordingly. I don't have strong feelings about it.

Adding a warning about trailing underscores is a good idea.

I updated the Developer's Guide accordingly.

I did not look through every single modification you did, but I did some random sampling and it looks good. And it compiles on my machine. Did you do this manually or did you use scripts?

Please set to positive review if you accept my changes.

Best, Johan

comment:17 Changed 3 years ago by jhpalmieri

  • Authors changed from John Palmieri to John Palmieri, Johan Rosenkilde
  • Reviewers set to Johan Rosenkilde, John Palmieri
  • Status changed from needs_review to positive_review

This looks great to me!

comment:18 in reply to: ↑ 16 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to jsrn:

Did you do this manually or did you use scripts?

I did it manually, by the way.

comment:19 in reply to: ↑ 18 Changed 3 years ago by jsrn

Cool!

Replying to jhpalmieri:

I did it manually, by the way.

That's what I thought: I didn't see how that could be done automatically. Very impressive :-)

comment:20 Changed 3 years ago by tscrim

You should make an announcement on sage-devel about this to make sure there are no objections about this major structural change to Sage's documentation.

comment:21 Changed 3 years ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:22 Changed 3 years ago by jhpalmieri

Merge conflicts are almost inevitable with changes of this magnitude. Should we wait until the next beta, or rebase on 7.4.beta5?

comment:23 follow-up: Changed 3 years ago by jmantysalo

Sounds reasonable.

A minor question: What about papers written by people with strange letters in name? "Möbius" is spelled as "Moebius" in function names, but can we use Unicode for references?

comment:24 in reply to: ↑ 23 Changed 3 years ago by dimpase

Replying to jmantysalo:

Sounds reasonable.

A minor question: What about papers written by people with strange letters in name? "Möbius" is spelled as "Moebius" in function names, but can we use Unicode for references?

Comments in UTF-8 are OK (one needs to set up encoding of the file), but it seems that labels should better be ASCII (i.e. [Moeb]_ rather than [Möb]_).

comment:25 Changed 3 years ago by egourgoulhon

  • Cc egourgoulhon added

comment:26 Changed 3 years ago by jhpalmieri

I merged this with 7.4.beta5 with no issues, so the merge conflicts must be with other tickets. I guess we wait until the next beta.

comment:27 Changed 3 years ago by jhpalmieri

  • Branch changed from u/jsrn/references to u/jhpalmieri/references

comment:28 Changed 3 years ago by jhpalmieri

  • Commit changed from 609571e4860c8d03aae5757c5c4bf28348775cbd to 00f588292ef703ba9e24aa1c6b5f8ca694065645
  • Status changed from needs_work to positive_review

Merged with 7.4.beta6.


New commits:

00f5882Merge branch 'develop' into t/21454/references.

comment:29 Changed 3 years ago by jhpalmieri

Volker: this branch is likely to cause merge conflicts. It would be helpful if it could be merged early in the transition from one beta to the next to avoid this.

comment:30 Changed 3 years ago by git

  • Commit changed from 00f588292ef703ba9e24aa1c6b5f8ca694065645 to 186922cd72d5da9fb69c8d41575cbd3acebc278c
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

186922crebased to 7.4.rc0

comment:31 Changed 3 years ago by jhpalmieri

  • Status changed from needs_review to positive_review

comment:32 Changed 3 years ago by vbraun

  • Branch changed from u/jhpalmieri/references to 186922cd72d5da9fb69c8d41575cbd3acebc278c
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.