Meta-ticket: Migration from Trac to GitHub

[Tobias Diez]

GitHub has many features that are superior to the ones trac provides. For example, pull request reviews, code view and navigation, issue management with projects and labels, GitHub actions for automatic code checks and other automation. Also, ​backlinks from any of the conversation branches are shown automatically. Moreover,

• it is ​by far the most popular site for hosting of open source projects, so potential new Sage developers are likely to be familiar with GitHub
• there are ​big network-effect benefits to being where everyone else is
• ​Moving to GitHub would be a motivating factor for people to contribute to sagemath as it gives a pathway to work in industry. github shows contributions made by a user to various projects. The GitHub contributions page has become the quintessential reference for job applications in the tech industry.
• ​it improves visibility of the project for potential sponsors/donors

Discussion:

• ​https://groups.google.com/g/sage-devel/c/ayOL8_bzOfk/m/8ljxgU--BgAJ (sage-devel 2016-01, 2020-12, 2021-01, 2021-03, 2022-09)
• ​https://groups.google.com/g/sage-devel/c/hX6ojxlNwOU (2022-09, 2022-10)

Pro/con overview: ​https://github.com/sagemath/sage/wiki/Github-vs-Gitlab-vs-trac

Vote: ​https://groups.google.com/g/sage-devel/c/7h5JoRgHpxY

The following needs to be done in this process:

• #33457 Meta-ticket: migrate from patchbot to GitHub Actions (completed in Sage 9.6)
• Documentation
  • Migration guide: ​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b (​beta-testing)
  • #34523 Add dummy package github_cli
  • #34648 Update developer guide
• Collect trac-to-github user name mapping in ​https://github.com/sagemath/website/blob/master/conf/contributors.xml
  • 100 top contributors added in ​https://github.com/sagemath/website/pull/266
  • (done) ​sage-devel post: Provide GitHub account cross-reference, deadline Oct 5
  • (done) Merge info from ​https://github.com/sagemath/sage-changelogs/blob/master/merger/contributors/9.7 into ​https://github.com/sagemath/website/blob/master/conf/contributors.xml (use contributors-from-sage-changelogs.py from ​https://github.com/sagemath/trac-to-github/pull/10); add comma-separated altnames attribute
  • (done) Merge info from ​https://trac.sagemath.org/#AccountNamesMappedtoRealNames into ​https://github.com/sagemath/website/blob/master/conf/contributors.xml; Trac 1.2 displays real names anyway
  • #34664 Your GitHub account name is needed by Oct 29 for our Trac->GitHub migration
• Migrate the Trac wiki to the Github wiki:
  • ​https://groups.google.com/g/sage-devel/c/xtwaueSUJI0
  • ​https://github.com/sagemath/trac-to-github/issues/18
• #34624 Backup Issues/PRs for projects hosted in ​https://github.com/sagemath/
• Issue and PR templates:
  • ​https://github.com/sagemath/sage-gh-templates-sandbox
  • ​https://github.com/sagemath/sage-gh-templates-sandbox/issues/new/choose
  • See ​https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository
• ​Convert Trac to GitHub Issues and set Trac to read-only, using ​Ticket-to-Issue/PR migration script
  • extract the username mapping using ​https://gist.github.com/mkoeppe/5da92ff1703e941c3d21a7a7187a23ba in the format needed by ​https://github.com/sagemath/trac-to-github/blob/master/migrate.cfg.example#L21
• Display a link from each Trac ticket to the converted GitHub Issue
• Need a plan to ​keep links from external websites to trac.sagemath.org working
  • either host all of trac.sagemath.org in static readonly mode (e.g via github pages) forever to preserve the links (suggested by @was)
  • or set up http redirects to the Issues (ticket numbers & issue numbers map 1:1)
• Dev training
  • PR practice 1: ​sage-devel post: Provide !GitHub account cross-reference
  • PR practice 2: ​Edit https://www.sagemath.org/development-ack.html
  • ​PR practice 3: Update ​https://github.com/sagemath/publications (fix open issues at ​https://github.com/sagemath/publications/issues, first interaction with a ​GH Actions check)
• Update the ​https://github.com/sagemath/sage-changelogs script

The proposed full migration to GitHub Issues in one shot makes the following tasks unnecessary:

• #30406: Make it possible to contribute to Trac via GitHub Pull Requests
• #33818 Ticket badges on Trac show "failing" when actually in progress
• #34524 Pip package git_trac_command
• #33687 Document developer workflow with GitLab Merge Requests
• #33683 Developer manual: Document setting up the trac remote within VS Code (instead of command line)
• #33406/#25980 other tasks regarding the broken/abandoned GitLab CI

See also:

• #33725 Migrate wiki.sagemath.org contents to more suitable places
• #28936 Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide
• #31164 Meta-ticket: Add external user packages as optional/experimental packages
• #34526 Move "mathematical software landscape" from Trac wiki to manual

[Samuel Lelièvre]

There once was a bot called sageb0t that would transform any pull request opened on the SageMath repo at GitHub to a ticket on the Sage Trac server. The bot was lost.

There is now a mechanism that will transform any merge request opened against the SageMath repo at GitLab into a ticket on the Sage Trac server. Read more about this mechanism:

• ​sage-devel, 2019-07, New contributor experience

• ​sage-devel, 2019-04, gitlab status

• ​sage-devel, 2018-08, Enabling Merge Requests from GitLab

[Samuel Lelièvre]

Note another discussion on this theme:

• ​sage-devel, 2018-07, talk

[Tobias Diez]

Thanks for these links.

I would argue that a simple bot that converts github pull requests and issues to trac is not enough in the long term. It might ease the transition period, but also duplicates the infrastructure. Moreover, you don't get to enjoy the above mentioned benefits if pull requests are still reviewed here on trac etc.

Are there any issues that prevent a clear one-time migration of the current issues and code contributions to github, say using tools like ​https://github.com/trustmaster/trac2github.

[Matthias Köppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[Christopher Swenson]

I wrote some of the sageb0t code, and it is still available here: ​https://github.com/swenson/sage-workflow/blob/master/sagedev/pr_export.py

[Erik Bray]

-1 we have already put a significant amount of work into migration to GitLab which doesn't take money from the US government to imprison migrant children: ​https://gitlab.com/sagemath

[Dima Pasechnik]

Replying to Erik Bray:

-1 we have already put a significant amount of work into migration to GitLab which doesn't take money from the US government to imprison migrant children: ​https://gitlab.com/sagemath

Let's not get bogged down in political discussions here - we're paying about US$4000 to Google for trac hosting, and Google is not an angel, either. Even if we create our own exclusively solar panel and wind power generator operating hosting site for trac, we still will be reliant on various arguably evil parties to be operational.

Besides, GitLab's free tier (and not only) is lacking in functionality compared to GitHub, and ​https://gitlab.com/sagemath is semi-broken, in part due to this, in part due to lack of attention.

[Matthias Köppe]

@vbraun, would it break any of your workflows if we protect the branches of the main Sage repo? (see ​https://github.com/sagemath/sage message "Your develop branch isn't protected")

[Matthias Köppe]

​​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b ready for broader review, what's missing

[Dima Pasechnik]

not sure why you changed the title; the plan should involve re-linking git.sagemath branches/commits to the corresponding branches/commits in ​https://github.com/sagemath/sagetrac-mirror as well.

[Matthias Köppe]

Replying to Dima Pasechnik:

not sure why you changed the title; the plan should involve re-linking git.sagemath branches/commits to the corresponding branches/commits in ​https://github.com/sagemath/sagetrac-mirror as well.

Sure, I've changed it back

[Matthias Köppe]

Replying to Matthias Köppe:

@vbraun, would it break any of your workflows if we protect the branches of the main Sage repo? (see ​https://github.com/sagemath/sage message "Your develop branch isn't protected")

Described now at ​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#proposed-permissions-and-protections

[Kwankyu Lee]

In the migration guide,

for trying the branch of a PR locally, instead of using git trac checkout TICKETNUMBER 
or git trac try TICKETNUMBER, 
use git fetch origin pull/PULL_REQUEST_ID/head:LOCAL_BRANCH_NAME

Is the "origin" right? A PR seems associated with "upstream" rather than "origin" (my fork). No?

[Matthias Köppe]

Thanks for catching this mistake! I've fixed it now.

[David Roe]

As a note about github permissions, I ​found that write access means "If required reviews are enabled and a collaborator with write, admin, or owner access to the repository submits a review requesting changes, the pull request cannot be merged until the same collaborator submits another review approving the changes in the pull request." I think this is good most of the time, but I worry about how we might resolve cases where different people disagree about whether a PR should be merged or how to resolve issues on it.

[Dima Pasechnik]

Replying to David Roe:

[...] I worry about how we might resolve cases where different people disagree about whether a PR should be merged or how to resolve issues on it.

well, with trac don't have a good procedure in such cases, either. That's a conflict between humans to resolve, something that no amount of computing tools would help you much.

[David Roe]

I completely agree that there is primarily a conflict for humans to resolve here, now and on github. My concern is a technical one. Currently on trac, if there's a consensus of people involved in the ticket, anyone can mark the ticket positively reviewed, the release manager can review the situation and merge it. It sounds like, if we give all active developers write permissions, then someone can just refuse to finish their review and maybe the only recourse would be to open another PR?

I bring this up just to figure out if we want to be explicit about some policy in advance that we can point to if such a situation arises. For example:

If a conflict arises in reviewing a PR, a developer should allow the PR to proceed by submitting a positive review if both of the following conditions hold:

• At least three Sage developers are in favor of the PR, outnumbering those who opose,
• At least a week of discussion has passed since any new concerns were raised.

Or maybe I'm misreading what options others would have in this setting. I think this may be an unusual situation for projects to be in since we're trying to grant Write access unusually broadly in order to preserve the ability for many developers to make changes to others' PRs.

[Matthias Köppe]

This is an important question. I'd think a solution would be that a Team such as Core would be able to override required review etc., but we should find out the details & document it.

[Dima Pasechnik]

Actually, I don't see a big reason to give all the active developers write permissions. One can work on almost all PRs without them just as well, with only a tiny bit of extra hassle.

[David Roe]

I just looked further, and found that ​Admins have the ability to "merge pull requests on protected branches, even if there are no approving reviews." So I think that takes care of my technical objection. It still may be worth codifying a social policy like I described to help in situations where people get stuck, but I'm fine putting that off for now.

[Matthias Köppe]

Replying to Dima Pasechnik:

Actually, I don't see a big reason to give all the active developers write permissions. One can work on almost all PRs without them just as well, with only a tiny bit of extra hassle.

I tend to agree with this too. We could say that write permission can be requested - as a convenience if that developer is very active and finds themselves inconvenienced by the extra hassle.

[David Roe]

I'm fine with having Write permissions be opt in, but I'd phrase it a bit differently. Perhaps "if you want to be able to make changes directly on PRs (when the author elects to allow edits from maintainers), please contact X and we can give you the relevant permissions." I think it's a reasonable workflow that we want to support, rather than having it feel like a special priveledge.

Of course, we may come up with other reasons why it's a bad idea. That's what I've been trying to think through the consequences of various permission settings.

[Matthias Köppe]

Yes, this is a great wording.

[Matthias Köppe]

David, do you want to add a bit along these lines to ​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b ?

[David Roe]

I added a bullet point to the "reviewing a change" section.

[Dima Pasechnik]

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

[John Palmieri]

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool ​https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

[Matthias Köppe]

Ticket numbers will be preserved as issue numbers

[Dima Pasechnik]

Replying to John Palmieri:

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool ​https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

the plan is that it should be 1-1, then only code handling the tag itself would need to be changed (in a trivial way).

[David Roe]

One thing that Jeroen pointed out in a ​thread about Gitlab a few years ago was that he "very much [liked] the fact that we have a single git repo where all pull requests appear. Checking out a pull request for reviewing is so much easier with Sage than it is with typical GitHub projects. If we ever move to GitLab, we really should keep this workflow."

If many Sage developers have write access to the repo, it is possible to have feature branches all in the main Sage repo (which we ​may need to do regardless as part of the transition). But there is the issue of clutter to keep in mind.

[John Palmieri]

Replying to Dima Pasechnik:

Replying to John Palmieri:

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool ​https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

the plan is that it should be 1-1, then only code handling the tag itself would need to be changed (in a trivial way).

We've been talking about :trac: directives, but there is a bigger problem: git grep trac.sagemath.org src/ yields lots of hits, and all of those links are going to be broken if we get rid of the trac server. Many of them are in comments rather than live documentation, but it would be good to not lose those links. One solution would be to just change them all to use :trac:.

[Matthias Köppe]

In the ticket description we have:

• Need a plan to ​keep links from external websites to trac.sagemath.org working
  • either host all of trac.sagemath.org in static readonly mode (e.g via github pages) forever to preserve the links (suggested by @was)
  • or set up http redirects to the Issues (ticket numbers & issue numbers map 1:1)

[Matthias Köppe]

Replying to David Roe:

One thing that Jeroen pointed out in a ​thread about Gitlab a few years ago was that he "very much [liked] the fact that we have a single git repo where all pull requests appear. Checking out a pull request for reviewing is so much easier with Sage than it is with typical GitHub projects.

I don't think this is true (nor was it true when Jeroen wrote it). The PRs will all be available in our main repo, and as documented in ​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#proposed-workflow-on-github-with-transition-guide-from-trac, they can be checked out using git fetch upstream pull/PULL_REQUEST_ID/head:LOCAL_BRANCH_NAME

[Matthias Köppe]

Replying to David Roe:

If many Sage developers have write access to the repo, it is possible to have feature branches all in the main Sage repo (which we ​may need to do regardless as part of the transition). But there is the issue of clutter to keep in mind.

I don't think we should create named feature branches in the main Sage repo.

The old branches from Trac are on sagetrac-mirror, and pull requests can be initiated from there.

[David Roe]

Creating pull requests from sagetrac-mirror sounds like a clean solution. Reading through more of the thread where Jeroen was commenting, I think he was in the minority. So I withdraw the proposal of having named feature branches in the main Sage repo.

We should make sure we're clear that this is the standard if we're giving Sage developers write access (which would allow them to create such branches).

[Matthias Köppe]

Replying to David Roe:

Creating pull requests from sagetrac-mirror sounds like a clean solution.

I've added this to ​https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#conversion-of-trac-tickets-and-the-trac-wiki-to-github

[Matthias Köppe]

As suggested in ​https://github.com/sagemath/publications/pull/142#issuecomment-1259001501 and ​https://groups.google.com/g/sage-devel/c/hX6ojxlNwOU/m/dup_Ywu1BQAJ, we should add to ​the transition guide how to model Trac's ticket dependencies in GitHub PRs. See ​https://github.com/orgs/community/discussions/4477 for solutions.