Opened 2 years ago

Last modified 40 hours ago

#30363 new task

Meta-ticket: Migration from Trac to GitHub

Reported by: Tobias Diez Owned by:
Priority: critical Milestone: sage-9.8
Component: misc Keywords:
Cc: Samuel Lelièvre, Dima Pasechnik, David Roe, William Stein, Kwankyu Lee, Sébastien Labbé, Vincent Delecroix Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Kwankyu Lee)

GitHub has many features that are superior to the ones trac provides. For example, pull request reviews, code view and navigation, issue management with projects and labels, GitHub actions for automatic code checks and other automation. Also, backlinks from any of the conversation branches are shown automatically. Moreover,

Discussion:

Pro/con overview: https://github.com/sagemath/sage/wiki/Github-vs-Gitlab-vs-trac

Vote: https://groups.google.com/g/sage-devel/c/7h5JoRgHpxY

The following needs to be done in this process:

The proposed full migration to GitHub Issues in one shot makes the following tasks unnecessary:

  • #30406: Make it possible to contribute to Trac via GitHub Pull Requests
  • #33818 Ticket badges on Trac show "failing" when actually in progress
  • #34524 Pip package git_trac_command
  • #33687 Document developer workflow with GitLab Merge Requests
  • #33683 Developer manual: Document setting up the trac remote within VS Code (instead of command line)
  • #33406/#25980 other tasks regarding the broken/abandoned GitLab CI

See also:

  • #33725 Migrate wiki.sagemath.org contents to more suitable places
  • #28936 Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide
  • #31164 Meta-ticket: Add external user packages as optional/experimental packages
  • #34526 Move "mathematical software landscape" from Trac wiki to manual

Change History (102)

comment:1 Changed 2 years ago by Tobias Diez

Component: PLEASE CHANGEmisc
Priority: majorcritical

comment:2 Changed 2 years ago by Samuel Lelièvre

Type: PLEASE CHANGEtask

There once was a bot called sageb0t that would transform any pull request opened on the SageMath repo at GitHub to a ticket on the Sage Trac server. The bot was lost.

There is now a mechanism that will transform any merge request opened against the SageMath repo at GitLab into a ticket on the Sage Trac server. Read more about this mechanism:

comment:3 Changed 2 years ago by Samuel Lelièvre

Note another discussion on this theme:

comment:4 Changed 2 years ago by Tobias Diez

Thanks for these links.

I would argue that a simple bot that converts github pull requests and issues to trac is not enough in the long term. It might ease the transition period, but also duplicates the infrastructure. Moreover, you don't get to enjoy the above mentioned benefits if pull requests are still reviewed here on trac etc.

Are there any issues that prevent a clear one-time migration of the current issues and code contributions to github, say using tools like https://github.com/trustmaster/trac2github.

comment:5 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.2sage-9.3

comment:6 Changed 22 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:7 Changed 21 months ago by Christopher Swenson

I wrote some of the sageb0t code, and it is still available here: https://github.com/swenson/sage-workflow/blob/master/sagedev/pr_export.py

comment:8 Changed 21 months ago by Samuel Lelièvre

Cc: Samuel Lelièvre added
Description: modified (diff)

comment:9 Changed 21 months ago by Erik Bray

-1 we have already put a significant amount of work into migration to GitLab which doesn't take money from the US government to imprison migrant children: https://gitlab.com/sagemath

comment:10 Changed 21 months ago by Erik Bray

Milestone: sage-9.4sage-wishlist

comment:11 Changed 8 months ago by Matthias Köppe

Description: modified (diff)
Summary: Migration to GitHubMeta-ticket: Migration to GitHub

comment:12 Changed 8 months ago by Matthias Köppe

Cc: Dima Pasechnik added
Description: modified (diff)

comment:13 Changed 8 months ago by Matthias Köppe

Description: modified (diff)

comment:14 Changed 3 months ago by Matthias Köppe

Cc: David Roe William Stein added
Milestone: sage-wishlistsage-9.8

comment:15 in reply to:  9 Changed 3 months ago by Dima Pasechnik

Replying to Erik Bray:

-1 we have already put a significant amount of work into migration to GitLab which doesn't take money from the US government to imprison migrant children: https://gitlab.com/sagemath

Let's not get bogged down in political discussions here - we're paying about US$4000 to Google for trac hosting, and Google is not an angel, either. Even if we create our own exclusively solar panel and wind power generator operating hosting site for trac, we still will be reliant on various arguably evil parties to be operational.

Besides, GitLab's free tier (and not only) is lacking in functionality compared to GitHub, and https://gitlab.com/sagemath is semi-broken, in part due to this, in part due to lack of attention.

Last edited 3 months ago by Dima Pasechnik (previous) (diff)

comment:16 Changed 3 months ago by Dima Pasechnik

Description: modified (diff)

comment:17 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:18 Changed 3 months ago by Matthias Köppe

@vbraun, would it break any of your workflows if we protect the branches of the main Sage repo? (see https://github.com/sagemath/sage message "Your develop branch isn't protected")

comment:19 Changed 3 months ago by Matthias Köppe

comment:20 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:21 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:22 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:23 Changed 3 months ago by Matthias Köppe

Cc: Kwankyu Lee added

comment:24 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:25 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:26 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:27 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:28 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:29 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:30 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:31 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:32 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:33 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:34 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:35 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:36 Changed 3 months ago by Matthias Köppe

Summary: Meta-ticket: Migration to GitHubMeta-ticket: Migration from Trac to GitHub Issues/PRs

comment:37 Changed 3 months ago by Dima Pasechnik

not sure why you changed the title; the plan should involve re-linking git.sagemath branches/commits to the corresponding branches/commits in https://github.com/sagemath/sagetrac-mirror as well.

comment:38 Changed 3 months ago by Matthias Köppe

Description: modified (diff)
Summary: Meta-ticket: Migration from Trac to GitHub Issues/PRsMeta-ticket: Migration from Trac to GitHub

comment:39 in reply to:  37 Changed 3 months ago by Matthias Köppe

Replying to Dima Pasechnik:

not sure why you changed the title; the plan should involve re-linking git.sagemath branches/commits to the corresponding branches/commits in https://github.com/sagemath/sagetrac-mirror as well.

Sure, I've changed it back

comment:40 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:41 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:42 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:43 Changed 3 months ago by Matthias Köppe

Cc: Sébastien Labbé Vincent Delecroix added

comment:44 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:45 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:46 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:47 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:48 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:49 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:50 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:51 Changed 3 months ago by Dima Pasechnik

Description: modified (diff)

comment:52 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:53 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:54 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:55 in reply to:  18 Changed 3 months ago by Matthias Köppe

Replying to Matthias Köppe:

@vbraun, would it break any of your workflows if we protect the branches of the main Sage repo? (see https://github.com/sagemath/sage message "Your develop branch isn't protected")

Described now at https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#proposed-permissions-and-protections

comment:56 Changed 3 months ago by Kwankyu Lee

In the migration guide,

for trying the branch of a PR locally, instead of using git trac checkout TICKETNUMBER 
or git trac try TICKETNUMBER, 
use git fetch origin pull/PULL_REQUEST_ID/head:LOCAL_BRANCH_NAME

Is the "origin" right? A PR seems associated with "upstream" rather than "origin" (my fork). No?

comment:57 Changed 3 months ago by Matthias Köppe

Thanks for catching this mistake! I've fixed it now.

comment:58 Changed 3 months ago by David Roe

As a note about github permissions, I found that write access means "If required reviews are enabled and a collaborator with write, admin, or owner access to the repository submits a review requesting changes, the pull request cannot be merged until the same collaborator submits another review approving the changes in the pull request." I think this is good most of the time, but I worry about how we might resolve cases where different people disagree about whether a PR should be merged or how to resolve issues on it.

comment:59 in reply to:  58 Changed 3 months ago by Dima Pasechnik

Replying to David Roe:

[...] I worry about how we might resolve cases where different people disagree about whether a PR should be merged or how to resolve issues on it.

well, with trac don't have a good procedure in such cases, either. That's a conflict between humans to resolve, something that no amount of computing tools would help you much.

comment:60 Changed 3 months ago by David Roe

I completely agree that there is primarily a conflict for humans to resolve here, now and on github. My concern is a technical one. Currently on trac, if there's a consensus of people involved in the ticket, anyone can mark the ticket positively reviewed, the release manager can review the situation and merge it. It sounds like, if we give all active developers write permissions, then someone can just refuse to finish their review and maybe the only recourse would be to open another PR?

I bring this up just to figure out if we want to be explicit about some policy in advance that we can point to if such a situation arises. For example:

If a conflict arises in reviewing a PR, a developer should allow the PR to proceed by submitting a positive review if both of the following conditions hold:

  • At least three Sage developers are in favor of the PR, outnumbering those who opose,
  • At least a week of discussion has passed since any new concerns were raised.

Or maybe I'm misreading what options others would have in this setting. I think this may be an unusual situation for projects to be in since we're trying to grant Write access unusually broadly in order to preserve the ability for many developers to make changes to others' PRs.

comment:61 Changed 3 months ago by Matthias Köppe

This is an important question. I'd think a solution would be that a Team such as Core would be able to override required review etc., but we should find out the details & document it.

comment:62 Changed 3 months ago by Dima Pasechnik

Actually, I don't see a big reason to give all the active developers write permissions. One can work on almost all PRs without them just as well, with only a tiny bit of extra hassle.

comment:63 Changed 3 months ago by David Roe

I just looked further, and found that Admins have the ability to "merge pull requests on protected branches, even if there are no approving reviews." So I think that takes care of my technical objection. It still may be worth codifying a social policy like I described to help in situations where people get stuck, but I'm fine putting that off for now.

comment:64 in reply to:  62 Changed 3 months ago by Matthias Köppe

Replying to Dima Pasechnik:

Actually, I don't see a big reason to give all the active developers write permissions. One can work on almost all PRs without them just as well, with only a tiny bit of extra hassle.

I tend to agree with this too. We could say that write permission can be requested - as a convenience if that developer is very active and finds themselves inconvenienced by the extra hassle.

comment:65 Changed 3 months ago by David Roe

I'm fine with having Write permissions be opt in, but I'd phrase it a bit differently. Perhaps "if you want to be able to make changes directly on PRs (when the author elects to allow edits from maintainers), please contact X and we can give you the relevant permissions." I think it's a reasonable workflow that we want to support, rather than having it feel like a special priveledge.

Of course, we may come up with other reasons why it's a bad idea. That's what I've been trying to think through the consequences of various permission settings.

comment:66 Changed 3 months ago by Matthias Köppe

Yes, this is a great wording.

comment:67 Changed 3 months ago by Matthias Köppe

David, do you want to add a bit along these lines to https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b ?

comment:68 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:69 Changed 3 months ago by David Roe

I added a bullet point to the "reviewing a change" section.

comment:70 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:71 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:72 Changed 3 months ago by Dima Pasechnik

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

comment:73 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:74 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:75 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:76 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:77 in reply to:  72 ; Changed 3 months ago by John Palmieri

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

comment:78 Changed 3 months ago by Matthias Köppe

Ticket numbers will be preserved as issue numbers

Last edited 3 months ago by Matthias Köppe (previous) (diff)

comment:79 in reply to:  77 ; Changed 3 months ago by Dima Pasechnik

Replying to John Palmieri:

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

the plan is that it should be 1-1, then only code handling the tag itself would need to be changed (in a trivial way).

comment:80 Changed 3 months ago by David Roe

One thing that Jeroen pointed out in a thread about Gitlab a few years ago was that he "very much [liked] the fact that we have a single git repo where all pull requests appear. Checking out a pull request for reviewing is so much easier with Sage than it is with typical GitHub projects. If we ever move to GitLab, we really should keep this workflow."

If many Sage developers have write access to the repo, it is possible to have feature branches all in the main Sage repo (which we may need to do regardless as part of the transition). But there is the issue of clutter to keep in mind.

comment:81 in reply to:  79 Changed 3 months ago by John Palmieri

Replying to Dima Pasechnik:

Replying to John Palmieri:

Replying to Dima Pasechnik:

One more thing to take care is to fix :trac: tags in our manual, to point at the correct URLs (possibly more like this?).

Can the tool https://github.com/sagemath/trac-to-github build a dictionary of trac ticket numbers <-> Github issues/URLs/whatever, and then we can use that to fix these tags?

the plan is that it should be 1-1, then only code handling the tag itself would need to be changed (in a trivial way).

We've been talking about :trac: directives, but there is a bigger problem: git grep trac.sagemath.org src/ yields lots of hits, and all of those links are going to be broken if we get rid of the trac server. Many of them are in comments rather than live documentation, but it would be good to not lose those links. One solution would be to just change them all to use :trac:.

comment:82 Changed 3 months ago by Matthias Köppe

In the ticket description we have:

  • Need a plan to ​keep links from external websites to trac.sagemath.org working
    • either host all of trac.sagemath.org in static readonly mode (e.g via github pages) forever to preserve the links (suggested by @was)
    • or set up http redirects to the Issues (ticket numbers & issue numbers map 1:1)
Last edited 3 months ago by Matthias Köppe (previous) (diff)

comment:83 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:84 in reply to:  80 Changed 3 months ago by Matthias Köppe

Replying to David Roe:

One thing that Jeroen pointed out in a thread about Gitlab a few years ago was that he "very much [liked] the fact that we have a single git repo where all pull requests appear. Checking out a pull request for reviewing is so much easier with Sage than it is with typical GitHub projects.

I don't think this is true (nor was it true when Jeroen wrote it). The PRs will all be available in our main repo, and as documented in https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#proposed-workflow-on-github-with-transition-guide-from-trac, they can be checked out using git fetch upstream pull/PULL_REQUEST_ID/head:LOCAL_BRANCH_NAME

comment:85 in reply to:  80 Changed 3 months ago by Matthias Köppe

Replying to David Roe:

If many Sage developers have write access to the repo, it is possible to have feature branches all in the main Sage repo (which we may need to do regardless as part of the transition). But there is the issue of clutter to keep in mind.

I don't think we should create named feature branches in the main Sage repo.

The old branches from Trac are on sagetrac-mirror, and pull requests can be initiated from there.

Last edited 3 months ago by Matthias Köppe (previous) (diff)

comment:86 Changed 3 months ago by David Roe

Creating pull requests from sagetrac-mirror sounds like a clean solution. Reading through more of the thread where Jeroen was commenting, I think he was in the minority. So I withdraw the proposal of having named feature branches in the main Sage repo.

We should make sure we're clear that this is the standard if we're giving Sage developers write access (which would allow them to create such branches).

comment:87 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:88 in reply to:  86 Changed 2 months ago by Matthias Köppe

Replying to David Roe:

Creating pull requests from sagetrac-mirror sounds like a clean solution.

I've added this to https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#conversion-of-trac-tickets-and-the-trac-wiki-to-github

comment:89 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:90 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:91 Changed 2 months ago by Matthias Köppe

Last edited 2 months ago by Matthias Köppe (previous) (diff)

comment:92 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:93 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:94 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:95 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:96 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:97 Changed 2 months ago by Matthias Köppe

Description: modified (diff)

comment:98 Changed 7 weeks ago by Matthias Köppe

Description: modified (diff)

comment:99 Changed 7 weeks ago by Matthias Köppe

Description: modified (diff)

comment:100 Changed 7 weeks ago by Matthias Köppe

Description: modified (diff)

comment:101 Changed 7 weeks ago by Kwankyu Lee

Description: modified (diff)

comment:102 Changed 40 hours ago by Kwankyu Lee

Description: modified (diff)
Note: See TracTickets for help on using tickets.